Welcome to the Samoyed Software blog! For our inaugural post, I’d like to share a solution we developed for a project that required embedding an RSS feed into an existing web page.
The Requirements
- The client wants to embed a third-party RSS feed into the middle of an existing web page.
- The web page is an ASP.NET Web Form, so server-side code is an option, but client-side code is also acceptable.
- Support for older, non-standards-compliant browsers (I’m looking at you, IE) is not essential.
An Easy Solution?
The problem boils down to two tasks:
- Fetch the RSS feed’s XML.
- Transform the XML to HTML and embed it.
Most modern browsers include objects in their APIs that can handle both XML retrieval (XMLHttpRequest) and XSL transformation (XSLTProcessor), so we can just build a simple client-side solution, right? Let’s try a test, using an RSS feed that resides locally.
<html>
<head>
<title>RSS Embed Test</title>
<script>
var xhttp = new XMLHttpRequest();
// Read our local XSLT file
xhttp.open("GET", "rss-to-html.xsl", false);
xhttp.send();
var xsl = xhttp.responseXML;
// Our callback function: when the RSS feed has been successfully read,
// use the XSLT to transform it to HTML then insert into our page
xhttp.onreadystatechange = function() {
if (xhttp.readyState == 4 && xhttp.status == 200) {
xsltProcessor = new XSLTProcessor();
xsltProcessor.importStylesheet(xsl);
document.getElementById("W3News").appendChild(xsltProcessor.transformToFragment(xhttp.responseXML, document));
}
};
xhttp.open("GET", "W3CNews.xml", true);
xhttp.send();
</script>
</head>
<body>
<div id="W3News"></div>
</body>
</html>
That works great! So, what happens when we replace the local RSS feed,
xhttp.open("GET", "W3CNews.xml", true);
With the remote RSS feed?
xhttp.open("GET", "https://www.w3.org/blog/news/feed", true);
Oops:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://www.w3.org/blog/news/feed. (Reason: CORS header 'Access-Control-Allow-Origin' missing).
We’ve been stymied by the same origin policy. That restriction is there for a good reason, but it means that our “easy solution” is not an option.
A Server-Side Approach
Our web page is really an ASP.NET web form, so let’s try a server-side solution. The following function fetches the feed and transforms it into HTML:
protected string FetchRSSFeedAsHTML(Uri url, string xsltFile)
{
if (url == null)
return "";
try
{
/*
* Read the feed's XML from the URL
*/
string strXmlText = "";
using (WebClient wc = new WebClient())
{
using (Stream strm = wc.OpenRead(url))
{
StreamReader reader = new StreamReader(strm);
strXmlText = reader.ReadToEnd();
reader.Close();
}
strXmlText = EncodeTitleAndDescription(strXmlText);
}
if (strXmlText.Length == 0)
return "RSS Feed is empty.";
/*
* Create an XmlReader for the string containg the feed's XML,
* then use an XslCompiledTransform to apply the XSLT sheet.
*/
XmlReaderSettings settings = new XmlReaderSettings {
IgnoreWhitespace = true, CheckCharacters = true, CloseInput = true, IgnoreComments = true, IgnoreProcessingInstructions = true, DtdProcessing = DtdProcessing.Ignore
};
using (StringReader srReader = new StringReader(strXmlText))
{
using (XmlReader reader = XmlReader.Create(srReader, settings))
{
StringBuilder sbHTML = new StringBuilder();
using (StringWriter writer = new StringWriter(sbHTML))
{
XslCompiledTransform xslTransfrom = new XslCompiledTransform();
xslTransfrom.Load(xsltFile);
xslTransfrom.Transform(reader, new XsltArgumentList(), writer);
writer.Close();
}
return sbHTML.ToString();
}
}
}
catch(Exception ex)
{
return String.Format("Could not retrieve RSS Feed. Error details:<br/>{0}<br/>{1}", ex.Message, ex.StackTrace);
}
}
EncodeTitleAndDescription called by FetchRSSFeedAsHTML just handles feeds that have failed to properly encode HTML tags in the <title> or <description> elements. You can find the code for it in the sample project.
The RSSEmbedServerSide project in the sample solution shows this function in action.
This solution does what we need it to do, but with one big caveat: the web server must be able to make outgoing HTTP requests to the site hosting the RSS feed. If the web server is Internet-facing, there is a good chance that the firewall sitting between the server and the rest of the world is filtering such outbound HTTP traffic.
That is exactly the situation with our client, and while there are very good reasons for this type of filtering, it means that our server-side approach is not feasible.
A Client/Server Approach: AJAX and a RESTful Web API
Fortunately, in addition to the server that hosts the page in question, our client has a separate Internet-facing web server that they use as conduit to a number of third-party services, and that server is permitted to make HTTP requests to designated hosts. The client is willing to add the RSS feed’s host to the permitted list, and that gives us the opportunity we need!
We can use the second server to fetch the RSS feed, but how do we get it back to the original page? Our first thought is to create a service on the second server, then call the service from the server-side code on the first web server. However, the client is not comfortable with the security risks of allowing direct communication between the more-protected first server and less-protected second server.
What if we called our new service from the page’s client-side code instead? But wait a second – the two servers have different host names, so aren’t we going to run into the same origin policy again? If we tried to call the service directly using XMLHttpRequest, then yes, but there is another way…
JSON-P to the rescue!
json-p.org provides an excellent primer on JSON-P, so I won’t attempt to duplicate that effort, but the crux of the solution is this: if we write our new service to return the RSS feed as a JSON-P response, we can accomplish our goal without fear of the same origin policy.
Server-Side: a RESTful Service
The first step is to create the service that will run on the second web server. A few years ago, we might have created a SOAP-based web service, but now we can leverage the ASP.NET Web API 2 to create a RESTful service.
Our new service will retrieve the RSS feed using code similar to our earlier server-side approach, but instead of transforming the XML into HTML, we need to transform it into a JSON object, which we’ll then wrap as a JSON-P response. As luck would have it, Microsoft provides a sample that has just what we need: a JsonFeedFormatter class that inherits from SyndicationFeedFormatter, which we can use with DataContractJsonSerializer to format our feed.
To use the JsonFeedFormatter class, we first need to create a SyndicationFeed from the feed’s XML, for which we can use Atom10FeedFormatter and Rss20FeedFormatter classes. We could use Rss20FeedFormatter only, but by using both we can support feeds in both RSS format and Atom format.
SyndicationFeed feed = null;
// Read from the XML string that we populated earlier
using (StringReader srReader = new StringReader(strXmlText))
{
using (XmlReader reader = XmlReader.Create(srReader, settings))
{
if (reader.ReadState == ReadState.Initial)
reader.MoveToContent();
Atom10FeedFormatter atom = new Atom10FeedFormatter();
if (atom.CanRead(reader))
{
atom.ReadFrom(reader);
feed = atom.Feed;
}
else
{
Rss20FeedFormatter rss = new Rss20FeedFormatter();
if (rss.CanRead(reader))
{
rss.ReadFrom(reader);
feed = rss.Feed;
}
}
}
}
Now we can use our JsonFeedFormatter class:
/*
* Use the DataContractJsonSerializer with JsonFeedFormatter to transform
* feed's content into a JSON object
*/
string strFeedContent;
using (MemoryStream ms = new MemoryStream())
{
DataContractJsonSerializer writeSerializer = new DataContractJsonSerializer(typeof(JsonFeedFormatter));
writeSerializer.WriteObject(ms, new JsonFeedFormatter(feed));
ms.Position = 0;
StreamReader rd = new StreamReader(ms);
strFeedContent = rd.ReadToEnd();
}
Then we wrap the JSON object in a function call to create our JSON-P string. (In this snippet, the name of the callback function is hard-coded, but we’ll change that in a moment.)
string strJSONCallback = "myCallbackFunc";
strFeedContent = String.Format("{0}({1});", strJSONCallback, strFeedContent);
Putting it all together, we create the action method for our new service. Our action will take two parameters:
- url (required)- the URL of the RSS feed
- callback (optional) – the name of the callback function for the JSON-P string
public HttpResponseMessage GetFeedByUrl(string url, string callback)
{
if (String.IsNullOrEmpty(url))
return Request.CreateResponse(HttpStatusCode.NotFound);
/*
* Get the callback name from the paramater, using a regular
* expression to ensure it's a valid JavaScript identifier.
*/
Regex rgxJavaScriptFunctionName = new Regex("^[a-z_$][a-z0-9_$]*$", RegexOptions.IgnoreCase);
string strJSONCallback = "processFeedContent";
callback = callback.Trim();
bool bCallbackRequested = callback.Length > 0;
if (bCallbackRequested && rgxJavaScriptFunctionName.IsMatch(callback))
strJSONCallback = callback;
try
{
/*
* Read the feed's XML from the URL
*/
string strXmlText = "";
using (WebClient wc = new WebClient())
{
using (Stream strm = wc.OpenRead(url))
{
StreamReader reader = new StreamReader(strm);
strXmlText = reader.ReadToEnd();
reader.Close();
strXmlText = EncodeTitleAndDescription(strXmlText);
}
}
if (strXmlText.Length == 0)
return Request.CreateResponse(HttpStatusCode.NotFound);
/*
* Create an XmlReader for the string containg the feed's XML,
* then attempt to use Atom10FeedFormatter or Rss20FeedFormatter
* to read the feed into a SyndicationFeed object.
*/
XmlReaderSettings settings = new XmlReaderSettings
{
IgnoreWhitespace = true, CheckCharacters = true, CloseInput = true, IgnoreComments = true, IgnoreProcessingInstructions = true, DtdProcessing = DtdProcessing.Ignore
};
SyndicationFeed feed = null;
using (StringReader srReader = new StringReader(strXmlText))
{
using (XmlReader reader = XmlReader.Create(srReader, settings))
{
if (reader.ReadState == ReadState.Initial)
reader.MoveToContent();
Atom10FeedFormatter atom = new Atom10FeedFormatter();
if (atom.CanRead(reader))
{
atom.ReadFrom(reader);
feed = atom.Feed;
}
else
{
Rss20FeedFormatter rss = new Rss20FeedFormatter();
if (rss.CanRead(reader))
{
rss.ReadFrom(reader);
feed = rss.Feed;
}
}
}
}
if (feed != null)
{
/*
* Perform content negotiation, based on the Accept header value:
* 1. If the preferred media type is application/javascript,
* or if the callback parameter was explicity specified,
* format the feed as a JSONP function call, with Content-Type
* application/javascript.
* 2. Else, format the feed as a JSON object, with Content-Type
* application/json or text/plain.
*/
MediaTypeWithQualityHeaderValue mediaType = GetBestMatchMediaType(Request.Headers.Accept);
if (mediaType == null)
return Request.CreateResponse(HttpStatusCode.NotAcceptable);
string strContentType = mediaType.MediaType;
if (bCallbackRequested && strContentType.Equals(_mediaTypeJSON, StringComparison.CurrentCultureIgnoreCase))
strContentType = _mediaTypeJS;
/*
* Use the DataContractJsonSerializer with JsonFeedFormatter to
* transform the feed's content into a JSON object
*/
string strFeedContent;
using (MemoryStream ms = new MemoryStream())
{
DataContractJsonSerializer writeSerializer = new DataContractJsonSerializer(typeof(JsonFeedFormatter));
writeSerializer.WriteObject(ms, new JsonFeedFormatter(feed));
ms.Position = 0;
StreamReader rd = new StreamReader(ms);
strFeedContent = rd.ReadToEnd();
}
/*
* If JSONP is requested, wrap the JSON data with the specified
* callback function
*/
if (strContentType.Equals("application/javascript", StringComparison.CurrentCultureIgnoreCase))
strFeedContent = String.Format("{0}({1});", strJSONCallback, strFeedContent);
/*
* Prepare the response
*/
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.RequestMessage = Request;
response.Content = new StringContent(strFeedContent, System.Text.Encoding.UTF8, strContentType);
return response;
}
return Request.CreateResponse(HttpStatusCode.NotFound);
}
catch (UriFormatException)
{
return Request.CreateResponse(HttpStatusCode.NotFound);
}
catch (FileNotFoundException)
{
return Request.CreateResponse(HttpStatusCode.NotFound);
}
catch (Exception)
{
return Request.CreateResponse(HttpStatusCode.NotFound);
}
}
The complete code for the service can be found in the RSSFetchService project in the sample solution.
You may notice that we also added some simple content negotiation, so that our service can optionally return the feed as a plain JSON string, instead of a JSON-P string. This isn’t strictly necessary in this case, but it’s a nice feature to add.
It’s also worth noting that instead of putting everything directly into the service’s code, we could have created a custom media formatter (or maybe two: one for JSON and one for JSON-P) that would accept the feed and use DataContractJsonSerializer and JsonFeedFormatter to format the content. That would have allowed us to take advantage of the Web API’s built-in content negotiation, but it seemed overkill in this case.
Client-Side: AJAX, with the assistance of jQuery
Our service is ready, so now we need to call it from the page’s client-side code. We could just insert a <script> tag, somewhere near the bottom of the page:
...
<div id="W3News"></div>
<script type="text/javascript" src="http://server2.mycompany.com/rssfetchservice/api/feed/?url=http%3A%2F%2Fwww.w3.org%2Fblog%2Fnews%2Ffeed&callback=formatFeed"></script>
</body>
</html>
But we’ll let jQuery make the AJAX call instead:
$(document).ready(function () {
$.ajax({
url: 'http://server2.mycompany.com/rssfetchservice/api/feed/',
data: {
url: 'https://www.w3.org/blog/news/feed'
},
dataType: 'jsonp',
})
.done(function (data) {
formatFeed(data);
});
});
The formatFeed function receives the JSON object and creates the HTML (with help from a couple more functions):
function formatFeed(feed) {
var maxItems = 5;
var feedTitle;
var feedDesc;
var feedUri;
var itemList = [];
if (feed != null && 'feed' in feed) {
if ('Title' in feed['feed'] && 'Content' in feed['feed']['Title'])
feedTitle = feed['feed']['Title']['Content'];
if ('Description' in feed['feed'] && 'Content' in feed['feed']['Description'])
feedDesc = feed['feed']['Description']['Content'];
if ('Links' in feed['feed']) {
var links = feed['feed']['Links'];
for (i = 0; i < links.length; i++) {
if ('RelationshipType' in links[i]) {
if (links[i]['RelationshipType'] == 'self') {
feedUri = links[i]['Uri'];
break;
}
else if (links[i]['RelationshipType'] == 'alternate')
feedUri = links[i]['Uri'];
}
}
}
if ('Items' in feed['feed']) {
var curItem;
var items = feed['feed']['Items'];
for (var i = 0; i < items.length && i < maxItems; i++) {
var itemObj = { title: "", summary: "", uri: "", pubDate: new Date() };
curItem = items[i];
if ('Title' in curItem && 'Content' in curItem['Title'])
itemObj.title = curItem['Title']['Content'];
if ('Summary' in curItem && 'Content' in curItem['Summary'])
itemObj.summary = curItem['Summary']['Content'];
if ('Links' in curItem && curItem['Links'].length > 0 && 'Uri' in curItem['Links'][0])
itemObj.uri = curItem['Links'][0]['Uri'];
if ('PublishDate' in curItem) {
if (curItem['PublishDate'].indexOf('/Date') >= 0)
itemObj.pubDate = new Date(parseInt(curItem['PublishDate'].substr(6)));
else
itemObj.pubDate = new Date(curItem['PublishDate']);
}
itemList.push(itemObj);
}
}
}
var feedDiv = $('#W3News');
if (feedDiv.length > 0) {
for (var j = 0; j < itemList.length; j++)
buildFeedItemDiv(itemList[j]).appendTo(feedDiv);
if (feedUri.length > 0 && feedTitle.length > 0) {
$('<div/>', {
'class': 'feedSource',
text: 'Source: '
})
.append(
$('<a/>', {
href: feedUri,
text: feedTitle,
target: '_blank'
})
)
.appendTo(feedDiv);
}
}
}
function buildFeedItemDiv(itemObj) {
var itemDiv =
$('<div/>', {
'class': 'feedItem'
});
$('<div/>', {
'class': 'feedItemTitle'
})
.append(
$('<a/>', {
href: itemObj.uri,
text: itemObj.title,
target: '_blank'
}),
$('<div/>', {
'class': 'feedItemLastUpdated',
text: formatFeedItemDate(itemObj.pubDate)
})
)
.appendTo(itemDiv);
$('<div/>', {
'class': 'feedItemDescription'
})
.html(itemObj.summary)
.appendTo(itemDiv);
return itemDiv;
}
function formatFeedItemDate(itemDate) {
var weekday = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"];
var month = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"];
return weekday[itemDate.getDay()] + ', ' + month[itemDate.getMonth()] + ' ' + itemDate.getDate() + ', ' + itemDate.getFullYear();
}
In the sample solution you’ll find a project named RSSEmbedAJAX that includes the complete client-side implementation, along with the server-side RSSFetchService project described above.
And Bob’s your uncle, we’re done!