From the Blog

Jan
22

Does your US Representative have an RSS feed?

Posted by wubbahed on January 22nd, 2009 at 5:49 pm

I read earlier this week about a new application contest being sponsored by SunlightLabs called Apps for America. You can read more details on their site, but the idea is to encourage developers to create open source applications that increase awareness and transparency in government. I started thinking about a few ideas for applications, and while the SunlightLabs API provides details for each legislator like website, email, Twitter, and YouTube URLs, what I really wanted to know was my representatives’ RSS or Atom feed.

Turns out, that information isn’t provided by the SunlightLabs API, but that doesn’t mean we can’t find it. As it turns out, there’s a function in the Google AJAX Feed API to submit a general URL and then receive information back if there’s an RSS or Atom feed associated with the URL. So let’s mashup those two services and find our answer.

In this example, I’m going to do it just using Javascript running on a webpage. The webpage will contain a single textarea element named ‘output’ where we’ll display the results.

The Javascript for this is very straightforward. First, we’ll initialize our variables and Google services.

var legislators;
var currentRep = -1;
var sunlightAPIcall = "legislators.getList.json";
// replace this with your full API key for Sunlight Labs

google.load("jquery", "1");
google.load("feeds", "1");

Then using jQuery, we’ll grab the details about ALL the legislators from SunlightLabs and store the result in a variable.

function initialize() {
   $.getJSON(sunlightAPIcall, function (data) {
      if (data.response.legislators && data.response.legislators.length > 0) {
         legislators = data.response.legislators;
         checkNextFeed();
      }
   });
}

Next, we need to check each representative to see if they even have a website.

function checkNextFeed() {
   do {
      currentRep++;
   } while ((currentRep < legislators.length) && (legislators[currentRep].legislator.website == ""));
   document.getElementById("output").value = currentRep + " / " + legislators.length;
   setTimeout(lookupNextFeed,100);
}

If the legislator has a website, then we'll use the Google API to lookup the feed, or if we're at the end of the list, then we'll print out the results.

function lookupNextFeed() {
   if (currentRep < legislators.length) {
      google.feeds.lookupFeed(legislators[currentRep].legislator.website, lookupFeedDone);
   } else {
      outputResults();
   }
}

Once Google is done checking for the feed, it returns the results to our lookupFeedDone() function and we store the result in a new field of the object. Then, we start the process all over for the next legislator.

function lookupFeedDone(result) {
   if (result.url) {
      legislators[currentRep].legislator.feed = result.url;
   } else {
      legislators[currentRep].legislator.feed = "";
   }
   checkNextFeed();
}

Finally, here's the function to print out the results.

function outputResults() {
   var csv = "bioguide_id,website,feed\n";
   for (var i=0; i < legislators.length; i++) {
      csv = csv + legislators[i].legislator.bioguide_id + ","
         + legislators[i].legislator.website
         + "," + legislators[i].legislator.feed + "\n";
   }
   document.getElementById("output").value = csv;
}

So what do the results look like? Well,

  • There are 540 legislators
  • There are 473 legislators with websites
  • There are 109 legislators with data feeds

That's helpful, but it's still pretty sad that only 20% of legislators provide content in a feed format. Perhaps you should send them an email and ask them to update?

In the meantime, here's some links for you:

UPDATE: Turns out some of the feeds are invalid (pointing to htmlfixit.com). Here's the full list of feeds as rendered by the Goog, I'm sure as comments come in, we'll update the list some more:

Leave a Reply

4 Responses to Does your US Representative have an RSS feed?

  1.  

    |

  4. Joe Germuska |

    February 2, 2009 at 1:30 pm

    I was trying to do some stuff with your data and found that there are ten cases where you have apparently valid feed URLs in your CSV file, but parsing that feed results in 0 entries. Checking further revealed that 9 of the 10 have problems with the autodiscovered RSS feed, and the tenth (Conyers) just should have been more careful with the redirects when moving the site to “conyers.house.gov”

    Burr[B001135] feed URL is different [http://burr.senate.gov/public/index.cfm?FuseAction=RSS.Feed], but all entries have relative URLs which don’t load.
    Conyers[C000714] website URL and feed URL have changed. With updated values, all is OK. [http://conyers.house.gov/, http://conyers.house.gov/index.cfm?FuseAction=RSS.Feed
    Costello[C000794] autodiscovered feed URL is 404
    Cuellar[C001063] feed URL returns invalid XML
    Davis[D000599] autodiscovered feed URL is 404
    Ensign[E000194] same as Burr, feed URL is different [http://ensign.senate.gov/public/index.cfm?FuseAction=RSS.Feed] but entries have relative URLs which don’t load
    Hensarling[H001036] autodiscovered feed URL is 404
    Holden[H000712] autodiscovered feed URL is 404
    Johnson[J000174] same as Cuellar: feed URL returns invalid XML
    Tierney[T000266] autodiscovered feed URL returns 403 Forbidden

    So after taking out those, the “undefined” and the “htmlfixit” feeds in your datafile, I’m down to only 78 legislators having feeds. Still, this is a great way to draw attention to the issue.

  5. Dan |

    March 30, 2009 at 10:43 pm

    I enjoyed reading the article walking through the approach you used.

    You should consider deleting that initial comment, listed as comment 1 from 1/22. Its not constructive.