Pulling in 3rd party data to your website? Be sure to shield yourself from 3rd party problems

Published in Contingency Design on Wednesday, February 18th, 2009

This post comes a bit late in the whole web 2.0 cycle. I feel that it bears repeating because I have come across sites that don't follow some basic principles when pulling in 3rd party data from sites such as flickr, twitter et. al.

APIs and data portability

The blessing of popular and easy to use APIs and the data portability of web 2.0 applications has had an unfortunate side effect, and that is that some implementations that use these services do not integrate appropriate contingency design should these 3rd party services fail.

Caching data calls to APIs is a good bit of contingency design. Many APIs will require caching - like that of Amazon - but I suspect this is intended to help limit resource use of the API host, not the site using the API. The reasons a person using API accessed data on their website would want to cache the data are:

  1. To speed up the load time of their website
  2. To have a back up plan if the API call fails

A simple implementation to handle those two cases would be one that caches an API call for a given amount of time and one that freshens stale cached data and triggers an error should an API call fail.

Caching is good contingency design practice

As I said above, this post is a bit late to the party but it is worth writing as recently I have come upon at least three sites where firebug and other widgets have revealed issues retrieving API fetched data and the site loading times have been horrible.

A decent implementation idea would be to roll your own caching wrapper and agnostically plug it in to a stable caching tool, perhaps something like Cache Lite for PHP. In this manner you have a reusable, caching library independent piece of code that can handle caching/flushing and refreshing of data which could function to handle the two cases discussed above.

And that's it. It's been 541 days since my last post. Wow. I hope this is a re-start of a new phase of blogging. Right, and it looks like I had not built the commenting functionality into this version of the site. What a surprise. I'd still like feedback so if anyone has any email me at mike at this domain and I'll pop a comment right into the database. Off to build some commenting functionality... Comments should be working now.

Comments and Feedback

Mike Wed, 18th of February, 2009

Posting a test comment.

Julian Wed, 18th of February, 2009

Whow, 541 days. The web sure has changed since then.

Jorge Thu, 19th of February, 2009

5411 is back!!!!

Mike Thu, 19th of February, 2009

Thanks Jorge, those exclamations are encouraging.

sam Sun, 22nd of February, 2009

This is great! Thanks so much. Hope it will continue.

you can get interesting post about web usability and technology on my <a href=

Sammi Thu, 26th of February, 2009

thanks for this hope it does continue :D

Jeff Tue, 16th of June, 2009

I have to look into implementing a caching system, it definitely seems useful especially since it saves on bandwidth and increases page load times...only thing is that some people like the idea of having real-time, up-to-the-second information (especially for something such as stock quotes), perhaps a free service could implement a caching system and get away with it, while a paid service would be the better option for displaying real-time information at a reasonable price, for those who require the most up-to-date information.

cet Fri, 31st of July, 2009

Whow, 541 days. The web sure has changed since then.\n

Lesia Holmes Thu, 12th of November, 2009

[url=http://tm33rxdphca7c0gc.com/]sse2m40v1yusyy98[/url]\n[link=http://kisib4ebtk47y3js.com/]mw3pt7oehjq6nilx[/link]\nx620m3xrezcoi5uq\nhttp://foyjg1jpgs8k1c7x.com/\n

This is used for spam checking; we will not share or use your e-mail address for anything else. Your address should look like your_email@something.com.

If you have a website share it with us!

Click this box and we'll set a cookie on your computer so that the next time you want to comment you don't have to type your details in again.

Home » Blog » Contingency Design

Check out the blog categories for older content

The latest from my personal website,
Mike Papageorge.com

SiteUptime Web Site Monitoring Service

Sitepoint's web devlopment books have helped me out on many occasions both for finding a quick solution to a problem but also to level out my knowlegde in weaker areas (JavaScript, I'm looking at you!). I am recommending the following titles from my bookshelf:

The Principles Of Successful Freelancing

I started freelancing by diving in head first and getting on with it. Many years and a lot of experience later I was still able to take away some gems from this book, and there are plenty I wish I had thought of beforehand. If you are new to freelancing and have a lot of questions (or maybe don't know what questions to ask!) do yourself a favor and at least check out the sample chapters.

The Art & Science Of JavaScript

The author line-up for this book says it all. 7 excellent developers show you how to get your JavaScript coding up to speed with 7 chapters of great theory, code and examples. Metaprogramming with JavaScript (chapter 5 from Dan Webb) really helped me iron out some things I was missing about JavaScript. That said each chapter really helped me to develop my JavaScript skills beyond simple Ajax calls and html insertion with libs like JQuery.

The PHP Anthology: 101 Essential Tips, Tricks & Hacks

Like the other books listed here, this provides a great reference for the PHP developer looking to have the right answers from the right people at their fingertips. I tend to pull this off the shelf when I need to delve into new territory and usually find a workable solution to keep development moving. This only needs to happen once and you recoup the price of the book in time saved from having to develop the solution or find the right pattern for getting the job done..