Dreaming of Web Hooks for Twitter

In case you’re not familiar with the Web Hooks, here’s a simple explanation from Jeff Lindsay, web hooks evangelist :

Web hooks are user-defined callbacks over HTTP. They’re intended to, in a sense, “jailbreak” our web applications to become more extensible, customizable, and ultimately more useful.

Why Web Hooks for Twitter?

Twitter provides a solid API for its service. Even though calls are rate limited, they’re pretty generous about it, and with 20,000 calls per hour, there’s a lot you can do without being throttled. Over the years, they have heavily invested in caching mechanisms to be able to handle such a load. Given the insane numbers of services built on top of this API, we can say that they managed to build a scalable platform. Yet, in some cases, using the API doesn’t seem to be efficient.

Let’s take a look at a real world example : tracking unfollowers.

The Twitter API provides a method to retrieve the ids of your followers. The doc states:

cursor. Required. Breaks the results into pages. A single page contains 5000 ids. This is recommended for all purposes. Provide a value of -1 to begin paging. Provide values as returned to in the response body’s nextcursor and previouscursor attributes to page back and forth in the list.

http://api.twitter.com/1/followers/ids.json?screen_name=pims&cursor=-1 outputs:

{"previous_cursor_str":"0","next_cursor":0,"previous_cursor":0,"ids":[86977748,38477048…]}

If you want to know who stopped following you, you have to make another request at a later date, and by diff-ing both lists, you’ll see which ids were part of the first set but not part of the second. Easy. You’ll notice that we don’t exactly know when it happened, we just know it was between the two requests. A daily request is probably plenty enough, after all, they’ve stopped following you — it’s not really an emergency.

With roughly 250 followers, I’m far from being a Twitter celebrity, and the API handles my situation fairly well. Of course, I wouldn’t mind knowing exactly when they stopped following me. Why? Well, I may have hurt their feelings and would like to apologize, but with a 24h timeframe, it’s difficult to know which one of my tweet is to blame.

The situation gets a bit more complicated, when, you’re @ev and have 1,182,333 followers. You’ll only be able to find out if somebody — and who — unfollowed you, once you’ve made 24 requests. At roughly 48kb each request, that’s 1,152kb of data transferred to find out that today, nobody stopped following @ev.

Now, imagine if, when someone stopped following @ev, Twitter would notify our little app. They’d push that information once to us, instead of us pulling it repeatedly. Wouldn’t that be an efficient — and elegant — solution to this problem? Well, that’s how web hooks work. @ev could tell Twitter that when someone stops following him, Twitter should notify http://mylittleapp.com/ev/unfollowers and push the ids of the offenders who committed a Lèse majesté crime. And knowing exactly when it happened, well, that’s the icing on the cake.

The following events would benefit from being web hook enabled:

• you have a new follower
• you’ve lost a follower
• you have a new direct message
• your tweet has been favorited
• your tweet has been retweeted
• you have a new @reply
• you’ve been added to a list
• you’ve been removed from a list

Web hooks work as an instant notification system, and would be a great addition to the Twitter API. If you’re going to Chirp, maybe you could ask them why Twitter isn’t web hooks enabled yet?!