Superfeedr : XMPP over websockets.

XMPP over websockets.

By Julien, on 04 Apr 2011

You know we’re great XMPP fans : it’s by far the greatest protocol to “push” content from a server to client. It comes with stuff like identity, authentication, presence, and many awesome features. Yet, it is not as widespread as HTTP.

HTTP’s massive adoption was driven by some great server software : Apache, but also by the ubiquity of web browsers. Unfortunately, XMPP never had clients that could be hacked easily to build apps that go beyond chat. XMPP clients make way to many assumptions about what the user wants to do…

HTML5 comes with an awesome push technology : websockets. They open the door for support of XMPP in the browsers. Of course Bosh exists, but it’s not nearly as elegant, fast or scalable as the server needs to keep some kind of state. Whoever played with Bosh for a while knows how dirty it feels.

Today, we’re proud to release an Ejabberd module that builds Websockets in the XMPP server, and makes it a first class citizen in terms of underlying protocol. It’s probably not perfect though, and it’s definitely a first attempt, but feel free to install on your own servers. We also had to branch Jack ’s amazing Strophejs to split the core.js and bosh.js part, so we could easily replace bosh.js with websocket.js. This also needs more work and love, so feel free to branch and improve that code!

I sincerely want to thank Nathan for his awesome work on this. Let us know what you think!

YouTube API Blog: PubSubHubbub for YouTube Activities

PubSubHubbub, for those not in the know, is a server-to-server protocol for notifying interested parties of events they’re interested in. Notifications are pushed out to subscribers via HTTP web hooks, which offers efficiencies over polling-based solutions. With PubSubHubbub, your server finds out about events in near real-time, without having to determine the optimal polling interval or repeatedly fetch individual activity feeds that haven’t changed.

We’re happy to announce that it’s now possible to subscribe to three types of YouTube user events via PubSubHubbub: video uploads, new subscriptions, and video favorites. The subscription requests need to be made on a per-user basis, so you will only receive updates for specific users that you’re interested in.

Your code can look for our PubSubHubbub hub address in the href attribute of the top-level <link rel='hub'> element in uploads, subscriptions, and favorites feeds. For example, the feed
exposes the hub address in the <link rel='hub' href='http://pubsubhubbub.appspot.com'/> element.

More details on the PubSubHubbub can be found in the specification document. If you’d like your server to receive YouTube user event updates via PubSubHubbub, this list of libraries is a good place to start.

Cheers,
-Jeff Posnick, YouTube API Team, in conjunction with the PubSubHubbub Team

Awesome news! Google is almost 100% PubSubHubbub now!; Who's next?

Superfeedr : Susbcriber Toolbox

Susbcriber Toolbox

By Julien, on 01 Oct 2010

Our XMPP console is great for very initial debugging but doesn’t fulfill a lot of our user’s need after they subscribed to a few dozen feeds. Additionally, the subscribers who use PubSubHubbub cannot benefit from it and they usually have a hard time debugging their endpoints. As a consequence of this, we have decided to re-factor most of our Subscriber toolbox. Here is a small review of it.

Like always, please remember that all these tools actually use our API and only our API, which means you can certainly build similar or better tools on your own to fit your specific needs. Check our documentation.

A PubSubHubbub Console

With this tool, you can easily build and debug your callback url. It lets you choose a feed url, and point to your callback to perform subscriptions and unsubscriptions, and shows the result in details, so you can fix your callback URL accordingly. It should be your first stop if you’re building a PubSubHubbub based application.
It can also be used to retrieve the status of a given subscription, as PubSubHubbub is idem-potent. Which means that if you want to know whether you’re subscribed to a feed you just have to actually make a subscription request. If it succeeds, it means that you’re subscribed. If it fails, it means that you aren’t!

A Feed Status Retrieval tool

If you don’t get any notification for a given feed, it may be that there is nothing in it, or that there is problem with this feed. It may have been deleted, it may not be parse-able… etc. At any point you can now check a feed’s status.
We will show the last HTTP status we got for it, a small log message, the next fetch, the last fetch, the period at which we fetch it… Again, all this data (and more!) is actually available thru the API, as well as upon notification. You should check it out too!

Our XMPP Console

The venerable XMPP console is still here, with a few bug fixes and improved performance. It’s the best tool if you’d like to build your own XMPP client and you need to see the detail of the traffic sent to and from Superfeedr.

Feed Export

No lock-ins. We really value your freedom and what happy users more than “locked” users. You can export all the subscriptions. To prevent obvious traffic jams, you will have to export them by pages of 100.

Of course, we’re committed to improve this, as always, your feedback is greatly appreciated. Go log-in your subscriber account and let us know what you think!

Superfeedr : Publisher's callback, track and firehoses

Publisher's callback, track and firehoses

By Julien, on 24 Sep 2010

Superfeedr’s hosted hubs come up with a lot of nifty things. We recently introduced a few more : track feeds, fat pings and publisher callbacks.

Track feeds for hosted hubs

Back in July, we introduced track. A nice and easy way to get realtime notifications of keyword mention across all the content that goes thru Superfeedr. Back on August, we improved track, by adding the ability to track boolean expressions and do some geofiltering. In September, we now allow track on hosted hubs. You want any mention of Apple on Tumblr? check this feed.

Fat pings

Several of our publishers complained about the fact that loop between the time they ping the hub and the time the hub polls their feed wasn’t the most efficient. We believe they’re right, and we have now setup a mechanism where publishers can do fat pings to indicate which feed has been updated and the new content of said feed. It’s much more efficient than having us poll the feeds right after they’ve been pinged to us.

Of course, it involves some kind of signature of the content, to prevent anyone from forging your content. You’ll find all the details in your hub’s settings.

Publisher Callbacks

Many publishers want to make their content available… but a lot of them also want to keep control of it. The publisher’s callbacks are a nice way of doing so. In your hub’s settings, if you’re using Superfeedr to make your RSS feeds realtime, you can now register a callback which shall be called upon each subscription request to your content.

Of course, this simple callback lets the publisher decide of their own policy regarding these. For example, a publisher is likely to let everyone subscribe to individual feeds, but may want to control who subscribes to fire-hoses or track feeds for their hub. We also prepared a small AppEngine template for this.

Additionally, these callbacks are smart, as they will echo back the answer to the original subscriber, who can then, take measures to see his subscriptions accepted. For example, a publisher may request for an API key to be submitted as part of the subscriptions parameters, Superfeedr will forward all the params posted to the hub, so the publisher can safely grant access based on their business decisions and own choices.

A PubSubHubbub-Auth pattern

We believe that these subscription callbacks can be the basis for an Authorization pattern (at least for the read part of it). Since the publisher can decide how they want to implement the subscription callback, the publisher may then ask the users if they allow for their content to be published.
Let’s take an example :

  • Google Buzz asks Superfeedr (the designated hub) for John’s Gowalla feed,
  • Superfeedr asks Gowalla if it’s ok to let Google Buzz access this content,
  • Gowalla asks John if it’s ok to send his data to Google Buzz.
  • If yes, Gowalla tells Superfeedr to accept the subscription, if not, Gowalla tells Superfeedr to refuse the subscription.

Of course, this cannot be considered as a spec of any kind, but I surely hope we can discuss this with the Open Web community, at FOO Camp, for example. Feel free to share your thoughts. It has the great advantage of letting the publisher and the users decide, rather than imposing them some kind of policy. It also requires that publishers trust the hub which they designate.

Client Push Services Open Up Real-Time to Everyone

Client Push Services Open Up Real-Time to Everyone

Phil Leggetter, September 14th, 2010

Real-time

The number of services offering real-time APIs is slowly but surely expanding and it looks like we’re going to have to add quite a few more. Since the start of the year a new type of service has started to appear–client push services, which help developers include real-time updates in their web apps.

Real-time client push APIs have actually been around for quite a while (around 10 years) as they are shipped with Comet servers but only recently have these been moved into the cloud and offered as a service. The service flavour of these APIs give the developer the ability to instantly push information from their web server, through their chosen push service and into a web browser viewing their website.

Scrabb.ly -- real-time multi-player word game

Real-time client push is intended to replace the previous pull, or polling, mechanism that has been used for many years to mimic live data on a website. Using push via a dedicated Comet server is generally more resource efficient than polling a web server, and by using a service the resource load and complexity involved in setting up and running a Comet service is completely taken away from the developer’s considerations. This means that the web server is under much less strain, the developer can concentrate on building a killer real-time application and the website user gets the benefit of a truly real-time experience.

These services have only recently started to pop up due to a number of technology advancements. To be able to use a real-time client push service the web browser needs to be able to maintain a persistent connection back to the web server so that the web server can push information to it as soon as it becomes available. This has been achievable for a number of years via what some developer have labelled as “hacks” but is now easier than it has ever been. Most of the new services use the JavaScript WebSocket object to achieve this and fallback to using Flash if WebSockets are not supported by the browser.

The web browser also needs to be able to maintain a cross domain connection from the JavaScript code running in the website to the service e.g. a connection from blog.programmableweb.com to www.example.com. In older browsers cross domain connections were not allowed but the introduction of client access policy files (crossdomain.xml and clientaccesspolicy.xml) and more recently the Access-Control-Allow-Origin HTTP header have made cross domain calls from JavaScript possible (You can find more information and a demo of this in action here).

All of the real-time client push services have adopted a data publisher subscriber model with the web server code generally acting as the data publisher and the JavaScript code running in the web browser acting as the data subscriber. Subscriptions are made to a channel (or topic), that either exists or will be created, within the service and are identified by a name e.g. “my_channel” or “/PW/CHAT”. The publishers then simply publish data to that channel or topic using a service API and the information is instantly received by all subscribers, again via an API.

The real-time client push services that we know of at the moment are:

And some examples of their use include:

You can also check out the demos on each of the services websites.

The real-time 'meh' button

Real-time is already a big topic and users are starting to demand data and results as fast as possible. There is also the expectation that they should be informed as soon as new data is available or the existing data changes. Google are already pressing ahead with new real-time advancements such as Google Real-Time search (which actually uses polling) and Google Instant but the good news is that with the availability of real-time client push services any developer can now add real-time to their website.

Let us know if you are interested in finding out more about these real-time client push APIs and services and we’ll cover each one in more detail.

Photo via Blake Patterson

Related ProgrammableWeb ResourcesLearn more

Kwwika Kwwika API Profile

Internet realtime data update service

Protocols
JavaScript,
Data Formats
Mashups
1 mashup

Pusher Pusher API Profile

Real-time push service

Protocols
REST
Data Formats
JSON
Mashups

Beacon Push Beacon Push API Profile

Real-time push service

Protocols
REST
Data Formats
JSON
Mashups

Nice overview of various realtime API in the browser.