In a perfect world, my syndicator (which happens to be ping.fm) would take care of all my syndication needs. It would take my micro-blogs and status updates and send them to all known social sites where I have an account. Unfortunately while ping.fm appears to have the best coverage, it’s not complete. Posterous, for example, is a significant omission (any site where Stephen Fry is active has to be considered a significant site, or anyway it soon will be).

As it happens, it’s very easy for me to update Posterous with my ping.fm outpourings since I’ve already implemented a ping.fm Custom URL for my own site – it updates the dynamic sections in the left-hand column.

All I had to do was arrange for the PHP page that receives the ping.fm updates to then mail the same update to Posterous. Job done.

Not sure why ping.fm haven’t done it themselves, except that pressure of other feature request probably prevents them doing everything they want to straight away.

More to come on my struggle to get an acceptable flow of location data in place.

So I’ve decided to add them to Tumblr. Easy. Tumblr is a great tool for doing this, it makes it easy to add videos, links, pictures, anything. I am very happy with this choice.

But this has now repurposed Tumblr in my social data flow from an output device to a content provider. Before, it just used to receive my status updates and micro blogs. Now it is the place where I link to things that have amused me.

Here’s how I needed to reconfigure my social data flow:

1. Add Tumblr to my aggregators. Now my Tumblr posts will appear in my All Activity feed.

2. Remove Tumblr from my syndication list. Now Tumblr will not receive my status updates or micro blogs. This is important because otherwise these updates would appear twice in my All Activity feed.

3. Er…

4. That’s it.

Simple.

Here’s the revised flow with notes below:

(Click on the image for a bigger version)

Notes:

1. Now I can allow ping.fm to update Twitter and identi.ca immediately so I have removed the latency problem from the previous configuration. Once again I can use Twitter in a conversational way instead of it being a mere receptacle for RSS updates every half hour.
2. You’ll note also that Twitterfeed can now update identi.ca and other Laconica-based micro-blogging sites. Thanks, Mario. This means I can promote identi.ca to be a peer with Twitter and receive all my updates including real-time micro-blogs (tweets).
3. The three subscription points are the same: Writing, Status Updates and All Activity. The URIs for those resources have not changed.
4. I have distinguished between blog posts such as this and the notification of those posts. To subscribe to the full post stream use the Writing feed. If you subscribe to the All Activity feed you will be notified about posts but only see the title not the whole article. Unfortunately if you subscribe to both you will see both the blog post and a separate notification of its existence – I can’t do anything about this yet.

So there’s an emerging gap between those select sites that I use to do stuff and those sites that merely receive notification of that activity. My strategy will be to keep to a minimum the sites I use for doing things, and maximise the number of sites that are notified about it. This way I can keep all my pictures, links, writing etc. in one place but all my disparate multiple friend communities will get equal notification of new activity in one of those upstream sites.

I think the new slightly less elegant social data flow is functionally better and I’ll try to stick with it for a while to see how it performs in practice.

(Click on the image for a bigger version)

The new data flow, using ping.fm to syndicate my status updates and micro-blogs and FriendFeed to aggregate my other activity and consolidate everything into a coherent set of feeds, is working quite well. I would like to be able to have more control over the format of what both ping.fm and FriendFeed do for each target system, but the defaults are adequate for now.

Here are the issues I’m experiencing:

1. Using Twitterfeed to take my FriendFeed RSS and pipe it into Twitter introduces a degree of latency that’s a bit inappropriate for Twitter. Twitter should be near real-time and conversational (whilst not being an instant messaging system), but Twitterfeed only reads my FriendFeed every half an hour. If I get ping.fm to update Twitter directly then I’m going to get some updates twice, which is exactly what I’m trying to avoid.

2. Some of the connected systems (identi.ca for example) are Twitter-like, and I’d like to treat them the same way as Twitter: as an output device for all my activity. But Twitterfeed is Twitter-specific and I haven’t found an equivalent for identi.ca etc. yet. In any case I don’t want a different Social ETL tool for each output system; I want Twitterfeed (or something) to do this for a number of target systems.

3. There’s no class solution for location-based systems yet. Obviously I’d like to be able to update my location and travel plans in one place and have it propagate to TripIt, Dopplr, BrightKite etc. I think FireEagle may be trying to do this but it’s short of two things at the moment: co-operation from the other systems and an invitation for me. If anybody has access to FireEagle I would appreciate a way in.

4. Now I need to do the other thing – deduplicate my profligate friends’ updates. When Myrto uploads a picture to Flickr, for instance, I get notified about it four times. I can rationalise this a bit but I can’t ignore her FriendFeed or Twitter because I would miss some of her updates, so I’m condemned to hearing about her new pictures multiple times unless (a) she adopts a version of my social data flow or (b) I find a solution to deduplicating the same update coming through multiple channels. This duplicate filter is necessary until everybody adopts my architecture, i.e. for ever. Does anybody know of a solution out there?

This all seems quite difficult to manage and well beyond anybody who is either busy or technically challenged. There’s no real way of developing a packaged solution until all the social systems adopt a single sign-on technology like OpenID, and many of them have sound technical reasons for not doing so (OpenIDs can be created by anybody anywhere so it’s a bit like expecting system owners to trust a digital certificate that has no trusted root certification authority).

Anyway, that’s the state of play at the moment. More to come I’m sure.

(Click on the image for a bigger version)

Having done this I’m fairly sure of three things:

1. There are no feedback loops: no update is going to propagate to another site that feeds it back where it came from
2. There is a specific feed for three appropriate levels of details that does not omit anything relevant.
3. There is no need for any other feed unless you’re interested in particular asset classes, e.g. photos.

There is one publisher, one syndicator and one aggregator. Each of them can be replaced at any time without affecting the principle of how this works.

So here is the guide to subscribing to updates about me:

Give me everything!
I love hearing about what you say and do and I can’t get enough of you
OK: subscribe to this feed

I want to read what you write about
I’m not interested in where you are or what you’re doing today but I am interested in your thoughts
OK: subscribe to this feed. And to this feed for what people are saying about my writing

I’m interested in keeping track of you but I don’t want to read your boring writing
I’m a stalker not an bleeding intellectual
OK: subscribe to this feed.

Hopefully that will cover everybody. If you’re interested in specific stuff like photos then go to my FriendFeed page and choose the source you want.

Notes on how I rationalised this:
1. I now use Twitterfeed simply as a mechanism for posting RSS to Twitter, specifically my FriendFeed stream. I’ve classed it as “Social ETL” in the diagram :-)

2. Twitter is now just an output device and a way of feeding my updates to mobile phones, for example.

3. I have tried to remove all Facebook apps that update my status except for ping.fm

4. Sorry for the multiple and egregious posts while I sorted this out today

5. Thanks to Myrto for the ping.fm invite.