Scripting News

Monday morning stuffff

Mon, 14 Sep 2009 18:13:32 GMT

Chuck Barris used to announce The Gong Show as just some stuffff.

Of course our stuffff is very serious.

A new Rebooting The News with guest Dan Gillmor. One of our best. Dan drills into just what rssCloud is and realtime. He's one of the best interviewers out there. It was my turn this week to name inspirations, and I chose Young People, as exemplified by Blake Ross, Joe Hewitt, Matt Mullenweg and Joseph Scott. You can skip to the last five minutes of the podcast, it's worth listening to. Usually we choose older folk as inspiration, but we have to remember that youth, in the right hands, is itself inspiring.

I posted a proposed addition to the rssCloud walkthrough.

Typepad announced support for Pubsubhubub. I predict on Twitter that we will bridge it with rssCloud so support of one will get you compatibility with th'other. Earlier I admitted to being a dork and not seeing them as being in competition. After all, they're not commercial products. What I care about is decentralizing the realtime web, so we're not dependent on one company. Both methods accomplish that. The real problem is centralization.

Andy Oram at O'Reilly wrote a stirring ode to decentralization. At one time O'Reilly was a big proponent of P2P. Maybe they will be once again?

Zee is spelled Zed Eee Eee

Sun, 13 Sep 2009 17:40:48 GMT

My FriendFeed friend Zee is in town for a conference, and he's worried about how Americans will take to his name. At first I was puzzled. What's so hard about the name Zee, I wondered. I had only seen it spelled out -- Zee -- cause we've never talked verbally only digitally.

He explained: "I got a blank stare when I first said it in Starbucks. The barista asked, how do you spell that? I said 'Zed', 'E', 'E'. Received a blank stare. Then I said 'Zee', 'E', 'E', which then got him a little more confused."

Ahhh I get it now. His name is the same as the first letter in his name when you say it in American English. In British English there's no such confusion.

After a bit of back and forth I came up with a suggestion.

Make up some long incomprehensible name that begins with Z (how about Zarathustra). When you get the puzzled look, say "My friends call me Zee." They'll like that for two reasons: 1. They don't have to remember the name and 2. You said you want to be a friend. Americans generally like this.

Just a slice of life on the Internets.

Tornado

Fri, 11 Sep 2009 04:14:33 GMT

The former FriendFeed company now owned by Facebook did something very interesting today. They released Tornado which is the customized web server that runs the backend of FriendFeed.

I speculate in a thread on FF: "Just thinking out loud if there were a REST interface for the backend that worked like the REST interface for the client, I would be able to program both ends without having to learn the internals of your system. It would be really elegant, and probably wouldn't cost that much in overhead. I was able to create an interface to the client side of your realtime API in an hour or two. If I could sneak into the backend the same way that's all I'd need to at least put together a proof of concept. Does this make any sense?"

We need what their backend does to make the connection from rssCloud to desktops. This is something the FriendFeed guys mastered, and there's reason to believe it scales to the level we'd need since they are the guys who did GMail and Google Maps.

Interesting times we live in.

I also reminded people that when cool technologies are shipping everywhere it's not a time of death it's a time of life, as long as we have the web to connect our work, there's nothing exclusive about it. The engineers don't think we're wiping each other out, only the pundits and the hangers-on do.

Interesting times we live in.

And that's a good thing.

BTW, elsewhere on Facebook, our friend Blake Ross shipped Facebook Lite, which we heard was wonderful and are not surprised to find out is. Congrats all around!

Bad Hair Day at 7PM

Fri, 11 Sep 2009 01:00:55 GMT

We're having a special Bad Hair Day podcast which is a mini-reprise of last night's meetup in Berkeley.

Marshall is off tonight. I'll have two guests -- Doug Kaye of spokenword.org and Joseph Scott of Automattic.

Doug is working on a podcast aggregator that supports rssCloud.

Doug also founded IT Conversations that was a focal point in the podcasting bootstrap at the beginning of the decade. The Gillmor Gang got its start on Doug's network; we were inspired by his work.

Joseph developed the plugin for Wordpress that shipped at the beginning of the week. I haven't known Joseph a very long time, but we've already had a spectacular success, imho.

We'll talk about many of the things that were discussed at last night's meetup.

And for this podcast we'll have an IRC chatroom:

irc://irc.freenode.net/#badHair

Tune in at 7PM Pacific!

Twitter updating terms of use

Fri, 11 Sep 2009 00:21:09 GMT

Biz posted a list of changes to the terms of service.

They also published a draft of a short list of rules for developers.

Based on a quick read the changes seem reasonable, they reflect how the service is used and the role Twitter the company plays in it.

Clues for testing your rssCloud app

Thu, 10 Sep 2009 20:55:29 GMT

This is a frequently asked question.

If you're doing a cloud-aware app, a feed reader, skimmer or aggregator of any flavor or genre, you need feeds to test with. That was an issue a few weeks ago, but today it's not an issue at all. There's a huge variety of cloud-aware feeds updating all the time, for you to test with.

The original feed is one of mine that's announcing three new states every 15 minutes, day and night rain or shine. Not very interesting, but quite reliable.

Then there are the cloud-enabled feeds I've created from the people I follow on Twitter. Robert Scoble is the Old Faithful of this class. He's updating a lot, all the time. I have a changes.xml for all of these feeds, so you can see who's been updating.

Any of my feeds will also show up on the log page on my server, as will your registration. If you're not getting through you won't see anything there. It's very important for debugging.

And the new exciting way to generate a test feed is to create a wordpress.com blog, and post to it. Your feed has a cloud element and it will notify your app when you update. You control when it updates, so this makes it easy for testing.

InBerkeley.com, a Wordpress blog we host ourselves is cloud-enabled.

The feed for Scripting News is cloud-enabled as is the feed for Rebooting The News and Bad Hair Day.

Of course you should refer to the Implementor's Guide as you implement.

And finally, the punchline, the reason for all this michegas -- CNN has the first real news feed that's cloud-enabled. And it's a gem. When I got this one running in River2, I had to stop and pause and say, we got there.

Phil Jones on how things connect

Wed, 09 Sep 2009 23:23:58 GMT

Phil Jones and I agree on how bootstraps work.

He wrote a beautiful piece in 2006, and just re-ran it with links to 2009 bits that illustrate his points.

It's a case study in how Internet bootstraps work. They're about 10 percent technology and 90 percent working with people, trying to figure out what they want and getting it for them. In the process something builds out that has a cohesive whole, and another layer is formed.

A few years go by and we do it again.

I'm certainly not the only person who understands this process, I'm a student, and I've learned from many others that come before. I love reading books about how this works, and the latest inspiration was the Connections series by James Burke. He goes all the way back to the beginning of civilization and shows how ideas interconnect and build on other ideas.

In the end it really is all about working together. And I'm glad that Phil is there. It's nice to have someone watching who sees how it all fits together.

Then, this evening, a really insightful Webmonkey piece came out. It's the same insight that William Mougayar had, in a comment here yesterday. When this bootstrap plays out it will all be seen to have happened at the workstation. What Matt and Wordpress did over the weekend was the nuclear fuel that lit the fire. But the big winners will be the readers, skimmers and Twitter clients that will, as Webmonkey puts it so well: "We'll just have to stop calling them Twitter clients and start calling them what they should be referred to as: news clients." Amen.

Am I a hypocrite?

Wed, 09 Sep 2009 15:08:12 GMT

Sure. Of course. I am a totally f*cked up human being.

Now that that's out of the way, let me explain.

Last night on Twitter, Staci Kramer of PaidContent asked what's the difference betwen the default list in River2 and Twitter's suggested user list.

There's a qualitative difference and a quantitative one.

I use the default list for two purposes: 1. To provide an initial user experience that isn't blank. In this sense it is like Twitter's list. 2. To highlight interesting uses of RSS and clouds and reading lists, things I want to encourage people to support. By throwing them a bit of recognition, I hope to create an incentive to support the features that River2 is leading the way with. I did this with Radio too. I do it with Scripting News. I'm unabashed about it. It's how you bootstrap new stuff. It's a good thing.

No one has paid for position on the list, but I don't guarantee that I will never sell a position on the list. But I will never put a feed on the list that I wouldn't put there if they didn't pay.

Now for the differences.

1. My list is like the default list in Tweetdeck or Tweetie or Google Reader. I don't have a monopoly. I am not the only game in town. If people dislike my choices they can vote with their feet. Twitter is the whole ballgame. As I said yesterday, it's as if Google favored their friends in search results. Or if Tim Berners-Lee made it so that 1/4 of every web page had an ad for Om, ReadWriteWeb, Tim O'Reilly or TechCrunch.

2. I have a shopping list in my pocket. On it I list products I'm going to buy when I go to the supermarket. That's also like the Suggested User List. The products on the list profit from being there. But Chef Boyardee won't notice whether or not he's on my list (he's not). River2 is a teeny weeny little product compared to the mighty Twitter, which delivers hundreds of thousands of followers to people on its list.

3. I try not to influence editorial content through my choices. I've gone with big pubs like Reuters, BBC, the Guardian, CNET, NYT. Their techies hopefully will appreciate the respect, but if it influences the writers I will remove it immediately. In TwitterLand, the problem isn't so much that Twitter tries to influence, that's understandable, it's that the reporters don't object.

There will continue to be a default list in River2.

My code is a blog

Wed, 09 Sep 2009 14:42:26 GMT

Here's a screen shot of one of the objects that makes up the River2 aggregator.

The top of the script is a blog.

We've always done it this way. There are scripts in the system that are still in use with "posts" at the top that date back to the mid-90s. From lots of team members, most of whom have gone on to other things. I imagine they've taken some of our practices with them, as we inherited practices from teams we came from and code we used to work on. There's a culture to programming that's mostly invisible to non-programmers.

Another example of Narrate Your Work, the philosophy of the blogger and reporter.

Happy baby is happy!

Wed, 09 Sep 2009 00:06:58 GMT

Nothing more to say than what a hugely perfectly pretty happy baby!

She made me happy.

That's all.

What does rssCloud mean to you?

Tue, 08 Sep 2009 15:08:58 GMT

Okay I think it's time to explain what rssCloud means to people who use the web, as opposed to people who create it at a technical level.

The idea is to deliver news faster, without relying on a single company to do all the work.

Until now you could have one or the other, but not both.

You could have the news delivered via RSS, but if you wanted it fast you had to go to Twitter or Facebook or FriendFeed.

The problem with going to a company is two-fold: 1. The company might not be able to handle it. 2. The company might screw with it.

We saw both 1 and 2 with Twitter. Last summer they had trouble keeping the system up. They still "fail whale" a lot, and it could get worse, if they grow. Certainly if they grow to a billion users and become the "pulse of the planet" -- as they've said they want to. There are other reliability problems, like no good record of what's posted on Twitter. And all the trouble scaling has meant that new features show up in the center at a snail's pace, while the pace of innovation at the edges is furious, and limited by the slow growth at the center.

But even worse is that the company has not stayed out of the editorial flow of the community. At the beginning of the year, the basic metric, the Twitter equivalent of page-rank was follower count. Then all of a sudden some user's counts starting going up at an incredible rate. It came out that the company was recommending these people to new users. They were giving this flow to their friends, and to reporters who cover them, but not to people they don't know or don't like. They've even taken it away from reporters who have written stories they don't like. This is totally unacceptable, we've said so repeatedly. They ignore the concern.

Think about it another way. Imagine if Larry and Sergey's friends pages always showed up at the top of search results on Google, even when their pages had nothing to do with what people were searching for. Now you understand how un-Internet-like this is.

People say they're entitled to do this, and I suppose they are. But then we are entitled to take this idea and use the Internet to implement it, instead of all relying on a company.

That's how a techie like me, who loves the open-ness of the Internet, thinks.

To understand why, you have to understand that in 1993, I gave up on tech, because it was so utterly messed up by big companies fighting to control the users. A guy like me couldn't make software and sell it, because they were always screwing with us. They weren't doing it on purpose, they just didn't care. If you said "You should care" they'd call you a utopian or Mother Teresa and laugh. But in the end they got screwed by their own screwing around. The Internet straightened it all out, and newly charged with optimism, I learned something important. The tech world is cyclic. First it's wonderful, then it gets f*cked, then it gets wonderful again.

Our goal with rssCloud is to take what was wonderful about Twitter, that got f*cked by their Suggested Users List and the Fail Whales, and make it wonderful again.

I want everything fast with no company in the middle. That doesn't mean Twitter goes away, not at all. They just have to stop being in the middle.

That's what rssCloud is about. Fast news updates without the company in the middle. Small pieces loosely joined.

PS: And now maybe you understand why there was so much crap flying around RSS for the last few months. It always happens when the dam is about to break. No big deal. Just go right through it. The bark is worse than their bite.

Someone give Om an award

Tue, 08 Sep 2009 13:19:12 GMT

I just took a quick look at the RSS feeds of the major tech pundit blogs to see which had <cloud> elements.

I found one at Gigaom!

http://feeds.feedburner.com/ommalik

If you View Source you'll see it.

Something else worth noting, Feedburner preserves the <cloud> element. So two things are working!

I subscribed to his feed in River2 and it grokked his realtimeness. Oh life is sweet.

For techies, you might wonder why we received notification of an update after subscribing. That shows that Om's Wordpress installation is following the Walkthrough. You're supposed to test the notification handler before you register it. If it passes the test go ahead. If it fails, throw it away. This keeps the conversation real.

2002 != 2009

Tue, 08 Sep 2009 22:23:41 GMT

There's been concern expressed today in various Twitter messages that rssCloud might not scale to support all the people who might want to use it.

I'm not an expert in massive centralized systems, that's why I designed this to be decentralized, like RSS is.

I set up an rssCloud server, fairly confident that it would scale to meet the demand, and with a fallback if it shouldn't. I'm not risking anything, because we know that polling works for RSS. rssCloud is an optimization, its purpose is to make RSS faster. But if it fails RSS still works. As I wrote earlier, even though many people predicted in 1999 that RSS would never work, it's actually never failed, there is no RSS fail whale.

This is not my first time with rssCloud. My team at UserLand Software implemented it in 2002 in Radio and Manila. We had problems, but I've factored in what we learned in 2002 in the 2009 implementation. If you're interested in the details, I've spelled them out in the Implementor's Guide to rssCloud, which was published in mid-July and has been reviewed by dozens of programmers, and implemented by more than a few, including the people at Automattic.

I'm sure we'll talk about this at tomorrow's rssCloud meetup in Berkeley.

Any Wordpress blog can be cloud-enabled

Tue, 08 Sep 2009 02:48:40 GMT

This is worth a special post.

Wordpress did two things today.

1. They enabled rssCloud support on wordpress.com. This means that any weblog hosted on their server can publish real-time. This is the announcement that got all the attention.

2. But equally important is that you can install the rssCloud plug-in on any Wordpress blog that you host and it adds a cloud element to your feed and handles notifications for subscribers. That's how we got InBerkeley.com to be cloud-enabled. It takes a couple of minutes and you're ready to go.

http://wordpress.org/extend/plugins/rsscloud/

Tease! Tease! Tease!

Mon, 07 Sep 2009 19:57:57 GMT

This is one of many millions of blogs on wordpress.com.

Here is its RSS feed.

View source on that.

Notice anything interesting?

ReadWriteWeb: "All blogs on the WordPress.com platform and any WordPress.org blogs that opt-in will now make instant updates available to any RSS readers subscribed to a new feature called RSSCloud. There is currently only one RSS aggregator that supports RSSCloud, Dave Winer's brand-new reader River2. That will probably change very soon."

Apparently for some people this is the first time they're hearing about the <cloud> element in RSS. It first appeared in January 2001 and was part of RSS 0.92, and of course RSS 2.0.. It was fully supported in Radio UserLand 8.0 and Manila.

Also, many thanks to Matt Mullenweg and Toni Schneider at Automattic, and the rest of the guys. It's so cool to have them in the tech industry. We all owe them a lot for their support of open formats and protocols.

InBerkeley is cloud-enabled. And it works! Real-time, baby.

Any Wordpress blog can be cloud-enabled, not just the ones on wordpress.com. I wonder which major tech blog is going to be first to go cloud.

Also it would be great if Twitter clients, such as Seesmic and Brizzly, would start thinking about supporting rssCloud.

Don't forget there's an rssCloud meetup at UC-Berkeley on Wednesday at 7PM.

Test post

Mon, 07 Sep 2009 22:43:16 GMT

please ignore

An rssCloud case study: Brizzly & Seesmic

Sun, 06 Sep 2009 14:46:35 GMT

There's a lot of sides to a bootstrap. The idea is to take a something that's highly integrated and break it into pieces. Connect the pieces with open formats, and then show people how to compete. Then each of the pieces becomes a market where users have choice. And when users have choice, competitors must work hard to please them.

It's how we got RSS going in the first place. Netscape got a few content companies to create feeds. They created an aggregator, which we competed with. Two aggregators, that meant more, for sure. And then we made our blogging software produce feeds and worked with lots of publishers. Now blogging software updates millions of feeds every day. But at one point there were just a half-dozen feeds and two aggregators.

There's lots of competition in the market for Twitter clients. There's a raging battle between a dozen teams all of whom are vying for your attention. Each of them wants to produce the product that attracts the most users. However at the center of the market there is no competition, so improvements come slowly. Our goal is to change that.

Now, two excellent examples of clients are Seesmic and Brizzly. There are many others but these are the two that I use, so I am most familiar with them. Here are screen shots of the two products. (Click on the thumbs to see larger versions.)

There are two ways they can support rssCloud, and help the bootstrap give more choice to users, and potentially free themselves to create more features without having to wait for Twitter.

1. Allow your users to subscribe to cloud-enabled feeds. Right now, most of those are the 1151 feeds of the people I subscribe to. But soon (knock wood) there will be many thousands more, content that you can't get from Twitter. So supporting cloud-enabled feeds now could buy you a head-start on your competition in October or November.

2. Publish each user's stream of 140-character messages as cloud-enabled feeds, in addition to pushing them through Twitter. Several reasons to do this. First, you provide them a backup, which may be a feature that's of value to them. And they might be able to share their content with 140-character networks formed by others, including Facebook, Yahoo, Google, who knows who. Open formats and protocols create lots of options. Maybe you want to start your own little network of users independent of Twitter. I can tell you -- I do! (And will.)

Brizzly and Seesmic may not want to do this now. After all it is a holiday weekend in the US. But you never know what the future holds, and it doesn't take much time to think about it.

RSS has no Fail Whale

Sat, 05 Sep 2009 14:35:20 GMT

First, thanks to Fred Wilson and Bijan Sabet for standing up for RSS. You don't have to be a tech scholar to know that RSS is like XML or HTML or HTTP or text files. It's fabric, permanent, it ain't going anywhere. It's not like the Norweigian Blue, it isn't pining for the fjords or pushing up daisies. Let's keep a sense of humor and a sense of perspective. Some people look at life and see death. I look at life and try to be happy, cause yeah someday we all die, but that day hasn't come yet. (Knock wood and Praise Murphy.)

William Mougayar suggested in a comment on Fred's post that Twitter could have a button that allowed you to add any RSS feed to your Twitter stream.

I said "if they did that they'd have to absorb a significant share of the RSS flow, and that's the problem and why it's ludicrous to think that RSS is anything but the elephant in the room and the 800-pound gorilla combined."

"You can see it in the famous TechCrunch leak piece of the internal Twitter docs. They know they can't handle the load.

"That's why a distributed approach is the only one with any hope of working. The reason RSS could grow so huge is the same reason HTML and HTTP could, it's not centralized. That was the mistake Feedburner made. They thought 'Oh we can make a killing by snarfing up all the RSS.' No way Jose. That's a losing proposition. Luckily they got Google to give them $100 mill before the house of cards collapsed. They too put the brakes on growth."

Talking with Scoble last night, he offered that Twitter would have to grow to the size of Google to handle all of RSS. I agreed, but said it would be worse. Twitter would have to grow to be the size of Google overnight and without a revenue stream. Google got to grow with the web. Twitter wouldn't have that luxury relative to RSS. It became a giant long before Twitter even existed.

Which brings us to the Fail Whale.

RSS has grown in a fairly orderly fashion, quietly, without daily articles in the NY Times, or appearances on Oprah, or proclamations by athletes and movie stars. It also grew to huge size without a Fail Whale. RSS, in over ten years, has never gone down. Think about that for a moment. That's because it was designed for growth from day one. Getting on the RSS bus can be as simple as putting a file on your Apache server. It's just another rendering of your content flow. It requires a fairly small commitment, you don't need tens of millions of dollars of venture capital to build out an RSS network. You can rent it from Amazon at pennies per gigabyte.

Anyone who thinks Twitter could replace RSS doesn't understand the scale of the two things. Twitter is a fairly new company that has had trouble meeting the remarkable growth it has achieved. I use Twitter all the time, and I love it. I find that RSS and Twitter are a good combination. And I think news people create controversy for the same reason journalists have always sold war -- it gets people to read their journals, and their ads, and it makes them money. That's all that's going on here.

Interestingly, rather than Twitter absorbing RSS, it may go the other way. Perhaps RSS will absorb Twitter. That's the idea behind rssCloud. That a lot can be gained by creating a loosely-coupled 140-character network. Sure there will be tradeoffs, it'll take up to a minute for you to see your friends' updates. But there would be a lot of advantages, for example -- while one component of the network can fail, the whole thing is as resilient and distributed as the Internet.

All the angst over Atom

Sat, 05 Sep 2009 15:55:26 GMT

Yesterday at around 5PM, I added the code to the OPML Editor to support Atom 1.0 in River2, my new River of News aggregator. The coding took about an hour.

I tested it on some feeds from Blogger and Google News, fixed a few bugs, and burned it in overnight. It appears to work perfectly. So I released it this morning a little before 9AM.

The point? There were years of strife in the RSS world over this. In the end it took less than 24 hours, beginning to end, to support the new format. We could have saved all that angst. A new format isn't that big a deal.

1151 cloud-enabled feeds

Fri, 04 Sep 2009 18:59:41 GMT

I just added a changes.xml for my rssCloud server.

http://static.scripting.com/rsscloud/changes.xml

Now you'll see a whole bunch of cloud-enabled feeds that weren't there before. They're feeds for the 1151 people and organizations that I follow on Twitter.

They're automatically produced by an app that I've had running since the beginning of 2009 that keeps an XML-based backup for everyone I follow. This is just another form of that stream.

If you follow any of these people in River2, and you have notification turned on, you'll get updates within a minute, knock wood.

This is another piece of the loosely-coupled 140 character network.

Another way of saying this is if I follow you on Twitter, you now have a cloud-enabled feed. Try updating on Twitter, and refreshing changes.xml in a minute. If all goes well you should see your feed at the top of the list.

For a lot of users looking in, this probably means as much as the pictures of the Bay Bridge being taken apart and put back together. But when it's done, you'll be able to drive a car across it! :-)