Scripting News

Twitter clients could help with backup

Thu, 04 Jun 2009 14:09:13 GMT

I don't know to what extent Twitter archives my posts. For example, here's a blog post from January of this year. It links to a tweet, which is still there. Not sure if it keeps around older stuff, or how I would browse them if I wanted to see what I had written. The search command in Twitter stops at a certain point, exactly where that is -- I don't think anyone knows.

Uncertainty about what's backed up is a sure sign of a problem with backups.

Because I want a record of whatever I post to Twitter, I wrote an app that archives all my posts and those of people I follow. It's a very easy bit of code to write, since Twitter has an API call that returns all the recent tweets of all the people I follow, every client has to make this call regularly, so it has to be efficient, on both ends; and it is. I have shared the code, anyone can download it for free. It runs in the OPML Editor. But I think more developers should add this to their Twitter clients as a service to their users, as a competitive advantage, and as a way of making the work we do safer.

Right now we're all running without a safety net. Or more accurately, only Twitter knows how much of a safety net we have. And as we saw in the financial meltdown, it's not wise to assume that people we depend on to understand how complex systems work actually understand how they work.

In software, it's always a good idea to back up your work. And the people who make the popular Twitter clients could do a lot to help us there.

Discussion:

1. Some environments allow apps to write to local disks, and others don't. I don't know if Air, the platform many of the Twitter clients run on, allows this. If so, then I recommend that the clients simply maintain a calendar-structured folder of XML files containing each days' tweets, one file for each user. If not, then the backup has to be maintained in the cloud.

2. The size of these files is negligable in the age of MP3 and AVI. Text files are tiny and disks are relatively huge. Size isn't an issue.

3. Neither is performance. The file systems of today's computers are incredibly good at saving small text files.

4. It might add a little complexity to the Prefs user interface. At least it would require a panel that allows the user to choose a folder, and to enable or disable the feature. I would have it enabled by default.

5. You might want to allow the user to save his or her backup in Amazon S3 or to use FTP to upload to another server. Again, the overhead is negligable. I have the software running on my desktop system in the background. It's just an ordinary iMac. I don't notice any delays. Honestly.

6. What format to use? The simplest choice would be to use the XML-based format that Twitter itself uses. Other choices include RSS, Atom, OPML, or something of your own invention. I think RSS is the most rational choice, but I used OPML. I'm beginning to think that was a mistake, though I had good reasons for that choice at the time.

7. I also dereference short URLs and store both the long and short version. Wouldn't want to go to all the trouble of backing up the tweets only to find out the URLs broke because tinyurl (or whatever) went away.

8. The most basic reason to do this is backup, and that was the original motivation in my suggestion, in the summer of 2008. I suggested to the client vendors I could reach that they support RSS-based backup. That way, when Twitter went down -- as it was doing regularly then -- their users would not go down. But then Twitter started becoming more reliable so the urgency of this decreased

9. However, storing user backups first on the desktop, then in the cloud, those are the first steps towards an open, low-tech, simple form of federation that doesn't depend on a central node. If for no other reason, we as a community, should start down that road, asap. Murphy's Law says that at some point we will wish we had.

10. I'm sure there are other considerations, please post comments if you think of them, and I'll add to this list as I think of them.

Older people get to me too

Thu, 04 Jun 2009 05:13:07 GMT

Just read a wonderful post by Jackie Danicki, about how she chokes up in the presence of older people. Me too. I have tears streaming down my face as I read her piece, as I write this one, as I experience the memories it evokes for me.

A few weeks ago I was walking in my neighborhood in Berkeley, a hot day, and I came up behind an old man, all bundled up, walking slowly. The sidewalk was narrow, so I walked around him, and as I passed I said, gently "Excuse me." He jumped, startled and said reflexively -- "I'm sorry." In an instant I felt protective and sorry I hadn't found some other way to handle this. I did the best I could, and said something like Oh no, smiled and continued my pace.

He was skinny and fragile, exposed and vulnerable, and reminded me of my father, who will turn 80 this month. I felt protective for this man as I would if he were my own family. And was instantly reminded of something my father says often, that's worth remembering: Growing old isn't for sissies.

No doubt. But we, whose bodies still work, more or less, as they were designed to, can easily overlook that in every old body is a person who remembers well what it was like to be young. We are at a disadvantage, we don't know their experience, we get little inklings of it when we get sick, but we expect we'll get better. At some point, you no longer have that to depend on, and the quality of life must change.

Why do old people reach my heart this way? It could be their courage, or the inner strength it takes to compensate for the weak body. I don't know why they get to me this way. But they do.

Greetings to Chinese bloggers!

Thu, 04 Jun 2009 01:10:30 GMT

Twenty years ago today -- the Tiananmen Square Uprising in China. There were no blogs that day, but things have certainly changed.

Now so many blogs and social media sites are blocked in China.

Here's a tool that helps you determine if your site is blocked.

http://www.websitepulse.com/help/testtools.china-test.html

If you discover anything interesting, feel free to report in a comment here.

Suggested User List, reloaded

Wed, 03 Jun 2009 14:13:50 GMT

When the Guardian ran a piece over the weekend on the Suggested User List, my interest, and others' re-stoked. Then I got an idea from Sarah Delman, pondered it, wrote some code, published the results. Now we know more about the source of the SUL. More back and forth, some of it heated, then it comes together for me -- we lack leadership. I can't provide it, I'm not trying to. I'm trying to be an provocateur, a role I often cast myself into.

As I've said before, I believe Twitter is a chapter in a story that's been playing out for a long time. It's both the best news system and the worst. No pictures, no video, limited metadata, and it has an increasingly confining 140-character limit. But it connects people like no system before has. It's both the backroom for journalism and the delivery mechanism. A lot of power there. But imho it's not the last word.

The remaining news organizations will move onto twitter-like systems over the next few years. The news system of the future is electronic and real-time. The stakes are huge now. If I were in their shoes, I'd be thinking very hard about how I want these systems to evolve as environments for journalism. I'd stop worrying about squeezing Google for a handout and start thinking about how to grab some of the PE-ratio for myself.

But the lions of the news industry lack imagination and chutzpah. Where are the strategists, the bizdev people of the news industry? They're plotting paywalls, when they should be creating and linking new conduits with graphics, sound and movies.

The business model? The same one that served Google in its early years. People are so excited about what you're doing that they pour cash all over you.

The SUL is a small piece of the big picture, but it's an important one. For all the reasons I've said. No need to repeat it. Now I'm going to let everyone else worry about this for a while.

How Twitter is not simple

Tue, 02 Jun 2009 15:50:15 GMT

Early-on in the life of Twitter I tried to figure out how it works behind the user interface, and was told I couldn't, that it was very complex, and I shouldn't try. This bothered me then, and it bothers me even more now. After all, I'm an experienced software engineer, with a number of products under my belt, some quite complex. If I can't understand it, who can? And if they say I can't, do they?

We know the networks we're building on Twitter are very fragile things, but they're more fragile than most users have been willing to comprehend, because we can't find out how it works. We have to trust that the company knows, and that they will continue to operate it as they have in the past. However we don't have any right to that service, and if they restricted it or took it away we would have no recourse.

Of course it would be a disaster for them as a company, but such disasters have happened before and they certainly will happen again. And what do we actually know about the people who run Twitter? Not all that much. They don't tweet a lot, and when they do, they tend to be short messages, and often cryptic.

So while Twitter is useful and fascinating, even intriguing, is it simple?

It it not simple.

Who do the people of the NY Times follow on Twitter?

Tue, 02 Jun 2009 15:31:44 GMT

Yesterday I posted the results of a study of who the people who work at Twitter follow on Twitter. This is important because it is the source of their Suggested Users List, which in turn determines who new users will follow. People and organizations who are placed on this list see a substantial surge in followers, often in the hundreds of thousands. How much this is worth was the subject of a recent Guardian story.

It was easy for me to adapt the crawler to also generate a table for the people of the NY Times. I don't know exactly why this is interesting, but it seems people are curious to know if they make it onto the radar of the people at the Times.

If there's continuing interest I'll refresh both these tables from time to time.

Why simplicity matters

Tue, 02 Jun 2009 13:46:42 GMT

Jay asked, in a recent RTN podcast, why simplicity is so important. I don't think I had ever been asked this question, and it had been a long time since I considered any other way, so I didn't have an answer prepared. I fumbled.

It has bothered me ever since, but not as a front-burner thing -- I still accept the value of simplicity as a given. As I build layers of software, the simpler I make each layer, the higher I can build. If you don't design to hide complexity behind interfaces, you get overwhelmed by the complexity sooner, and your project can't do as much. Early in my career I often scrapped multiple levels and went down to the roots and rebuilt. Once I understood what the higher levels looked like, it exposed deficiencies in the lower levels. Reworking the lower levels allowed me to build higher. (It happens in other technologies too. For example, the design of office buildings changed radically once elevators had been invented.)

The ultimate interface is where you pass off to another user, and a whole new human brain comes into it. As a designer, I try to present a conceptual model to the user that's as simple and predictable as possible. This allows them to focus on their work without mine getting in their way. In the movies they call this "suspension of disbelief." The user gets so wrapped up in the story they forget it's a movie. It starts feeling like life. Even so, I like to create things that can be explored beneath the surface. If you want to "lift the hood" I know you'll see a puzzle, but I hope it's approachable.

As I discussed in this week's RTN, I'm working my way through Season 1 of James Burke's Connections series. It aired on PBS in 1978. Yesterday I watched the final episode in the series. It was even more fascinating than the previous nine. Its message is so important that it's worth taking time away from the innovations of our day and to talk about the process of innovation itself, and why it's dangerous. Yes, that's right -- dangerous. I've known this all along, and I've tried to publish warnings, but Burke does it with a breadth I've never approached.

There was a time, not too long ago, when anyone who wanted to could understand any invention that our lives depended on. Look around and look past the computer monitors and hard drives. Look at a door, for example, a hinge, a door knob, the door itself. All of this technology is transparently simple, and at one time, not that long ago, it was leading edge. I suppose paint might be hard to understand, but if you really wanted to understand how it was made, you could. It might take some time, and you would have to talk to an expert or read a book, but it wouldn't take much time to understand it.

But why bother understanding it? Burke asks. Isn't it enough to just use it and enjoy it and let it sustain your life? Yes, it is, until you have to make a decision about it. Or until it fails.

If you had asked me a year ago if we would live to see the financial system melt down, I would say yes, it's the one thing I'm most scared of. Like many others, I listened to experts talk about the aftermath of the collapse, reporters from famous newspapers, economists, radio show hosts. One thing I heard over and over was that no one actually understood how the system worked. Yet we had built our economic system, our lives, on it. Let me repeat that so it sinks in. No one understood how it worked.

It's only in its unraveling that we're learning how our economy works.

But that's far from the only system that no one understands.

Think about all the technology you depend on to live that you don't understand, and ask yourself if anyone understands it.

What if you went to the ATM and all your money was gone? What if everyone went to the ATM and all their money was gone? Think it can't happen? It did happen in post-Katrina New Orleans.

That's just one vector. Think about food supply, sewage treatment, water, transportation, education, law enforcement, health care, defense. We're making decisions about all these things every day whether we know it or not, and like the financial system, some people understand parts of these systems, but it's likely that no one understands the whole thing.

At the end of the series Burke lays out a set of choices we faced in the late 70s. We could continue along as we have been through all of history, or we could change course. It's now 30 years later and we didn't change course, and it seems we're not likely to. But if you're an innovator, you have a choice. You can create new things with an emphasis on being understandable to the people who use them, not just at the surface level, but under the hood too. It often takes more work to make it hard to understand. But even if it takes more work, it's worth doing it simply, helping create the sense that technology is understandable, because imho that's the key to moving forward in a way that might just work.

Update: Example of perfect simplicity. View Source in web browsers.

Who do the people of Twitter follow?

Mon, 01 Jun 2009 18:34:44 GMT

During yesterday's discussion of Twitter's suggested user's list, I got an excellent suggestion from Sarah Delman, CEO of the news-oriented startup, Of Record, Inc.

Here's the story...

We've heard that the SUL is like the list of books that employees of a book store recommend to customers. So this raises the question -- which "books" do the employees of Twitter read -- i.e. who do they follow, and how does that correlate to the list of Twitter users on the SUL?

I loved the idea -- it's my favorite kind of investigative reporting, because it involves programming.

Thanks to the Twitter API, it was actually possible to write a crawler that:

1. Generates a ranked list of who they follow and

2. Highlights the ones who are on the SUL.

I've written and run that crawler and the lengthy table on this page provides the result:

http://scripting.com/sul/twitterCorp.html

You can judge for yourself who influences the people of Twitter, and how that in turn influences the SUL.

There's a wealth of other information in the table, I've already spent a couple of hours pondering it, but it's time to share it with everyone else, and see what you all come up with.

Also thanks to Tom Reynolds for getting me the contents of the SUL late last night in response to a query I posted on Twitter. I didn't know where to find it, and it turns out it's hiding in plain sight on a page on twitter.com.

Also note that people come on and off the SUL. So there are people with high follow counts who are not on the list, who were at one time.

Update #1: There are four accounts on the SUL that have 0 followers at Twitter: Ali_Nejad, AstrobiologyNAI, LIVESTRONGCEO, PEOPLEPets.

Update #2: Who do the people of the NY Times follow?

Rebooting The News #11

Mon, 01 Jun 2009 14:25:49 GMT

Last night's podcast is up this morning, bright and early!

A theme from last week continues this week: Bug catching as a key practice in a re-booted system of news. Jay unfolds an example from this week: the AP's coverage of the Twitter TV show.

The TechGuardian asks How much is it worth to be one of Twitter's suggested users?

Dave discusses BitTorrent and why he put RTN 1-10 on it.

CheckBox News, Dave's mock-up of a re-booted user interface for television news where you can uncheck the streams you don't want and check the ones you do, and program your TV set that way.

For sources of inspiration (it's his turn) Dave returned to three: James Burke's public television series Connections, about the history of science and technology (inventions are usually the result of a synthesis of things created by earlier inventors); the Cluetrain Manifesto (ten years old and great); and VisiCalc, the first spreadsheet program for personal computers-- the demo for which was ~~almost~~ a spiritual experience. With a response from Jay about the common thread: distributing power outward from the insiders to the users.

Two views of the announcement this week that the New York Times had hired a social media editor, Jennifer Preston.

Dave argues that the great news organizations should be the operators and originators of systems like Twitter. It's not too late, but soon it will be, he warns.

We close with a short reading from Barbara Ehrenreich's commencement speech to the UC Berkeley School of Journalism. "A recession won't stop us. A dying industry won't stop us. Even poverty won't stop us because we are all on a mission here."

A question for journalists

Sun, 31 May 2009 20:01:32 GMT

I asked a question on Twitter: "An example of a non-monetary gift you couldn't accept from a company you cover? Why?"

Some of the answers...

Michael Calore: "copy of shrink-wrapped software i didn't review, comp tickets to a show/concert sponsored by a company i write about often"

Janet Ginsburg: "at businessweek (a while back) strong rule re no gifts. kept things clean. sm conf swag (pens, bags) but that's it."

Doug Levy: "just as physicians are inadvertently biased by trinkets like drug co pens, reporters need to vigorously avoid potential bias."

Freda Moon: "I was taught to accept no gifts from sources. None. Not tickets. Not a meal. Not even a cup of coffee. That last one can be hard."

Megan Taylor: "Tickets to sporting events. Reporters are supposed to be objective and accepting gifts compromises that."

There seems to be a consensus here.

Now a followup question.

Can you accept placement on Twitter's Suggested Users List if you are a journalist who covers Twitter?

Please this question is only for journalists.

The Times attitude

Sat, 30 May 2009 18:34:23 GMT

I don't know why I remembered this story last night, but that's when it happened.

When I was a kid, in sixth grade in PS 32 in Queens, there was a deal where we could get the NY Times for students. It was a special edition, with some of the big words edited out, or perhaps explained. I got it even though I had been reading the full adult version for a few years.

We also had a weekly quiz that was written by someone at the Times. It was multiple choice and it indicated (I guess) whether or not you had read the Times that week. There was one question on one test that really caused me trouble. What's the largest state in the US? There were four choices, and two of them were Alaska and California. That's a nightmare because they didn't say what the criteria was. Is it largest by area or population? So I chose Alaska, because I felt if you didn't say, it must mean area. Well that's not what they meant and I got the answer wrong. I protested to the teacher, who must have thought: Here's a teaching moment. So she asked me to write a letter to the Times, which I did. She mailed it for me.

Now here's the part that taught me a lot about the Times, and how adults can be ridiculously rude to children. I decided at the time to remember this so when I was an adult I would treat children with respect, which I really try to do.

The Times response came to my teacher, not me. They didn't like something about how I wrote my letter. And they didn't respond to the substance of the challenge. You may think it was a small point, but I was small then, and I wanted to know.

To this day, the Times has remained consistent and so have I. They generally talk over our heads, and respond to the manner in which the challenge is raised, not to its substance. I still keep trying to find new ways to approach this. I guess what I've been trying to say to the Times since sixth grade is this -- We are real people out here. Just like you. And we're smart. So let's talk, without the attitude.

An afterthought: Everyone is inside some things and outside everything else. I have a funny story to go with this. In 2004, I was at Harvard and Scoble was at Microsoft. I don't know how it came up but my longtime friend threw a bit of tech industry FUD at me, so I threw some Ivy League FUD back at him. I said that Harvard has been around since 1636 and pretty likely would be around long after Microsoft was gone, so don't try to push me around bud. We had a good laugh. (I think I also boasted that we had more cash than Bill Gates, but on reflection I doubt that was actually true.)

I also became friends with Chris Lydon, who had been a Times reporter in the 1970s. He clued me into some of the culture from an insider's point of view. I also got to know Rebecca MacKinnon who had been a reporter for CNN in Japan, Korea and China.

Rebooting the News 1-10

Sat, 30 May 2009 17:25:01 GMT

A lot of change is coming in the Rebooting The News podcast series.

The last episode was so good, it seemed a shame there wasn't a site specifically for the podcast. So I bought a domain and spent a couple of days creating a site for it. (Along the way creating some interesting new desktop software for managing podcasts.)

http://rebootnews.com/

There's a new feed just for RTN podcasts.

http://rebootnews.com/rss.xml

They will continue to appear in the Scripting News feed, so if you're subscribed to that, no need to change anything.

I also made a package of the first ten episodes and uploaded a torrent to Mininova.

http://www.mininova.org/tor/2637891

So if you've missed any episodes, or would just like to have a complete collection through last week, please go get it. BitTorrent is such a rational way to distribute content, and it's under attack by the entertainment industry. This is a perfect non-infringing application of BT, Jay and I want you to have our recordings, so go get em!

If you're new to BitTorrent, it's really easy. On the Mac, I recommend Transmission and on Windows, I use uTorrent. Just follow the instructions on the site to install, then click on the mininova.org link above and click on OK to the prompts that appear. It may take an hour or two to get all 135MB, so just leave it running in the background. Once it's finished, leave it running a little longer so that the next people can download it from you. It's a peer-to-peer system.

And don't worry -- I'm allowed to do this -- I created the podcasts (along with Jay).

Is there value in being on the SUL?

Fri, 29 May 2009 22:49:29 GMT

I think it's self-evident that there is a lot of value in being on Twitter's Suggested User List, especially for publications that run ads on the pages they link to from posts to Twitter. And many of the most-followed Twitter users do that. You can see that if you look at the main 100TWT site, the posts of the 100 most popular Twitter users.

http://100TWT.com/

I wrote this app so I could get a sense of how these feeds were being used. Some even run outright ads in their feeds, not links to pages that have ads. AdventureGirl, with 493K followers, is an excellent example.

Some people have said that being on the SUL is like being linked to, but I don't think so. There is no web equivalent to the SUL. It's as if Google seeded their search engine so that every web newbie, when they searched for anything, got 20 of 200 sites in every response. There would be no correlation between the sites returned and the query. Further, the power of these initial sites in recommending other sites would be almost absolute. New users wouldn't have any other way to find things on the web other than the first few sites they got in their very first Google search. It's so out-there that it's hard to even explain, the web could never have worked that way, people simply wouldn't have gotten the point.

Another experiment would be to walk around the office and ask every Twitter user if they would mind having more followers. Don't say why you're asking, just ask. If they all say they don't care how many people follow them, I'll buy every one of them a bagel next time I'm in town.

I couldn't discuss this on Twitter because there's no way to explore any subtle subject 140 characters at a time. But I'm willing to discuss it here, as long as we're uncovering new ideas, and not rehashing stuff.

Note: I originally wrote this piece to reference a specific organization, but on reflection I think it works better if it it's general. These are my opinions, I don't present them as gospel, as always your mileage may vary.

Discussions in Twitter, day 2

Fri, 29 May 2009 16:32:21 GMT

Yesterday I soft-launched a little project I'm working on called twdsc.us. Today the service gets a little more real, because now you can create discussions around tweets too, and not just your own tweets, anyone's. In other words, this could get interesting.

For example, I just started a small discussion around a post by WNYC's Brian Lehrer. Apparently they had a power outage, and that's interfering with all kinds of things, including their ability to post a podcast. Maybe someone in the community can help.

And if you look in the right margin of that page, you'll see a very brief HowTo explaining how to create your own discussion pages. If you have any questions, post them here, and I'll build how this blog post and turn it into documentation.

PS: A million thanks to the guys at Disqus who provide a wonderful and flexible tool, and support it like pros.

Google's killer app

Thu, 28 May 2009 19:38:22 GMT

In May 2001 I went for a visit to Google, which was then a very young company. I offered a list of ideas that we could work on together that would help bring their search engine closer to the blogging world, which was also then, quite young.

Then as now I saw the challenge as shortening the distance in time between a post and its appearance in the search engine. I called the idea then just in time search, it's the same idea that people call "real time" today. The term "just in time" was borrowed from the manufacturing world, where the goal is to have the components needed to build a product ready at the exact moment they are needed.

The meeting was cordially breathless, we loved Google then, it felt like it was our company, they were on our side in keeping the web open and the playing field level. But the ideas discussed that day weren't implemented. I don't think this was anyone's fault because then it wasn't so easy to see exactly what was necessary. This was quite a bit before RSS solidified behind version 2.0, and weblogs.com hadn't fully bloomed yet. It may not have been obvious how to do it then, but today -- I believe it is.

I described it to Michael Gartenberg in a phone conversation this morning. It took me about five sentences and less than a minute. That deserves a blog post, I thought.

Here's what I'd like to see them do:

1. When my blog updates I ping their server with the address of the RSS feed.

2. They read the feed, note the changes from the last time they read it, update their index.

3. On the other side, they allow users to subscribe to a set of RSS feeds they maintain, either on Google or elsewhere (using OPML subscription lists). Google provides an RSS feed that can be read every minute to get all changes to the aggregated feeds.

I know they already have elements of #3 implemented in Google Reader, that's good!

The whole thing needs to be tested, burned in and tweaked with clients that do more or less what Twitter does. How it works on the back-end is Google's business.

If they want to do more, I don't have a problem with that. This is the part my software will use.

I would like to see #1 based on the weblogs.com ping protocol, which included a REST interface. I'm not wed to any particular format other than it be as simple as it possibly can be and not dependent on toolkits or having any particular software installed. No opportunities for lock-in, nothing that isn't completely transparent and simple.

Then Google gets to do what Google does best, and what we depend on them for -- run a great search engine. They get all the updates flowing through their servers. Believe me, everyone will support this. And the advantages for users are manifold. Wide choice of software to use. You can have a 140-char limit if you want, or not, if you don't, again -- your choice. You can include media objects, RSS fully provides for it, or you can choose not to. The whole point is to keep it super lightweight and give you all the choices. Let the market figure out where it goes, so we don't have to wait for any one person or group of people to figure it out. That's the Internet way of doing things.

I can be happy that at least I've written down the idea where all can see it. I don't know if anyone from Google reads my blog, or cares. Microsoft or Yahoo could do it, I would be happy to work with anyone with the search and scaling ability.

Twitter as coral reef, cont'd

Thu, 28 May 2009 15:03:28 GMT

I've been feverishly experimenting some new ideas this week, the most interesting of which is a mashup between Twitter and Disqus.

This should make Fred Wilson happy, since he is an investor in both companies. But that's not why I did it.

Here's why I did it. The 140-character limit is driving me crazy.

I need a place to express ideas that just don't fit into 140.

There are some, believe it or not.

So here's an example. Early this morning Mike Arrington posted a teaser on his exclusive personal Twitter account. "Get ready for a very, very big news day." Well, Mike ought to know, everyone's telling him stuff they tell no one else.

So I wanted to post a comment asking What's up? What does everyone think this means.

So that's what I did.

http://twdsc.us/2.html

Note the URL. Cute, huh? It's a pre-shortened url. No need to push it through any of the commercial shorteners. New trend started by my friend Andrew Baron with his new superhot beta startup mag.ma.

Anyway. So far there are 11 comments with some very interesting theories about what's up. If nothing else, it's an inventory of ideas out there that people are expecting as announcements any day now. Google's realtime search engine (would be great if it supported RSS both ways and weblogs.com compatible pinging). Microsoft's new search engine Bing (for which expectations are really low, so it should be easy for them to impress). And on and on...

Add your own two cents.

And my next sub-project is to create a bookmarklet that makes it super-easy for anyone to start a comment thread on any Twitter post they like.

So Fred, whatcha think?

PS: Proof again that Twitter is the great coral reef of the latter part of this decade. It's so easy to attach something to it, that might turn into a branch or perhaps an entirely new species!

PPS: I'm working my way through James Burke's fantastic series Connections. Just watched episode 4, which ends with the beginnings of the modern computer. Hollerith, who invented the famous card that many people used to program Fortran and Basic (such as yours truly) decided to make them the same size as the dollar bill of the day. Because there was already so much machinery that existed to process them. Oh yeah. That's the kind of tech I love. Build on what's out there. More coral-reef thinking! Yehi.

Update: The big announcement is Google Wave. How much you want to bet in 5 years it'll be as famous as OpenDoc is today.

Rebooting the News #10

Mon, 25 May 2009 03:51:46 GMT

We got this one folks!

Topics include: Maureen Dowd of course, the Church of the Savvy, One year of Twitter for Jay. Why is user interface so damned hard? 10 years since Edit This Page. And an inspired choice for Inspiration of the week, Elvis Costello's recording of Nick Lowe's classic What's So Funny 'Bout Peace Love and Understanding.

One of the best Reboots yet, imho.

PS: As usual subscribe in your podcatcher or iTunes.

What are reading lists?

Sun, 24 May 2009 20:44:08 GMT

First an update on today's earlier blog post. Apparently Google Reader does not support reading lists. I think when I asked the question, the 140-character limit on Twitter made it impossible for an accurate answer.

No matter what, maybe it would be a good idea now to try to give a complete technical explanation of what a reading list is.

1. There are many kinds of "feed consumer" apps, all of them are capable of supporting reading lists, not just feed readers or aggregators. In the rest of this piece I'll use the shorthand "FC" to refer to a feed consumer app.

2. When you subscribe to a feed you're telling the FC that you want it to periodically read the contents of that feed and somehow act on the new items in the feed.

3. A reading list contains a set of feeds. The format of a reading list is exactly the same as the OPML-based subscription list format that's supported by many FCs.

4. When a FC subscribes to a readling list it does not import the feeds.

5. When the FC checks for updates, it checks for new items in the feeds in the reading list. Therefore it must keep a record of the feeds in the reading lists it is subscribed to.

6. If a new feed appears in the reading list, it does whatever it does for a new feed. Many FCs will consider all the items in the feed as "old" the first time the feed is read, esp if it's a podcatcher.

7. If a feed that was in the reading list has been removed, then the feed is not read and all record of the feed is erased from FC database, with the following exception. If a feed appears in two or more reading lists, a reference count must be maintained, and the feed is erased only when the reference count goes to zero.

8. Obviously the user can subscribe to as many reading lists as he or she likes.

9. The behavior I've described is how the NewsRiver aggregator that runs in the OPML Editor works. I suppose it's possible that other FCs work differently. If so, it would be great to hear about them.

I think that pretty much covers it.

Google's incomplete support of reading lists

Sun, 24 May 2009 15:24:00 GMT

Kevin Marks, who works at Google, tells anyone who will listen that Google Reader has a new feature that's exactly like reading lists, and that's a good thing -- because they are powerful and useful, and likely a key to making news reading work for more people.

Reading lists allow you to delegate subscription to feeds to experts. So for example, I could let Lance Knobel, an economist who I trust, choose the feeds I follow in his area of expertise. That way, when a new feed comes along, instead of sending me an email saying "Hey Dave you might want to subscribe to this feed" he can do it for me simply by adding it to his reading list.

Similarly, if a feed is no longer being updated, when Lance unsubs from it, so will I, automatically.

You can think of reading lists as a mutual fund of feeds. Busy people don't have time to research which feeds to follow and unfollow, so they delegate that to experts.

Another application -- the BBC has a large number of feeds, some for special events like the Olympics or elections. They could have a reading list for all their feeds, and when one falls off, they'd remove it, and when a new one comes on, they could add it.

It's an obvious extension to RSS, and to the ability to import and export OPML subscription lists. You can subscribe to a list of feeds in addition to individual feeds.

Now I'd love to provide reading lists for users of Google Reader, but I can't because they're using an incompatible format. This is absolutely the wrong thing to do. When asked to explain why, Marks gives a nonsense answer about the OPML Editor, which this has nothing to do with. It's always a shame when technologists, who have to answer precisely to the computer, use political spin when talking to users.

Further, if Google plans to challenge Twitter, as I've said I hope they do -- they will not get my support if they respond to Twitter's locked trunk with their own locked trunk. They must use RSS, OPML, Atom, everything they can find that there is even a bit of consensus for, including Twitter's API. They must achieve a remarkable level of compatibility to make the barrier to entry as low as it possibly can be and to send a signal that they just want to be a player in the market, not the dominator of the market.

Google's attitude in this area has been very unfortunate -- they've tended to be incompatible with existing formats whenever they can get away with it.

If Google had not invented a new format for reading lists, this would be a very different post. I'd be offering some reading lists of my own for their users to subscribe to, and encouraging my colleagues to do the same. I'd have written a howto that shows people what they need to do to create a reading list for Google Reader if they don't use Google Reader.

It's bad strategy to be gratuitously incompatible. It's also bad manners. Google was given a market for their reader built on open formats. They ought not consume that open-ness, they ought to at least preserve it, if not enhance it.

However, I will sing their praises if they fix their implementation to use the same format we use for their implementation of reading lists. If not, we'll wait to see what their efforts to compete with Twitter look like.

Maureen Dowd

Sat, 23 May 2009 07:00:00 GMT

I don't want to make a federal case about it, but I don't think the press did enough checking into Maureen Dowd's explanation of the plagiarism that appeared in her column last week.

Had it been a major political figure, say a Governor or the Speaker of the House, I doubt if a vague explanation about quoting a friend would have stopped the questions. (I'm thinking of Spitzer, McGreevy and Pelosi, just a few recent examples.)

Doesn't it beg the obvious question -- who was the friend? We should know if he or she corroborates Ms. Dowd's explanation. Clearly there was plagiarism, someone committed it. If this friend is a reporter or columnist, don't his or her readers have a right to know who they are?

And by the way, Dowd hasn't admitted to plagiarism. So if we're to forgive her, if this is one-time thing, doesn't she have to say: 1. She did it and 2. She's sorry. She's done neither.

This is all part of the problem with journalism today. Maybe it has always been this way and we haven't had the tools to communicate about them without going through them. Maybe they've always been lifting copy from other writers, and only now do we have the ability to report on them instead of reporting through them.

We haven't gotten the facts from the Times and Ms. Dowd. We ought to press for them, the way a reporter would press a political leader. We, the public, their readers, are entitled to know what happened and what their standards are for columnists. If plagiarism is okay, then who can do it, and how much. Guidelines, public, open and transparent -- are a minimum requirement. Then we can decide for ourselves how much we want to trust the Times and their columnists.

I didn't read a single report from another journal saying that what Dowd did is wrong and that her explanation is unacceptable, and that the Times is stonewalling, all of which seem obvious to me. I don't know about other readers, but it's this casual attitude, the inside-dealing I see both within the press and with the people they cover that makes me unenthusiastic for ideas meant to "save" them. I'm more into reformation. I want a new kind of journalism that sees incidents like this as bugs to fix. An opportunity to make journalism even more excellent, instead of ever more mediocre.