Scripting News

Boostrapping a decentralized Twitter

Mon, 05 May 2008 15:45:15 GMT

Overnight Mike Arrington weighed in on the decentralized Twitter discussion. I'm glad he is getting involved, he's a smart guy and is now using Twitter as an integral part of his communication system. But I have to disagree with the way he characterized my thinking.

I always work in bootstrapping mode, addressing the first big issue, solving the problem, then advancing to the next one. It's why so many of the ideas I've worked on have become popular modes of communication. Big-bang approaches always fail. I've spent decades arguing with people who want to reinvent the world in one stroke. They always try anyway and always fail. Bootstrapping is the only way that works.

BTW, I'm not the only one who believes in bootstrapping. Doug Engelbart, who invented many of the things we take for granted today works that way as well.

So the first step in decentralizing Twitter is to get our data safe and stored off twitter.com. Then we need discovery, a way to find people through Twitter, and then without Twitter. There are many ways to do this that work and scale (DNS for one, Skype does it too, without a central server).

It's also important that we work with Twitter and that they be rewarded for being the primary bootstrapper of this network. I think it's important because it's right, and also because we need to incentivize others to do the same.

Also, while others believe the conversational aspects of Twitter are primary, I'm not one of them. I buy into the original vision -- "What are you doing?" -- and also see it as a link-blogging environment. I have of course used it conversationally, I've replied on Twitter to others, but I don't depend on it because I think this is going to "spam out" -- in fact it already is going that way. Just in the last few days I've gotten replies from users whose Twitter streams look totally like splogs, and at least a few of them are clearly automated. I block every one of them.

Steve O'Hear: Respect what already exists. Amen!

Is Obama black?

Mon, 05 May 2008 14:35:06 GMT

Yesteray at breakfast at the Sunnyside Cafe in Albany, we arrived late, all the indoor tables were taken so we sat outside. It was frigid cold, for California, in the low 50s. I wasn't really dressed for it.

We started talking with the man at the next table about how cold it is, and I said it's nothing, I grew up in NYC and went to school in Wisconsin. The man, who was black and wearing an Obama for President button, said he was from the Bronx, we started talking about the hometown and the good old days (we're about the same age) and after a while talk turned to politics and he volunteered something that I found jarring. You know Barack Obama isn't black like I am. Hmmm.

He said Obama was raised by his white mother in Indonesia and his white grandparents in Hawaii and his father who was from Kenya was not an American. I've been to Hawaii, it's not like the Bronx or Chicago, LA or the Deep South where most black Americans live. If you look at a map you'll see how far away Hawaii is from the US mainland.

So what's the point? I don't know, but if if the tables were turned and we were electing the first Jewish president, but his father was from Israel, and his mother was Christian, and he was raised far away from the cultural centers of Jewish life in the US, I'd wonder how much like me he was.

That's all. I'm still voting for him.

IRC for Indiana/North Carolina

Tue, 06 May 2008 04:36:25 GMT

I started a chatroom for tomorrow's primaries.

irc://irc.freenode.net/#indianaNorthCarolinaPrimary

Please join if you want the firehose conversation!

If the Dems didn't have Superdelegates...

Mon, 05 May 2008 18:58:46 GMT

I was wondering if the Democrats, like the Republicans, didn't have superdelegates, where would the race stand right now. Here are the numbers...

At this moment, Obama has 1491 and Clinton has 1337.

There are 404 delegates remaining in 8 primaries and caucuses.

So with a difference of only 154 delegates, the nomination would not be decided.

However, to take the nomination by 1 delegate, Clinton would have to win 70% or the remaining delegates.

Why decentralizing Twitter is so important

Sun, 04 May 2008 20:22:29 GMT

At dinner last night, Scott Rosenberg, researching his history of blogging book, said he couldn't find any trace of the original version of Tim Berners-Lee's original site, info.cern.ch. I found this amazing.

When I was maintaining the What Are Weblogs page on weblogs.com, in 2000, I said up-front that TBL's site was also the first weblog. The crazy thing is I remember looking at the site, with my own eyes, and realizing that I was looking at history, like listening to the first telephone conversation or watching Thomas Edison turn on his first electric light bulb.

Today, in 2008, the network we're building with Twitter is imho as historic as any of these things, we're all creating artifacts and connections that are even more fragile than the early web, because, unlike the web, it's 100 percent centralized. We all trust the owners of Twitter, but they're human, even with the best intent, we all are taking a risk that the network could disappear at any time. And unlike the Internet which has huge amounts of redundancy built-in, if there's any redundancy in Twitter, none of us outside the company know about it.

This is just plain unacceptable.

I'm on the case because I care so much about this medium, and if it were to disappear, I would feel partially responsible if I hadn't raised a huge red flag warning about this very unreliable architecture we're building on.

And, if you know where there's a backup of the original info.cern.ch, please post a link here, in a comment.

Update #1: A new web service for Twitter clients.

Update #2: Marc Canter checks in.

Put this one on the calendar

Mon, 05 May 2008 00:35:17 GMT

We had RSS Awareness Day, that was fun, so let's have another new holiday, next Thursday, May 8 is Chickens Come Home To Roost Day.

You have to fit the phrase into conversation at least once during the day. Example. "It's bad design to put all your eggs in one basket. One day your chickens will come home to roost."

Sunset over the bay

Mon, 05 May 2008 00:16:40 GMT

Taken last night on Indian Rock.

A view of the back of Indian Rock on Google Maps.

A new web service for Twitter clients

Sun, 04 May 2008 19:00:45 GMT

Yesterday I wrote about a way to prepare to decentralize Twitter, in the event of a lengthy outage. The goal is to create no extra work or complexity for users. I think this is the responsible way for developers to help because it's 1. Not a good idea to build a centralized system around a for-profit company and 2. Users generally won't do anything extra to decentralize to prepare for an outage, but when one happens, they blame us (technologists) for not protecting them. Right or wrong, this is the way it is. So I'm working on a step-by-step bootstrap that, if enough developers go along, will have us reasonably protected against a prolonged Twitter outage. It's not to say that it's the only way to do it, but it seems to me that it's one way.

I said I might put up a web service to store user's RSS feeds on Amazon S3, and I'd pick up the hosting bill, to help the bootstrap. One developer took me up on the proposal, so I went ahead and implemented it. Here's how it works.

1. There's a new XML-RPC service at this address: xmlrpc:\//rpc.twittergram.com/RPC2

2. The name of the procedure is twittergram.saveFeed.

3. It takes three params: The user's Twitter username and password, and the text of the feed. The password is only used to authenticate, it is not stored on the server.

4. It returns the URL of the feed as its stored on feeds.twittergram.com.

5. Code (in UserTalk) that works.

local (server = "xmlrpc://rpc.twittergram.com/RPC2")
local (username = "davewiner", password = user.twitter.prefs.password)
local (feedtext = tcp.httpreadurl ("http://twitter.scripting.com/daveRss.xml"))
local (url = [server].twittergram.saveFeed (username, password, feedtext))
webbrowser.openurl (url)

6. You can call the routine at most once a minute. This may be increased if it becomes a popular service. My server is limited to 70 calls per hour. Again something will have to be done if it becomes popular.

Microblogging should be decentralized

Sat, 03 May 2008 21:07:04 GMT

Scott Hanselman asks the question that should be on all our minds, as we come to depend more and more on Twitter. We need to do something about our over-reliance on a centralized system run by a for-profit company.

Read his whole post and think about what you can do.

One thing right off the bat -- if you make a desktop tool for Twitter, you can offer the user the option to store their twitstream as an RSS feed. Just do it in parallel, transparently for the user (although it's a preference). You can key off their Twitter ID. If you want I'll set up a service for free hosting of the feed on Amazon S3 (it's not a very expensive thing) or it's something you could provide as a bonus feature.

It's a step in a positive direction for decentralizing. It's not the whole thing, but it's a big part of it. And should Twitter ever go off the air for more than a few days, it'll be the way we put the network back together.

Update #1: I took a walk, and thought some more about this. If there were a way to point, from your Twitter account, to an alternate feed, then your desktop client could cache a pointer to the feed for each person you're following. If Twitter were to go down, then the desktop client would fall back to polling the feeds. It would probably be slower, but it would work.

If one of the people you follow didn't have an alternate feed, you'd have to wait for Twitter to come back up to find out what's new with them. But if they used a desktop client, and the client was maintaining the feed automatically for each user (subject to a pref), then it would notify Twitter where it was storing the feed. Twitter would just have to maintain one more string for each user, alongside the user's location, link to their website, one-line bio, etc.

However: It would require support from Twitter, Inc. to work.

Rev Chickens-Come-Home-To-Roost

Sat, 03 May 2008 19:43:52 GMT

We wasted another week on Rev Wright, hopefully the last one. No it didn't drag Obama down, though the right wing is spinning their wish that it would. If only. Keep dreaming.

Yes, Obama is a gifted politician, and that's why we like him. We need political leadership, we've done without it for the last four terms, sixteen years, and we hope that this guy really gets that the power is with us, that we pay the bills, fight the wars, and it's our hard work and innovation that drive the economy that makes America powerful when we are powerful.

Wright? He's a deer that got caught in a headlight none of us have ever been in, so we don't really have a right to judge him. He spun around looking for friends, and found he was radioactive, and it really wasn't his fault. He wasn't running for office, in a rhetorical way he totally did inhale and exhale and inhaled again, and again. The only people who really liked his radioactivity were people like Louis Farrakhan and his followers who want to disrupt the US political process.

So what to do with the Wright legacy? I feel sorry for him, but I hope he finds some good people who aren't out to destroy other good people and he skips writing the book, and he kicks back and lets the member of his flock take the leadership we want and need him to.

See also: Bill Moyers, yesterday, on Rev Wright.

What is a Dual-WAN Router?

Sat, 03 May 2008 18:23:11 GMT

Imagine that water pipes were a new thing, and therefore not reliable. They work for most people most of the time, but sometimes they go down, and then if you want to take a bath or wash the dishes or cook a meal, no luck.

Then you learn that there are two types of water pipes, run by different companies, and it's very rare that both go down at the same time. So, if you can afford it, you get both water suppliers to pipe into your house, and when one goes down, you go to the street, lift a manhole cover, go down a ladder, and disconnect the one that doesn't work and hook up the one that does.

It's a hassle for sure, but you always have water. Then someone invents a T-shaped pipe with two inputs and one output. You hook the two suppliers each to a branch and your house to the third. This way when one goes down, you don't even know it. The water keeps flowing, you're happy.

It works the same way with the Internet. Some days Comcast goes down, and others AT&T DSL, but they rarely go down at the same time. I have both because the Internet is still young and unreliable, and I've lived with its lack of reliability before and it's worth a hundred bucks a month to have the luxury of uninterrupted service when one or the other goes down. But until yesterday, if I needed to switch over, I'd have to rewire my network just to switch to the other vendor. But, it turns out they make T-pipes for the Internet, they're called Dual-WAN Routers. And I got one.

It took a fair amount of fussing-with to get it working, but it has a nice fractional horsepower HTTP server to configure the router, a good help system, and I kind of intuitively get what it does.

Comcast is going to like this, their service is so much faster than AT&T's that I set it up so that AT&T is the backup. I won't actually send any packets over their line unless Comcast goes down, or gets pissed at me (it happens) -- but they should note that turning me off will no longer get my attention.

And thanks to my parents for buying this lovely gift for me. How did they know I wanted one? Another innovation, Amazon Wishlists. This Dual-WAN Router thing isn't something I would likely buy for myself, too speculative, I wasn't at all sure it would be simple enough to set up, or that I really needed it. There is a recession going on after all. And they never would have had a clue that I was interested. But now I have one and I'm very happy to have it.

Why did I decide on Xincom? I read the reviews on Amazon and it seemed the most likely to work.

Sunset at Indian Rock

Sun, 04 May 2008 03:40:12 GMT

A bunch of us went up to Indian Rock to see the sunset.

Click here for the set.

Yes Virginia, there's oil in Iraq

Sat, 03 May 2008 02:24:24 GMT

And we wouldn't have our military there if there wasn't.

Everyone knows the war in Iraq is about oil, but if a Democrat were to say it the Republicans would challenge his or her patriotism. But what if a Republican said it? What if John McCain, the Republican candidate for President said it? In public, on camera, with mikes, clearly, unambiguously?

You might think it was major news, at least on the same order as whether Obama wears a flag pin or properly salutes the flag?

Would it be news that over 4000 Americans and at least $1 trillion has been piddled away on a really bad deal? That $1 trillion could have bought a lot of oil. Or it could have built some great mass transit. Or it could have paid for Republican tax cuts. Or whatever, who cares -- it's news dammit.

No the press isn't likely to report this latest McCain truth-leak, that's why it's so important that we give generously to the DNC so they can run ads that do their work for them.

I gave them $100 today. For now, I think my donations to Obama aren't going to change things very much, so I'm switching my donation-flow to the DNC. Their ads defining McCain are good, they're accurate, and they work. They just have to be run in more states more times.

BTW, MSNBC is being way too soft on McCain when they say he "seemed to suggest" -- the usual press BS re McCain. He said it, so report it, without the double-qualification. He didn't "seem to" and he didn't "suggest." He said it, so say he said it. They're going to soft-pedal it for sure. Arrrgh.

Update: Cross-posted at Huffington.

Local fiber

Sat, 03 May 2008 02:15:58 GMT

Lots of information in the comments on yesterday's post about city-owned fiber in Berkeley.

UPS truck came

Fri, 02 May 2008 23:48:07 GMT

And it delivered two geekish presents.

I'm installing the Eye-Fi card right now. Wish me luck!

It's going pretty well. Now they want me to connect my card to one of the photo utilities, like Facebook, Flickr, TypePad, Photobucket, Windows Live, Costco (!) and many many more. What they don't tell you is if every picture you take will be uploaded. That could be pretty embarassing and since my pictures flow to Twitter through TwitterGram, that could be a problem too. A Help option here, explaining, would be useful -- but I'm going to assume that only some of my pics get uploaded and wire it up to Flickr.

They also don't tell you whether you can create a new account from this page.

Answer to the second question -- you can create a new account.

Okay, I followed their instructions, took a few pictures of myself, while I was away the browser (Firefox) crashed. I'm trying to get back to where I was but keep hitting this error page.

Ooops, it seems to upload all pictures. That's not good! I'm going to turn that one off.

Looking for the folder on my hard disk. Nope. Not there.

I'm still the best guy around for breaking software.

Okay slow down. You have to leave the camera on so it can upload the pics? I guess so.

Postscript: I don't think the Eye-Fi is designed to work well if you take high resolution pictures, which I do with my Canon. I'm going to try using it with my Nikon, which I have set up to take low-rez pics.

Tom Hunt's FlickrFan discovery

Fri, 02 May 2008 17:11:02 GMT

The other day fellow Berkeleyite Tom Hunt, who uses FlickrFan, came up with a neat way to solve a common problem that I thought I should share.

Here's the problem. You're watching pics go by and you see a picture of President Bush speaking at a conference on Children and Faith-based schools and think, man, that's an interesting picture, I want to show that to a friend.

So you open up the pics folder and see a lot of pics with names like pic091337.jpg, pic091338.jpg and pic091339.jpg. How do you find the one you're looking for?

Well, there is a way, believe it or not...

Open the screensaverpics folder and type Faith-based schools in the Spotlight box in the upper right corner of the window. 13 pictures show up. Select-All, right-click, Open in Preview. Review them and voila, there's the one I was looking for.

How did that work?

Well, AP and AFP put metadata on their pics, and the FlickrFan process preserves it, and Spotlight can search it. Here's the Get Info box for the pic I was looking for. The red arrow points to the metadata.

Tom is a smart guy, and a generous one. Thanks for passing this tip along!

Government could help us use less oil and save money

Fri, 02 May 2008 16:27:21 GMT

McCain and Clinton propose a moratorium on Federal gasoline tax for the summer to give the "average American" a break. It must test well with focus groups, but it's meaningless, because the prices would immediately adjust. If you lower the price, people will take longer trips, drive instead of fly, increasing consumption, driving the price up. There might be a bit of a benefit to drivers for a very short period of time, but in the end it would be a wash. This is what economists say, and it makes sense.

President Bush wants more exploration, and to build more refineries, these are medium to long-term things that likely wouldn't do anything for us right away.

But wouldn't switching to smaller European style cars do more to ease the problem than increasing exploration or creating more refineries?

And the money we'd give up for Federal gasoline tax could be better spent on putting high capacity network lines under our streets to increase communication. Some of the car trips must be to exchange information that coud be replaced by moving packets around at gigabit speeds. It wouldn't cost much to retrofit a few cities with really high speed lines, then we could get to work on developing the services that would make life more interesting, fun and efficient.

Interesting meeting today

Fri, 02 May 2008 04:23:53 GMT

I had a chance meeting with some people today who wanted to talk about bringing fiber to Berkeley. There's an idea that it could be done in a very Berkeley way, which is to say, the city owns it, and the service providers can tap in, but they don't get to dictate terms, as we know many of them like to do. I don't know how exactly I got enlisted, but I guess we've got enough Internet celebs in Berkeley to maybe get this done right, politically. Not sure what comes next, but it sure is interesting.

Speaking of Berkeley celebs, I tentatively have a podcast interview with George Lakoff tomorrow.

Snicker snicker heh

Fri, 02 May 2008 04:09:55 GMT

Twitter broken?

Fri, 02 May 2008 02:38:53 GMT

It's been hard to get through to Twitter today.

Calls through the API fail.

Most accesses to the site get an internal server error.

Nothing about it on the Twitter blog.

Oy.