Scripting News

A way for Twitter back in the pink?

Tue, 01 Jul 2008 03:18:31 GMT

I'm not sure how much of the stress in Twitter is caused by the services that poll its API on behalf of thousands of users, but it's got to be a lot of work to service all those requests that are constantly coming in.

Here's why it has so much work to do. When I post something to Twitter, within a couple of minutes it shows up on FriendFeed. I don't know for sure, but I bet that it's calling the Twitter API every few minutes to ask if Dave has posted something over there. Most of the time the answer is no. And it's asking for each of the thousands of FriendFeed users that have connected their Twitter accounts to their FriendFeed accounts. Wouldn't it be simpler for FriendFeed to say to Twitter: "Here's a list of all the FriendFeed users who want to have their twits reflected over here." Then Twitter could call FriendFeed saying "Yo, Dave just updated and here's what he said." Don't call us we'll call you. It's often more efficient.

Back in the old days when I used to work on much larger systems known as mainframes, they had special-purpose computers whose only job was to offload work for the main computer, much the way a booster rocket or a tugboat help a space ship or an ocean liner. In computers they were called TIPs which is an acronym for Terminal Interface Processor. Each user sat at a terminal, a sort of dumb computer that behaved like a printer, and typed away, and then the TIP would talk to all the terminals, and then talk to the mainframe in a language only the two computers understood. It was much more efficient for the mainframe. Seems Twitter could use that kind of efficiency.

There's lots of this kind of connecting going on these days, and it is costly. It slows systems down. Probably the way the problem is going to be solved is through something like the TIPs, adapted to the 21st Century.

Just a thought for a possible way to make Twitter a little more perky.

PS: In 1997 I knew Apple was going to fire its CEO, I had been brought in, in confidence. The morning of the announcement, I wrote a Wired column (published on the web) calling for his resignation. It ran two hours before the announcement. Some people mistook it for cause and effect.

Looking for a few good feeds

Mon, 30 Jun 2008 22:28:47 GMT

I'm revamping my feed reading.

FriendFeed has made me (and apparently others) much more aware of how I get my news.

I've also learned a ton from the NewsJunk project. I get much better political news now than I ever have, and it's getting better all the time.

Something I've learned...

The thing that makes the difference: GOOD FEEDS.

Behind those feeds of course are honest, smart people with a passion for information.

I started NewsJunk because I was getting terribly spotty news about politics. I asked how other people get their politics, and everyone said the same thing, they hunt and peck. Now I get a steady stream of great stuff. It's like the briefing books political candidates get from their staff, but open to everyone. When a story breaks I get a bunch of perspectives. If I'm not interested, I don't click, but in an instant I have a sense of what's going on.

And it's a level playing field. If a story breaks via pro or amateur, we get it. Fast. No waiting. (When we're doing our job.)

Now, I want to straighten out my access to news about technology.

In a word, it sucks!

I want it not to suck.

As much.

Tech news is different from politics though, most people in the tech world, the insiders, hit TechMeme at least a few times every day, I do, at least 20 or 30 times. I don't want it to change, it serves a very useful purpose. But it isn't enough.

What I want is what I've always wanted: News about products. New products. What people think about products, but features added to popular products. And not just the really huge products, like GMail and Amazon. I use lots of stuff. You should see my bookmarks and my system tray. And some of the products I'm interested in aren't even in my Bookmarks. Earlier today Steve Rubel wrote about Summize and a neat new feature they just added. It's a really small thing, but I care about really small things. I make and products for a living. Ideas are important. And someday I might meet the guy who did that, and I'd like to know about it so I can congratulate him. The personal touches matter. People care that you notice. I certainly do!

You know what else I like -- hearing about products from the person who implemented it. What were they thinking? What were their goals? What were they surprised by when people used the product? What questions do they have? You can learn a lot by listening to the person who wrote it.

Anyway, I want to know about products. Today I found two blogs that are devoted to reviewing tech products. I added their feeds to my mix.

I want to know what you rely on for product news, and I want to start reading what you read, voraciously. And I don't just want to read it, I want to consume it.

So please, if you feel so inclined, either post a URL of a favorite product-related feed in the comments here or send it to me at scriptingnews1 at gmail dot com.

Thanks!!

"cheesecake"

PS: If we can improve the flow of news about tech products we can create more opportunities for tech products. I'm sure there are niches we're missing, big ones, but they're hard to see because the picture has been muddied up by all kinds of peripheral stuff.

PPS: One of my inspirations for this work was a post by Fred Wilson where he said he wanted a TechMeme for inspiration. I don't think it'll end up looking like TM, and your source of inspiration might look very different from mine. We've gotten too centralized, imho -- we'll now get more decentralized. Pretty sure I see how it could work.

Web 2.0 gas prices

Mon, 30 Jun 2008 03:44:00 GMT

Earlier today we were having a hot debate about how John McCain doesn't know how much a gallon of gas costs. A Republican thinks we're being too hard on old John. I thought not, what single fact could you expect someone running for President to know? It's like asking the manager of a baseball team their percentage (the number of games won divided by the total number played). Or asking a batter how many RBIs he has. A president should know what gas costs, as would the CEO of an airline or car company. It's a very basic indicator of what's going on. You can't even go to war (something McCain is proud to say he knows something about) without gas. Lots of gas.

You could forgive him for not knowing what a gallon of milk goes for, you'd have to actually go inside a store to find out, but the price of gas is displayed prominently on street signs. All he has to do is look out the window of the famous Straight Talk Express.

Anyway, we did a little checking, found an MP3 of the interview where the question came up, verified that the transcript was accurate. (Yeah, if you want to split hairs, he wasn't asked if he knows the price of gas today, literally, just if he knew the price of gas at any time in the past. Lawyers everywhere.)

Then I went looking on Google Maps for a Street View of a gas price sign on a station at San Pablo and Marin Ave in Albany, an intersection I go through frequently on my way to San Francisco or the South Bay or the movies. Later I was waiting at a red light at that exact spot and thought to take out the camera and take a picture of the sign today. Uploaded it to Flickr. The prices had changed quite a bit!

What a world we live in. Gas is ridiculously expensive. But the Internet keeps getting more interesting.

More than meets the eye

Sun, 29 Jun 2008 16:18:00 GMT

This morning, the story I've been tiptoeing around here appeared for the first time in the business press.

Guardian: Shel Israel puppet show bites the dust.

There's an undercurrent to the story that insiders will understand that I don't want to explain here at this time.

People need to do some soul-searching, now, and then do some damage control before this gets much worse.

State of the Twitter, June 2008

Sat, 28 Jun 2008 16:22:00 GMT

June was a terrible month in TwitterLand. The service was down a lot. It's basically down right now, has been for days -- since the Replies tab doesn't work.

I've never seen anything like it. A service so many people use that can't stay up.

Yesterday I got an email from Jay Rosen asking if this was the day Twitter died. It had completely gone off the air. No whale, no features taking a rest, the server wasn't responding at all. I posted a message on FriendFeed linking to the Don McLean song American Pie that's about "The Day The Music Died." Yeah, yesterday might have been the day that Twitter died.

Fact is, Twitter as it was conceived was never meant to live.

It's very possible with better engineering its architecture might have gone on for a few more years, but eventually it would have hit this wall, where there were too many people posting too many twits to too many followers. The scale of the system as conceived rises exponentially. Just look at the spewage report for a sense of the scale.

So I started arguing for a decentralized system, and the engineers at Twitter sniffed that you would never be able to recreate Twitter in a decentralized fashion. I still doubt that's true, but now we have a counter-argument -- you couldn't keep it running in a centralized fashion either. It may just be too rich an application for today's computers. To a user this seems ridiculous -- it doesn't look rich. I guess sometimes appearances can be deceptive.

So the conversation moves to FriendFeed. True, I am ignoring the flow I have on Twitter. Easy come easy go. The flow there is pointless. It's like trying to make a baby by having sex with a rock. First, it's hard to get excited. And second, no baby.

And FriendFeed is a much better place for conversation than Twitter. No 140-character limit (they do have a limit, but it's much higher, so high I haven't had a reason yet to figure out what it is). And most important, with 10K-plus followers on Twitter, when I respond to one person's question, all 10K see the response and some get annoyed (a certain percentage say so) or ask what we're talking about. If I answer their question, I'm annoying and confusing a bunch more people. Conversation was awkwardly grafted onto Twitter as an afterthought. It seems to fit in better with FriendFeed.

However, before we all move to FriendFeed and think we've solved anything, this underscores the problem with putting all our eggs in one basket. We just move the problem into the future. FriendFeed may be able to scale where Twitter can't, but there are other problems with centralization, putting all your trust in a corporation, esp one so young and unformed. Instead, we should start bootstrapping a decentralized Twitter-like thing immediately, building off the base of clients that connect to Twitter. It can connect to any service we want to connect to, and if one should go away, we do the thing the Internet does so well, route around the outage. I wrote about this, extensively, in early May.

PS: I implemented my own suggestion. Here's my RSS feed of today's Twitter posts.

PPS: At 7:40PM, replies in Twitter are back. Now we get to find out if our fling with FF is the real thing, or just a summer love.

Classic geek video

Sat, 28 Jun 2008 05:44:11 GMT

Much-discussed today on FriendFeed, with art.

10 great movies in 10 genres

Sat, 28 Jun 2008 03:09:07 GMT

From today's Fresh Air, a selection of 10 great movies in 10 genres from the American Film Institute: Animation, Romantic Comedy, Western, Sports, Mystery, Fantasy, Sci-Fi, Gangster, Courtroom Drama, and Epic. I love resources like this, cause there are bound to be some movies among the hundred that I haven't seen. Maybe you'll find some too.

To Obama: I'm not an ATM

Fri, 27 Jun 2008 15:16:06 GMT

I just got an email from David Plouffe, the campaign manager for Obama. Click on the image below to read the email.

I've underlined in red the part of the email that got me to write this angry blog post.

When I saw the email in my inbox entitled Strategy Briefing For You I thought for a brief instant that the Obama campaign had figured out that I have a mind, that I have an education and a resume, and I might be someone worth briefing. Three paragraphs later the disappointment hits. Watch the video then give us money.

I (like to) think Obama needs more than my money. I think Obama needs my mind and my influence and experience. My creativity. I think Obama might, from time to time, want to brief me, without asking for money. I think Obama might want to invite me to a meeting of people from Berkeley or Northern California or the tech industry, or academia, or any number of my other affiliations (Bronx Science alumni?) where people put their minds together and think about ways to co-create a new America.

The primaries are over and he won. There's one more hurdle and he'll be President. Yes, he's got my vote. He probably will get my full $2300. Does he want anything more? My guess is that honestly, no more than Clinton or Bush did. Sorry to say, but that's how it seems to me. Still a little time to turn it around. But the voter as ATM thing is wearing pretty thin.

PS: ATM stands for Automated Teller Machine. Someday soon some kid will ask "What's a teller?"

PPS: I had the same epiphany about public radio in 2003.

What is whoisi.com?

Thu, 26 Jun 2008 22:46:47 GMT

This site showed up in my referrer logs.

http://whoisi.com/p/216

http://whoisi.com/p/755

Not sure what to make of it. Looks quite interesting.

I hadn't heard of it till today, but I see it's being discussed on Twitter.

Here are some other people...

http://whoisi.com/p/459

http://whoisi.com/p/1

I think I get it -- it's a wiki-like FriendFeed?

http://whoisi.com/p/683

http://whoisi.com/p/141

The opportunities for abuse abound (but there are obvious ways to fix things, if you claim your own person, and correct the links). It's very clever. Why didn't I think of it??

One of the things I love about it is that it does the right thing with RSS descriptions. Bravo!!

"cheesecake"

Some things shouldn't be joked about

Thu, 26 Jun 2008 22:32:03 GMT

It's been suggested that McCain made a good choice in hiring a comedian to write about Barack Obama for their campaign webiste. I humbly disagree.

There are some things that you shouldn't joke about. For example, tech support. How would you feel if your server had crashed and it turns out your ISP was playing a joke on you. Come on lighten up! Read the Cluetrain.

Or suppose your doctor was playing a joke on you when you went in for your prostate exam and hid a little treasure for you to find. Relax! It's a joke!

Presidents have buttons that launch missles that destroy the world. Their power is even greater than doctors and tech support people. It's better if they stick to telling us what they think without misdirection. Imho.

Good news or bad?

Thu, 26 Jun 2008 20:25:07 GMT

I just signed on to Twitter and there were two status messages waiting for me and that was it. The entire Twitterverse had shrunk down to Charlene Li and Josh Bancroft. This is a new idea. An interesting plot for a science fiction movie? Or a sad comment on the times? I hope they like each other? Maybe one is a Republican and the other is a Democrat? I wonder what their offspring would look like?

John McCain's RSS feed

Thu, 26 Jun 2008 19:40:39 GMT

I forget how I stumbled across John McCain's RSS feed, but I've been reading it regularly for some time. Occasionally they have a post written in the candidate's name, but usually the stories are written by staffers.

I was hoping, when I found it, that I could learn more about the candidate, but mostly they use the feed to take shots at Barack Obama. A lot of it is very embarassing stuff, not for Obama, but for McCain. How could a candidate of his stoop to sarcasm bordering on bitterness. It sounds terrible, like they've already lost, know it, and all they have left to support their candidate is -- what? Do they think that undecided voters would be swayed by snark?

I'd like to see them use the feed to comment on current events, as they sometimes do. For example, today the Supreme Court released its first interpretation of the 2nd Amendment. No matter how you look at it, this is historic. It's not quite as big as overturning Roe v Wade, but it's in the ballpark. McCain ran a piece about the news that stated their position, and contrasted it to (their interpretation of) Obama's, without insulting the reader's intelligence. It was published in the candidate's name. Good use of the feed.

Later in the day they published a hard to parse piece that starts out as a criticism of Karl Rove and then attempts to defend Obama for rewriting the Declaration of Independence! And at first I thought they were serious in their denunciation of Rove, which would, imho, be a very smart thing politically. But that wasn't their point, at all. I don't think Obama has said anything about the Declaration, I guess this was a weak attempt at humor? If so, it didn't work. Screen shot. This piece was written by Michael Goldfarb. Not sure who he is.

I guess my message to McCain is this --> being President is serious stuff, and if you don't take it seriously, how could you expect anyone to support you?

BTW, the same criticism goes to Daily Kos and Redstate, two highly partisan blogs at opposite ends of the spectrum. I hardly ever refer people to either blog, because they always take cheap shots along with stating their interpretation of current events. But they're just blogs, two of many. McCain is just one of two major party candidates for President. There's a big difference.

PS: If anyone knows of an equivalent Obama feed, please let me know!

Silicon Valley as second grade

Thu, 26 Jun 2008 15:50:42 GMT

Posting a link to Shel Israel's piece here yesterday accelerated the discussion, of course. Most of the discussion that I've participated in has been on FriendFeed. I also talked for about 45 minutes last night with Mike Arrington. It was a surprisingly friendly conversation, I had forgotten how much I like and respect him. After sleeping on it, I've had a chance to distill my own thinking. Here's some of it.

First, when I became aware of how the videos were hurting Shel, I stopped watching. All I could think about is how mean this community had become. Most people had never heard of Shel before, he's not really a celebrity. That was until these people decided to make an example of him, and turned his name into a bad joke, which became more well known than the real person. Shel is far from rich, and this isn't just hurting him financially, it's breaking him, though he's too proud to say so.

Now they've gone after me too, but it's not so easy to hurt me. I've been trashed plenty, and I think most people whose opinions I care about know that I am not what they say I am, which can be pretty awful stuff.

As Duncan Riley said, one of the few bloggers who has been willing to come to Shel's defense publicly: "If I was Shel, I wouldn't be coping at all, in fact I'd probably fall to complete pieces." True. It's enough to wither your spirit. Not the satire itself, but the people who say they're friends who don't offer support. That's what really hurts. That's one of the things I tried to convey to Mike last night. I offer the same to several other people I'd like to call friends again.

When a friend is in trouble and asks for help, you don't turn your back. In fact, I'd go so far as to say that when a stranger is in trouble and asks for help you don't turn your back.

Satire that's based on hurting people stops being funny to most people pretty quickly. People who support it really need to stop and think how they're contributing to other people's misery, and whether it's still fun after realizing that. I believe that most people are good at their core, and when they give it some thought, will help us turn this corner and get to the next level. We've sunk really really low. Time to pull ourselves out.

PS: If you think writing this was easy, think again.

PPS: The First Amendment says you have the right to say (almost) whatever you want. But it doesn't say anyone has to listen.

California weather

Thu, 26 Jun 2008 03:10:47 GMT

Just a few days ago we were sweltering with near-100 degree heat, and today, it's so cold out we have the furnace on. I'll take the cold weather any day, but it would be nice to settle into a summer weather pattern at some point.

Arrington, Feldman & Israel

Thu, 26 Jun 2008 00:35:56 GMT

I'm forwarding this link to my readers, without comment, at this time. Please read it and give it your consideration.

Shel Israel: About Loren Feldman & Michael Arrington.

Thanks, Dave

Can Twitter become ubiquitous?

Tue, 24 Jun 2008 20:42:15 GMT

First today's news: Twitter announced investments from Spark and Bezos. Bijan Sabet will become a Twitter board member. They haven't announced a business model, their approach is to get big and stable and figure it out later. Unlike some, I don't see any problem with this approach. Lots of companies have made their investors very happy (and users) with such an approach.

Disclaimer: I have two friends on Twitter's board -- Sabet and Fred Wilson. Yet, I say exactly what I think about the company, as a user and a developer. I don't think Bijan or Fred would have it any other way.

Mike Arrington: "If they can get the platform stable, I believe they will eventually become as ubiquitous as email, instant messaging, sms and other forms of communication."

This is an interesting idea that deserves serious discussion.

I agree that Twitter is that useful that it could become as ubuiquitous and valuable as email, IM and SMS. However, they have to become a fully open platform before that can happen. I don't believe it will become ubiquitous in its current form. The platform owner has too much power. And there are disturbing indications that it may take more power. The fact that they can do this unilaterally is the big limit on Twitter's growth. It will be hard for investors to risk on new ideas that build on Twitter knowing that the company can foreclose on them at any time.

6/21/08: "I was forwarded an email yesterday posted by a Twitter employee to their developer mailing list that suggests that once Twitter is healthy the terms will change, requiring developers to get a license from the company to use data that previously was available without a license. This is exactly what developers hate, because Twitter gets to decide how much competition they want, they can reserve markets for themselves, even ones they're not serving. No one should have this power, it's not a healthy situation for anyone, not even Twitter, imho. Can't help but think they're killing the goose that laid the golden egg here. Also feels a bit screwy that we helped them build their network, for free (isn't it funny people only look at how they give stuff away) -- only to find that now they want to take back what was open about it."

It's actually worse than I said on Saturday. They can decide retroactively to take over markets that were once the province of developers. Now, that can't happen in email or IM -- there's no single vendor that has the power to destroy businesses without even launching a product. At this stage in the market development, that's too much power. Bijan and Fred, ask yourselves what guarantee you'd need from Twitter to feel comfortable investing in its aftermarket. I don't think you'd settle for anything less than complete freedom, upfront, before you invest a dime.

Getting NewsJunk through IM

Tue, 24 Jun 2008 18:37:46 GMT

I haven't tried it, but this comes recommended...

http://www.imfeeds.com/

Since there's a NewsJunk RSS feed, it should work.

I'm going to try it now. Please let me know if it works for you.

Update #1: It was pretty easy to join.

Update #2: I signed up to follow the NewsJunk RSS feed. It took a while but a new item showed up. And like FriendFeed, they ignore the RSS <description> element, which is a mistake, I think. Here's a screen shot.

Okay, there was a short description that went with that item. Why not push it down the line?

How to push stories to NewsJunk using del.icio.us

Tue, 24 Jun 2008 16:42:56 GMT

Underneath its simple user interface there's a lot of RSS that goes into NewsJunk. It's like flour to cake or ice to hockey. The product is more than RSS, but without it, it wouldn't be happening.

As you know I've been re-exploring del.icio.us, yesterday I asked if it could do what FriendFeed does, keep a feed synched with its internal structure, and found out that the only way to do this was to write a script. I decided not to do that, at this time, but I did write a script that made our FriendFeed support much better. It wasn't refreshing often enough, only once an hour, which makes the news not-so-fresh. And it bothered me that even though FF can display longish bits of text, more than Twitter, the descriptions in RSS feeds were ignored. Well, if you use the FriendFeed API, you can get around both of these problems. It took a couple of hours to connect NewsJunk to FF through their API, and it's really nice. Highly recommended.

Another dividend of this exploration is that I hooked up my inbound del.icio.us links feed to NewsJunk so there's a way for anyone, through del.icio.us, to add a story to the input flow, just send a link to "for/scripting".

However, there's no guarantee that it will appear in the output flow, we're a very focused site, our only interest is US national politics. We're willing to wander off-topic for a moment or two, for example when George Carlin died on Sunday, that story was added to the flow. When OPEC meets to talk about oil prices, that's grist for our mill, because oil prices are a huge issue in the 2008 election. And when a NASA scientist says that this, now, is the last minute to take drastic action to head off a global warming disaster, we put that in too, because while it isn't a major campaign issue, perhaps it should?

The basic qualification for inclusion is the same as for a blog -- would an informed person want to be aware of this fact or point of view? That's why we run outrageous claims from both sides, because even if you support the candidate they're defaming, you should still know what they're saying about your guy.

Also, the better FriendFeed interface gives us a place to discuss news events. I notice that people are starting to do that, and I think it's great! Use all the tools and learn from them and each other. This is how politics and the Internet move forward, we think we're right on the leading edge, and want to keep pushing.

I may have found Scoble's hook

Tue, 24 Jun 2008 01:30:21 GMT

Heh. I've been wondering when Scoble would discover NewsJunk. I think today I finally baited the hook, dropped the line, and he took the bait. Maybe!

I've been deliberately not trying to get my "friends" in Silicon Valley to write about my new offering because I want to see how well they pick up on things outside their own Beltway. I love these guys, Om, Arrington, Scoble, Gillmor, but I want to earn my flow, not be given it. I think NewsJunk is good enough for them to care about it all on its own, not as some kind of favor to me.

Now here's how I baited Scoble...

I was tuned in to his QikCast of his panel at PDF in NY, and heard him say that Memeorandum was the fastest way to get breaking political news just like Techmeme in the tech blogosphere. I posted a twit, disagreeing, I don't think Memeorandum is good at fast-breaking news, it has a 24-hour cycle, and top stories tend to stick there for the full cycle, keeping other less phenomenal stories that we see quickly in NewsJunk from showing up there at all.

Sometimes they show up 24 hours after they happen! That's just not good enough for news in a political year. That's why we started NewsJunk -- to scratch the itch that Nicco and I (and many others) had. We tried to imagine the news system that Chuck Todd deserved, or Joe Trippi, David Axelrod, Josh Marshall or even Barack Obama himself. (McCain, only being "aware" of the Internet is not in a position to use it).

The theory being that if it's good enough for a pro, it would also be good enough for a schmuck like me.

And Nicco is a pro. Not in Silicon Valley but inside the Beltway. And he's my buddy, and a programmer (he led the tech team at Dean For America in 2004).

So I hope it also works for Scoble.

Remember Scoble? That's who this story is about.

Scoble said to me he gets most of his news from Summize and FriendFeed.

Bing!

Now you gotta know that we designed NewsJunk in the post-Twitter world, and we use this stuff, seriously, so of course you can get NewsJunk in FriendFeed.

http://friendfeed.com/newsjunk

And what do you think Summize summizes? Twitter!

We supported Twitter on Day One.

We got you covered Scoble old dude!

"cheesecake"

del.icio.us question

Mon, 23 Jun 2008 17:36:49 GMT

I've started to use del.icio.us recently, as part of the editorial flow for NewsJunk, and it's making me think of ways of integrating the two.

Now, I understand how I can get an RSS feed out of del.icio.us, the question is -- how do I get one in?

I want all the stories that show up in the NewsJunk feed to become one of my bookmarks on del.icio.us, much as they flow through to FriendFeed.

Is there a way to do this?

Update: From the comments it's very clear they don't have the feature. I'm not going to write a script, it was only of passing interest, in no way is it mission critical. The functionality is already in FriendFeed. I just thought there might be an easy way to provide NewJunk headlines to people through del.icio.us. Thanks for all the great advice!