The Semi-Great Twitter Experiment
I stand before you today to confess perhaps my greatest clusterfuck of the year: the Skeptoid Twitter Experiment, which rendered your Twitter account nearly useless on September 14 and 15, if you follow me or anyone else who follows me.
I have an upcoming Skeptoid podcast episode for which I want to include some informal survey data (it was episode 173 on Astrology). I’ve also been thinking a lot about Twitter for its potential to virally spread information. So I thought it would be a clever idea to combine my survey with Twitter, which (I thought) would be a lot of fun for everyone and would accomplish two goals:
- Virally spread awareness of my podcast, Skeptoid
- Get a huge number of respondents to my survey
Well, it worked. The good parts worked better than I hoped, and unfortunately, undesired side effects were just as potent. Now, before I describe what happened, let me state outright that it was shockingly naive of me not to foresee what would happen. It was dumb, it annoyed a lot of people, and I have no excuse other than failure to think it through very well. So, my apologies, and I offer no defense of what turned out to be a giant mess.
Here’s how it worked. I posted questions to Twitter in this form:
Skeptoid Twitter Experiment, Question 7, is now live. Get it now at http://skeptoid.com/twitter
If you followed the link, it would take you to a page where you would answer some questions, and it would give you a code to post your response back onto Twitter, like this:
I’m doing the Skeptoid Twitter Experiment (http://skeptoid.com/twitter) – My answer is #skep_4_4 – Follow @BrianDunning to join
The effect was twofold. First, people unknown to me who follow my followers saw these posts, were intrigued, and followed me, boosting the numbers of my Twitter followers, and reaching a larger and larger number of people with the question announcements. Second, among groups of people who follow each other (like many of my followers who know me from Skeptoid and know each other from JREF, TAM, etc.), great colliding volumes of redundant answer messages from everyone they follow clogged their Twitter inboxes and made it virtually impossible to see normal messages. I heard from one guy who shares 80 Twitter friends with me, and if they each answered the 8 survey questions, that made 640 tweets.
Many called my attention to other less intrusive ways the survey could have been given, including a number of free web survey tools. Having a web development background, I’m aware of these, and even built a languishing one myself – http://GroovySurveys.com (a day’s work over Christmas break a few years ago) – but there’s a reason I didn’t do it that way. It wouldn’t have been a Twitter survey. And the whole point of my experiment was to see how well a survey worked where the entire thing – questions and answers – were sent via Twitter; and particularly to evaluate its viral effect.
There were some interesting observations to be made. Three times, before the survey started, I made one daily post to Twitter like this:
Please RT: Be a part of the Skeptoid Twitter Experiment: http://skeptoid.com/twitter
And, predictably, this enticing message began to spread. I started seeing it being retweeted (Twitterese for “repeated”) among people who did not follow me. My follower count began to increase. By Monday morning, when I posted the first question, the rise was on a very solid trend.
The first half of the questions were posted throughout Monday, and if things grew before, they exploded now. I use Tweetdeck to view Twitter, and I have columns to view mentions of my name (@BrianDunning) and the podcast (#Skeptoid). Both were absolutely clogged. I thought this was wonderful; but since nobody else is likely to follow either of those search words, it never occurred to me that anyone else was having the same problem.
Tuesday morning I awoke to a barrage of emails, direct messages, and public replies. Obviously, all was not well. Accusations of spam were flying everywhere – which hurt, I’ll admit, because I hate spam as much as the next person and it’s the last thing I’d ever want to be associated with. The growth in my follower count had stopped.
What it reminded me of was junior high school, when a few Kool Kidz wore Izod Lacoste shirts. Soon everyone had to have one. And then, once they became popular, the Kool Kidz wouldn’t be caught dead in them. My survey grew quickly when it sounded like fun, swamped the market, and everyone wanted out.
I posted some hasty apologies and did what little I could to reduce any perceived spamminess. I took my name and the site name out of all the messages, killed the invitation to follow me, and put instructions on the landing page for economizing posts. But then I made things worse: I publicly offered to terminate the experiment if 10 people asked me to do so, but I suggested that they do so by Twitter direct message. I was unaware that direct messages can only be exchanged between people who follow each other, and the handful of personal acquaintances whom I follow did not happen to be the same people as those out in the real world that I’d annoyed. Oops. At least I can boast to having received no such direct message requests, but it’s a pretty thin boast. ;-)
My stuffing of fingers into the dike seemed to help a lot. Adoption rose again, and through the end of Tuesday, my follower count resumed its growth (though I can’t figure out why, since there was no mention of my account anymore), and answers continued flooding in.
So, as far as providing the survey data I wanted, it was a massive success. As far as being user friendly, it was a massive failure, and would probably get my account banned if I did it again. So take what lessons you can from this – I certainly have.
It has taken me 35 years to learn that failure is one of the best things we can do to make progress. Thanks for sharing this story, and backing up your skeptical chops with admission that you didn’t know some stuff, and now you do. The same can be said for me, now.
This reminds me of something that happened to a student network administrator at my college back in the day. There were two primary systems for students to use that operated completely independently of each other. This guy was an admin on both, with email on both. Well, to keep track of the happenings when he was on one or the other, he set up his email to autoforward his email on network A to network B, and vice versa. Then he sent a test message.
It didn’t take long for him to figure out the problem. A quirk of our email system at the time was that, upon receipt, it threw up a notice at the bottom of your screen that you had to manually clear before you could do anything else. He cleared his initial test notice, and then another popped up…Then another. He couldn’t clear them fast enough to actually do anything. He had to charge to the server room across campus and unplug the machines in order to get it to stop, and by the time he got there, over 80% of all the server space was taken up with copies of his test email.
So, yeah, sometimes technology bites us in the ass. It’s still often funny, though. Don’t sweat it, Brian.
Turning Twitter’s echo chamber into a denial-of-service attack. How fiendishly clever.
I think the unintended spamminess makes an interesting sidebar to this survey. As I don’t share 80 followers with you, I didn’t get slammed like some others, but I did notice when some of my friends also responded.
But, like Ransom above, it makes me want to relate a story from 1989, back in my halcyon college days, of the unintended consequences of a cool idea.
I was doing some schoolwork on our UNIX system (I was a total UNIX n00b at the time), and somehow managed to go up one folder from my home folder, and found that every one of the 500+ users on the system had a folder there.
“Huh”, I thought. “I wonder…”. So I typed this command…
mail *
And sent this innocuous little message that said, “Hey, just curious if this works – reply if you get this message.”
What I didn’t plan for was that the common way to reply was with the r key. Unfortunately, r is reply-to-all, whereas R is reply-to-sender.
So suddenly the emails in the system started growing pretty quickly, partially because people were replying to all to say “don’t reply to all”!
All this is happening unbeknownst to me. A little while later, I can’t log in. When I call for support, I’m told I need to come down and talk to the sysadmin. He’s not happy. Apparently it took him hours to clean out the mess of emails I had made.
I think he put something in place to avoid a repeat occurrence in the future, so I like to think that I helped them.
It happens whenever someone inadvertently sends an email to an entire department. Some people reply to all to say, “Unsubscribe me from your list.”
I am secretly delighted by the people who “reply to all” in order to tell everyone not to “reply to all”. The recursive stupidity amuses me.
OK I forgive you, so long it was a worthwhile cause :)
Are you sure your messages were “retreated” instead of, perhaps, “retweeted”?
Darn spell checker, I’ll fix it, thanks.
I didn’t think it was that bad. None of my followers complained.
I saw a lot less spam from this twitter experiment than I saw from that dumb “Spymaster” game a while back.
Its a real testament to the technology that it can handle this type of load. Note Rob T and Ranson’s posts about a system in the late 80’s. Just 20 years ago, something like this would take down the system. Now, this was probably less then a blip on the overall twitter radar.
Of course, try this on an exchange server and the result is worse then the Unix systems in the 80s, but thats a nice little anti-microsoft rant for another day.
Yup. In those days, any user with some savvy could launch a DOS attack on any other user on the college network with three lines of code. That little quirk about the email notice could also be used as an instant messenger on the network, without an email attached. Sending to “all” was possible, but that was only done by admins to warn of shutdown, people who made a mistake, or idiots trying to be funny. One .bat file later, and the person was effectively locked out of all function. We’d usually fill up their email limits with a similar tactic, with an email script linked to a random content generator.
Hell, our little podunk college had so much infighting between admins, they’d launch DOS attacks against each other’s servers with nothing but ping floods. That would usually go on until one guy walked three steps across the lab and hit the other.
We really need new etiquette books for things like Twitter.
DO
– Create freely accessible and interesting content in the hopes that it will be shared.
DON’T
– Ask others to post contentless links as part of a survey, registration, contest, etc.
I only answered two of the questions anyway. No offense, but I thought they were pretty meaningless. I couldn’t figure out what kind of information you were trying to gather and what you planned to do with it. It seemed like just an attempt to get people to tweet about you.
Which I guess it was.
I guess you didn’t read this very closely…. :-)
I agree w/ Liz’s dos&don’ts for the most part – it just takes time to learn them w/ twitter. Don’t fret too much
not to be spammy myself but
if you do run another survey, email me- I can set you up w/ an account where I work- I think our tool’s better than that groovy site
“Now, before I describe what happened, let me state outright that it was shockingly naive of me not to foresee what would happen.”
That’s the thing about experiments. You can’t always forsee how they will work! Interesting lessons were learned by all, no biggie.
twitter is annoying. and it sux.
Personally I find Twitter, facebook, and yahoo groups nothing but spam for every interesting thing I have dozens of “thanks for that”, “Nice Photo’s” or “I’m just scratching my bum” Which means that a useful tool is ignored because I can’t be bothered to wade through the junk.
This article shows just where the problems lie with this type of program.
So Brian, will you be at the next BlackHat conference telling people how to pwn! twitter? Hehehe.
It is how you respond to criticism and your mistakes that mark you out as decent or not. I think once you realised your mistake you did everything in your power to amend it.
I might add that I don’t see anything wrong in a bit of selfless publicity, especially when it is promoting skeptism.
Heh, well one thing was proven by this “survey”. I made the right choice when I decided not to drink the twitter kool-aid.
If everyone is sharing their stories… I used to work for a company in the building maintenance department. We had just rezoned the parking lot and I was to send a picture of the new parking map to all users. I sent a .bmp. Apparently, those are quite large. To make matters worse I typed the keyboard randomly to name the file. The file name was something like hdjdudjndjxjzujdnfkfkgjfugsgrjfmrirurjrjfndjdidin.bmp. My sysadmin was not amused.
Somewhere a village is missing its idiot.