Remember When Your Followers Were Deleted? Twitter’s Done it Again!

whale.pngEarlier in the year you may remember Twitter “accidentally” deleting a large portion of a majority of their users’ followers and having to spend days restoring those from tape backup due to their mishap. Well, it would appear that Twitter has done it again. This time may not be as obvious to users.

On SocialToo.com I was noticing we were getting “Rate Limit Exceeded” errors for our whitelisted @socialtoo user. Such errors shouldn’t be happening for whitelisted users so I looked into it. Sure enough, on the Twitter development mailing list I noticed an email from the founder of MrTweet, Yu-Shan Fung, stating they too were seeing the same issue with their @mrtweet whitelited user.

Alex Payne, Twitter API Lead, confirmed the issue, stating, “It looks like some database maintenance inadvertently truncated our table of whitelisted users.” He then continued in a follow-up e-mail making the problem seem worse: ‘The updated estimate I’ve just received from our ops guys is “more
than 15 minutes and less than 12 hours”. They have to restore from a
nightly database backup. Said backups are quite large, and take some
time to get through.’ It would appear that Twitter has inadvertently deleted an entire table of all the whitelisted users – this is the table that enables many of the Twitter Apps you use to continue to talk to the API without breaking. This is what we as developers are relying on to work or our Apps go down.

It was the exact issue of database deletion and insecurity of Twitter being competent enough to not do it again that caused me to go on hiatus from Twitter earlier in the year. Problems like deleting an entire table from a database, especially twice in a year, are inexcusable Twitter! I recommend finding someone that can ensure this doesn’t happen again. It’s time to upgrade your Operations team. As developers, we simply can’t rely on this, and it’s why I called my service SocialToo and not TwitterToo.

In the meantime, many of the Twitter services you use until this gets resolved may have issues. This includes SocialToo.com autofollow being down until the table in the Twitter database is up again. I’ll keep you updated on the @socialtoo Twitter account when this comes up again.

Advertisements

13 thoughts on “Remember When Your Followers Were Deleted? Twitter’s Done it Again!

  1. So let me get this straight. You're expecting someone's software to be completely bug free? As a software developer, I am sure you can understand the difficulty of this, especially in something as complex as Twitter is. Sure, they deleted a table in the users database earlier this year. It happens. They recovered the lost data and haven't had that happen since. Now they've deleted a table from their whitelisting database. Whose to say these two problems were caused by the exact same problem? Conversely, whose to say these were caused by two separate and distinct issues. From my POV, I see it as the latter. We're talking about two separate databases, likely running separate code from each other. Could Twitter have done better? Sure. But holding them to a higher standard just isn't fair to them. We're all human (and some of us software developers). We've all done stupid things we wish we could take back. The bigger issue would be if Twitter were to have the same mistakes over and over again. I just don't see this happen. I see mistakes, but then they are addressed and don't reappear at a later date.

    Like

  2. Unfortunately that won't fix this problem since this is related to
    whitelisted developers and not really followers. It's the same concept
    though – a DBA at Twitter accidentally deleting data from their database.

    Like

  3. “No Thankyou”, I'm always amazed at the length people are willing to go
    through to defend Twitter, despite all its problems – and I'm curious why
    you're trying to remain anonymous here. As a business trying to make money
    off this stuff I have every right to demand that Twitter remain reliable.
    When Twitter goes down like in this instance, so does my business. I don't
    care if their code has bugs in it – so long as they're going to be allowing
    access to an API like this they have a responsibility to protect data and
    ensure the service is reliable.
    Bugs or no bugs, DBAs “accidentally” deleting entire tables in a database is
    inexcusable. At every reputable place I've worked at you can lose your job
    over mistakes like that. One time would be one thing, but twice (and I
    argue this isn't even the second time this has happened – I remember it
    happening several times before) shows you have a problem with your DBAs.
    Something needs to be fixed, and I have every right as a business user of
    Twitter to demand that. Deleting tables in the database seems to be a very
    common occurrence at Twitter – it's inexcusable.

    Like

  4. I'm not defending Twitter – I'm playing the devil's advocate. I don't see where people get off criticizing something they are getting for free.

    “As a business trying to make money off this stuff I have every right to demand that Twitter remain reliable.”

    No. You don't. If you paid for access to the Twitter API, then yes, you would have a voice. But you don't. You pretty much mooch off of the hard work the Twitter folks do to keep the service up and running for everyone and then expect them to bend over backwards for you. Twitter has no “responsibility” to you simply because you've chosen to leach off their service.

    Now, is it a bad thing Twitter deleted these tables? You bet it is. All I'm trying to get across is that mistakes happen. Provided Twitter fixes the problem in a timely manner (15 minutes to 12 hours seems timely to me), then I'm OK with that. If Twitter refused to acknowledge the problem, or in any other way tried to deflect responsibility for causing the problem I wouldn't be taking this stance. But from what you've posted, it sounds as if they have been pretty darned open about it.

    Furthermore, put yourself in Twitter's shoes. From what I'm seeing, you are not running any kind of service that's comparable in size to what Twitter is. Things change A LOT when things get big. Things happen that you simply cannot anticipate and I suspect this is one of those.

    Lastly – I choose to be anonymous because this is the internet, afterall. I am simply not comfortable providing any real information about myself in a public space. I don't go about in public spewing my private information, and I don't do it on the internet, either. Simple as that.

    Like

  5. I'm not defending Twitter – I'm playing the devil's advocate. I don't see where people get off criticizing something they are getting for free.

    “As a business trying to make money off this stuff I have every right to demand that Twitter remain reliable.”

    No. You don't. If you paid for access to the Twitter API, then yes, you would have a voice. But you don't. You pretty much mooch off of the hard work the Twitter folks do to keep the service up and running for everyone and then expect them to bend over backwards for you. Twitter has no “responsibility” to you simply because you've chosen to leach off their service.

    Now, is it a bad thing Twitter deleted these tables? You bet it is. All I'm trying to get across is that mistakes happen. Provided Twitter fixes the problem in a timely manner (15 minutes to 12 hours seems timely to me), then I'm OK with that. If Twitter refused to acknowledge the problem, or in any other way tried to deflect responsibility for causing the problem I wouldn't be taking this stance. But from what you've posted, it sounds as if they have been pretty darned open about it.

    Furthermore, put yourself in Twitter's shoes. From what I'm seeing, you are not running any kind of service that's comparable in size to what Twitter is. Things change A LOT when things get big. Things happen that you simply cannot anticipate and I suspect this is one of those.

    Lastly – I choose to be anonymous because this is the internet, afterall. I am simply not comfortable providing any real information about myself in a public space. I don't go about in public spewing my private information, and I don't do it on the internet, either. Simple as that.

    Like

  6. Unfortunately that won't fix this problem since this is related to
    whitelisted developers and not really followers. It's the same concept
    though – a DBA at Twitter accidentally deleting data from their database.

    Like

  7. “No Thankyou”, I'm always amazed at the length people are willing to go
    through to defend Twitter, despite all its problems – and I'm curious why
    you're trying to remain anonymous here. As a business trying to make money
    off this stuff I have every right to demand that Twitter remain reliable.
    When Twitter goes down like in this instance, so does my business. I don't
    care if their code has bugs in it – so long as they're going to be allowing
    access to an API like this they have a responsibility to protect data and
    ensure the service is reliable.
    Bugs or no bugs, DBAs “accidentally” deleting entire tables in a database is
    inexcusable. At every reputable place I've worked at you can lose your job
    over mistakes like that. One time would be one thing, but twice (and I
    argue this isn't even the second time this has happened – I remember it
    happening several times before) shows you have a problem with your DBAs.
    Something needs to be fixed, and I have every right as a business user of
    Twitter to demand that. Deleting tables in the database seems to be a very
    common occurrence at Twitter – it's inexcusable.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s