What just happened to Pocketables, what have we lost?
Here’s a quick timeline of where our week went.
My dearest Site5: If you are reading this, you’re not using your own nameservers and are looking at a different host. Try using your own name servers.
On January 31st something broke inside of Pocketables. The running nightly VaultPress backup disconnected and I received no notifications that they had. I’m not sure if we were hacked at this point or later.
On February 1st, Site5, our website hosts, claim they took a full backup.
On Thursday Feb 13th, a plugin called UpDraftPlus that I added for potential disasters and was running once a week happened to activate, made a backup of the site, and moved it to my home computer.
Wrote a couple of award winning articles on Friday.. I’d show them to you but…
On Saturday, Feb 15th a plugin for WordPress called ThemeGrill importer was found to be an easy target. Basically anyone could wipe a database if there was an admin account named “admin” which, sure enough there was. It was deactivated, from the CG days.
On Sunday Feb 16th Site 5, our website hosts, made two backups of the website. One three hours after the other.
The morning of Monday Feb 17th Pocketables was up at 8 something CST. I know because spammers hit the Contact us links and the website was functioning at that time.
About 10AM I get notification something is afoot with the site. Database reset. Just saying “Hello World” which is what happens when WordPress initializes. Pocketables was one of over 200,000 sites from some stories I read.
I open a ticket with Site5 at 1:06pm Monday, the people who host the website. At 3:50 not having heard anything and with no phone number to call and no response to the ticket email I ask on Twitter what the turnaround time is on tickets as I don’t want to bug them.
They respond that there’s no estimated turnaround time but they can escalate the ticket. I go ahead and do this as the last incident I had with them their techs were not touching the ticket for days and the only way I got in touch with a human was to go hard and heavy on the Twitter. Should have been an indication then.
At almost 20 hours since ticket creation and 15 hours since elevation I ask if I can pay someone to actually take this case. I’m not trying to get in front of anyone, but nobody’s handling it, this is a restore job. I’m told nope, they’ll update the ticket and email. Nope.
I get a notification from the Twitter people on Tuesday that the ticket was updated last night and I was emailed and that they did a restore and it was completed. Nope. At this point I strongly suspect it’s the Themegrill hack and send over information on that.
I’m going to compress this down as here’s what happened from Tuesday to Thursday. Twice I was told the site was restored via email, once I checked 12 minutes later it wasn’t, once an hour later and it wasn’t. The Pocketables UpTime Robot we’ve got claimed it was down during those times.
I got a notification Tuesday night on my unlimited storage unlimited files Pocketables account that I was nearing the limit. This was because they had copied folders and didn’t delete them. This was also because unlimited suddenly didn’t mean anything. I was told the thing was restored, and when it wasn’t working that it was because I was nearing the limits of my unlimited and that’s obviously why the database wouldn’t work. They chose to delete a local backup file to free some space and when that was done, told me the site was working and sent a link to a picture that didn’t work.
I was told to bypass CloudFlare and change the DNS to Site5’s because they were seeing something different… er, no, they weren’t
I was sent notifications that the site was up when only the ads were showing. Given an “all good” as the database was inaccessible. And was told Wednesday at 8:33pm that the issue was that the wp-config file had a script in it and that the table names and database names were wrong and asked if I had a backup because none of their worked.
Er yeah people, you just deleted the local backup. Killed it Wednesday because of my limited unlimited account and the last thing I heard from Site5 was at 8:33pm Wednesday night. I hit them up on Twitter one last time, a different rep was manning the thing and didn’t have the notes evidently. I believe I left it escalated.
Last I left it on Wednesday night was that they had multiple backups, that was part of the service, so perhaps restore a backup and if theirs didn’t work then use the local plugin backup which should be contained in any of their backups. That was the last I heard from them.
I was approached by another hosting provider. I was responsive, happened to have a copy of my 27 gig website from Feb 13th, and sent it to them. I’m not entirely sure how difficult it was for them, but we were up and running again this morning about 8 hours from the time I signed up.
We’ll talk a bit about them later. Not going to gush until I see the website standing a month later and all the sites I’m moving from Site5 still working.
In the meantime, my ticket with Site5 is still open, in theory they’re still supposedly trying to work it (they’re not working it,) and we’re at 76 hours and 10 minutes since opening said ticket and I’ve not been able to speak with a human once, been told it’s fixed three or four times, looks like i’ve had 5 different people work this and claim it’s done, and they have now claimed none of their backups are good and asked me for the backup file they deleted, but have in backups.
This is pretty much on par with the whole months-long episode in which I finally managed to get in touch with a sales guy and told him exactly what the problem with Site5’s servers were and how to fix it for every one of their customers and miraculously they were fixed the next day. (This was a DNS lookup fail issue)
So yeah, that’s been the week. We lost some articles because I didn’t know Vaultpress had disconnected, I’d put in UpDraft plus just as a backup to the backup to the backups and thought I was paranoid so I only had it running once a week.
I mean seriously, in this day and age how many backups does one need? Evidently 3. Thank you UpDraftPlus, I really thought I was being super paranoid with this many levels of backup, but yeah, I had no idea VaultPress had disconnected for two weeks at that point.
Update Friday 2/21
Last night I received an email
As we’ve moved hosts, I went to my hosts file, popped in the IP address that Site5 has for Pocketables, and went to see how Pocketables was fixed.
As of this point the ticket’s been open for 91 hours. I’ve been told it was fixed five times now.
