Hi, I’m Paul and I’m an IT guy for my day when I’m not busy making my Pocketables tens of dollars. I don’t claim I know everything, I’m an expert in a few areas but yeah, this Christmas I had tech issues come at me pretty much nonstop and I’m pretty sure it’s because I faltered on the message of the season: solid consumerism.
TL;DR – how tech spent my holiday.
Anyway, my tech woes starts with the remote on my Yaber Pico T1 projector dying. I’m assuming it’s the battery but we sort of had an ice storm of death and I just decided to order a replacement on Amazon.
The ice storm caused some issues with the state’s power grid and we had our first rolling blackouts in the TVA’s history as I recall. No problem thought I, I’ve got UPSes on everything and the machines, they’re set to gracefully shut down in the event of a power outage.
Yeah, evidently not.
The home machine was dead when I noticed it and the UPS was drained and appeared dead. My servers at work were half down. I ran to Staples because part of our work setup involves my home storage and I really didn’t want to spend Christmas dealing with a ransomware attack or something without a 4 hour old snapshot on physically rotating drives. Yeah, I unplug those beasties.
Turns out the home machine’s UPS had just been drained so many times and cycled through on and off 15 minute windows of the rolling blackouts that the battery was just dead and I’m not used to dealing with a drained APC UPS battery… just takes a really long while to charge and there was no indication it was charging.
OK, bought an extra UPS… replaced, realized 4 hours later that the original was charging and holding a charge… oh well. I manage 12 at work, there’ll be one blowing out in the next few months I’m sure.
Work went down, servers went offline. We had people working from home so I headed in when the roads were good and using my 30+ years in the IT industry pressed the power button. I had a POE switch showing a fault (it’s dead, other reasons though,) one server down that had shut off gracefully, and my work computer which I sat and watched for 20 minutes as it struggled to load.
The work machine had not shut down gracefully. It got told there was going to be no power, and it decided that was a perfect time to not shut down and also write a bunch of stuff to the user profiles.
I ended up with a corrupted Windows 11 user profile from a machine that was sitting there doing nothing, on a RAID card/array that should have handled anything short of a lightning strike. It didn’t.
Went through all the user profile fixes I could locate and gave up when the event logs were saying it was just corrupted so yeah… tried about everything. Due to corrupt profile, but literally everything on the drive other than that directory being good, I decided to go with the “local admin profile, delete other profile, recreate other profile.”
This of course did not work. Running MMC was not getting me into the local profile editor, I couldn’t add a snap in, and trying to add or remove profiles was not functioning.
Turns out pressing start and typing MMC vs pressing start, run, and typing MMC were both bringing up MMC, but one was one I could add snapins, and the other was not. Great. Learn something new.
Add local admin as I forgot what my admin was for this box (I’ve got an envelope of rando passwords and accounts I could have gotten but that involved tearing open the emergency envelope.) Logged in as that admin, deleted profile, and viola! I recreated it and that should have been that.
Started in the newly done profile and everything seemed pretty good… except Outlook… it was in IMAP mode as opposed to Google Workspace… ok, I set it up wrong, let’s install Google Workspace and nope… it says Outlook isn’t installed.
It was, however it had been installed on a previous profile and I checked and the only thing showing in add/remote was Office. I chose to repair that. Nope. Reset it. Nope. No changes. No ability to uninstall anything other than “Office” and when I did that Outlook, Word, Etc were all still there and not visible via Google Workspace Sync.
As all of them had been installed on the now-corrupt profile I had to go and download an Office repair tool from Microsoft. It’s from MS, this isn’t random spam to get you to download something. Took forever but worked. I mean, it took something like 5 minutes to even display a screen that it was going to do anything and then I though 20 minutes in that it had failed. It got there about 40 minutes in however and had uninstalled everything.
Reinstalled Office 365, it registered properly, Google Workspace Sync worked now, bam…
One corrupted directory. Three days of trying to get it back up and running.
As part of my work and having unlimited bandwidth I have a really good test for when there’s something wrong at the office. That test involves an HDHomeRun, a VPN, and streaming local over the air TV to my house. While the entire Christmas break I never dropped more than a ping or two on the office, on Christmas day trying to watch a parade it became unwatchable.
I checked the stats, no real ping drops, jumped into one of the servers and did a speed test on our dual gigabit connections and was getting 2mbit down and 60 up. That was terrible. I bundled up and headed into the office because I’d just added a POE switch to the rack (replacing the one with the fault,) and we’d had the rolling blackouts and I assumed it was my issue.
It wasn’t. This one was AT&T… and I’m not really going to harp on them here for the outage, but this was the first time I have had to contact support for a business account ever and… it was stupid… like remarkably a waste of everyone’s time and money.
I called the number I could find… had to go through authentication, pulling up the account, verifying things you know, the usual… stuff you’d need to do if you were talking to the right freaking department… no, I was in the wrong place. Rather than tell me I’d called the wrong AT&T number I had to be verified before they’d tell me that or point me to the right one.
So, I jump through the verification hoops and am both given the business number I should have called and also connected. I’d be happy if the story ended here, but no… none of that verification carried forward… it took another three minutes of repeating the same info from the previous time and when they pulled up the account they could only locate one of our two fiber modems.
OK, fine… both of them were sucking… I ask for diagnostics on the one they can see. They start on diagnostics and tell me they see a bunch of wrong IPs showing on the thing… ok, someone plugged a switch in incorrectly whatever that’s not the issue. Six minutes into diagnostics I’m told that oh, there’s an outage, I’m affected, do I want to continue talking to tech support on Christmas? No… I don’t… tell me that there’s an outage when you pull my account up people.
They send me a link to track it… it’s the most useless link I’ve ever seen… outage reported with no ETA.
I mean, they fixed it, but no ETA no ticket number…