weeks months back we abandoned hope. Not that “fiber is coming,” it’s here, we’ll get into that in a bit, but that we would ever trust Google Fiber as an ISP in terms of reliability when/if it finally gets working. Below is a month by month account of what happened, and I will stress some of this is from memory and may be incorrect but most is backed by email, when I took photos, etc. I didn’t think before the install I was going to be writing anything but “yay, we got Google Fiber and it’s GREAT!” but here we are.
Let’s set the stage here. Client was completely failed by Lumen/Level 3/Whatever they’re calling themselves these days previously after nearly a year long attempt at getting fiber, Google came along at the right time and said they could get it in at our AT&T cutoff deadline that Lumen was going to miss because Lumen abandoned the project and didn’t tell anyone and wouldn’t return calls.
I have been writing this article for the past five or so months about events of the past nine and a half months. As such treat each month starting in April as that was probably me then.
TL;DR version of this is: Scheduled an install at the start of Dec 2020 for a Feb 2021 cutover and it’s August 2021 and just reached the point where it would work.
Hi, I’m Paul – I used to work at an internet service provider aimed at businesses. I used to configure all sorts of ISP equipment and assisted with multisite customer networks building out a new network from effectively nothing. At one point I worked at an IT firm where we did network wiring, which included fiber optic. I have, however, never trenched a fiber optic cable or dealt with that.
I currently am the IT tech for an office building (among my many other jobs.) I mention this because I am not complaining about things I do not know the basics of. I’ve dealt with planning, deployment, implementation, customer service, and have seen most possible IT fails as well as making a lot of them myself.
2015: Google Fiber enters the Nashville market, fights years-long battle to get One Touch Make Ready so they can run fiber on the poles. After winning, losing, winning, and possibly losing again, they decide shallow trenching is the way to go.
2018, April – Google Fiber trenches outside the building. We’re told in 2018 it will be available shortly. Shortly to the ability to sign the initial contract was about 29 months. We’re told it’s available in our area sometime in September of 2020, and manage to get someone to talk to us in November/early December 2020 (first email exchange I can spot is 12/2 but we’d talked before that). We weren’t in a rush at that point because Lumen/Level 3 was in theory installing fiber for us.
At the time of contact, we’d had AT&T fiber for a couple of years and our gigabit costs were about $500 a month for a business connection. Google business fiber at $250 a month was a real no-brainer. Our AT&T contract was up at the beginning of February 2021 and the rep we talked to seemed confident this was not going to be an issue having it lit by Jan and ready to go and tested by Feb when our AT&T contract expired. Email even in January expresses optimism that by our Feb deadline we’ll have been up for a couple of weeks because all they have to do is run a fiber optic cable up our driveway.
I’m going to gloss over this next part because it doesn’t matter (this may encompass some Feb also with the ghosts of many contractors,) and it’s subcontractors. The install was simple on our end – run a fiber optic cable up from an already existing pipe and equipment buried right next to our building, mount your equipment in this closet that has plenty of space and an existing properly installed outlet, that’s it – building side happy up to codes easy to install and fish the fiber in a pipe in a data closet.
We already have 6 cable connections running through, telephone system (t1), and existing fiber run through the 5″ access pipe, and plenty of space. There’s a whole wall. We’re doing the office/network wiring. This should have been a one time meeting and ended up being three or four with multiple subcontractors because none of them share information with each other. I was later required to be on site during a pandemic (mostly pandemic having to deal with kids schools being remote, and not full on lockdown levels,) to unlock a door so the next set of contractors could take the same photos of the walls. None of them would accept emailed photos or talk to the other subcontractors.
Whatever, they’re subcontractors and not Google. Things could have been smoothed out if there was someone coordinating the project.
I am contacted multiple times, I say the same things – run fiber here, mount wall equipment here, you want 5-10 potential connections here for tenants. That’s it. This is all going on in the aftermath of the Lumen/Level 3 fiasco as a note. I’m asked to provide blueprints by one subcontractor who doesn’t understand they don’t need them, but won’t proceed without them. They’re sticking a fiber line in a pipe and pulling it out the exit hole. Without blueprints (which don’t really show the pipe) they won’t proceed. Whatever. Blueprints located and provided.
I am assured that we’ll be ready for February at the beginning of January.
February happens. We don’t have Google Fiber but AT&T’s being cool, says the worst that would happen is a $20 rate increase was possible, end seems in sight.
A huge weeks long fiasco happened with the Google Fiber paperwork.
I’m going to skip details because I was already having a really bad Google day and managed to look a bit like a Twitter ass picking on them because their Docusign game was off, which of course came to their attention because Google does read, just not respond generally. This takes a while. Signatures keep getting kicked back and I kept getting bad docs. Whatever, moving on past this. They didn’t have a way for us to sign the paperwork electronically, tried rigging it, failed rigging it, caused us to be delayed.
Paperwork level screwed up massively, knocked us back a couple of weeks. I’m told legal kept kicking it back. Not my problem.
Site survey by planning is finally done. I ask planners to use right side of building or use the left side through the easement. I’m told no. They won’t go easement, they’ll trench or go under a neighboring business’ driveway, skirt across a small access area that will require half our front parking to be roped off during construction, finally going across a gas line, water pipes, sewer main, trenching 90 feet of gravel parking lot on the side. My way was nothing but dirt and 6-18 feet of gravel. They refuse to go my route.
March happens. AT&T still being cool, drops our rates post-contract. We’re not terribly worried at this point which is freaking great because had we been this would have been nightmare start. Let me tell you if AT&T has not been being so great we would have told Google that they had to put up or shut up at that point.
Google subcontractors keep showing up unannounced. This is during the end of a pretty active wave of the pandemic with schools closed half open, out, etc, and everyone’s been working from home. There were very few people at the office, I had to arrange kid care or drag them with me, I was working mostly remote. When someone showed up I had to find out who was at the office who might have a key to let them go and look at the closet. It was always the closet. That closet had more photoshoots than (insert overly photographed celebrity here)
One day the installers for the interior equipment showed. They were unannounced if I recall correctly and told that we’d been informed. I had to leave while they were there. I showed them where the wall was, told them they were installing to there and we would handle the rest. I don’t know what other information they were working from but when I came back the next day they installed 56 fiber jacks taking up the majority of a wall now mounted in our space.
Basically it was nearing one fiber jack per office. While the building is multi-tenant, this was extraordinarily optimistic of them. 10 would have sufficed. We have a wall dedicated to Google now. I stress we were going to use one jack (for 15 or so offices,) and maybe a few of the tenants will subscribe for their own needs/HIPAA/whatever requirements. There’s no chance they’re getting over 10 customers out of that building.
April happens. AT&T has dropped our business cost to half and doubled our building’s bandwidth at this point. We’ve got 2gig asymmetrical on two connections for $240 total a month. This now beats Google Fiber by $10 for twice the product.
Techs keep showing up unannounced. I’m getting calls such as “we’re heading there, we were told you were informed, I’m calling from the end of the driveway” – yeah, I come in a couple more times because there’s literally nobody to open the door for them because we weren’t expecting them and why do they need in the closet yet again? No subcontractor would accept photos I had on my phone of the exact same thing they were going to photograph.
I ask for the fifth time for nobody to show without calling because I have to arrange for access every single time. I’m assured nobody else will show without calling first.
The next day a guy shows up at the front desk asking for me, once again unannounced.
He’s the head of one of the subcontactors, inspecting their work…. whatever.
I’m not entirely sure at this point if the trenching installers showed last day of April or first of May. As I recall I was not really expecting to see them and we’d not informed anyone to move vehicles at this point.
I meet with the installer who is looking at the plans I kept fighting, and he asks whether I wanted those, or I was forced to accept the plans. I tell him no, that’s not what I wanted, that’s not what anyone wants except the planning/design team. I show him the two routes we’d requested and he agrees either of them was better than digging through the neighboring business and evacuating and excavating under portions of the parking lot.
He says the plans they had in hand would cost them a whole lot more than the route I showed (which was 30 more feet or so and did not impact anything other than 6 feet of gravel parking).
He said had I not been there they were going to do the right side of the building any way. This concerned me. Last second subcontractor buried route change with no checking property boundaries… uh, not easing my tensions, however it’s my client’s property, we discuss that the trees there are more valuable than cars, the softscaping shallow trenches they’re going to dig cannot come near the trees which he agrees to. Shouldn’t matter anyway because of how they’re placed you avoid them by going right next to the parking lot, or on the other side of the trees. There’s no reason you would have to come near any of the 20 or so trees.
He swears it shall be done, the trees are safe, etc. This install takes a while. If I remember correctly two or three days. Let’s see if I can find the pictures of the zig zag cable that somehow intersects multiple trees, and the newly dug up tree roots before I publish this. The damage didn’t kill anything, yet, we report it and are told it’s been recorded and anything will be addressed after the install is complete.
A week or so later I was given a heads up / advance notice and arrive on site to meet a light tech who was going to verify all the fiber. If I recall correctly this is the second person who’s given more than a 10 minute heads up. He had something explode, delayed him almost 4 hours getting here, I’m not harshing on him, I just had a meeting scheduled on the other side of town so I asked him to give me a call if he needed anything. He told me he was probably the last Google person I would see.
I don’t hear from anyone for a few days, finally I shoot an email asking what was going on / next steps. I’m informed that the light tech discovered there was something wrong, another tech was dispatched. Nobody had contacted me.
A day or two later I get a call from the building owner that two Google trucks were blocking access to our building and the company next door (they share a driveway). He asked them to move. They claimed they were almost done and would leave when that was done. He was irate. I got to hear all about it. Evidently they dug up some equipment, checked the light levels there, and reburied it. All while half blocking the only access driveway.
That done, and promises that I would be updated, things go dark for a while. 10 days after my last Google Fiber encounter I get a survey from the Google Fiber Team asking how I like the service and what could they improve. Oh man I fill that survey out. Crickets.
At this point AT&T has dropped our business fiber price to about $120 a gigabit (think there’s 10 or so tags, fees, etc) – we’re rocking along with dual gig fiber connections while waiting on Google. The plan had been to load balance between them and drop one of the AT&T lines (we wanted redundancy,) and the new plan is… no idea. At this point Google for business fiber is over twice what we’re paying for twice the AT&T line and if the customer experience is anything like the install experience… no.
We still want network redundancy though, the choice was made in May to go for it. I get no updates.
Since the fiber team evidently reads my Twitter, I start asking on Twitter if I’ve got Google Fiber yet. I do this for a few days because at this point, I feel I’m entitled. This time the Fiber team isn’t reading me and relaying to local fiber associates who email and call me 20 minutes after a tweet about them.
End of May, 23 days of silence and no updates from Google Fiber, I finally decide to end my public questioning and ask the people who hadn’t given me any update. I get a note back from the initial sales contact finally that the problem they had identified at the end of last month/beginning of this was damage not on this property, and that repairs were slated in approximately 3-4 weeks from the day I asked.
This would put the repair time from when the damage to the fiber infrastructure was found to when the repair is scheduled at 8-9 weeks, or about two months.
I’m asked at work what the status is on Google Fiber. The short is at post contact month 8, we suspect by month 10 it will be working. At potentially two months to identify and repair a damaged three year old buried fiberoptic cable, Google Fiber has moved out of anything we’re pursuing for reliable main internet.
(going to preface that I was under the impression here that all of Google Fiber was stopped from working on their buried lines, and not just the install division and their subcontractors when I wrote this)
Imagine your internet down… for two months. That’s what we’re imagining now. Oh there was a wreck on Allan St and a car caught on fire superheating the fiber cable buried 6 inches down and it’s out now? We’ll be back up in 60 days or so.
So you’re selling the fastest most reliable internet and there’s no network monitoring that told you the general area you’re selling fiber in was not working? This raises red flags. I mean it’s been a few years since I worked at an ISP but we generally knew when a node/CO/piece of equipment was malfunctioning.
(I’m told months later install and repair are separate and as such had this been a break they could have repaired, also that live networks are monitored and this one wasn’t live)
More June happens
Third week of June rolls around, I get an email from the sales liaison that repairs have been halted by the city. It takes a couple of days to find out that the city has added restrictions that require additional city management to be involved in before the fiber gets repaired. My assumption is after the disasters of the Fiber rollout over the past couple of years that have involved wrecks, cut gas lines, damage to metro property, etc. that the city is now requiring a whole lot more from Google than they did before.
Le sigh… not a damned thing happens. No contact. Nobody checking in.
As I’m leaving the office there’s a Google Fiber truck out at the street and someone working. Of course this happens while I’ve got to get back home and grab kids to get them away from the house for an important meeting my wife has, so no chance to stop and talk.
I got a call later that the light tech from many months ago was headed that way and was hoping I was there to let him in. Nope, I was headed out and 10 minutes “heads up I’m here” was once again not something I could accommodate. Luckily there was someone at the office who was able to let him in to the closet.
And that was the last I heard from him.
Friday August 6th I get a call on my personal cell phone. I don’t recognize the number, and T-Mobile’s been pretty much only ringing for people who wanted to sell me a car warranty or buy my home for cash. It goes to voicemail. It’s the original sales person I talked to 9 months ago saying that the lines are fixed and we have light again.
At this point I’m assured that the install and the operations are two separate things and that repairs are their own thing. Network monitoring only happens when the network is live, which our area was not.
I’m asked by the client whether I’m going to trust a company that missed the install target by six months.
I go back and ask about if there’s any discounts or price adjustments since they missed the deadline by six months (of which only 2 were city related) – nope. Did get offered a couple of months free for missing that install by six months, or about $500 off the bill for a year.
I ask what business backup options are available since evidently the last two month delay was caused by a contractor building a driveway access and slicing through fiber and then cementing it and I suspect that will happen again in the future as Nashville is nonstop construction. No backup options.
No written guarantees on service either.
I’m leaving this open.