Anyone who’s worked in networking long enough will have at some point dealt with the Cisco TAC (prepend dreaded for more of an effect). I remember clearly an issue back in 2009, in which the company I was working for at the time had a pair of 6513s with Sup7203Bs and first generation 10GE cards (which also ran VERY hot. The two onboard ASICs could fry an egg). Whilst slurping on a canteen coffee one afternoon, the NOC screens lit up like a Christmas tree. The NOC urchins quickly identified the fault as a “we’ve been disconnected” case which prompted some investigation. It was revealed that the 10GE card randomly rebooted to which Cisco’s response was that of expediting equipment to the location. I was instantly relieved by not having to argue with an Indian engineer who claimed it to be a spanning-tree error (another story).
This relief was short lived as Cisco immediately shipped out a whole 6513 with associated line cards….to the wrong address. After clearing up some issues around equipment location and it being in NL not the UK (which the Smartnet correctly referenced), Cisco redirected the package to the NL location. Great, back on track. Not so quick Speedy McSpeed. My office phone chirped and I was greeted by a call from an engineer who was arranging the site visit for the very next day. My somewhat flustered response>“Cisco were made aware this has to be done out of hours with enough notice to alert the company to the emergency maintenance window. Why are you trying to get in tomorrow?”. “I don’t work for Cisco, I work for a spares company and they sent me your number”. Yikes. A third call to Cisco in the same day for the same ticket was made. At this point, another engineer asked me the root cause of the issue. I proceeded to run through everything that happened and sent him a copy of the logs. His response was “Ok, I can see really, you don’t need the chassis and the other cards. It’s a known issue. We’ll send you the 10GE cards”. My response “Well, there will be a pair of 10GE cards on site tomorrow, can’t we just use them?” Cisco engineer “No, it’s part of a different RMA. Just leave them in the warehouse and a courier will collect them”. I proceeded to explain that the swap over had to be executed out of hours with enough time to declare an emergency window. Happy that he understood, I hung up. Later on I was informed by the warehouse that not only did they recieve one sparee chassis with linecards, another empty chassis also arrived with the 10GE cards.
Next day: An engineer called on site to say he’d been let in and was on site. Much to my somewhat twisted amusement, the engineer had found his own means of access to the site and wanted the access code to our core racks. I explained he did not have permission to be on site and he did not have authorization to proceed with the line card swap and to his credit he mumbled something in Dutch and hung up. This was finally resolved one evening when the cards were swapped (as scheduled) by the same engineer who had been caught in the midst of the chaos. Moral of the story? We’ve all been that engineer at some point caught in the chaos, laugh with him, not at him. If you need to moan, do it at the vendor directly. They have departments setup to deal with it.
Also, +1 for any messages of the TAC “wishing you luck” when you have an issue they can’t solve. Not quite what you want to hear 🙂
David Gee
Attempting to share visions and experience with a wider community for the greater good of our industry, covering topics such as: Golang, network device programmability, automation, DevOps, SDN, NFV, NRE, silicon developments, instrumentation, telemetry and programming/coding education for networkers. This blog is full of personal opinions that do not represent those of any employer. All thoughts and words are my own even if I occasionally have to eat them. Currently working as a consultant engineer for Juniper in the UK. I do not make any money from this blog and do not write in exchange for goods or cash.
Follow Me:
