Anyone that's ever worked in maintenance-level tech support knows what a miserable job it can be. The profession, such as it is, mainly involves two things: 1) listening to frustrated and disgruntled customers who are whining about their service interruption and demanding to know what you are doing about it and when it will be fixed, and 2) fixing things when they break. I've said it before: entropy is king, baby; things break all the time. While the job wouldn't be quite so stress-inducing if it only involved step two, an optimist would argue that at least this usually equates to some level of job security for tech geeks.

Dealing with idiots customers who do not know how to operate their computer or connect to the Internet is often child's play compared to the challenges associated with technical problems in the telecommunications industry. I've done both for many years, and while you may not be needing my wisdoms bad, I'm gonna share some of them with you anyway. Bear in mind that this advice pertains to my realm of experience in North America while working for a radio common carrier, and is probably nothing compared to the horrors of dealing with British Telecom or any other overseas telecom provider. If you are a tech for an RCC, you probably know all this stuff already. This writeup is for those of you who aren't in the phone business, and who might find our work day slightly interesting.

Dealing with the largest computer on Earth

I've long held the belief that anyone who looks up the definition of ubiquitous in the dictionary should find an illustration of a telephone. You probably own many of them, including wire-line POTS handsets in your home and cellular phones, and odds are high that you take them for granted every moment of every day. You may not have ever given it much thought, but that telephone on your desk or on your hip is a network node in the largest computer system on Earth. Anyone can dial a series of digits and directly connect, in fractions of seconds, to any other telephone anywhere in the world at a moment's notice. How it all works, so quickly and affordably, is nothing short of magic. Not only is your phone reasonably durable and cheap to own and use, but it is exceedingly reliable. If your personal computer was as dependable as your telephone, much of the PC tech support industry as we know it today would dry up and disappear.

It's no coincidence that the Internet's famed reliability, based on redundancy and multiple paths available to a given destination, is similar to the telephone network: they are one and the same. But as great and wonderful as the global telecommunications network is, it is just as fraught with technical problems as your local ISP. The major difference is that the voice traffic side of the network is so much bigger than the data traffic side, it boggles the mind. Still, things break... a lot. But the network is so unbelievably huge and fail-safe on so many levels that most folks hardly notice. My role as a cog in the great machine is to handle those times when folks do take notice.

Identifying the problem

As any seasoned telco tech can tell you, identifying the true nature of the problem is one of the most significant obstacles faced in the path leading to resolution. Reports of trouble from customers or customer service representatives are usually vague and misleading. "All of our numbers are ringing busy!" can often mean that a certain small block of phone numbers is experiencing a problem. "No calls are going through!" usually means that a single, isolated radio transmitter is down somewhere out in the woods. Customers invariably tend to extrapolate their particular, momentary service interruption into a widespread outage of catastrophic proportions. While I have experienced a few of these (9-11 was one such event), they are few and far between. It usually takes several trouble calls and numerous pointed questions to the callers before you can begin to reasonably identify the possible nature of the problem at hand.

Contacting the service provider

As a technician, if your particular telco problem involves local phone numbers, this step is usually easy. Contacting regional and local telcos like Verizon, Sprint, any of the Baby Bells, or even tiny private telco providers usually means dialing a toll-free trouble reporting number that's been the same for ages. For some reason however, long distance carriers like Sprint, AT&T and Worldcom like to change their trouble reporting numbers on a regular basis. The number you once dialed for assistance can often be disconnected, changed, or terminate into a call center that handles a completely different segment of their business operations. It can sometimes take a half hour and a half dozen phone calls and transfers just to get you to the personnel who are qualified to record your problem. Having patience is helpful, as is the willingness to repeatedly explain your problem in detail.

Opening a trouble ticket

Once you get a carrier's customer service rep on the line who handles the type of problem you're reporting, the next step is strictly clerical. This process is referred to as "opening a trouble ticket." The person you will be speaking with is almost always not a technician of any sort. (If you call after regular business hours, you might reach a tech in the central office, but even then their initial function is the same.) They will have only rudimentary understanding of the technical problem you are describing to them, and their job is to 1) use their training, experience and judgement to determine if you should even be speaking to them in the first place or if you should be sent to someone else, and 2) if valid, record exactly what you say, including all applicable circuit ID numbers and your description of the problem. Once they have taken all of your information, you will have to wait (sometimes as much as ten or fifteen minutes) for them to issue you a trouble ticket number. Their computer terminals are usually slow and sometimes overloaded. Once you get your number, you thank the operator (that's all they are, really) and hang up. Then the real waiting begins.

The wait

Let's face it: Your carrier has you by the short hairs. You might have 20,000 customers without service, but they are going to take their sweet time in getting around to your problem. You're not their only customer with problems, after all. If you have authorized intrusive testing of your circuit(s) by the carrier, automated testing by the carrier's maintenance computer system will begin almost immediately after you have opened your ticket. Occasionally this process will find and identify your problem, but usually not. If more than an hour passes without a return phone call, it is usually a good idea to call the trouble reporting number again and check on the status of your ticket. If you have a major issue requiring expedited resolution, you can request ticket escalation if you've not gotten a call back within an hour or so.

Dealing with carrier technicians

When you finally do get a call back, it will be from a technician who has picked up your trouble ticket and has the unenviable task of figuring out what to do next. They may or may not have useful information about a major outage or related equipment failure that is affecting your issue. Regardless, never make the mistake of assuming that the carrier tech has the first clue about what they are doing or what may be causing your problem. These are union people, after all. They generally have a great deal of knowledge about their job and how to perform the necessary tests and inquiries it takes to research and identify your trouble, but they are also marking time until shift change just like the rest of us. My experiences have more or less been like this:

  • 30% of the time, the tech will be 20 minutes or less from the end of their shift, and will dance around your problem in order to pawn it off on to the next shift, or will try to convince you that the problem is related to customer equipment (i.e.: it's your fault that your circuits are down).

  • 25% of the time, the tech will seem competent and earnest in their attempt to root out the source of the trouble. If the issue is something less obvious or more complex than router or connectivity problems, or if translations or third-party carriers are involved, you will get one of a myriad of excuses as to why the problem is not related to the carrier's equipment or infrastructure. Fighting resistance of this sort can be time consuming and frustrating, but the onus is upon you to prove that the problem lies with the carrier. Innocent until proven guilty is the logic at work here.

  • 22% of the time, the tech will perform a series of tests and ask a lot of questions, yielding no results. These techs are usually nice people who sincerely want to help you, but they are dumbasses. They will overlook obvious sources of trouble, disregard simple checks and test routines, and fail to look at the problem from a holistic view (after you have explained to them the various troubleshooting steps you have already done on your end of the pipe). You can shoot yourself in the foot going round and round with these folks, wasting days of time - the best thing to do when you encounter one of them is to escalate your ticket or speak to their supervisor.

  • 20% of the time, the tech is a whiz kid who knows exactly what you are talking about, and can identify the trouble in short order. They think quickly on their feet, analyze the issue by taking all possiblilites into consideration, and never presume fault when their testing turns up no helpful results. They will work with you for as long as it takes, and do whatever is necessary to insure that the source of your problem is identified and fixed. I love these people - they are worth their weight in bandwidth.

  • 3% of the time, the tech was just promoted or hired into this department, and can not find their ass with both hands. Escalate your issue immediately, or ask to speak with their supervisor.
The important thing to remember when dealing with carrier techs is that they are working slobs just like you. Sometimes they haven't had enough coffee to function normally, or maybe their substance abuse problem is getting out of hand, or perhaps they are having a really good day and are on top of the world. Your mileage may vary. Stay frosty and keep thinking around and ahead of the technician you are dealing with - it will save you time, and make your customers happy sooner.

Doing your part

This doesn't necessarily go without saying, so I'm going to say it: As an RCC tech, you must take an active role in troubleshooting your problem to the fullest extent possible before contacting your carrier. If you haven't done the necessary alarm checks and testing on your end, you will have no leg to stand on when the carrier tech starts asking the hard questions. Carriers are quick to bill your company for wasting their time, even though they have no worries about wasting yours. Once you have their undivided attention, it is paramount that you continue to work with them every step of the way, bringing your equipment down or up as may be warranted, doing hard power resets, or swapping hard wire connections on site as need be in order to eliminate possible sources of trouble. But always, always get your shit together first, then call your carrier.

The last word

Carrier-level telco problems suck. The customers hate them. The employees hate them. The boss hates them. But they are a fact of life just as sure as the sun will rise tomorrow. Sometimes you will be juggling many of them at once, even into the wee hours of the morning, but you can't close that trouble ticket until the problem is fully resolved and your customers' service is restored. If the carrier closes your ticket without your consent (oh yeah, this happens quite often), you must open another one and press on. Sometimes the problem can drag on for weeks, and if the issue is so isolated as to affect only one customer, perhaps months. Still, you must persevere. It's your job. When you're done, you can put your feet back up on your desk and take another nap until the phone rings again. You know it will.

Good luck.

Log in or register to write something here or to contact authors.