Troubleshooting Theatrical Systems: Part 2

The seven steps to troubleshooting your theatrical systems:

#7: Break your system into a pyramid of priority using a Failure Mode Effects Analysis

A great place to break down your system into sections is by layering your system into a giant pyramid scheme. Start with the larger requirements at the top, such as, complete system failures (main power, servers, main switches, etc.), sub-system failures and generally anything that could be considered a safety concern or show loss possibility. Next, find the mid-level systems that could cause severe show alterations. Last, work your way down to the smaller requirements such as individual component and scenic element issues.

There are many different ways to identify areas within your technical systems that will require eventual troubleshooting. A thorough review of Standard Operating Procedures (SOPs), examination of system schematics (electrical, hydraulic, etc), and a Failure Mode Effects Analysis or FMEA are just a few to get you started. The FMEA is a step-by-step approach for identifying all possible failures within a design. In more basic terms, an FMEA is a risk assessment for all of your equipment with the outcome being the impact to a performance should an individual component fail. It can be a manpower heavy and a very lengthy process, but this process can continue throughout the life of the system for continuous monitoring and improvement. Once the FMEA is complete and in place it will be much easier to use a half-splitting technique to get to the failed component at a much faster rate.

#6: HMI/Console Feedback

In most theatrical systems today, technicians will find the use of servomotors and closed-loop feedback control that will give indication to the server, console and/or Human-to-Machine Interface (HMI) on position, condition status, velocity, or any other feedback that has been designated and wired throughout the system. This makes the control desk a great place, if not the best place, to start your troubleshooting.

Most theatrical systems using motion controllers or PLCs will have a BOP or Basic Operation Panel on the front to indicate any kind of fault or failure with the equipment that it is controlling; however, some of these fault codes are very complex and specific, making it extremely hard to determine the fault without the use of a pre-determined fault code list or searching on the Internet. For this reason, most show control design companies will place pre-set lists of faults in data blocks within their system so that when a normal occurring fault appears, it is easily seen on the console for an operator to react.

The console can usually not only determine the fault, but also indicate the fault’s exact location. These pre-set fault lists, constantly monitored condition status of hundreds of inputs, and the location of those inputs is what makes the HMI or control console the first port of call when it comes to troubleshooting any theatrical system.

#5: Cheat Sheets

Cheat sheets are the easiest to establish, least expensive to establish, easiest to understand, and require the least amount of training to technicians for reliable information to be relayed during a troubleshooting situation.

What are cheat sheets and what can they be used for? Cheat sheets can be anything that can quickly help a technician determine what is in a system or what a system should look like during normal operations. This could simply be laminated schematics, a list of fuses, breakers or any other piece of equipment that might need to be located quickly or it could even be a printed snapshot of an I/O module’s LEDs during normal condition status. If a system should be in fault, there is a good chance that one of these lights will be the opposite of normal conditions and can easily be determined without the need of any expensive test equipment and time consuming half-splitting techniques.

For cheat sheets, you can use as much imagination and innovation as you want, however, keep in mind that your biggest limitation may be indoor cabinet space.

#4: Redundant/Spare Parts

Redundant parts can be utilized to determine a failed part, while exchanging the failed part with a working spare, killing two birds with one stone. If the part swap is successful, then you have both identified the failed part and remedied the issue at the same time. However, using redundant spares for troubleshooting can come with its own complexities.  In one of my previous articles called “Technical Redundancies: Finding the Balance,” the many different types of redundant systems that be used for troubleshooting are discussed and described in detail.

The last thing to be said when using spare or redundant parts for troubleshooting purposes is that you save an immense amount of time by testing and replacing at the same time, but you can also lose a significant amount of time should you be changing the incorrect part. Even though this is a great troubleshooting technique, you run the risk of being wrong and can put your initial time frame in jeopardy, so just make sure that if you are going to take the time to start disconnecting parts that you have identified the correct component.

One last comment on the use of redundant or spare parts. It is a common mistake while being under the pressure of  such time-constraints to start changing parts and disconnecting wires without documenting wire numbers or input ports. Any technician can save themselves the headache of going to reconnect a part and not knowing what goes where by either having schematics readily available or simply take a quick photo.

Either one of these are simple to read and make an easy review of connections that can put the technician’s mind at ease when they are second guessing themselves prior to turning the system back on. Last thing you want to be responsible for is damaging the system even further due to an incorrect wire job. Take your time and check your connections.

#3: Testing Equipment

Testing equipment is the obvious go to for the expert troubleshooter in the time of crisis, not to mention, it is always highly recommended to have the right gear available for troubleshooting a system safely. Testing equipment can be a very quick and easy way to test for 24 VDC, fuse continuity, pneumatic/hydraulic pressure, or signal flow. This can be the quickest and easiest way to know that a component has failed and not outputting the correct voltage or information. Get the part replaced, tested and move on. Just be wary that any type of testing equipment can also be confusing and worse of all time consuming for the unprepared or untrained technician. Confusing? What is confusing about testing equipment?

Well, first, let’s talk about what type of testing equipment could be used for initial troubleshooting in a failed system. There are voltmeters, ammeters, digital multi-meters, infrared thermometers, or even pressure gauges to name a few. What could possibly be confusing about how to use a digital multi-meter? You just switch it to the correct setting and test, right? Well, sure, I would be willing to bet that unless you are really having a bad day and the meter has failed that you are going to get an accurate reading. However, in the heat of the moment it is easy to not know what the correct reading should be.

For most theatrical systems, there are a lot of electrical contacts that have either been configured for Normally Open (NO) or Normally Closed (NC) circuits. This can be easily confused in a time-sensitive situation and make a signal flow analysis with testing equipment extremely time-consuming and full of false-positive readings. A false-positive reading could send you in the wrong direction and have you replacing the wrong part. It doesn’t matter if you’re talking about an electrical system, pneumatic system or a hydraulic system, unless you know exactly where or when there should be electrical or hydraulic pressure or have schematic readily available (see #5: Cheat Sheets), then it is best to leave the testing equipment for continuity, residual capacitance/pressure checks or safety checks only.

If you still insist on the use of testing equipment in your arsenal of troubleshooting techniques, then the most important piece of advice that can be given is practice, practice, practice! A technician definitely needs to practice the use of testing equipment, but make it interesting. You can encourage and entice any technician’s motivation to practice their use of testing equipment and troubleshooting thought processes by putting in place, what I like to call ‘The Gauntlet.’ ‘The Gauntlet’ is a list of systematic errors that have been intentionally put in place for the purpose of troubleshooting training and system analysis. This can be anything from pushing multiple E-stops throughout the system to putting pre-blown fuses in the system to symbolize a failed component. For additional complexity and fun, you can even put technicians in the field in pairs with the added stress of a time-clock set at your discretion.

Make it fun, but keep it safe. Try and stay away from high voltage areas and brake contacts that could cause free falling scenery, if you catch my drift.

#2: Diagnostic Software

Diagnostic software is pretty self-explanatory and doesn’t require too much background. Most large-scale theatrical systems designed within the last ten years, utilize robust closed-loop feedback systems, Programmable Logic Controllers or PLCs, and/or Distributed Control Systems or DCS. These types of systems usually have integrated topology built within the system for the use of diagnostic software. What does topology mean? In the most basic of terms, it means that the software can see the condition status and functionality of each component of the system individually as well as a part of the system as a whole.

Most name brand automation and PLC companies used in industrial theatrical control systems such as Siemens, Beckhoff, and Pilz will have PG computers, control software with integrated diagnostic software. Any type of diagnostic software that can connect to the system in fault while it is still running, is not only the fastest way to determine which component is in fault, but it can also allow you to troubleshoot while the show is still in progress. Yes, the show may need to eventually come to a pause, but hopefully by that time, the faulted component has already been identified and a replacement is on stand-by ready to go.

Now a word to the wise on this subject. If technicians are not trained to a significant level of competency and do not know how to use the control and diagnostic software, then do not allow them to attempt to try and troubleshoot with it. There is one major setback to the orgy of information that a PG computer can give to the end user. The extremely powerful PG computer contains much more than just the main interface software. It can include, but is not limited to, expert lists, diagnostic software and possibly hundreds, if not thousands of parameters used to properly set anything from the Variable Frequency Drives to the angular commutation of the absolute rotary encoders.  Any data that is changed or downloaded to the system via these parameters can cause the entire system to stop or put the system into an unsafe condition.

In other words, the troubleshooting of your system through the use of diagnostic software can be extremely quick and powerful, but it should only be attempted by a highly trained and competent technician or with the direct guidance of the experts. If you do not have the monetary resources to purchase a version of the diagnostic software described in this step, thanks to technology, you can receive this type of assistance from a vendor using remote software over the Internet from the opposite end of the Earth should it be required. It may be costly, but necessary to get back up and running.

#1: Simply Knowing Your System Inside and Out

The easiest and most powerful step has been saved for last.

This step only requires one simple task and that is that technicians spend as much time within their systems as they possibly can. That’s it. Simple right? The philosophy goes like this. The longer that is spent in one particular system and with the system, the easier and faster any technician is going to be able to troubleshoot it.

This time does not physically have to be completely spent in or at the system but instead, can be spent in the newly developed FMEA documentation, electrical/hydraulic schematics, or OEM manuals. However, the rest of the time can be used observing how the system operates under normal conditions, including the use of test equipment on all components while they are known to be in good working order. This will give any technician the edge required during a breakdown to know the correct voltage/pressure a system should be giving for easier and quicker analysis. Refreshers at regular intervals are definitely recommended for maximum retention over time. You’d be amazed how fast you forget the details.

Long story short, most technicians have a hard time troubleshooting their equipment, mainly due to the fact that they do not spend a lot of time with their system when the components of that system are working properly. Oppositely, they only make a journey to their system when it is either breaking down or has broken down at the most inopportune time. The best thing technicians can do is spend as much time with their equipment to understand things such as, but not limited to, the normal environmental conditions, the sounds a cabinet normally makes, when a light should be on instead of off, or even the smell of a cabinet. Once each technician becomes that ingrained with the inner workings of their equipment, it will be hard not to know when something is either down or on its way out.

Round-Up

As I mentioned at the beginning of this article, there are no systems that are created equal, so troubleshooting of each system must be taken on a case by case basis. Each one of these steps is a great place to start troubleshooting any system that you can encounter, but I must re-iterate that not one of these steps is going to help you without the correct amount of information or preparation.

Whether it be cheat sheets, use of testing equipment, or knowing your system inside and out, every single one of these steps requires a significant amount of time and effort in practice, setup, and understanding. There was nothing at the beginning of this article that mentioned that any of these steps were going to be quick, but no one said that all of the preparation has to be boring and monotonous. Remember, ‘The Gauntlet.’ Training exercises do not have to be as complex as this, but making it interesting for the technician is definitely going to give you an edge when it comes to getting their attention and retaining the information for later use. If you put in the right amount of practice, preparation, dedication, and energy towards the system, then faulted components are going to be much easier to identify when the time comes.

Lastly, no matter what happens during your period of troubleshooting, try and remember 3 things. Stay safe, don’t get discouraged and never make a situation worse than the condition in which you found it. Good Luck!

Troubleshooting Your Theatrical Systems: Part 1

Also by Jay Withee:

Safety: The Show Must Go On – Or Does It

Cover photo: Janusz Ciechowski

Join TheatreArtLife to access unlimited articles, our global career center, discussion forums, and professional development resource guide. Your investment will help us continue to ignite connections across the globe in live entertainment and build this community for industry professionals. Learn more about our subscription plans.

Love to write or have something to say? Become a contributor with TheatreArtLife. Join our community of industry leaders working in artistic, creative, and technical roles across the globe. Visit our CONTRIBUTE page to learn more or submit an article.