Redundancy in Theatrical Systems

“Two is one and one is none.” This famous phrase describing redundancy, coined by the United States Navy Seals, was used to remind themselves of the basic concept that having one of something is the same as having nothing; leaving the operation open to failure and putting the military operation or people’s lives in the hand of chance.

Redundancy, to many of us, is a word used in an opposite manner to the military; either to describe a person, procedure, or asset that is no longer useful and can be relinquished of existence within the system. However, in many industries like entertainment and theatre, this word could actually be the very key to success.

Over the years, there has been a lot of documentation pertaining to venues around the world that have introduced innovative and creative ways of designing redundancy into their technical systems to keep the performance running, whilst ensuring the safety of their cast and crews. What is so important about redundancy within our technical systems? Redundancy is not just for safety anymore; it is also an important element of efficiency and show continuity. Redundancy can give a technical team the ability to isolate a problem and keep things running. It has even become so popular that most technical design and installation companies will actually implement a redundant system into your initial system design, at your request and an additional cost.

If there are venues that still do not have redundancy available within the technical systems, maybe there should be and it is never too late.

Most redundancy in theatre is out of necessity, not luxury. For example, any theatre that uses large-scale theatrical automation equipment is going to have an emergency backup system or rescue procedure to get an artist out of the air safely should the system lose power. However, this particular type of backup or redundant system is a necessity, mainly due to safety regulations and industry standards. What about the luxury of keeping a show running? Are there theatres or shows out there that put redundant systems in place purely to keep a specific scenic element or entire performance at peak efficiency? Absolutely. Compromising an integral scenic element for too long or losing too many shows is not good for the reputation of any roadhouse, resident show, or tour. However, this type of redundancy comes with some serious forethought and could possibly be accompanied by a hefty price tag.

So what exactly would be considered a redundant system within the entertainment industry? Redundancy can include any piece of equipment or procedure that can be put in place of the failed equipment/procedure to get the same desired effect as the original, perform a rescue procedure for failed equipment or get failed equipment out of the view of the audience in order for a performance to continue within a specified period of time. Redundancy can be identical components within a system or a set of functions put in place to mimic the original system to increase system efficiency, reliability, and by extension, safety. This can include backup equipment, spare equipment, rescue plans, training, and even show contingencies. This could be anything from a spare sound console with swap over capabilities or a procedure that requires a follow-spot operator to take over for a failed moving light during a dangerous act.

This sounds like something every theatre should have in all of their entertainment systems and on demand, right? Why wouldn’t everybody have these backup systems in place? There are many different reasons depending on what part of the industry one works in, but money, time, equipment, manpower, and even inexperience with a particular system’s failures, are a few that come to mind.

According to Rexander Gonzaga, the Automation Day Maintenance Lead at The House of Dancing Water (THODW)  Theatre in Macau, “[In Theatre], we must design a way to keep the show running efficiently without putting the performance, cast, crew and audience at risk. This can be not only difficult but highly complex and expensive.” Rexander speaks from experience, as he is in charge of five automation maintenance technicians specifically tasked to ensure the efficiency and safety of the performing flying system within the THODW theatre. Rexander is mainly focused on the automated systems there. However, there are many other systems involved within the venue that must be accounted for to ensure a complete show night after night. The cost of redundant systems for every single system, as you can imagine, can be astronomical.

Throughout the life of a theatre, the concept of standards, safety, efficiency and redundancy become the most monetarily demanding and time consuming.

So how do we manage to incorporate efficiency and redundancy into our systems without putting standards and safety out of balance?

The initial step of installing redundancy into our systems is to first find the need for it. Redundancy is great to have but only if it makes sense financially and goes to the greater good of the production.

A great place to start your search for redundancy requirements is to turn your system into a giant pyramid.

Start with the larger requirements at the top, such as, complete system failures (main power, servers, main switches, etc.), sub-system failures and generally anything that could be considered a safety concern or show loss possibility. Next, find the mid-level systems that could cause severe show alterations. Last, work your way down to the smaller requirements such as individual component and scenic element issues.

There are many different ways to identify areas within your technical systems that require redundancy. A thorough review of Standard Operating Procedures (SOPs), examination of system schematics (electrical, hydraulic, etc), and a Failure Mode Effects Analysis or FMEA are just a few to get you started. The FMEA is a step-by-step approach for identifying all possible failures within a design. Basically, an FMEA is a risk assessment for all of your equipment with the outcome being the impact to a performance should an individual component fail. It is a manpower-heavy and lengthy process but can continue throughout the life of the system for continuous monitoring and improvement. Another identification process for system redundancy is the use of a Computerized Maintenance Management System (CMMS). A CMMS used correctly will identify a system’s inefficiencies and breakdown trending through a compilation of faults and failures within a specific functional location.

In general, the more time you invest in the system itself within your specific environment, the more familiar you are going to become with where redundancy is required.

As discussed prior, there are many different types of redundant systems. The most obvious of choices is going to be a backup system. A backup system is typically a similar system (if not exact) to the main system that is already in place and ready for action at a moment’s notice. A backup system should take only a few minutes, if not seconds, to take over after an issue is deduced. Can a spare part be considered a redundancy? It can be argued that a spare part would not be considered a redundancy due to its position on the shelf awaiting installation. However, a spare part could be considered a redundancy if it is pre-programmed (if required) and pre-installed within the system for an immediate switch over, putting the system back online to its original working state. Other redundancies could be battery packs or UPS (Uninterruptable Power Supply), as these can be used to power a system in the event of a power failure.

There are both the pros and cons of implementing redundancy into your technical systems.  The positive side of implementing redundancy within your technical systems is increased safety, increased show efficiency, less downtime, decreased show losses and immediate restoration of your system. The negative side can come with high costs due to increased component procurement and additional manpower for preventive maintenance requirements for both the main and redundant systems.

If we are looking for redundancy in our systems, what is to keep us from going to the extreme and putting in double or even triple redundancy? Additional redundancy is ideal but obviously at your discretion and the discretion of technical supervisors. Is it necessary? Is it worth the time? Is it worth the cost? How many times must this redundancy be put in play before it covers the overhead costs? These are the questions that must be asked prior to the installation of all redundant systems. The possibility of being 100% efficient with zero show losses for the year is a great goal but the overall balance between redundancy, standards, safety, cost and many other aspects of the performance must remain.

Join TheatreArtLife to access unlimited articles, our global career center, discussion forums, and professional development resource guide. Your investment will help us continue to ignite connections across the globe in live entertainment and build this community for industry professionals. Learn more about our subscription plans.

Love to write or have something to say? Become a contributor with TheatreArtLife. Join our community of industry leaders working in artistic, creative, and technical roles across the globe. Visit our CONTRIBUTE page to learn more or submit an article.