Realtime Music Solutions

1) "Towards a New Form of Opera"

TOWARDS A NEW FORM OF OPERA PRODUCTION
November, 2003

Dr. David B. Smith
Dr. Frederick Bianchi

              Introduction: Need for a New System
                1.    The Nature of Opera's Expense
                2.    Essentials
                3.    Developing Efficiencies
                4.    New Capabilities
                5.    Advantages
                6.    Amortization
                Conclusion

Introduction: Need for a new system

The viability of opera, as in any art form, depends on maintaining and increasing the size and appetite of its audience. The way to do this is to create additional opportunities to experience one of the great art forms of western civilization.

Unfortunately, opera has become expensive. If there is an advantage towards building the audience for opera, more performances must be made available. However, limited funding exists. This paper discusses some of the issues around this problem, and suggests strategies using new and emerging technologies that may help effect a solution.

Because of the increasing cost of opera production, the performance of opera has become more difficult, with most resources focused into a small number of large institutions. It has also contributed to the demise of the opera tour. Outreach initiatives are usually shadows of the fully staged version and can not demonstrate the full impact of opera.

Under these circumstances, it becomes important to increase the efficiencies of producing opera without sacrificing quality. Only by lowering the costs of producing opera will it become possible to provide more opera performances in more venues.

1.    The Nature of Opera's Expense

It is important to identify the factors of opera performance that add to its great expense. It is important to note that opera combines all the aspects of traditional theatrical productions with all the aspects of symphonic performance. Because of this, rehearsal and production costs become immense. Sets, scenery, costumes, lighting, communication, transportation, administration make their own contribution. A large amount of space is required. Cast, orchestra and crew sizes create a significant payroll. Is it possible to increase the efficiency of opera without sacrificing its impact?

2.    Essentials

Opera cut to its bare bones consists of vocalists singing and acting a story to a flexible musical accompaniment. It is the multimedia experience, however, that makes opera compelling. The question becomes how best to enhance the scintillating experience of great singing with the trappings of theatrical performance?

3.    Developing Efficiencies

In the history of performance, a number of different techniques for dealing with the cost issue have been developed. Often, a concert version will eliminate the theatrical trappings. Smaller organizations may resort to reduced ensembles or the use of a single piano. In all instances, it can be seen that the one component that always remains is the cast.

Emerging systems within the world of entertainment technology have the capability of providing powerful visual and sonic environments within which opera performance can occur. These systems have efficiencies many times greater than more traditional theatrical techniques and in synergy with each other can create an entirely new experience.

4.    New Capabilities

In opera, events take place within the metrical time of the music. Minutes and seconds are replaced by measures and beats. This elasticity of time is of critical importance to artistic expression. Systems synchronized to the metric time of the musical score offer a tremendous advantage towards increasing the efficiency of performance.

Once control of metrical time is established, it becomes simple to provide musical accompaniment. Recent advances in computer music technologies have enabled the realistic simulation of acoustic instrumental performance. Realism continues to increase exponentially and will do so until eventually it will be impossible to differentiate between acoustic and electronic performance.

A single musician can play a Sinfonia orchestra system. Alone or in accompaniment with other instrumentalists, Sinfonia also provides the ability to provide the exact musical time to any of a variety of theatrical systems.

One such system is image projection. Scenery can be replaced with a minimal set of platforms backed by one or more large rear projection screens and attached to image projection systems. This allows for a visual palette that can be changed by modifying or replacing the image at any time. Because it is synchronized to the underlying music, extremely accurate and complex transformations can occur. The resulting images can be dynamic and descriptive in ways that traditional scenery can never hope to match. It has the potential of being a powerful artistic tool.

Lighting systems, as well, can be controlled with the same level of accuracy and without additional human operation. Because the resolution of control is very small (10 ms or less), lights can be programmed to pulse their design along with the music.

Although much development remains, these systems are not experimental. Sinfonia has been used successfully in thousands of musical theatre performances in productions of all sizes. Video projection technology can be seen in practically any modern commercial music concert. Moving lights and their control are sophisticated, and there are small packages that can create powerful effects. The cost of all systems is plummeting while their capabilities increase.

5.    Advantages

There are many advantages to the use of such a system. Performances are easily repeatable. Once programmed, they will perform consistently. A single staging system can be programmed to support many different operas without any additional material expense. The size of the package is much smaller, and viable performances can occur in spaces otherwise too small or unusual to fit a traditional production. The package can also be transported and installed more easily. Personnel costs can be reduced. The ratio of upfront expense to performance expense is greater, making it easier to maintain a production over a longer period of time. Because the package is identical, a single company could load into a theatre and perform multiple operas over different nights without changing anything but the underlying show data in the system. Storage costs are much less: cumbersome sets are eliminated, and a single production can be revivified without having to invest in the space constraints required with more traditional technologies. Maintenance of mechanical elements is reduced or eliminated, though of course electronic systems have their own maintenance cycle.

More importantly from the artistic standpoint, these proposed systems also creates a new vehicle for artistic expression. The seamless integration of lights image and music within the flow of metric time creates a powerful multimedia environment that, in its essence, is the epitome of the opera experience.

6.    Amortization

Are these systems inexpensive? Not at the present moment. It is true, also, that the amount of programming and rehearsal that is required would make their use in single or small number of performance environments more expensive than traditional solutions. However, once a show is programmed, the cost per production drops in ways that are impossible with more labor-intensive solutions. In addition, the costs of storing multiple sets for different productions is eliminated, since the same projection system can be used for an unlimited number of productions: it is the programming that changes. Because the overall size is also much smaller than traditional sets, an entire production can easily be shipped from location to location.   This allows for touring possibilities that would be otherwise impossible.

Once a production has been developed, it can be sold or leased to other companies. Indeed, because of the nature of software, an entire production can be cloned and performed simultaneously in multiple places. Because overall costs of reusing productions are drastically lower, other companies can take advantage of adding productions to existing seasons with the probability of actually generating a profit. This will not only allow for a longer season, and therefore more opportunities for audience growth and outreach, but also provide income streams with which to fund operas using more traditional techniques, if desired.

Conclusion

The Sinfonia+ system does not intend to replace existing forms of the art of opera. Instead, it makes possible the creation of many new productions using efficiencies impossible with more traditional approaches. In order to create a demand for larger market share, it is the duty of the opera community to commit to providing opera in as many different flavors as possible. Only by providing opportunities for greater audience can audience become greater.

2) Arguments for Enhancement Technologies in Live Performance

Arguments for the Developing of Enhancement Solutions for Orchestras in Live Performance
Dr. David B Smith
Dr. Frederick Bianchi

(Adapted from a post to the Theatre Sound Mailing Mailing List in response to a thread on use of synthesis in the pit: look for “Musical Director at this link:
http://www.brooklyn.com/theatre-sound/archive/older/AUGUST-2005/thread.html)

This paper discusses current efforts to create more convincing and musically interesting solutions to problems related to the economics of modern performance environments

Part 1: Historical Solutions to Orchestral Enhancement

I think that we all can agree that for a show traditionally scored for acoustic instruments, the original intentions of the composer/arranger should be honored when feasible. However, for reasons of budget, space, time, or instrumentalist availability, this very frequently cannot happen. Productions must then rely on a reduced instrumental set. When this occurs, a variety of compromises have been used to rectify the situation.

1) Reduced orchestration. Creating a smaller ensemble by hiring an orchestrator to redo the parts and fit it into the provided mix. This is time consuming, expensive, and ends up with a smaller, thinner sound. Sometimes the increased intimacy works to the benefit of the show: often it does not.

2) Prerecorded tape (or "click"). Obviously, a set of acoustic musicians can be hired to perform the show in the studio, and a high quality tape can be generated from this. However, this is also expensive, and creates problems in that the tempo of the show is fixed, forcing any accompanying musicians and performers to follow the original tempi, destroying one of the very important components of live performance. In addition, unless heavily multi-channel outputted, the front of house mix is unable to make internal adjustments on a show-by-show basis. Also, any changes in rehearsal, vamps, additions, etc., require a revisit to the recording studio, at extra expense, and with the problem of attempting to match the mix of the original recording session.

3) Synth/Sample players. This is a very common way to increase the richness of the orchestration, and has in fact been used at all levels of production for several decades. Most productions on Broadway and the West End use some form of this type of enhancement. However, this solution creates the problem that a collection of monophonic instruments has been combined into a single polyphonic instrumental part. Once this occurs, no matter how virtuosic the performer, the performance nuance of the individual lines becomes lost. This is impossible to rectify, and creates an organ type sound, no matter how accurate the sample set being used.

4) Sequencing. Sequencers are powerful tools, and are being used more frequently in performance situations. However, most sequencers are designed for studio applications, and therefore do not address many of the needs of the live performance situation, such as flexible tempo, arbitrary vamps, instrumental levels, and other types of performance flexibility.

It is important to note that these solutions have been used to varying degrees of success in performances at all levels. Many times, the choice is between either a reduced orchestra or outright canceling the production. It is impossible for any except the largest organizations to perform the original Roger and Hammerstein scores, and even these can no longer fit these orchestras into existing pits.

This is an age of computer-based technology. Every other area of human endeavor is using computers and their applications to increase productivity. This application can transform the art form and our entire culture is seeing this transformation. Why is the orchestra pit considered sacrosanct from this? Certainly the sound and lighting designers on live theatrical productions rely heavily on these new technologies to perform their art. If the need arises, why not utilize the best technologies available to give the audience a higher quality sounding musical experience?

Part 2: Analysis of the Problem

As stated in a previous post:

>The whole "Live" experience IMO includes the interaction of the performers, the acting, the singing etc. within the environment of the stage set design, lighting and sound,

Obviously, if requiring some sort of enhancement solution, it is important to be able to integrate into the theatrical environment with as little disruption as possible. This means that more modern innovations are required to address these issues. If the reduction of the orchestra size is a given (and in most cases it is), then the interaction between the human performer and the performance system needs to be studied very carefully. We are faced with two important considerations here.

1) The nature of musical performance indicates that this human needs to be a musician, sensitive to the needs of the performance, and able not only to follow the conductor’s baton, but also to blend with the various instrumental sections in a convincing and musical way. They need to be able to read notation, and the system must be flexible enough to be modified during both rehearsal and performance situations. Otherwise, the musicality of the system remains limited to predetermined input, and this is antithetical to the live performance situation.

2) However, each human has only a fixed mental bandwidth upon which to draw. This poses certain limitations when asked to play the work of multiple instrumental parts. This is true with traditional instruments as well. Note that monophonic instruments can generally be controlled during the entire generation of an individual note: a violinist is responsible for all aspects of each note produced. Multiphonic instruments, however, which require the performer to generate many notes simultaneously, require less responsibility for the production of each note. A piano player is responsible for the attack and the release of each note, but not any of the internal generation: sustain, decay, and other features are created automatically by the acoustics of the instrument. This relationship between polyphony and responsibility is obvious, and created by the limitations of the human bandwidth.

So, by continuing to increase the polyphony of a single performer, and at the same time retain the musical individuality of the independent monophonic lines, new instruments with new paradigms need to be created. Fortunately, in the world of musical theatre, an additional consideration can help to resolve this problem.

Limitations of Live Performance. The fact is that most of what an instrumentalist does is preordained. Whether notated in the score, decided upon in rehearsal, or created by the limitations of the specific instrument, pit musicians actually have very little choice about what they do. If they change a pitch, dynamic level, rhythm, or other information, what they have done is just plain WRONG. So much of what a musician does is mechanical. The musicality comes out of HOW they perform these structured events.

This insight provides us with a way to overcome the human bandwidth problem, and still to allow for musical nuance while increasing the number of instrumental parts a single performer can handle. The technique is to analyze the music, determine those aspects that are fixed, and assign that responsibility to the computer instrument. Those aspects that require human expression can then be given to the human performer, who has now been freed from mechanical reproduction requirements. Thus, in the same way that a pianist can perform more notes than a violinist because they are not responsible for the internal production of those notes, this new instrumentalist can perform many more events than either by further reduction of the mechanical aspects of performance.

This is the direction of our research, and has produced some amazing results and ramifications for modern musical enhancement of acoustical performances.

Part 3: Moving Forward with a New Instrument

Many of the objections to use of computers in pits stem from their use within shows that use traditional acoustic ensembles. However, as stated, we live in a world of transformation, and this has not been limited in any other area of musical performance. Styles of music rely more and more upon electronic timbres. Performers rely more and more upon electronic reinforcement or amplification. More and more of the music we hear has passed through the filter of sound systems, recording studios, processing, etc. etc. etc. Our audience accepts this as part of the world within which we live. It is natural that this mix of acoustic and electronic continues within the pit.

Many modern musicals follow in this tradition, and rely heavily on electronic sounds as their source, thus jettisoning the traditional orchestral pallet for more modern flavors. And this is not new: from the earliest musicals of A. L. Weber, this has been gaining momentum (and his stuff is now over thirty years old!). Recent hits such as Rent (I guess Rent can't be considered recent either, huh?) and Movin' Out display the effectiveness of these smaller ensembles.

What is important is not the source of the music, but the ability of that music to provide an authentic artistic experience for the audience and the performers. This, as always, comes down to the skill of the composers, arrangers, and performers. It is easy to denigrate the use of electronic instruments in the pit by pointing to bad examples, and there are many bad examples at which to point. This does not mean there are not good examples of intelligent and artistic use of these new instruments: usually, their effectiveness is proven by lack of criticism.

It is also easy to criticize these new instruments because some are proving inadequate or inferior to existing solutions. However, the current state of the art is not the ultimate state of the art. Traditional acoustic instruments have had the benefit of hundreds of years of development and innovation. In fact, little, apart from the introduction of newer materials and manufacturing, has changed in the last hundred years. The most recent acoustic instrument added to the pallet is the saxophone, and that was invented in 1842, over 150 years ago. Computer based instruments, however, are much newer, and they are still developing at an exponential rate. It is crazy to imagine that state of the art will remain where it is today, and dismissal of these instruments is similar to early equestrian's disdain of the automobile. At the time, they were noisier, slower, and more problematic than the horse, but what is the predominant mode of land transport today? I am sure that the maker of the final buggy whip made an amazing buggy whip, but do not expect much of a return on your buggy whip stock these days.

I just came back from Austria, and when I was there, I saw the Lipizzaner horses perform at the Spanish Riding Academy at Wien Hofburg (I have a ten year old daughter). They were absolutely gorgeous; performing beautiful synchronized ballet. They are also the only remaining example of this art form. We then left and took the UBahn, a train, and an airplane back home. I am glad I did not have to ride a horse.

Aesthetics