Overtone Tracking in Music and Performance Systems
Overtone Tracking in Music and Performance Systems
Demonstration 4: Stage and Musical Instrument Lighting Effects Controlled by Measured Time-Varying Overtone Amplitudes
Continuing, the NRI patent-pending technology [1],[2] provides for real-time amplitude measurements of the fundamental and portions of the overtone series of a pitched audio-frequency electrical signal and the use of these measurements to create real-time control signals for controlling of light in performance situations. The pitched audio-frequency electrical signal may be produced by a microphone, the pickups of a musical instrument fitted with transducers, electronic synthesis, etc. In the case of a musical instrument fitted with transducers, the vibrating elements of the musical instruments may be sent into vibration via traditional means, or alternatively by electronically control drive transducers [3] wherein the harmonic variation of the vibrating element can be richly varied. This demonstration page illustrates some exemplary lighting arrangements responsive to measured real-time overtone amplitude control signals of a provided audio signal. The example assumed for the provided audio signal is assumed to have to fundamental and overtone amplitude dynamics of the first five harmonics that was depicted in the earlier harmonic demonstration page, for example the fading note of a string or synthesized sound. In the first example application, the five control signals responsive to the measured real-time overtone amplitudes are used to control red, green, and blue lighting components shining on stage performers or objects. In the animation below, the red component of the left downward light beam is fixed at 70% of maximum, while its green component diminishes from a 100% initial value to lesser intensity levels responsive to the decaying measured amplitude of the second harmonic and the blue component diminishes from a 100% initial value to lesser intensity levels responsive to the decaying measured amplitude of the third harmonic.In the right downward light beam, the green component of the light is fixed at 80% of maximum, while its red component diminishes from a 100% initial value to lesser intensity levels responsive to the decaying measured amplitude of the fifth harmonic and the blue component diminishes from a 100% initial value to lesser intensity levels responsive to the decaying measured amplitude of the audio signal fundamental (first harmonic).
As a variation of this example, the animation below maps the levels of the red, green, and blue components of each downward light beam into a hue light color representation with constant brightness and constant saturation. The resulting hue model is capable of sweeping out colors of the rainbow as a single parameter is varied. The animation below comprised four such hue model beams, two left of center and two right of center, respectively responsive to the measured amplitudes of the second, third, fourth, and fifth harmonic of the provided audio signal. The center light is a white light responsive to the amplitude of the fundamental (first harmonic) of the provided audio signal. Here the provided audio signal is depicted as the fading note of a guitar string recently struck by the rock, jazz, or country music performer depicted.
In each case, the lighting respond to the note-to-note nuances in harmonic structure as the performer providing the audio signal source varies the tone production. Additionally, the mappings between light and the control signals may be themselves be time-varying so as to provide variability in how the lighting behavior varies in response to any nearly-repeating harmonic structure dynamic. Such variable control signal mappings may additionally be responsive to the pitch of the note (via, for example, pitch-to-MIDI conversion), Such variable control signal mappings, as well as other complex mapping which may be used for realizing the afore described the hue model transformation, are covered in other pending NRI patent application [5]-[7].
The variations in relative harmonic amplitudes may also be used to control lighting physically integrated into a musical instrument. The general use of control signals to control lighting physically integrated into a musical instrument is covered in a pending NRI patent application [7]. In one example implementation, the brightly illuminated intensity and or color of an instrument string may be varied (employing RGB, Hue/Saturation/Brightness, or other color model) responsive to the decaying measured amplitude of the various harmonics in the overtone structure of the original or processed sound produced by that instrument string. The strings themselves may be controllably locally illuminated or even emit controllable white or colored light. A forthcoming NRI patent [7] provides systems and methods for controllable localized illumination of individual instrument strings suitable for performance applications. These systems and methods include support for stroboscopic signals that can be used to create opto-mechanical standing wave and beat-frequency visual effects.
Returning to the control of string illumination by overtone tracking control signals, the animation below depicts four example applications to the strings of a bass guitar. For the sake of this artificial demonstration, all strings are depicted as being struck at the same instant and possessing the same dynamic change in the relative amplitudes of the harmonics. In practice this situations is rarely if ever the case, but this assumption is useful here as the illuminated behavior of various lighting models and harmonic mappings may be compared. The top-most string is illuminated by white light responsive to the amplitude of the fundamental (not the overall amplitude) of that string's audio signal. The next string down is illuminated with an RGB model with red, blue, and green respectively responsive to the second, third, and fourth harmonic amplitude of that string's audio signal. The third string down is illuminated with a Hue/Saturation/Brightness model with hue, saturation, and brightness respectively responsive to the fourth, second, and fifth harmonic amplitude of that string's audio signal. The bottom-most string is illuminated with a Hue/Saturation/Brightness model with hue, saturation, and brightness values respectively responsive to the negative second, positive third, and positive fifth harmonic amplitude of that string's audio signal. Note that the Hue/Saturation/Brightness model provides the most dynamic color variation, while the RGB model starts with a whitish light and evolves smoothly into a fading color with limited variation in hue. Of course, various mappings may be used to modify how the brightness and color behavior respond to variations in measured relative harmonic amplitude.
In another example implementation, the brightly illuminated intensity and or color of light sources on an instrument body may be varied (employing RGB, Hue/Saturation/Brightness, or other color model) responsive to the decaying measured amplitude of the various harmonics in the overtone structure of the original or processed sound produced by that instrument. Further, the mappings may include animation processes such as low-frequency modulations of amplitude and offset, sequencing, etc.[5]-[6]. Examples configurations producing such effects are shown below. In (a), an overtone tracking control signal multiplies the output of a low-frequency oscillator or control signal sequencer. In (b) an overtone tracking control signal is added to the output of a low-frequency oscillator or control signal sequencer. In (c), both of these approaches are combined in an exemplary composite chain. In (d), an overtone tracking control signal modifies the speed or other parameter of a low-frequency oscillator or control signal sequencer. Each of the above may be applied to various lighting control signals, and much more complex arrangements may be implemented. Combinations of an overtone tracking control signals may be employed: for example the arrangement of (c) employs two control signals. In another very general approach, (e) depicts and arrangement where one or more control signals influence the behavior of a control signal sequencer. The ranges of behavior may also be controlled by additional control signals unrelated to overtone tracking, for example, as from a foot switch or lighting control console. These additional control signals may be used for a wide range of purposes, ranging from simply turning the animation effects on and off to more sophisticated variations in how the control signals are interpreted or sequencers respond to their variation. Additionally, the string and stage lighting described earlier may also be animated in similar fashion.
The animation below emulates arrays of light sources on an instrument body controlled by animation processes that are responsive to the measured amplitudes of various harmonics of the original or processed sound produced by the instrument. Although a wide range of other configurations are possible, the animation shows arrays of extreme high-brightness LEDS with animated patterns assigned to different locations depending on the string plucked or the pitch of the associated sound. In this example, the bass string is first plucked resulting in an overtone-sequence controlled animation in an upper area of the instrument body, and then a treble string is plucked resulting in an overtone-sequence controlled animation in a lower area of the instrument body. In this suggestive example, the animated light patterns rendered are configured so that initial rich overtone structures produce whitish (unsaturated) light which differentiates into colors as some overtones drop away in amplitude. As the relative harmonic amplitudes change with respect to one another, modulated spatial patterns in the animated light patterns evolve. As the overall amplitudes of the overtone series fall to zero, the brightness of the animated light patterns drop into darkness.
REFERENCES
[1] U.S. Patent Application 10/676,926, "Derivation of Control Signals from Real-Time Overtone Series Measurements," published April 15, 2004 as Pub. No. 2004/0069128.
[2] NRI Whitepaper "Musical Instrument Lighting For Visual Performance Effects," 2005.
[3] U.S. Patent 6,610,917, "Activity indication, external source, and processing loop provisions for driven vibrating-element environments," August 26, 2003.
[4] NRI Whitepaper, "Activity Indication, External Source, and Processing Loop Provisions for Driven Vibrating-Element Environments," NRI Whitepaper, 2005.
[5] U.S. Patent Application US-09/812,400, "Processing and Generation of Control Signals for Real-Time Control of Music Signal Processing, Mixing, Video, and Lighting," published January 24, 2002 as Pub. No. US 2002/0007723.
[6] U.S. Patent Application US-11/004,449, "Musical Instrument Lighting For Visual Performance Effects," published June 16, 2005 as Pub. No. US 2005/0126373.
[7] Pending NRI patent application, "Musical Instrument String Illumination for Visual Performance Effects," U.S. PTO publication forthcoming.