I wrote a waterfall-style spectrogram program in MATLAB for a Fourier Analysis class I took a few years ago. It was the most involved MATLAB program that I had or have ever written, and it even has a graphical user interface (GUI). I call it tDfD, which means “time-domain, frequency-domain”. This is what the program does: it takes a time-domain signal (audio) and converts it into the frequency domain via the Fast Fourier Transform (FFT). The image above is a sample from the output while playing an Aphex Twin song. The artist took an image of his face and used an inverse FFT to convert it into the time domain and put it in his song. It sounds very strange, but it looks cool. You’ll notice that the face appears to be stretched out on the top and squished at the bottom. This is because the artist used software that had a logarithmic scale for frequency and I used a linear scale for mine.
Code on Github
Fourier Analysis Code
tDfD: Realtime Audio Signal Spectral Analyzer
Abstract—This document is a user’s guide and a descriptionof operation for the tDfD Realtime Spectrum Analyzer.
INTRODUCTION
The fast Fourier transform (FFT) is a useful tool in analyzing the spectral content of a signal. tDfD evaluates real-time audio signals that have a frequency range of 10Hz-100 kHz using an algorithm that employs the FFT and plots the output on a waterfall-style scrolling graph. tDfD has an assortment of settings designed to allow the user to betteranalyze a variety of input signals.
TRADEOFFS
During analysis, the sample rate, fs, is fixed; therefore, the window length will determine the frequency and time resolution of the output. A longer analysis window will increase the frequency resolution while decreasing your time resolution, and a shorter window will have the opposite effect. Another tradeoff was the decision not to allow the user to change the ratio of attempted FFTs taken per screen refresh. This was dueto time constraints.III.
USER GUIDE
Familiarity with the Fast Fourier Transform will make interaction with the program intuitive. There are two main buttons to start and stop the analysis, various user settings to tune the analysis to a particular signal, and two outputs: the waterfall graph and a simple strongest tone frequency output. In most cases, tDfD is resilient to changes of the audio input device during analysis, but may need to be restarted after a change. The Interface window may be maximized or resized to any desired size.
User Settings
Table I summarizes the available user settings, their respective minimum and maximum values, units, defaults, and themethod of selection.
Restart Required Settings
As indicated in the GUI, the first four settings require that the analysis is restarted before taking effect. Sample rate is selected from a pull-down menu that has a predefined number of sample rates based on hardware capabilities. The other three are text input fields thatperform input sanitation to ensure that the input is (1)a number(2)real and (3)within the predefined range. If the input is nota number (NaN) or not Real then the START button will be disabled and a message will be displayed indicating the errant field. Inputs above or below the minimum or maximum areautomatically set to their respective limits.
Dynamic Settings
The last three settings may be changed during analysis and are chosen from a predefined range or list so that only valid input data will be selected. The Intensity slider controls the contrast of the spectrogram display. The Window pull-down selects from a list of available windowing functions and will allow for different sections of the spectrograph to be windowed independently. The ColorRange pull-down chooses the color bar settings that may help different features to be highlighted in the spectrogram graph.
Note: the ’Lines’ setting is computationally expensive.
EXAMPLES
This section showcases a variety of signal analyses and tDfD’s outputs. First, a linear frequency sweep is analyzedat different sample rates. Second, various audio inputs are analyzed with various settings.
Example – Linear Frequency Sweep
Three example analyses are shown to highlight the effectof the hardware’s anti-aliasing filter. All three examples use a linear frequency sweep from 1Hz to that example’s samplerate over a 10s period. A Jet colormap and a Blackman filter were used for all examples. In all three examples, aliasing of the signal can be seen where the input frequency exceeds the Nyquist frequency; subsequent samples are “folded” back around the Nyquist, or folding frequency. For example, with a sample rate fs=8000Hz, the Nyquist frequency is 4000Hz. When the input sweep is at 0:55 ∗ fs, or 4400Hz, it is has an alias at 0:45 ∗ fs or 3600Hz. Other aliases exist at higher frequencies that are 0:55 ∗ fs + n and 0:45 ∗ fs + n, where n is a natural number. The first alias is shown in the graph. It appears as a reflection of the signal but is heavily dampedby the anti-aliasing filter. In many instruments, the effective bandwidth, Eb, is fs=2:56 instead of fs=2 due to AA filter rolloff.[2] For the fs = 8000Hz example, this would be 3125Hz. Oversampling is recommended to counteract these effects.tDfD’s logo displayed in the GUI was chosen to exemplifyaliasing. It has a half-tone background that is slightly too fineof a pattern to be displayed properly on most resolutions. Also, resizing the GUI’s window resizes the image which, changes the appearance of the half-tone pattern drastically as seen in Fig. 8.
Example — Various Inputs & Settings
- Spectrogram Encoded Music: The artist Richard D. James used an inverse FFT to produce a time-domain audio signal based on a desired spectrogram picture of his face, shown in fig. 4. The face appears stretched out at the top due to the linear scale of tDfD and the logarithmic scale of his picture.
- Voice: In fig. 5, a capture of the words “Hello World” are analyzed. The highly harmonic nature of the human voicecan be seen in this example.
- Piano Music: Franz Liszt’s “Hungarian Rhapsody No. 2” is analyzed in fig. 6. The piano notes are clearly discernible in the spectrogram. It may be possible to score the output to produce sheet music or to synthesize a MIDI file.
Future Ideas
tDfD could be used for a variety of applications, from aiding with speech pathology to identifying individual whales by their calls. Some novel uses may include a MIDI scorer that would produce a .MIDI file based analysis of an audio file for use in sampling and music production. This would likely only work with a limited set of instruments that are discreet in nature, such as a piano. I would like to develop a videogame that utilizes this software. The user would control a character’sposition by whistling. The movement would follow the trace of a sweeping tone (whistle) and would be relative to the character’s position before the tone event. When there was no input, the character would simply move forward as the level scrolled left. Referencing motion to the character’s starting position before every maneuver and referencing acceleration to the beginning tone of a tone input would allow relative input commands and would not require that particular whistling frequencies be met. Only the shape of the tone event wouldmatter, presumably making control easier.
References
[1] Alan V. Oppenheim, Alan S. Willsky, and S. Hamid Nawab. 1996. Signals& Systems (2nd Ed.). Prentice-Hall, Inc., Upper Saddle River, NJ, USA.
[2] “FFT Fundamentals (Sound and Vibration Measurement Suite).” Soundand Vibration Measurement Suite 7.0 Help. National Instruments, Nov. 2008. Web. 13 Mar. 2015.