Audio Visualization for Learning Musical Instruments | ECE 445, Study Guides, Projects, Research of Electrical and Electronics Engineering

Download Audio Visualization for Learning Musical Instruments | ECE 445 and more Study Guides, Projects, Research Electrical and Electronics Engineering in PDF only on Docsity!

i

Audio Visualization

for Learning Musical Instruments

by Brian Liu Nick Fila ECE 445 Senior Design Project Spring 2008 TA: Scott Anderson 4/29/ Project Group 3

i

ABSTRACT

We designed and attempted to implement a device to create a visualization of the audio signal from a guitar. The visualization was based on our research of current music teaching methods and early childhood learning patterns. The visualization utilized the concepts of color, shape, and position to represent the frequencies present in the audio signal. This provided a more intuitive method of learning music. The prototype device consisted mainly of an Altera Cyclone II DE2 Development Board which took an amplified guitar signal as input and outputted a video signal to a VGA monitor. Within the Cyclone II, an FFT function provided the frequency-domain data which was converted into an image and output to a monitor. The device also consisted of a basic LED Tuner so that the user could make sure the guitar was in-tune before using the device. This tuner consisted of a series of first-order high-pass and low-pass filters that were optimized for a guitar signal. Nearly every aspect of our design was functional, except for the FFT function. Because we were unable to properly implement the FFT function, our overall device did not function as desired. Future work would be done to create a working FFT function, either improving upon the original design or using a different chip to implement it. Our final cost was fairly high for what the device accomplished. We have discussed ways to reduce costs on future models in our conclusions.

1. INTRODUCTION

We planned to convert an inputted audio signal from a musical instrument, and utilize a Fast Fourier Transform to output a compatible signal to VGA. The image displayed would be a visualization that would utilize color, shapes, and a system of visual effects which would indicate the tones played, the correctness of the tones, and the intensity of the tones played. This essentially would be an alternative to traditional note notation.

1.1 PURPOSE

Common pedagogy for learning to play musical instruments relies heavily on aural learning and highly abstract ideas, which may not be the preferred methods of learning for many students. Focusing on visual learning, we planned to develop both an audio visualization and pedagogy which may be used as an alternative to or be taught in conjunction with traditional methods. To democratize music education, we designed a simpler, more intuitive method of learning, to benefit younger children and people with little musical background. Furthermore, inherent properties of music visualization would also enable the hearing-impaired to play instruments. The actual use of the product is extremely broad: It could be used as a therapy technique, for psychological development, for performances, and as a visual analysis of audio data.

1.2 SPECIFICATIONS

The recognizable frequency range will be from 82.41 Hz to 1174.66 Hz which corresponds to the fundamental frequency of each note on most guitars (open 6th^ string to 22nd^ fret of 1st^ string). Some guitars may have 23 or 24 frets, but these are very rarely used, especially in instructional settings. The frequency resolution is important but varies since the frequencies of musical notes are on a geometric scale (each semitone is 1.0595 times greater in frequency than the last). Thus, our frequency spectrum will include the frequency of each semitone that the falls within the set frequency range of 82.41-1174.66 Hz. Latency needs to be small since the visualization is meant to correspond to the current note(s) being played. 100 ms latency is the most that is allowable. This is considering 120 beats per minute is a relatively high musical tempo. This corresponds to a beat every 500 ms. If the latency is 100 ms, there will be enough time for 5 visualization changes per beat, which seems more than adequate.

1.3 SUBPROJECTS

The design was broken into many modules, which each perform specific tasks. The block diagram at the time of the design review has been included as Figure A.1. The block diagram of the finished project has been included as Figure A.2. There is some difference between these diagrams as some of the major modules have been altered. There was a major change in design due to including the Fourier Transform and subsequent audio processing in the FPGA. By replacing the DSP, the end user would benefit with less hardware issues and reduced cost. The modules we decided to implement are as follows:

1.3.1. GUITAR

This is the instrument that was to provide the original audio signal. The device user was to play a series of notes or chords on this and a signal was to be passed to the FPGA (Signal Processing block) and the LED tuner. The output was to be passed using a ¼ inch instrument cable which was to be input to a

2. DESIGN PROCEDURE

2.1 GUITAR AND AMPLIFIER COMBO

We had originally planned to input an electric guitar directly into the LED Tuner and the Signal Processing block. Through initial tests, we found that the guitar did not produce a voltage of sufficient amplitude to power the LED’s or be read accurately by the ADC within the FPGA board. Our simple solution was to plug the guitar into a small, 15-Watt Fender guitar amplifier. This configuration gave a signal with a voltage amplitude range of 0 V to 13.44 V, zero-to-peak. The adjustable voltage output allowed the guitar to interface well with both the LED Tuner, which required a high input voltage to function well, and the FPGA board which reached a saturation point at an input voltage amplitude of 0.3 V, zero-to-peak. The guitar amplifier added another hardware component to the project. Although the user would most likely have access to an amplifier, the voltage amplification would ideally have less bulk and be on board the actual device. One alternative we considered was a non-inverting op-amp circuit. We designed and built a simple op-amp circuit with a theoretical gain of 25 V/V. With a guitar input, we achieved a gain of 22 V/V. Unfortunately, this circuit did not function well with the LED Tuner circuits. When we plugged the output of the op-amp circuit to the input of the LED Tuner circuits, the gain was significantly diminished to the point that the circuit was not functional. In addition, a static-gain circuit would not have been ideal due to the differing voltage thresholds for the LED Tuner and the FPGA board. Thus, the use of an op-amp circuit would have been more practical but less functional.

2.2 LED TUNER

The LED Tuner ended up being a greater challenge than we initially anticipated. Our original design called for twelve first-order RC filters. Six of the filters were to be low-pass and six were to be high-pass. Each of the guitar’s six strings was to have one low-pass and one high-pass filter, each with a cut-off frequency at the string’s fundamental frequency when in standard guitar tuning. LED’s were to be connected to the output of each of the filters. The goal was to have both of the LED’s for a string light up with equivalent brightness when that string was played while in tune. Our initial calculations were based on the -3 dB cut-off. We built some test filters for the two lowest-frequency strings, the E2 and the A2, since they provided the slowest-attenuating voltage output. Initial tests on the filters provided less-than-desirable results. Function generator and guitar signals with frequencies outside of each filter’s pass-band did attenuate, but the frequency response was not as sharp as we desired. We decided to try some higher-order filters to attain quick transition-bands for our filters. We attempted second-order RC filters, which were just two of our first-order RC filters in series. This provided greater precision, but also too much attenuation of signals that were in the pass-band. From this point, we attempted a number of higher-precision filters that provided gain with operational amplifiers. The filter types we attempted were State-Variable and Sallen-Key. Despite checking our design both manually and with online, interactive design tools^1 , we could not get these filters to operate with an adequate amount of gain, or with the desired frequency response. (^1) http://www.analog.com/Analog_Root/static/techSupport/designTools/interactiveTools/filter/filter.html

After the higher-order filters did not function as desired, we were forced to implement our original design of first-order RC filters. Since we had numerous resistor values available, but few different capacitors, we decided to use constant capacitors and change the resistance values if necessary. We began with values as close as possible to the theoretical values for both the low-pass and high pass filters. We input a guitar signal from an in-tune guitar to test our filters, one string at a time. We noticed that in all cases the high-pass filter’s LED was brighter than LED of the low-pass filter. This can most likely be attributed to the harmonic frequencies, which only resonate at frequencies that are higher than the fundamental frequency. The high-pass filters passed these frequency components through, but the low- pass filters attenuated them. We decided to decrease the resistance of the low-pass filter to increase the cut-off frequency on these filters. This shifted the transition band and put the operational frequency of the filter at a higher gain level. We lowered the resistance value for each low-pass filter until the low-pass and high-pass LED’s had the same brightness for each in-tune string.

2.3. FPGA

After our redesign, both signal processing and visualization were controlled by the FPGA. Due to the support needed for the audio input, an additional two main component blocks for the audio codec initialization and data processing handled the header/routing settings needed for audio function and prepared the data to be analyzed by the FFT. See Figure A3 for FPGA Block Diagram.

2.3.1. AUDIO CODEC

Due to lack of documentation, we did not discover that routing settings were required for device function and the method of routing required an i2c controller until late into the development. We had been successful with passing audio through the Line-In input and outputting to the Line-Out output, but this particular function was merely the default "bypass" routing scheme of the audio device. Instead, the routing scheme needed to be set so the Line-In and Line-Out inputs/outputs were active and all corresponding timing, AD/DA path control, A/D selection, power, sampling, and control settings needed to be defined (See Figure A5). These considerations affected the performance and the operation of the device, so it was important to only enable the components that were absolutely required. The most important choices were the sample rate, bit rate, and sampling frequency. Since space and processing power is limited on the DE board, accuracy needed to be compromised (see Signal Processing section for elaboration). The FPGA employs the I2C interface, where all chips on the board are controlled by a single scheme. This makes it extremely difficult for parallel chip function and impossible for input and output to occur at the same time. Using an I2C controller, we had to isolate the audio codec and send input data based on the timing defined by the scheme. Therefore, we had write a separate module to divide the 50 MHz clock to 1MHz for proper I2C function. See Fig A6 for I2C Scheme. We needed to follow the rates of the clocks for both parallel and serial operation. Some subcomponents needed to be operated in serial and would need shorter clock cycles to operate within the larger clock cycle. Originally, we had a single 50 MHz clock controlling all the components but were required to create additional 12.5 and 1 MHz clocks.

2.3.2 DATA PROCESSING

3. DESIGN DETAILS

3.1 LED TUNER

As we noted in section 2.2, our LED Tuner consisted of 12 first-order RC circuits with identical LED’s driven by each circuit’s output. The six low-pass filters, each representing a different guitar string in standard tuning, adhere to the design schematic present in Figure B.1. The circuit is simple, consisting of a single capacitor and an equivalent resistance. The equivalent resistance consisted of a variable number of resistors, from one to four, connected in series with each other. The six high-pass filters, each representing a different guitar string in standard tuning, adhere to the design schematic present in Figure B.2. As with the low-pass filters, the circuit is simple, consisting of a single capacitor and an equivalent resistance. Once again, the equivalent resistance consisted of a variable number of resistors, from one to four, connected in series with each other. Tables C.1.1 and C.1.2 show the capacitor, equivalent resistance, and individual resistor values for each of the twelve filters by string and fundamental frequency. We’ve also included the theoretical value for the -3 dB cut-off frequency for a reference. The high-pass values are very near the theoretical value, but the low-pass show significant difference. We also decided to use different capacitor values for the two highest- frequency strings, B3 and E4. The 10 μF electrolytic capacitors that were used for the E2, A2, D3, and G strings gave a somewhat slow response time. Since the voltage signal from the B3 and E4 strings attenuated rapidly, the LED’s never lit up brightly enough to make an accurate reading when the 10 μF capacitors were used. Instead, 100nF ceramic disc capacitors were used for these strings. Each circuit was built on the same protoboard. The high-pass LED’s for each string were placed to the left of the low-pass LED counterparts. For each string, the representative LED’s were within 1 cm of each other. The filters for the E2 and A2 string share a common voltage input and ground. Each of the sets of filters for each of the other strings has its own separate voltage input and ground. FPGA Block Diagram

3.2 FPGA DETAILS

The FPGA is designed to provide two functions: The Audio Filtering and Image Construction, which correspond directly to the “FFT” and “VGA” components. The “Codec” and “Data” components are required for proper audio codec function. See A3 for FPGA Block Diagram and A4.1, A4.2 for FPGA Schematic

3.2.1 CODEC

The subprocesses within this component refer to the function of the Wolfson WM8731 audio codec. Essentially, it initializes the audio codec and reroutes the audio input data.

3.2.1.1 Audio Settings Controller

The Audio Settings Controller writes the audio codec settings onto the ROM to be sent to the Audio Codec. The header settings comprise of 16 of the 24 bits of the full audio signal. It also converts the onboard 50 MHz clock to 1 MHz for the I2C Controller.

3.2.1.2 I2C Controller

The Altera DE2 follows an I2C scheme, where all devices are centralized and data is received and sent via the I2C bus. The controller manages the clock, and retrieves the audio settings from the ROM and sends them to the Audio Codec.

3.2.1.3 Audio Codec

The audio codec modifies its function based on the audio settings, and receives the analog audio data from the guitar through the Line-In input. The onboard ADC controller converts the analog data to a digital signal. It also sends the clock and frequency settings to the FFT Controller.

3.2.2 DATA PROCESSING

The data must be processed so that data format is compatible with all components.

3.2.2.1 Digital Audio Data

The data is received from the AD conversion through as a signal digital input, but must be converted to readable digital format. The clock is modified through this subprocess so that it matches the clock for the Serial-to-Parallel and FFT Controller components.

3.2.2.2 Serial-to-Parallel Converter

Since the Digital Audio Data is a single signal sent over a period of time, the Serial-to-Parallel Converter converts the signal to a 24 bit audio signal.

3.2.3 Signal Analysis

The signal analysis converted the parallel audio data to frequency domain. We had also written a Filter Bank but decided that a FFT would be more appropriate.

3.2.3.1 FFT Controller

Since the MegaCore FFT only included the calculations of the data, the processes had to be controlled through the FFT Controller.

3.2.3.2 MegaCore FFT

We had decided to use Altera’s FFT MegaFunction under the MegaCore IP, which we found out was the most appropriate solution for the FFT. If we had written our own FFT, the data processing would have been too slow to achieve the resolution and timing required for the visualization and the design most likely would not have fit the device unless we used alternative memory on the FPGA device. The MegaCore FFT is optimized by the manufacturer to function on the device.

3.2.3 VISUALIZATION

The final image was created dynamically using the frequency domain signal as a basis.

3.2.3.1 VGA Controller

The VGA Controller interfaced with the VGA State Machine and Pixel Mapper, controlling timing and device settings.

3.2.3.2 VGA State Machine

The VGA State Machine controlled the states of the different functions of the VGA. The lines and resolution were taken in account so that the image had 480x640 resolution.

3.2.3.3 Pixel Mapper

The Pixel Mapper plotted the visualization according to the frequency-domain data. The magnitudes of the signal change the sizes of the “pie slices” and the color was mapped to the frequency.

This test was performed with sine-waves of 50, 100, 200, 400, 800, and 1600 kHz. We also input square-waves. The expected/desired results were not achieved. Instead, the output appeared as a slightly distorted, reduced-amplitude version of the input signal.

4.1.3 IMAGE CREATION

We wanted originally wanted a test to see that the proper image is created based on a known input frequency. A sine-wave function was to be input directly to the FPGA and the FPGA’s visualization output was to be passed to the monitor. We would then have compared the image created to the desired image. This test was essentially a test of the overall system rather than a test of the Visualization block alone. We formulated a new test to examine the functionality of the Visualization block. For this test, we mapped the one quadrant/octave of frequency slices and their magnitudes to 15 of the FPGA’s on-board switches. Twelve of the switches controlled which of the twelve slices would be displayed. The three remaining switches controlled the magnitude of the slices that were displayed. For the actual test, we sent the Visualization output to a VGA monitor and noted any inconsistencies with what was expected. We ran through about two hundred switch combinations making sure to test every slice at every magnitude and a number of combinations. All results produced were correct. There were a few pixel errors that were present. For instance, there were a few green single-pixel-wide vertical dashes through the left side of the visualization. These had always been present and are caused by an inherent flaw within the device. From a distance, they are hardly noticeable. (An example screenshot is included as Figure D.3)

4.1.4 LED TUNER

We performed two tests to determine whether the LED tuner functioned properly. The first test was the one outlined in our design review. For this test, we sent a sine-wave from the function generator directly to each of the 12 RC circuits. For each of the six pairs of filters, we swept through the frequency range of about 40% above and below the operational frequency at static intervals and recorded the amplitude of the filter output. For each pair we took ten measurements above the operational frequency, ten measurements below the operational frequency, as well as one measurement at the operational frequency. Ideally the curves for paired low-pass and high-pass filters would meet at the operational frequency. This was not the case for any of the pairs. This test essentially shows that conventional filter theory does not work because the guitar signal is a superposition of sine-waves. The multiple frequency components affect the results. The results of this test are included as Figures D.1 and D.2. The second test we performed was essentially a user test. For this test we tuned our guitar with the LED tuner and then checked the guitar with a commercial tuner.

4.2 TESTING CONCLUSIONS

Overall, we were not pleased with our test results. The FFT function was crucial to the overall functionality of the device. Because the FFT did not produce accurate results, the overall device could not accurately represent the input guitar audio. Despite the fact that the FFT function did not work, all other functions and major block worked as desired. The LED tuner, while providing poor results in a theoretical test, produced good results in an operational test. The tuner wasn’t as precise or easy-to-use as we would have liked, but it produced solid, serviceable results. It allowed the guitar to provide manageable input for the overall device, which was the overall goal of the tuner.

The entire Visualization block functioned almost exactly as we wanted it to. The visual “image creation” test showed accurate results. Outside of some minor glitches with the board we used, we were very pleased with this aspect of our project. We were also pleased with the accuracy of converting the analog input signal to an internal digital signal and converting it back to an analog output signal. This test showed that at least one aspect of the Signal Processing block was functional.

6. CONCLUSION

Overall, we have mixed feelings about the outcome of this project. We were extremely pleased with the outcome of the actual visualization. We believe that the display system we have developed can be very beneficial to music students, especially young students. It presents audio information in a way that is informative, is visually-pleasing, and utilizes two of the concepts that children pick up on first (color and shape). Beyond the theoretical aspects, we also were able to implement the visualization with control over the position, shape, and magnitude. We were also able to achieve proper functionality with most of the other aspects within the FPGA board, including the Audio Codec and the Analog-to-Digital Conversion. We were moderately pleased with the LED Tuner. It certainly is not commercially viable by itself, but it works with an acceptable degree of precision and is relatively simple to use. Given more time, the tuner could potentially have been implemented within the FPGA board itself, utilizing frequency-domain information from the FFT. Our biggest concern was that the FFT was incompatible with the rest of the system. Since this function was the basis of our analysis of the audio signal, the overall project did not function properly. In the future, a DSP might be used for any FFT function or filter-bank. This could add cost, but the benefit to the overall product would certainly outweigh it. There were no ethical considerations that needed to be considered regarding the development of the device. We believe the device would have nothing but positive effects. A properly functioning device would give deaf people an opportunity to learn to play an instrument. It would give younger learners and people with little musical background an alternative, potentially more effective, way to learn music. Though the development was expensive due to the use of FPGA boards, the actual components used (audio codec, video encoder, SDRAM, RAM) are relatively inexpensive. We recommend that future designs utilize this fact. Using dedicated chips with programmable interfaces would serve to minimize cost. It would also allow for great customizability within the design. Using a DSP chip to compute the FFT would eliminate the problems with the FPGA’s FFT function. As for the tuner improvements, we already mentioned a software-based method. Another option would be to implement a tuner similar in design a small, inexpensive, commercial tuner within the device.

7. APPENDIX

7A. BLOCK DIAGRAMS

Figure A.1 – Design Review Block Diagram Figure A.1 – Design Review Block Diagram Figure A.2 – Finished Project Block Diagram

Fig A.

• FPGA Block Diagram

• 1. INTRODUCTION.................................................................................................................................................................................. CONTENTS
  • 1.1 Purpose...........................................................................................................................................................................................
  • 1.2 Specifications................................................................................................................................................................................
  • 1.3 Subprojects....................................................................................................................................................................................
    • 1.3.1. Guitar......................................................................................................................................................................................
    • 1.3.2 LED Tuner.............................................................................................................................................................................
    • 1.3.3 Signal Processing (formerly DSP)................................................................................................................................
    • 1.3.4. Visualization (formerly FPGA).....................................................................................................................................
    • 1.3.5. Monitor/Computer...........................................................................................................................................................
• 2. DESIGN PROCEDURE........................................................................................................................................................................
  • 2.1 Guitar and Amplifier Combo...................................................................................................................................................
  • 2.2 LED Tuner......................................................................................................................................................................................
  • 2.3. FPGA................................................................................................................................................................................................
    • 2.3.1. Audio Codec.........................................................................................................................................................................
    • 2.3.2 Data Processing...................................................................................................................................................................
    • 2.3.3 Signal Processing................................................................................................................................................................
    • 2.3.4 Visualization.........................................................................................................................................................................
• 3. DESIGN DETAILS................................................................................................................................................................................
  • 3.1 LED Tuner......................................................................................................................................................................................
  • 3.2 FPGA Details..................................................................................................................................................................................
    • 3.2.1 Codec.......................................................................................................................................................................................
    • 3.2.2 Data Processing...................................................................................................................................................................
    • 3.2.3 Visualization.........................................................................................................................................................................
• 4. DESIGN VERIFICATION....................................................................................................................................................................
  • 4.1 Testing.............................................................................................................................................................................................
    • 4.1.1 Analog-to-Digital Conversion........................................................................................................................................
    • 4.1.2 Frequency Data Analysis.................................................................................................................................................
    • 4.1.3 Image Creation.................................................................................................................................................................... iii
    • 4.1.4 LED Tuner.............................................................................................................................................................................
  • 4.2 Testing Conclusions...................................................................................................................................................................
• 5. COSTS...................................................................................................................................................................................................
  • 5.1 Parts...............................................................................................................................................................................................
  • 5.2 Labor.............................................................................................................................................................................................
• 7. Appendix.............................................................................................................................................................................................
  • 7A. Block Diagrams.........................................................................................................................................................................
  • 7B. Configurations and Schematics..........................................................................................................................................
  • 7C. General Tables...........................................................................................................................................................................
  • 7D. General Graphs.........................................................................................................................................................................
  • 7E. Costs..............................................................................................................................................................................................
• Fig A.4.
• – Final VHDL Schematic P
• Fig A.4.
• – Final VHDL Schematic P