26-04-2011, 12:17 PM
[attachment=12821]
INTRODUCTION
1.1 Motivation
Automatic Speech Recognition (ASR) has developed into a popular technology and is being deployed in a wide variety of everyday life applications, including personal dictation systems, call centers and mobile phones. Despite the various benefits a conventional speech-driven interface provides to humans, there are three major drawbacks: Firstly, the audible speech signal prohibits a confidential conversation with or through a device. Besides that, talking can be extremely disturbing to others, especially in libraries or during meetings. Secondly, speech recognition performance degrades drastically in adverse environmental conditions such as in restaurants, cars, or trains. Acoustic model adaptation can compensate for these effects to some degree, however the pervasive nature of mobile phones challenges this approach. Performance is also poor when sound production limitations occur, like under water. Last but not least, conventional speech-driven interfaces cannot be used by speech handicapped people, for ex those without vocal cords.
To overcome these limitations, alternative methods are being investigated, which do not rely on an acoustic signal for ASR. Chan et al. [Chan et al., 2002b] proved that the electric signal (MES) from articulator face muscles contains sufficient information to discriminate a given set of words accurately (>90% word accuracy on the ten English digits). This holds even when the words are spoken non-audibly, i.e. when no acoustic signal is produced [Jorgensen et al., 2003].
The potential of electromyography (EMG) based speech recognition lies primarily in the fact that it does not rely on the transmission of an acoustic signal: it allows private, on-disturbing communication in any situation and could possibly be deployed by speech handicapped people.Moreover; it is robust to environmental noise.
Chapter 2
BACKGROUND
2.1 Surface EMG measurement
Electromyography is the process of recording the electrical activity of a muscle. Muscle fibers generate small electrical currents as part of the signaling process for the muscle fibers to contract. There are two basic methods to measure the signal called Electromyogram: invasively using fine wire electrodes that are inserted directly into the muscle or non-invasively by applying the electrodes to the skin surface.
Fine wire electrodes allow the testing of deep or small muscles and have a more specific pick-up area than surface electrodes. However, the needles may cause discomfort and the measurements should only be carried out by a medical doctor. Moreover, it is extremely difficult to identify the same point of insertion in consecutive recording sessions. As a result, surface EMG (semi) is the more common method of measurement.
2.1.1 Equipment
The following equipment is necessary for surface EMG recordings:
Electrodes: Generally speaking, surface electrodes convert the ionic currents generated by muscle contraction into electronic currents that can be fed into electronic devices. While the detection electrodes serve to pick up the desired signal, the ground electrode provides a common reference to the differential input of the preamplifier. Refer to section 2.2.2 for more details on properties of surface electrodes.
Differential Amplifier: When detecting an EMG signal, amplification is necessary to optimize the resolution of the digitizing equipment [Scott, 2003]. Moreover, an amplifier can also be used to maximize the signal-to-noise ratio - that is, the ratio of the energy of the wanted EMG signal to the energy of unwanted noise contributions of the environment. For that reason semi recordings generally involve a differential detecting configuration as schematically shown in figure 2.1. The EMG signals are represented by “p” and “m” and the noise signals by “n”.
Figure 2.1: Equipment required for semi measurements.
A differential amplifier subtracts the signals from two detection sites and amplifies the difference voltage between its two input terminals. As a consequence, signals common to both electrodes - such as noise originating far away from the detection sites - should ideally produce a zero output, whereas local EMG signals are amplified. The Common Mode Rejection Ratio (CMRR) is a measure of the degree to which this ideal is realized in practical designs. It is defined as the difference signal gain divided by the common mode signal gain. An ideal differential amplifier would thus have a CMRR of infinity, yet, in practice, only amplifiers with a
Maximum CMRR of approximately 120dB are available. As a result, it is not possible to obtain a signal free from noise, however, a CMRR of 90dB (CMRR(x) [dB] = 20 _ log10CMRR(x)) normally results in sufficient noise suppression [Overberg, 1992].
Electrical Isolator: The failure of any electrical device that has galvanic contact with the subject can cause a potentially harmful current to pass through the skin. In order to ensure safety the subject must therefore be electrically isolated from any electrical connection to the power source. This can be achieved by placing an optical isolator between the amplifier and the devices that are connected to the power point (e.g. the computer) [Luca, 2002].
A/D-converter: EMG signals usually need to be digitized for further processing and data analysis. The analog-to-digital converter transforms an analog signal into a discrete number of data points representing the amplitude of the input signal at particular instances in time.
Recorder: The purpose of the recorder is to generate a time record of the input EMG signal that can be reviewed later for data analysis.
2.1.2 Electrodes
Electrodes serve as converters of the ionic currents produced in muscles into electronic currents that can be manipulated in electronic circuits. There are two main types of surface electrodes: dry electrodes that have direct contact with the skin, and gelled electrodes, where an electrolytic gel is placed between the metallic part of the electrode and the skin to decrease the skin-electrode impedance.
Dry vs. Gelled Electrodes
Dry electrodes are typically used when the constitution of the electrodes does not allow the use of gel (e.g. bar electrodes). Due to the high electrode-skin impedance it is common to have the preamplifier circuitry at the electrode site. This makes the dry electrodes considerably heavier than gelled electrodes (about 20g vs. 1g) such that electrode fixation becomes an issue.
Gelled electrodes are therefore the common choice. Oxidative and reductive chemical reactions in the contact region of the metal surface and the gel allow an exchange between the ionic current generated by muscle contraction and the electron current flow of the recording instrumentation. It is worth mentioning here, that the quality of an electrode depends almost exclusively on its ability to exchange ions for electrons and vice versa [Overberg, 1992]. A general explanation of the mode of operation of surface electrodes from the chemical point of view may be found in [Meyer-Waarden,1985] and [Overberg, 1992].
2.1.3 Signal Characteristics
The raw EMG signal detected by a differential amplifier using surface electrodes is a “bipolar signal whose random fluctuations, if summed over a significantly long time period, would produce a zero result” [Lamb and Hobart, 1992]. Its amplitude typically ranges from 0.01 to 5 mV [Lamb and Hobart,1992]. Figure 2.2 shows an example of an EMG signal and the corresponding frequency spectrum. The usable energy of the signal is contained in the 0 to 500Hz frequency range, that is, the signal energy is above the electrical noise level in that frequency band. In fact, the dominant energy lies in the 50-150Hz frequency range [Luca, 2002].
Chapter 3
SYSTEM OVERVIEW
In this chapter we give an overview of our EMG speech recognition system. Section 3.1 describe the hardware we deployed while section 3.2 introduces the software we have written and used for this project. The workflows for data collection, system training and recognition are presented in section 3.3.
3.1 Hardware
3.1.1 EMG Equipment
Section 2.1.1 introduced the equipment necessary for sEMG recordings. We used two different physiological data recording systems for data collection which we will refer to as VARIOPORT II and VARIO-PORT VIII depending on the number of EMG channels they provide (two and eight channels respectively) [Becker, 2003b].
The VARIOPORT II data recording system was used for initial experiments on EMG based speech recognition.
Name EMG Chanel Frequency
Range A/D Conversion Resolution Range
Varioport II 2
0.9Hz..500Hz 12 bit ±500μV
Per bit ±500μV
Varioport VIII 8
19Hz-295Hz 16 bit 0.033μV
per Bit ±1070μV
Table 3.1: Properties of data recording systems VARIOPORT II and VARIOPORT VIII
3.2 Computers
Data collection and online recognition were generally performed on a Pentium III laptop (1000MHz,512 MB RAM) with a Microsoft Windows operating system. For offline recognition and training wedeployed different Linux based machines provided by the ITI Waibel of the Universit¨at Karlsruhe. It is worth mentioning here that recognition rates varied slightly for different operating systems.
Figure 3.1: Physiological data recording system
3.3 Software
The software we designed and implemented for this work consisted of two parts: (1) a Visual C++ project for data collection and demonstration purposes and (2) a JRTk based Tck/Tk script collection for recognizer training and classification.
3.4 Data Collection
All signal data used for our experiments was collected in so-called recording sessions. A recording session is defined as a set of utterances collected in series by one particular speaker.All settings (channels, sampling rate, speech mode) remain constant during a session. Data collection consists of four steps:
. Choosing settings: several settings have to be made prior to recording a session. Among other things a word list is selected containing all utterances a speaker has to record during that session. The list can optionally be randomized.
Data recording: The speaker records one file set (appendix B) for each word in the word list using the push-to-talk button of the speaker interface.
Generation of transcript file: When all utterances have been recorded a transcript is (automatically) created for the session.
Generation of settings file: All settings are stored in a file
3.5 Recognition
• Online recognition: Online testing was implemented for isolated words and phrase recognition only. First, a training session and a set of possible hypothesis is selected (subset of the set of training utterances). Next, the recorder and a janus recognition script are started by the VC++ software. Janus and the software communicate via semaphore files. When an utterance has been recorded (i.e. the push-to-talk “recording” button is released) a “start” file is created that is detected by janus. Janus deletes the file and reads the .adc file corresponding to the recorded signal data. It determines the Viterbi path for each hypothesis allowed in the vocabulary and the word/phrase yielding the best Viterbi score is written to a hypothesis file. A “done” file is then created by Janus which is detected by the VC++ software. The VC++ software reads the hypothesis and displays it on the speaker interface. When Janus detects a “stop” file (created by the VC++ software) it leaves the recognition loop. Figure 4.5 illustrates the communication between the different modules.
INTRODUCTION
1.1 Motivation
Automatic Speech Recognition (ASR) has developed into a popular technology and is being deployed in a wide variety of everyday life applications, including personal dictation systems, call centers and mobile phones. Despite the various benefits a conventional speech-driven interface provides to humans, there are three major drawbacks: Firstly, the audible speech signal prohibits a confidential conversation with or through a device. Besides that, talking can be extremely disturbing to others, especially in libraries or during meetings. Secondly, speech recognition performance degrades drastically in adverse environmental conditions such as in restaurants, cars, or trains. Acoustic model adaptation can compensate for these effects to some degree, however the pervasive nature of mobile phones challenges this approach. Performance is also poor when sound production limitations occur, like under water. Last but not least, conventional speech-driven interfaces cannot be used by speech handicapped people, for ex those without vocal cords.
To overcome these limitations, alternative methods are being investigated, which do not rely on an acoustic signal for ASR. Chan et al. [Chan et al., 2002b] proved that the electric signal (MES) from articulator face muscles contains sufficient information to discriminate a given set of words accurately (>90% word accuracy on the ten English digits). This holds even when the words are spoken non-audibly, i.e. when no acoustic signal is produced [Jorgensen et al., 2003].
The potential of electromyography (EMG) based speech recognition lies primarily in the fact that it does not rely on the transmission of an acoustic signal: it allows private, on-disturbing communication in any situation and could possibly be deployed by speech handicapped people.Moreover; it is robust to environmental noise.
Chapter 2
BACKGROUND
2.1 Surface EMG measurement
Electromyography is the process of recording the electrical activity of a muscle. Muscle fibers generate small electrical currents as part of the signaling process for the muscle fibers to contract. There are two basic methods to measure the signal called Electromyogram: invasively using fine wire electrodes that are inserted directly into the muscle or non-invasively by applying the electrodes to the skin surface.
Fine wire electrodes allow the testing of deep or small muscles and have a more specific pick-up area than surface electrodes. However, the needles may cause discomfort and the measurements should only be carried out by a medical doctor. Moreover, it is extremely difficult to identify the same point of insertion in consecutive recording sessions. As a result, surface EMG (semi) is the more common method of measurement.
2.1.1 Equipment
The following equipment is necessary for surface EMG recordings:
Electrodes: Generally speaking, surface electrodes convert the ionic currents generated by muscle contraction into electronic currents that can be fed into electronic devices. While the detection electrodes serve to pick up the desired signal, the ground electrode provides a common reference to the differential input of the preamplifier. Refer to section 2.2.2 for more details on properties of surface electrodes.
Differential Amplifier: When detecting an EMG signal, amplification is necessary to optimize the resolution of the digitizing equipment [Scott, 2003]. Moreover, an amplifier can also be used to maximize the signal-to-noise ratio - that is, the ratio of the energy of the wanted EMG signal to the energy of unwanted noise contributions of the environment. For that reason semi recordings generally involve a differential detecting configuration as schematically shown in figure 2.1. The EMG signals are represented by “p” and “m” and the noise signals by “n”.
Figure 2.1: Equipment required for semi measurements.
A differential amplifier subtracts the signals from two detection sites and amplifies the difference voltage between its two input terminals. As a consequence, signals common to both electrodes - such as noise originating far away from the detection sites - should ideally produce a zero output, whereas local EMG signals are amplified. The Common Mode Rejection Ratio (CMRR) is a measure of the degree to which this ideal is realized in practical designs. It is defined as the difference signal gain divided by the common mode signal gain. An ideal differential amplifier would thus have a CMRR of infinity, yet, in practice, only amplifiers with a
Maximum CMRR of approximately 120dB are available. As a result, it is not possible to obtain a signal free from noise, however, a CMRR of 90dB (CMRR(x) [dB] = 20 _ log10CMRR(x)) normally results in sufficient noise suppression [Overberg, 1992].
Electrical Isolator: The failure of any electrical device that has galvanic contact with the subject can cause a potentially harmful current to pass through the skin. In order to ensure safety the subject must therefore be electrically isolated from any electrical connection to the power source. This can be achieved by placing an optical isolator between the amplifier and the devices that are connected to the power point (e.g. the computer) [Luca, 2002].
A/D-converter: EMG signals usually need to be digitized for further processing and data analysis. The analog-to-digital converter transforms an analog signal into a discrete number of data points representing the amplitude of the input signal at particular instances in time.
Recorder: The purpose of the recorder is to generate a time record of the input EMG signal that can be reviewed later for data analysis.
2.1.2 Electrodes
Electrodes serve as converters of the ionic currents produced in muscles into electronic currents that can be manipulated in electronic circuits. There are two main types of surface electrodes: dry electrodes that have direct contact with the skin, and gelled electrodes, where an electrolytic gel is placed between the metallic part of the electrode and the skin to decrease the skin-electrode impedance.
Dry vs. Gelled Electrodes
Dry electrodes are typically used when the constitution of the electrodes does not allow the use of gel (e.g. bar electrodes). Due to the high electrode-skin impedance it is common to have the preamplifier circuitry at the electrode site. This makes the dry electrodes considerably heavier than gelled electrodes (about 20g vs. 1g) such that electrode fixation becomes an issue.
Gelled electrodes are therefore the common choice. Oxidative and reductive chemical reactions in the contact region of the metal surface and the gel allow an exchange between the ionic current generated by muscle contraction and the electron current flow of the recording instrumentation. It is worth mentioning here, that the quality of an electrode depends almost exclusively on its ability to exchange ions for electrons and vice versa [Overberg, 1992]. A general explanation of the mode of operation of surface electrodes from the chemical point of view may be found in [Meyer-Waarden,1985] and [Overberg, 1992].
2.1.3 Signal Characteristics
The raw EMG signal detected by a differential amplifier using surface electrodes is a “bipolar signal whose random fluctuations, if summed over a significantly long time period, would produce a zero result” [Lamb and Hobart, 1992]. Its amplitude typically ranges from 0.01 to 5 mV [Lamb and Hobart,1992]. Figure 2.2 shows an example of an EMG signal and the corresponding frequency spectrum. The usable energy of the signal is contained in the 0 to 500Hz frequency range, that is, the signal energy is above the electrical noise level in that frequency band. In fact, the dominant energy lies in the 50-150Hz frequency range [Luca, 2002].
Chapter 3
SYSTEM OVERVIEW
In this chapter we give an overview of our EMG speech recognition system. Section 3.1 describe the hardware we deployed while section 3.2 introduces the software we have written and used for this project. The workflows for data collection, system training and recognition are presented in section 3.3.
3.1 Hardware
3.1.1 EMG Equipment
Section 2.1.1 introduced the equipment necessary for sEMG recordings. We used two different physiological data recording systems for data collection which we will refer to as VARIOPORT II and VARIO-PORT VIII depending on the number of EMG channels they provide (two and eight channels respectively) [Becker, 2003b].
The VARIOPORT II data recording system was used for initial experiments on EMG based speech recognition.
Name EMG Chanel Frequency
Range A/D Conversion Resolution Range
Varioport II 2
0.9Hz..500Hz 12 bit ±500μV
Per bit ±500μV
Varioport VIII 8
19Hz-295Hz 16 bit 0.033μV
per Bit ±1070μV
Table 3.1: Properties of data recording systems VARIOPORT II and VARIOPORT VIII
3.2 Computers
Data collection and online recognition were generally performed on a Pentium III laptop (1000MHz,512 MB RAM) with a Microsoft Windows operating system. For offline recognition and training wedeployed different Linux based machines provided by the ITI Waibel of the Universit¨at Karlsruhe. It is worth mentioning here that recognition rates varied slightly for different operating systems.
Figure 3.1: Physiological data recording system
3.3 Software
The software we designed and implemented for this work consisted of two parts: (1) a Visual C++ project for data collection and demonstration purposes and (2) a JRTk based Tck/Tk script collection for recognizer training and classification.
3.4 Data Collection
All signal data used for our experiments was collected in so-called recording sessions. A recording session is defined as a set of utterances collected in series by one particular speaker.All settings (channels, sampling rate, speech mode) remain constant during a session. Data collection consists of four steps:
. Choosing settings: several settings have to be made prior to recording a session. Among other things a word list is selected containing all utterances a speaker has to record during that session. The list can optionally be randomized.
Data recording: The speaker records one file set (appendix B) for each word in the word list using the push-to-talk button of the speaker interface.
Generation of transcript file: When all utterances have been recorded a transcript is (automatically) created for the session.
Generation of settings file: All settings are stored in a file
3.5 Recognition
• Online recognition: Online testing was implemented for isolated words and phrase recognition only. First, a training session and a set of possible hypothesis is selected (subset of the set of training utterances). Next, the recorder and a janus recognition script are started by the VC++ software. Janus and the software communicate via semaphore files. When an utterance has been recorded (i.e. the push-to-talk “recording” button is released) a “start” file is created that is detected by janus. Janus deletes the file and reads the .adc file corresponding to the recorded signal data. It determines the Viterbi path for each hypothesis allowed in the vocabulary and the word/phrase yielding the best Viterbi score is written to a hypothesis file. A “done” file is then created by Janus which is detected by the VC++ software. The VC++ software reads the hypothesis and displays it on the speaker interface. When Janus detects a “stop” file (created by the VC++ software) it leaves the recognition loop. Figure 4.5 illustrates the communication between the different modules.
