Karl Nordstrom

MAGIC: Media and Graphics Interdisciplinary Centre, UBC
MISTIC: Music Intelligence and Sound Technology, UVic
karl(insert at sign)karlnordstrom.ca

Project

My PhD research started in collaboration with IVL Technologies and TC-Helicon, two voice processing companies in Victoria. TC-Helicon produces vocal effects products for the music industry with a focus on pitch correction and automatic harmony creation. IVL produces hand held karaoke products that produce a variety of vocal effects while being plugged into a TV. In recent years, vocal effects have become more common including chorus based effects and even distortion.

The goal of the research was to enhance a digital effect that adds noise to a voice to simulate breathiness. If a voice already sounds breathy, it is easy to add noise to increase the perception of breathiness. However, it is difficult to add noise to voices that exhibit high-effort (i.e. when the voice sounds strained). The added noise does not blend into the voice and instead sounds like a separate stream of noise. In my research, I developed a technique to adaptively manipulate the perception of vocal effort. This enabled the added noise to blend more effectively, thereby improving the breath effect.

An important outcome of this research is to highlight the fact that Linear Prediction (LP) (a common voice modeling technique) does not appropriately model the voice. I presented Adaptive Pre-emphasis Linear Prediction (APLP) as a technique to appropriately compensate for variations in vocal effort.

Academic and Industry Collaborators

• Peter Driessen, MISTIC, Electrical Engineering, UVic
• George Tzanetakis, MISTIC, Computer Science, UBC
• Glen Rutledge, 3dB Research (previously at TC-Helicon and IVL Audio)
• Kevin Alexander, TC-Helicon
• Brian Gibson, IVL Audio
• John Esling, Linguistics
• Mathieu Lagrange, MISTIC, Computer Science

Publications

• papers
• dissertation

I have now implemented adaptive pre-emphasis linear prediction (APLP) for phrases. The goal of this implementation of APLP is to transform high-effort voices into breathy voices.

Original voice:
I started with this sound sample: wav. With this kind of voice, it is difficult to add breathiness in a way that sounds natural.

Breath effect with constant pre-emphasis linear prediction (LP):
Constant pre-emphasis LP was carried out, noise was added to the residual, and the voice was resynthesized: wav. This is the technique used in current voice processors to apply a breath effect.

Breath effect with APLP:
APLP can be used to transform the spectral envelope of the voice to match that of a typical breathy voice. This reduces the perceived vocal effort and improves the blending of noise into the voice: wav.

Breath effect with APLP and breath modulation:
The amount of breathiness in a voice varies with the amount of effort. As such, it makes sense to add less breathiness during times of excessive effort. The APLP algorithm was further improved by modulating the quantity of added noise according to the quantity of vocal effort: wav.

Now listen to the original voice again.

With the breath effect, some voices work better than others. Here is how the breath effect sounds on another voice:

Original voice: wav.
Breath effect with constant pre-emphasis LP: wav.
Breath effect with APLP: wav.
Breath effect with APLP and breath modulation: wav.

________________
Technical Note:
Previous iterations of the algorithm included glottal closure detection to improve the blending of noise into the voice. Glottal closure detection was eliminated from this iteration of the algorithm. This ensures that the sound samples are representative of what can be achieved on real-time voice signals in a musical context.

• K. I. Nordstrom, G. Tzanetakis and P. F. Driessen, “Transforming Perceived Vocal Effort and Breathiness Using Adaptive Pre-Emphasis Linear Prediction“, IEEE Transactions on Audio, Speech and Language Processing, vol. 16, no. 6, pp. 1087-1096, August 2008.
• K. I. Nordstrom and P. F. Driessen, “Variable preemphasis LPC for modeling vocal effort in the singing voice“, Proceedings of the 9th International Conference on Digital Audio Effects (DAFx06), Montreal, QC, Canada, September 2006.
• K. I. Nordstrom, P. F. Driessen, and G. A. Rutledge, “Influence of the LPC filter upon the perception of breathiness and vocal effort“, IEEE Int. Symposium on Signal Processing and Information Technology (ISSPIT06), Vancouver, BC, Canada, August 2006.
• K. I. Nordstrom, “Eliminating the perception of vocal effort while synthesizing breathy voices”, Graduate Innovation Forum, University of Victoria, BC, Canada, May 2006. [abstact]
• K. I. Nordstrom, G. A. Rutledge, P. F. Driessen, “Using voice conversion as a paradigm for analyzing breath quality“, IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PacRim05), Victoria, BC, Canada, August, 2005.

K. I. Nordstrom, G. Tzanetakis and P. F. Driessen, “Transforming Perceived Vocal Effort and Breathiness Using Adaptive Pre-Emphasis Linear Prediction“, IEEE Transactions on Audio, Speech and Language Processing, vol. 16, no. 6, pp. 1087-1096, August 2008.

People often ask me why I chose my particular research topic. Why work on transforming high-effort voices into breathy voices? I typically respond by describing some of products for voice transformation, created by TC-Helicon, IVL Audio (now both competitors), and 3dB Research. I talk about how a breathy effect can be used in musical situations to enhance the voice. That said, there are some other motivations. I’m curious about the physiology and acoustics of the voice. I’m curious about how the acoustic signal from the voice can be manipulated to simulate physiological changes in the voice. I sometimes talk about these motivations. However, there is one motivation that I rarely describe, and that is of aesthetics.

In manipulating the voice, I am attentive to the particular sound texture of the voice (also known as voice quality). I want to find ways to manipulate that texture without creating something that sounds unnatural. This is very subjective. The voice is a complex instrument and it can create a diversity of sound textures. For example, voice quality has been described with more than thirty different terms as various researchers have attempted to better understand the voice. In my research, I am attempting to translate the subjective perceptions of breathiness and vocal effort into an engineering context where they can be quantified and manipulated. However, my deeper interest is in creating a pleasing sound that matches a subjective ideal.

This search for a particular sound texture in the voice parallels my search for what guitarists call “tone” in playing the guitar. It’s an aesthetic judgment about how good or bad the instrument sounds. It’s possible to analyze this “tone” on a spectrogram and artificially mimic the “tone” of a great guitarist, but ultimately the judgment about “tone” is an artistic judgment made by the listener. In theory, it may be possible to quantify “tone” but the point of creating a “tone” is not to be quantified and controlled. The point in the guitarist having “tone” is to create something aesthetically pleasing. It’s the same with my interest in the sound texture of the voice. I want to create voices that sound aesthetically pleasing.

While I find myself primarily in an engineering context, my motivation is to create sounds that are artistically appreciated. However, it’s not easy to do large transformations that sound natural. The complexity of the voice makes transformation difficult. After all, I’m working with an instrument that has no fixed dimensions. It is easy to make small changes, but most large changes sound bad. The techniques that I have developed are more effective than the prior technology, but there is still a long way to go before large voice transformations sound good. As a result, the artistic side of me is often disappointed. Yes, I find the work interesting, but there is still so far to go.

I carried out some listening experiments to evaluate the voice transformation algorithm. The results of this experiment have not yet been published but the algorithm is described in this paper that was presented at DAFX. The goal is to transform a high effort voice into a target breathy voice.

Here is the original high effort voice.

I want to transform that high-effort voice into this target breathy voice.

If I attempt to transform the high-effort voice by adding artificial aspiration noise while using constant pre-emphasis linear prediction (LP), the transformed voice sounds like this. The voice exhibits more aspiration noise but still retains the perception of high vocal effort.

A breathy effect from a commercial voice processor was also used to add breathiness to a high-effort voice. This LP-based effect uses some additional filtering to shape the voice spectrum and to shape the added aspiration noise. However, the core of the algorithm operates as a constant pre-emphasis LP algorithm.

The presence or absence of aspiration noise is only one of the differences between high-effort and breathy voices. When a voice changes between high-effort and breathiness, the spectral envelope of the voice also changes. High effort voices contain more high-frequency content than the corresponding breathy voices. Constant pre-emphasis LP does not modify the spectral envelope of the voice.

Adaptive pre-emphasis linear prediction (APLP) can be used to change the perceived vocal effort by modifying the spectral envelope of the voice to match the spectral envelope of the breathy voice. After the spectral envelope has been re-shaped, artificial aspiration noise is added. The results vary slightly depending on the order of the spectral emphasis filter that is used to shape the spectral envelope: first-order spectral emphasis filter or third-order spectral emphasis filter.

The transformation is not perfect but APLP results in a transformed high-effort voice that more closely matches the target breathy voice.

I have been developing a way to measure variations in the spectral envelope of the glottal source. Not long ago, I came across a old paper that describes much of what I had discovered on my own. Adaptive inverse filtering can effectively capture large variations in voice quality between breathy and pressed voices where more standard techniques of closed-phase, covariance LPC break down. Closed-phase techniques don’t work well for high-pitched or breathy voices because the closed-phase can be short to non-existent. Paavo Alku has written an excellent paper that effectively describes an adaptive inverse filtering technique (adaptive LPC) that can extract glottal pulses from breathy and pressed voices. If you’re interested in robust techniques for modeling variations in the glottal source, Alku’s paper is well worth reading.

• Paavo Alku, “Glottal wave analysis with Pitch Synchronous Iterative Adaptive Inverse Filtering,” Speech Communication, vol. 11, pp. 109–118, 1992.

You will find some samples, below, that demonstrate how the voice conversion algorithm sounds different depending upon the order of the adaptive pre-emphasis filter. The goal of the algorithm is to reduce the perceived vocal effort and to increase the perceived breathiness.

In the adaptive pre-emphasis algorithm, it’s necessary to choose an order of filter for the pre-emphasis. If the order is too low, the pre-emphasis does not have enough dynamic range. If the order is too high, the pre-emphasis will capture formant information.

I’m using LPC to estimate the pre-emphasis filter. As long as it is a first order filter, the pole of the filter is at zero hertz and the pre-emphasis filter looks like a spectral tilt. This is the typical configuration for the pre-emphasis filter. At orders higher than one, LPC can estimate a pre-emphasis filter with pole(s) at higher frequencies in the voice spectrum. This happens with high-effort voices and the resulting pre-emphasis looks like a spectral tilt plus a mid-range resonance. You can find a plot of this result in the DAFX paper.

I can make a number of arguments about whether the pre-emphasis should have a resonance in it or not. I’m not going to explain it now except to say that perceived vocal effort is the result of both changes to the voice source and changes to the vocal tract filter.

The above explanation was very brief. Whether you understand it or not, you can listen to some of the resulting samples, below:

We want to make the high-effort voice sound like this target breathy voice: wav.

Here is the original high-effort voice: wav.

One common way to try to simulate breathiness is to add aspiration noise to the LPC residual. This is what it sounds like when we do that with the high-effort voice: wav.

The voice conversion algorithm uses adaptive pre-emphasis LPC to reduce the perceived vocal effort in the voice before adding noise to simulate breathiness. Here is the transformed high-effort voice:

1rst order pre-emphasis filter: wav.
2nd order pre-emphasis filter: wav.
3rd order pre-emphasis filter: wav.

I have opinions about the sounds of these samples but I’m curious about your opinion. Which sample do you think gets closest to the target breathy voice? Which sample sounds the most natural to you? Which sample sounds the most unnatural?

I wasn’t entirely happy with the DAFX samples. There were a number of artifacts in those voice samples that I have now reduced. (You will need good speakers to hear the differences properly, i.e. they certainly won’t sound right through laptop speakers.)

My goal is to convert a high-effort voice into a breathy voice.

Here is the breathy voice that I’m using as my target: wav.

Here is the high-effort voice that I’m trying to transform: wav.

And here is the transformed voice with the pre-emphasis modified to simulate reduced effort. Pulsed noise has also been added to simulate breathiness: wav
Same thing with more aspiration noise added: wav.

The transformed high-effort voice sounds more relaxed and breathy even if it has not been fully transformed into the target voice.

The main problem with the DAFX voice samples is that there is too much gain and spectrum modulation in the LPC filter (the LPC filter bounces around). When there is just one LPC filter, this problem is not as large. However, my algorithm has two LPC filters in series. The filter modulation from the pre-emphasis filter (low-order LPC) exacerbates the filter modulation from the following vocal tract filter (high-order LPC), making the artifacts worse.

I reduced the artifacts by keeping the pre-emphasis filter constant for the short voice segments that I am synthesizing. The pre-emphasis still varies from sample to sample. The next step would be to use time-varying pre-emphasis but to smooth the filter coefficients in time.

I also reshaped the added noise to make it more similar to the breathy noise in the target voice.

PS: You can hear even more recent sound samples here.