During the great Renaissance of the 15th century Leonardo da Vinci said: "Study the science of art. Study the art of science. Develop your sense. Learn how to see and realize that everything connects to everything else.".
One person who seems to have listened to this bit of advice is Peter Neubäcker, founder of Celemony, the developers of Melodyne, who has a deep appreciation of musical instruments and a fascination with the science of sound. His two greatest interests, music and mathematics have guided his life and career. He has evolved from a blues guitar player, to guitar builder, to a Grammy winning software engineer.
Where are you from?
I was born in Northern Germany, but I have been living in Bavaria for a long time now. I think it was 1979 when I moved to Munich. I grew up in the Black Forest in the south of Germany, and later I've also been living in Switzerland for a few years – in the Mountains near Bern.
What were your first experiences with a musical instrument?
I started to play the acoustic guitar at the age of twelve when I fell in love with the music of the Beatles and tried to play their songs. Later I was into rock and blues, especially inspired by the British blues movement in the late 60s.
Playing in the band
It's an irony that the Rolling Stones (and others) popularized blues in the US. It took British bands to get Americans to listen to blues music….
It seems that the people in the US rediscovered their own blues roots that way. When I was twenty years old, I felt like a real musician, but also that being a musician was just a role that I had defined for me. It is much easier to live in such a predefined role than to verify the essential things of life all the time. That's why I lived three years in a kind of spiritual community in Switzerland and didn't touch a single musical instrument in order not to yield to the temptation of that "role".
Music has always been very important to me, but I realized that – for me – it's more important to understand what music really is and not to make music. Later, while searching for the essence of music, I intensely studied Indian and Arabic music.
The music of each culture is very unique and represents a particular aspect of what music can be as a whole. If I decide to study the music of a culture, I try to dive into it as deep as possible, to feel and understand the profound meaning behind it. In my opinion, it makes no sense to take certain parts of music just as exotic ingredients as it is often done in the pop genre.
At what point did you start to get interested in mathematics?
When I finished school, I had two things that I liked and was good at, music and mathematics. At that time, I didn't think they belonged together at all. And quite a bit later, maybe five years after school, I stumbled across some books about the relation between music and mathematics. It was described from a philosophical point of view, going back to Pythagoras. His idea – that music and mathematics are the substance of the world – fascinated me.
I became an apprentice of a luthier and started to build instruments, among them a Monochord (means literally "one string"). It was the instrument that Pythagoras used to show that numbers and musical intervals are in fact one.
Peter with his Monochord
A Monochord is a kind of musical measuring instrument for exploring the number ratios of intervals. It's fascinating to see one part of the string and the other one, let's say with the ratio two to three. Then you pluck the string and you'll hear a fifth. On the one hand that's just dealing with abstract numerical ratios on the other hand it touches your soul by creating something immediate – a musical experience. Music consists of abstract arithmetic and geometric relations and I'd like to find out how these mathematical relations must be to describe more than just single notes or intervals but to create real music. The development of Melodyne is based on that question.
And you started building guitars?
A couple of Peter's guitars
After school I thought at first I might go to a university, but for me it didn't make sense to do something that is expected to be done. So, I had my three years in Switzerland and then got back into the world, so to speak. Then I decided to do something concrete and ended up building acoustic guitars. The wood of an acoustic guitar is vibrating and you have to figure out how to make good vibrations. Being able to work on a piece of wood for a few weeks and it turns out that you can play it afterwards always fascinated me, but my heart belonged even more to philosophy and the relation of mathematics and music. That's why I never became an excellent luthier.
How did you become a programmer?
Guitar building students
I think it was in the mid-80s. At that time, I gave a lot of guitar building courses. One was at a German castle and there was someone who brought an Atari computer although he wanted to build a guitar. I thought: "Well, the way I'm dealing with music and mathematics, using a pocket calculator and just paper and a pencil, it may be a good idea to learn a little bit of programming to calculate things more comfortably. Also I could make a graphical representation of what I think about music and make the mathematics audible." So I taught him to build guitars and he taught me a bit of programming.
What happened next?
Well, at that same time, I was also interested in astrology and astronomy so I published working material for astrologers based on the celestial movements so they could make calculations for horoscopes and the like. The first things I did on the Atari weren't musical at all. I used it to make the graphics for the calendars I published. Then, I also experimented with the capabilities of the Atari to generate MIDI information from Fractals or algorithmic composition. The Atari was already quite good at outputting MIDI information, but you couldn't process audio at that time.
In the early 90s, everyone and every recording magazine said that the NeXT computer was the thing to have. They were very expensive, but I had the chance to get one at half the price and so I switched to NeXT, also with the hope of doing some sound processing on the built-in DSP. It was just learning by doing, with a little help from my friends of course. Working on the NeXT was so different from what I knew from the Atari. You had to learn a whole new philosophy and a new operating system. It was a big hurdle to get over.
Then I met Carsten Gehle, who wanted to make recording software for the NeXT computer. In fact the application was never released, but he told me more about object-orientated programming, and he's the one who started Celemony together with me.
And what year was that?
In the mid-90s. When I started to think about Melodyne, I asked Carsten: "Would you like to help me designing a program that can be used by other people as well?" Because at that time I was the only one who was able to use Melodyne.
I already had Csound running on the NeXT. Melodyne with its graphical user interface and the melodic analyzers sent its data for sound generation via Unix pipe to the Csound engine on the same computer and generated the first sounds. It was more like a laboratory environment and my friend Carsten helped me to set it up as a real program that other people could use as well.
The first version of Melodyne was for the Mac?
Yes, and it gave us some headaches because the NeXT hardware didn't exist anymore. Just the OpenStep system running on PCs and at the same time Steve Jobs went back to Apple….
That was convenient….
Yes, since we knew the NeXT system very well, we had high hopes that Apple would make a quick change. It was quite a hard time for the first few years because they were obligated to stick to Mac OS 9. It was not easy to convert everything because I didn't know Mac OS 9 at all.
Do you think Bach would do some programming today?
In fact, I don't think so. He was too much of a real musician, intuitive and well trained. At that time, all of the composers were technically skilled, but Bach was a real genius on top of that.
I would be curious what kind of music he would compose today. But I personally don't think he would do any coding because he was too much of a full-blooded musician.
What made me think of him was if you listen to his fugues, it's essentially mathematically correct music. I wonder how much of his thinking was in terms of the numbers or rules as opposed to just playing….
I don't think that it is possible to write an algorithm that is able to produce these fugues, but I think his thinking was very structured and that his musical abilities came on top. The people of today mostly think that the great composers relied purely on their intuition while composing. That they were sitting in front of the piano and played until they found something that sounded good. I think it was much more structured work and that a genius like Bach was excellent in constructing music.
What genre of music do you personally enjoy the most today?
I don't have a TV and I don't listen to the radio. I have a lot of CDs, but I think it's only every three days or once a week I listen to one of them. I'm searching for the essence of music and the existing music of the world is often frustrating me, because it just tries to get close to the "real" music. The music of Bach, on the other hand, has feeling on one side and is very close to the abstract ideas of music on the other side.
I think the biggest progress of the last century were the extended chords they use in Jazz music. To go back to Bach: He didn't use the seventh as a substantial part of the harmony, it was just used as a kind of spice to create tension.
Some extended harmonies sound arbitrarily constructed, others, despite of being very complex, sound very organic. The best examples may be found in the early Bossa Nova of the 50s and 60s. The harmonic space deforms organically. You just enjoy the beautiful music and don't even notice how complex the harmonies are. Other good examples are the vocal ensembles of that time, artists who inspired Manhattan Transfer, for example.
A good segue into the next question: People can experiment with your products in the creative process or they can use it as a crutch. And when you talk about singers that move you, let's give an example of Bob Dylan, who may have had his pitch fixed with something like Melodyne today if it had existed then. So I'm wondering what are your thoughts on this?
Melodyne was never intended to be a correction tool. The idea behind it was more about the vision to break up the recording process, to meld frozen audio up again and to work with it freely. You play a tune, record it and then you say: "Ah, it'd be nice to have that harmony." And you are able to continue composing after you have recorded your song.
Maybe 80% of our customers are using Melodyne for pitch correction, but it is rather used as a polishing tool and I think that's okay. If someone is playing live on a stage, you can be fascinated by what he is doing no matter if a note here and there is a little flat. But if you're producing music for a CD, you expect it to be rather perfect, like there is no picture being published without being touched by Photoshop. And still, a bad singer will never be a good singer just because he used Melodyne.
It might still need some time before people realize what you can do with Melodyne apart from pitch correction. Over time, they will be working with more freedom and creativity on the audio material. I would be very happy about that.
How many people work for Celemony now?
A bit over twenty. We are seven developers, the others work for example in Quality Assurance, Product Design, Support, Sales and Marketing. The company itself is located in Munich, but the people working for it live all over the world – two of them as far away as Japan and Los Angeles. In fact, only Anselm, who is our CEO, and I live in Munich. The majority of our crew is spread all over Germany.
Has your life changed profoundly since you started your company? I'm talking about how your time is spent….
To be honest, at first I was quite unhappy with how things were going because I couldn't continue to do what I was really interested in. For the first years, Carsten and I were the only people who worked on the program and I was a kind of CEO for our company. That's a job I'm not made for. I was totally eaten up by business things. I couldn't think about music and wasn't able to develop new ideas. But that has changed since 2005 when Anselm Rößler joined us to become Celemony's Managing Director. Gradually, I was able to shift to a new role as a kind of freelancer for the company. I'm still participating in meetings and our developer conferences, but I also have time for my laboratory. I'm still involved in the development of the actual products, but I can let my thoughts flow, I am able to read books on music theory and think about how music works. All that ends up in our products, but it is not aimed to become a product immediately. I think, this has been the main change over the last years: that I left the business behind and gained some freedom – to think.
If you would like to learn about Peter and his thoughts about audio check out this video.