New Types of Instruments

[nosuchtim]

"Roblox CEO David Baszucki says he wants both creators and players to be able to prompt experiences out of thin air."

https://open.substack.com/pub/bigtechno ... generation

I love this thought. People should be able to prompt musical (and visual) instruments out of thin air. We're very close. The Guthman contest entries almost invariably involve the creation of new physical interfaces. I'm looking forward to the more-virtual equivalents that don't require that much effort for the physical parts. These could take the form of 1) purely AR creations, 2) AR augmentation of always-present physical objects (a table, your arm, etc), 3) AR augmentation of 3d-printable objects that are usable as soon as they're printed. or 4) AR augmentation of purchased products. Having a physical object involved in the interface seems very desirable, even when using AR. It would be interesting if some mostly-unconstrained physical products (e.g. the Erae products) could become standard platforms for people to use in prompt-generated instruments.

...Tim...

[olavikkyvfx]

Hey, I’m very interested in creating a plug-in/vst thats kinda more synth based but also with pads, leads, detuned instruments, and then the best part a section that has up to 20 different keyboard stock sounds that are based off of real analog pianos that I have used over the years… In total im thinking possibly 80-100 sounds available. Would you be able to code it and also make it look a certain way if I designed it?

[JO512]

Nyle added a piezo electric transducer on the top for natural feeling vibrato like he had on his EVI it really felt like playing a string instrument.

I've done this with a cheap piezo microphone. I put my MIDI controller on top of three stacked pairs of those round, stiff foam furniture pads such that two support the back corners and one supports the front middle, a triangular support arrangement. I sandwich the piezo mic between the two pads that make up the pair that supports the front of the controller. Then I plug the mic into my audio interface. I made a Reaktor patch and alternately a Bitwig grid patch that processes that audio signal and turns it into a CC signal or pitch bend. I add an envelope to kill the effect for the first moment when you strike a key so that it starts out in tune. You can then vary the pressure on any of the keys and it creates a really nice, natural vibrato. The piezo mic being squeezed between the pads responds very nicely to slow pressure changes. I did that because I find it really hard to do natural vibrato with a pitch-bend wheel or touch strip.

[mjolnir]

this is amazing stuff you guys are doing.

[vurt]

The yearly Guthman Musical Instrument Competition always has interesting and though-provoking new musical instrument ideas. This year's 10 finalists can be viewed here, including videos of each:

https://www.popsci.com/technology/2025- ... finalists/

currently waiting for my chromaplane

which arrives tomorrow!

[zerocrossing]

I've been interested in this topic as well. In the early 90s, I was experimenting with a program called Arkaoss that would allow me to control any media that the Quicktime wrapper supported with MIDI. The biggest thing I learned at the outset was that for this type of thing to be successful, one had to have a performance that had very big and obvious gestures. So, as a guitarist, using a GK based pitch to MIDI converter was a flop. Keyboard was also not great. The only success I had was when I used a Roland Octapad and made sure my strokes were overly obvious, or else the audience had no idea that I was performing the visuals. Since I'm not really a percussionist, I gave up on Arkaoss.

In its place, I started playing movies and providing a live-improvised soundtrack to them. What was surprising, was that a lot of people asked me how I was able to get my music to match with the visuals so well when it was clear I wasn't watching them. Like my friend Morpheus says, "the mind makes it real." Human brains are really good at knitting together things to provide the you with a coherent world, even when it doesn't really exist. So people didn't get that I was directly trigging visuals when I was, and did think that I was playing along with a movie in an intentional way, when I wasn't.

I was also experimenting with music based installations, but I didn't really have the money to buy the gear that would have made it useful. Most of my stuff just ended up being the equivalent of a thumb piano made in Macromedia Director.

I stopped doing live shows, so I stopped following that kind of thing. Now I'm mostly interested in MPE controllers and what they can bring to a performance. A lot of most synthesizer sounds are fully automated. You hit a key (on switch) and you set a series of events in motion that play a sound that varies over time. With MPE, you gain a lot more control over how and when those events happen. With MPE, you can become the envelope generator, or the LFO. This makes for a much more expressive performance, which as a guitar player, I love. My favorite is the Roli Rise 49, though I wish they'd make a 61 key (lump?) version. I also had a Linnstrument for a while, and I found it a bit better for percussive type sounds, but terrible for voicing chords. I'd like to see some company make an MPE guitar style controller. No strings or anything, just like a Chapman Stick-Guitar type of thing, but narrow enough for a barre chord to be easily played.

Aside from that, I'm not sure what could be made that wouldn't be too difficult for the human form to easily manipulate. When I was working for Laurie Anderson, the tech was all cutting edge, but pretty conservative in terms of human ergonomics. While a tape-bow violin is a nice novelty, I imagine everyone, including Laurie, got bored with it fairly quickly. That's often the issue with all this stuff. It's kind of broad in one sense, but with very little depth. One avenue that I think could be ripe for experimentation is AR or VR. I suggested making a "Music world" for Meta's Horizon World platform when I was working for them, but no one seemed interested. Instead we made a shoot'n loot, which we all know the world needs a lot more of. PEW PEW!

[stoopicus]

In its place, I started playing movies and providing a live-improvised soundtrack to them. What was surprising, was that a lot of people asked me how I was able to get my music to match with the visuals so well when it was clear I wasn't watching them. Like my friend Morpheus says, "the mind makes it real." Human brains are really good at knitting together things to provide the you with a coherent world, even when it doesn't really exist. So people didn't get that I was directly trigging visuals when I was, and did think that I was playing along with a movie in an intentional way, when I wasn't.

Around that time, many of my favorite bands were electroindustrial outfits on labels like Nettwerk and Wax Trax, and many of them did things like this live. It is a cool technique and worked surprisingly well, for the reasons you mention.

(Many are still among my favorite bands, but they are no longer on those labels.)

[birdsoffire]

Nyle added a piezo electric transducer on the top for natural feeling vibrato like he had on his EVI it really felt like playing a string instrument.

I've done this with a cheap piezo microphone. I put my MIDI controller on top of three stacked pairs of those round, stiff foam furniture pads such that two support the back corners and one supports the front middle, a triangular support arrangement. I sandwich the piezo mic between the two pads that make up the pair that supports the front of the controller. Then I plug the mic into my audio interface. I made a Reaktor patch and alternately a Bitwig grid patch that processes that audio signal and turns it into a CC signal or pitch bend. I add an envelope to kill the effect for the first moment when you strike a key so that it starts out in tune. You can then vary the pressure on any of the keys and it creates a really nice, natural vibrato. The piezo mic being squeezed between the pads responds very nicely to slow pressure changes. I did that because I find it really hard to do natural vibrato with a pitch-bend wheel or touch strip.

I am currently using an Exquis just for adding natural vibrato. I use Max to just pass on the pitchbend to my DAW. I want to use the other MPE dimensions as well but find the sensitivity of the Exquis difficult to work with.

[JO512]

I am currently using an Exquis just for adding natural vibrato. I use Max to just pass on the pitchbend to my DAW. I want to use the other MPE dimensions as well but find the sensitivity of the Exquis difficult to work with.

You might try the piezo microphone method. Such mics are really cheap. It works better than you might expect once you get it set up properly.

[sarogrillkfc]

Dude, this is such a cool topic. I used to mess around with a Roli Seaboard Block and a Linnstrument (sold 'em both, though), so I’m stoked to dive deeper into this. Fingers crossed the responses here are way better than the last one. Back in the '70s, I had a Steiner Synthacon—Nyle slapped a piezo electric transducer on it for this super natural vibrato, like the EVI. It legit felt like strumming a violin or something. Anything that makes musical expression feel effortless is like a spark for creativity, you know?

[Supersonic20]

Tim, your post is a fascinating dive into the world of new instrument design! I love how you highlight the potential of technologies like pressure-sensing surfaces, haptic feedback, depth cameras, GPUs, and open-source tools to revolutionize both music and visuals. The concept of “casual instruments” is especially compelling—making creativity accessible with no barrier to entry while still offering depth for exploration is a game-changer. Your Burning Man experiences sound like a perfect testing ground for these ideas, and I’m excited to see the examples you’ll share in future posts. Looking forward to learning more about how audio and visual synthesis are merging in this space!

[nosuchtim]

If you're interested in the notion of "casual instruments", here's a talk titled "Adventures in Casual Instruments" that I gave a couple years ago when my Space Palettes (Classic and Pro) were shown at the Center for New Music in San Francisco:



Here's some video snippets from a showing this summer at the Tech Museum in San Jose:



I rarely perform with it, instead focusing more on other people's use of it, but here's a performance I did last year:



...Tim...

[vurt]

If you're interested in the notion of "casual instruments", here's a talk titled "Adventures in Casual Instruments" that I gave a couple years ago when my Space Palettes (Classic and Pro) were shown at the Center for New Music in San Francisco:



Here's some video snippets from a showing this summer at the Tech Museum in San Jose:



I rarely perform with it, instead focusing more on other people's use of it, but here's a performance I did last year:



...Tim...

no idea what a space pallete is yet (will watch vids later) but i think i want one!

[Hooskter]

I remember a young artist from Ukraine - The Maneken performing at the Kazantip festival in 2008.

[ATsoundstorm]

Long time ago (in a galaxy far, far away) there was an app made for BeOS which allowed mixing audio in simulated 3D. I think it was called 3DMix, but don't quote me on that.
You could literally grab audio source marker with your mouse and drag it anywhere within the plot. The position of the marker controlled pan, volume and proximity.
I wonder, if people could do that in the late 90s/early 00s, why don't we have more of this now? Such visual feedback could drastically simplify mixing.
The only software with such capabilities, that I can think of, is Audiocube. But that's a DAW. Can it be a plugin, similar to the Waves CLA Mixhub?