Photosounder: how to have two layers with two different sounds

[c_voltage]

Actually I think I'm starting to get the point of Photosounder, I always thought of it as a way to do specific practical things except it doesn't do them quite as well as dedicated tools do (like imagine doing equalisation of a track in Photosounder, which I unironically used it for), but now I think I get it, what it's great at is doing wild stuff, and I guess that's what people always liked in it, so that's a good direction to pursue, and the ability to deal with more than one file at once and do things like convolution is very important for that.

Frankly to say, recently came to mind some ideas for Photosounder, if we are talking about its tasks now.
I like his definition as a "synth" (that is really indicated on the site among other). Perhaps i not enough understand the mechanism of sound so deeply (in his digital nature), together with restrictions of spectral algorithm. But i remember the first thing that puzzled me when i first time start to explore Photosounder, this is - why it can produce only or near to sinusoidal signals together with noise. (I'm talking mainly about drawing from scratch). I wanted to get any sharp\crisp\ragged sound (not to be confused with postprocessing, like distortion) even like a simple saw or square but nothing happened.
I can admit that all sounds compose of short sinusoids (an vice versa - noise contain spectra of all sounds), and perhaps this is somehow related to the direction of thought of spectral processing. But why not go beyond the scope of this a bit.
We have full graphical editor in which the colors are analyzed and interpreted. Then, why not do something like this: red color - saw waveform, green - square, blue - triangle, white - noise, black - sinus, and brightness of each it's pitch level (for example) etc etc indefinitely. Any reaction to color at will.
Also can go further, since we have layers - it can be FM synthesis of superimposed signals (mixed between each layer).
You just imagine what a palette of sounds can get out of any image\photo.
And, ofcourse, such varieties of the algorithms can be dozens of different. It can be a list of the created by the author in advance. Or some customizer for creating a user algorithms of interpretation (or even some simple scripting ability to make algo in text view).
Would be cool.

[A_SN]

Frankly to say, recently came to mind some ideas for Photosounder, if we are talking about its tasks now.
I like his definition as a "synth" (that is really indicated on the site among other). Perhaps i not enough understand the mechanism of sound so deeply (in his digital nature), together with restrictions of spectral algorithm. But i remember the first thing that puzzled me when i first time start to explore Photosounder, this is - why it can produce only or near to sinusoidal signals together with noise. (I'm talking mainly about drawing from scratch). I wanted to get any sharp\crisp\ragged sound (not to be confused with postprocessing, like distortion) even like a simple saw or square but nothing happened.
I can admit that all sounds compose of short sinusoids (an vice versa - noise contain spectra of all sounds), and perhaps this is somehow related to the direction of thought of spectral processing. But why not go beyond the scope of this a bit.
We have full graphical editor in which the colors are analyzed and interpreted. Then, why not do something like this: red color - saw waveform, green - square, blue - triangle, white - noise, black - sinus, and brightness of each it's pitch level (for example) etc etc indefinitely. Any reaction to color at will.
Also can go further, since we have layers - it can be FM synthesis of superimposed signals (mixed between each layer).
You just imagine what a palette of sounds can get out of any image\photo.
And, ofcourse, such varieties of the algorithms can be dozens of different. It can be a list of the created by the author in advance. Or some customizer for creating a user algorithms of interpretation (or even some simple scripting ability to make algo in text view).
Would be cool.

Yes I'm aware of that. I see it like this, there's two basic types of sound components (actually you could say 3 if you count transients, like clicks), noises, which occupy areas of the spectrum, and pure tones (as in FM synthesis) that at any given time only represent one point in the spectrum. Noise can't occupy a single point (although if it's narrow enough it might sound like a sine with an envelope) and pure tones can't occupy an area (unless their envelope is sufficiently noisy). Here noises are best represented as raster images, and pure tones (as in FM synthesis, which may go up or down) are best represented as thin vector lines or curves. So it's thin vector graphics + raster graphics to synthesise anything you might need. But Photosounder only has raster graphics and therefore noise synthesis, so anything that might be vector/FM is only approximated by narrow bandwidth noise.

So one of these days (not too soon) I'll make something new based on this whole concept that will be more about synthesising actual music and music instruments, not just weird noises, and it will have the dual approach. So that's the plan, to go on on the current path for now, and later to make something more ambitious that would give us all the tools we need to synthesise actual music with great self-contained synthesis capabilities. This is why I've spent the last 3 years not developing current products but working on fundamental interface and groundwork stuff, because I'm gonna need a solid basis to do something like this on my own.

[XComposer]

Well, I see and use Photosounder as a great tool for transforming audio samples: making a sonogram out of an imported audio sample, then transforming it in many ways: stretching in time; stretching, inverting, slanting, thinning or smearing the spectrum; vocoding, convolving with other sounds… and the like. You may call it "sound design", but I call it "composing", because for me (and those like me) there is no difference between composing and designing sounds; I come from the "academic classical" tradition (not pop/rock/jazz) and, since electronic music was established in use among classical composers (think to Maderna, Berio, Stockhausen and the Darmstadt school, and pioneers like Varèse before them), it was already clear that one can "compose the sound", directly. This is why I love Photosounder, first. Personally, I feel no need at all to have any connection whatsoever to the tempered system or traditional scales and chords based on traditional tonality-based music and traditional keys. On the contrary, I appreciate strongly that this connection doesn't exist (I would experience it as a strong limitation, in fact!).

However, I would probably appreciate the possibility to have different waveforms for each horizontal line of the image; if you think each line as an oscillator, now we have more or less only approximated sinusoids for those "oscillators"; on other softwares (e.g. Metasynth) you can use "instruments", that are like simple samples or simple waveforms providing the waveform to be used in each "oscillator" (each horizontal line) instead of the default sinusoid. I think that this could be interesting.

[c_voltage]

A_SN Ok, got it, thanks. Well, then do not forget about file import to layer please ))

[A_SN]

However, I would probably appreciate the possibility to have different waveforms for each horizontal line of the image; if you think each line as an oscillator, now we have more or less only approximated sinusoids for those "oscillators"; on other softwares (e.g. Metasynth) you can use "instruments", that are like simple samples or simple waveforms providing the waveform to be used in each "oscillator" (each horizontal line) instead of the default sinusoid. I think that this could be interesting.

This would be totally contrary to the philosophy of my approach, which isn't to use whatever means to get to certain results but rather to have a very technical approach to a general problem, something that makes sense, I guess you could say on a scientific level, as a way to analyse sounds into an accurate graphical description and turn such graphical descriptions (an image of a sound) back into a sound, as accurately as possible, and make it a process that doesn't depend so much on my choices but that is the realisation of a simple (if unfamiliar to most) idea. Basically the idea is to have a clear and direct access to the sound's contents, either to modify it or create something new, so adding a layer of removal from that directness by using arbitrary waveforms without seeing what it does is very much contrary to what I want.

So I definitely would not use the Metasynth approach. However there is a way to do something like this, which will be even more feasible once we can open to a new layer, and it's using "vertical propagation" (convolution vertically between columns of pixels) with a layer that represents a continuous "waveform", as you suggested, which would pretty much achieve what you want, albeit still being limited to noise synthesis. Btw doing it for each horizontal line as you suggested would be quite horrible when you'd have several adjacent lines being on at the same time. The reason this works is that basically using an arbitrary continuous-sounded waveform instead of pure tones (or the noise equivalent I use) is like replacing each row of pixels with the transposed overtones of that waveform, in other words it's exactly vertical convolution in the log frequency domain, so pretty much the same thing. You can try experimenting with vertical propagation to achieve something like this.

[XComposer]

Thank you, I think you are right and I like your approach. When the "Open to a new layer" feature will be implemented, I will be very glad to experiment with propagation and convolution in general.

[c_voltage]

When the "Open to a new layer" feature will be implemented, I will be very glad to experiment with propagation and convolution in general.

Me too.

[A_SN]

Yep, fair enough. If I'm gonna showcase what convolution can do it would be better if I have that ready as well. Maybe I can do them both at the same time, I should start with a basic functionality that just adds a new layer with (by default) the same dimensions as the old project.

[A_SN]

Okay I got it working, I'm not exactly amazed by the speed of the 2D convolution, it's less than 1 frame per second with a New project (so the size is about 1740x571, size matters a lot), but at least it's consistent, for a given size it never gets slower. I could try throwing OpenCL at it (GPU acceleration), but I'm not sure if that would work for everybody. On the other hand if I start using OpenCL I could potentially use it in other places too, which could be good.

I use an interesting way to cache FFT and convolution results based on running checksums on layers to see if they match the cached results, so it only ever computes what it needs, which helps. It might make a huge difference when I get to vertical and horizontal convolution since only the changed lines will be recomputed.

Btw I'm thinking of making the drawing tools make flat horizontal lines when the Shift key is held, this should be useful, mostly for drawing anything vaguely musical.

[c_voltage]

Okay I got it working, I'm not exactly amazed by the speed of the 2D convolution, it's less than 1 frame per second with a New project (so the size is about 1740x571, size matters a lot), but at least it's consistent, for a given size it never gets slower. I could try throwing OpenCL at it (GPU acceleration), but I'm not sure if that would work for everybody. On the other hand if I start using OpenCL I could potentially use it in other places too, which could be good.

I use an interesting way to cache FFT and convolution results based on running checksums on layers to see if they match the cached results, so it only ever computes what it needs, which helps. It might make a huge difference when I get to vertical and horizontal convolution since only the changed lines will be recomputed.

Btw I'm thinking of making the drawing tools make flat horizontal lines when the Shift key is held, this should be useful, mostly for drawing anything vaguely musical.

Good.

[A_SN]

Great news everyone, the GPU-accelerated FFTs are really fast, that means the convolution updates several times per second (so basically you can draw with the convolution turned on, even though it's a bit laggy), and the FFT itself runs so fast (~50 ms, despite being 2D on millions of pixels) that it makes me think that I should make every FFT in Photosounder GPU-accelerated, that would make loading sounds but more importantly rendering sound (mostly in lossless mode) much faster, and in turn I wouldn't need to use all the cores of the CPU for things like this so it might make the whole program much more usable and snappy.

Edit: Actually nevermind about the last thing, it would take more than swapping out FFTs to make a real difference in speed. Still doable by GPU but it would take more work, so maybe later.

[A_SN]

So the convolution is done, the import is mostly done, and someone reported problems to me linked to SDL 1.2, so I transitioned to SDL 2.0 which among other things allows for drag-and-drop of files, which I just got working as the equivalent of an open. But since we'll also have the option to import, then it would probably make sense for drag-and-drop to be a way to add files to the project on new layers, right? But sometimes if let's say you're going through a folder of pictures then you probably wouldn't want this, so what should be done by default?

Another aspect of switching to SDL 2.0 is that it also allows me to have my new interface system I've been working on for the past few years, the one that I say will allow me to make a bunch of new interface things quickly, so it will be there in the next update. That means maybe when getting a drag-and-drop I can have some file dialog pop up and ask what to do?

[XComposer]

An idea would be: drag and drop means "open" if nothing is already open and means "open as new layer" if there is already at least one layer with existing content (problem: in some cases the existing layer could just be part of the current project, but in some other cases the existing layer could be an old project, to be closed and, if desired, saved).

An alternative: drag and drop means "open" (always as the base layer of a new project) and option (or alt)–drag and drop (if possible) means "open as new layer" (I like this).

A second alternative would be the one with the dialog, yes.

To make things clearer, please remember that a user could make a drag and drop also when an existing project is already on the screen, so also this case should be managed.

[A_SN]

I'm mostly done and you can check it out by downloading the beta in the full version download page. Opening to a new layer is there, works and has convenient dialog boxes:


As it is with this beta you can import as many files as you like and choose how they are added (the crop/fit/extend options). Opening normally still has the same lack of a dialog, I'm thinking of unifying it with the one above, but the internals have changed to the same way as opening to a new layer, the loading is also parallelised in the same way, I just need to link it to some GUI. And drag and drop still defaults to opening normally, which I will change once I've unified both ways of opening a file, so that upon drag and drop you'll get the same unified opening dialogs.

There's also fast convolution of layers (although on Mac the GPU-accelerated version is temporarily not there while I figure out how to statically link the library it depends on), 32-bit float TIFF import/export, a New file dialog, probably some other stuff too idk.