Embedding files in dll

[elena2]

Hi VST experts, I would like to hear your opinion on this ambiguous topic.
I am writing my vst plugins with SynthEdit. Despite being a very skilled coder (in fact many people know me for my SynthEdit modules and other projects), I never thought of coding a vst from scratch because it could be an incredibly long and complex job, in particular the graphics user interface part. At least with SE you can focus on actual DSP and it will take care of building the needed plugin skeleton for you. Unfortunately SE has one big and well known limitation: it cannot export one dll with all the needes resources embedded (namely gfx files, samples when it is the case and other dlls with custom components). The result is a folder containing the plugin .dll/.vst3 plus a myriad of other resource files. This is annoying and most SE users always agreed with me, because the result is not compact and anybody can easily copy your gfx stuff or modify it or anyway tamper with the content.

Now here is the point.
Jeff mc Clintock, author of SE, always stated that embedding files in a dll is not technically possible because Windows does not allow it (not sure if this is true starting from the x64 version or not), and apparently there are no arguments to convince him.
On the other hand I see that most non-synthedit-made plugins consist of a single dll containing all the plugin resources embedded - and no, they are *not* extracted after the first run as somebody says (at least not in the plugin folder)
So I wonder: where is the truth ?
Thanks

[Markus Krause]

https://stackoverflow.com/questions/189 ... s-in-a-dll

-----------

In some of my plugins I embeded patches or wav files to the dll. You can to convert the data of a fxp or wav file to a header (.h) and then you can include it directly to your code.

For this purpose I wrote a converter wav2header.exe and fxb2header.exe

[kirsty roland]

You can embed any file you like in a dll. Is is after all just a modified form of an exe.
I can't remember exactly but I believe it is a windows thing that windows has to load dll files from disk.
I suppose SE just offloads the 3rd party dll's at runtime so that windows can load them back in.

Also embedding files in a dll/exe does not have to be in the form of a resource and if you only want to make windows plugins
using Win32 GDI/GDI+ for the interface is not that hard. The guys here on kvr showed me how to do it a while back.

Kirsty

[Chaotikmind]

You can embed any file you like in a dll. Is is after all just a modified form of an exe.

There is no difference between an executable and a dll,the format is exactly the same.

[mystran]

There are plenty of ways to embed data into an binary file (whether an .EXE or .DLL which are basically the same as others pointed out).

These are the approaches I've used in the past:

• Use .rc resource file (easy, Windows only). You can see what's inside, even change the files with a resource editor. Whether that's a pro or con is up to you.
• Use a tool that converts binary files into C-files (eg. large constant array of hex-format bytes). Unix systems usually have "xxd" that can do this for you, but writing your own tool is also simple. This is rather foolproof in the sense that it works on every platform and there's nothing that can possibly go wrong, but it can make compile times a bit sluggish if you have a ton of data.
• Put everything into a ZIP and concatenate at the end of your .DLL file. This is sort of advanced approach and takes some effort (eg. you need some library to read that ZIP file and a bit of magic to figure out the binary path), but it's basically the same way you'd build a self-extracting archive: operating systems don't care if your binary contains extra junk at the end and ZIP has it's headers at the end of the file so you can have a file that is both an EXE/DLL and a ZIP file at the same time. Assuming you use standard headers (ie. not trying to hide the fact that there's an archive) you can also open such files in any standard archive tool.

There's probably other approaches, but generally these days I either use 'xxd' if the amount of data is reasonable (eg. at most a few megabytes). The ZIP approach is more effort and I'm not sure if I've ever tried it with a DLL (though really it should work the same as EXE), but it can be really nice for larger projects if you use one of the "virtual filesystem" libraries out there that can shadow local files and archives on top of each other (ie. your code doesn't care where it comes from, so you can use local separate files during development and then just ZIP it all up for the final build; note that this archive never needs to be extracted, plenty of libs let you load the files directly).

[kirsty roland]

You can embed any file you like in a dll. Is is after all just a modified form of an exe.

There is no difference between an executable and a dll,the format is exactly the same.

The op wrote "being a very skilled coder" so I was thinking about modified function naming conventions like WinMain() for exe, DllMain() for DLL etc...

[elena2]

Thanks guys. Yeah very skilled *at dsp coding* but perhaps you missed the point lol... however you pretty replied to my question nonetheless i.e that there is no excuse for not using a single .dll containing every file needed by the plugin and windows does *not* forbid that. Despite Jeff insists otherwise for not doing it, causing a general spirit of dissatisfaction between pretty *all* synthedit users.
The point you missed is that *I am pretty forced to use synthedit* to assemble my plugins. So I have no control on the plugin dll. It's not up to me to assemble it. Despite being a good coder, coding a whole plugin from scratch including the GUI would be an effort I cannot afford given my priorities and the fact I must focus on the dsp part and my time is not infinite! At least SE or similar tools spare you all the hassle involved. But I have *no control* over what SE does as for assembling the final dll and plugin! I can just code the dsp and gui components as accessory .dll's (named .sem's). But as I said you pretty replied to my question and I thank you

[mystran]

Despite Jeff insists otherwise for not doing it, causing a general spirit of dissatisfaction between pretty *all* synthedit users.

Ok, so I could have answered this as well if I had actually bothered understanding the question properly. The challenge here is not embedding and reading the files as such, but rather loading DLL files from memory. The standard LoadLibrary functionality wants a file, so the obvious thing to do is to extract into a file and load that.

Unfortunately there is no standard way to load a DLL from memory directly, so if you want to do that, you're basically stuck writing your own DLL loader. As far as I know that's technically possible, but it's absolutely not simple at all. I'm under the impression that LoadLibrary is basically a big black box and you can't really easily access any of it's internal functionality, so you'll have to redo it all from scratch. That's far from trivial and also a potential maintenance headache.

[elena2]

Thanks Mystran, you went to the point! Now I am *not* familiar with those OS specific aspects so I can't judge. I am not familiar with other plugin creation environments other than synthedit, like Flowstone or Juce and others, but I wonder if the same limitation applies to them aswell or not. I suspect not... I have here some plugins made by flowstone for example and they are single dll's without extra files !
So the truth is likely that Jeff judged the work involved to produce a single dll with everything embedded simply too complicated to put in practice... am I getting you right ?

[Zaphod (giancarlo)]

Despite Jeff insists otherwise for not doing it, causing a general spirit of dissatisfaction between pretty *all* synthedit users.

Ok, so I could have answered this as well if I had actually bothered understanding the question properly. The challenge here is not embedding and reading the files as such, but rather loading DLL files from memory. The standard LoadLibrary functionality wants a file, so the obvious thing to do is to extract into a file and load that.

Unfortunately there is no standard way to load a DLL from memory directly, so if you want to do that, you're basically stuck writing your own DLL loader. As far as I know that's technically possible, but it's absolutely not simple at all. I'm under the impression that LoadLibrary is basically a big black box and you can't really easily access any of it's internal functionality, so you'll have to redo it all from scratch. That's far from trivial and also a potential maintenance headache.

It's something I've been doing since 2006 and it's at the base of the functioning of antivirus, malware, protection systems and generally programming that is inherent to security especially in windows. A pe loader is a quite common operation/task and it's easy to have dll systems grafted with an "inception" style mechanism to the desired number of levels.
The problem is that the relative mechanism on mac for dylib, based on a single function, has been deprecated since at least 15 years and officially banned on silicon since at least two years - it's just something I was talking about here in this section of the forum. From what I understand synthedit runs on mac osx.
However I understand that embedding content into a PE seems a form of effective protection against prying eyes but a decently skilled programmer is able to extract this information very easily. There is no difference between having it there or in another location, the task to be done is at the limit the encryption of the content (and even here, it is easily reversible).

[ratchov]

https://stackoverflow.com/questions/189 ... s-in-a-dll

-----------

In some of my plugins I embeded patches or wav files to the dll. You can to convert the data of a fxp or wav file to a header (.h) and then you can include it directly to your code.

For this purpose I wrote a converter wav2header.exe and fxb2header.exe

I do so as well. Doing so works not only with .dll files but with any binary format on any operating system. One less portability issue to deal with.

[Markus Krause]

Despite Jeff insists otherwise for not doing it, causing a general spirit of dissatisfaction between pretty *all* synthedit users.

Ok, so I could have answered this as well if I had actually bothered understanding the question properly. The challenge here is not embedding and reading the files as such, but rather loading DLL files from memory. The standard LoadLibrary functionality wants a file, so the obvious thing to do is to extract into a file and load that.

Unfortunately there is no standard way to load a DLL from memory directly, so if you want to do that, you're basically stuck writing your own DLL loader. As far as I know that's technically possible, but it's absolutely not simple at all. I'm under the impression that LoadLibrary is basically a big black box and you can't really easily access any of it's internal functionality, so you'll have to redo it all from scratch. That's far from trivial and also a potential maintenance headache.

It could also make trouble with various virus scanners which might report false positives. That's why I do not longer use packers

[elena2]

Jeff kind of confirmed that the problem indeed is not with embedding samples or pictures but with embedding other component dll's. He also says that since on MacOs (SE also exports .au) plugins are however splitted in several resource folders, it would be worth nothing spending a lot of efforts to embed everything on windows using hacks when for mac os the resource files would be exposed anyway. Oh well... case closed thx anyway for your opinions

[AnalogGuy1]

I can't say for SE, but in the past I've used the free iLMerge to merge an .exe that I write with 3rd party .dll's so that I end up with a single, portable .exe. My use is different; I create many, small, interactive interactive programs to teach electrical engineering (and DSP!) concepts, and I don't want my students to have to deal with installation programs. They just download a single .exe and run it. I first embed all the graphical and data resource files that I create into the .exe using Visual Studio's resource manager, and then after compiling into an initial .exe I combine it with third-party .dll graphing packages (whose source code I don't have) using ILMerge. Then codesign it with an EV key.

Like you've found, I probably spend 10% of my time programming the DSP portion and 90% of my time dealing with headaches associated with everything else. And then deal with them again with the next Windows update. Also, my apps are free, so I can sidestep a huge amount of complexity by just trying to outsmart Windows, rather than having to outsmart other humans.

[JHellums]

It would be amazing if someone could write an add on to a dll that would show all of the presets on a single page.So many vst dll's have a split preset menu and I find that annoying. I know there's some of the older vsts that show the presets on a single page, so that would be awesome to upgrade that way!