Editing Sysex data dumps?

[DevonB]

Anyone edited a sysex dump? I have a Novation KS Rack that has 4 banks of 100. I also have several sysex dumps from other Novation synths that only use bank 1 and 2; 3 and 4 don't exist. So what I'd like to do is edit the Bank information to change it from 1 or 2 to 3 or 4 so I can dump directly into those banks. The manual on page 66-67 show you which bits to flip, but I'm not sure what program I would use to flip those bits. Any suggestions on what program would show that structure so I can modify it on the right bit/byte? Details below of what I'm looking at exactly -

https://resource.novationmusic.com/site ... ish102.pdf

When an all banks dump is transmitted, C=1 such that the receiving KSx will store the programs in the bank given in the B byte

PROGRAM DUMP REQUEST
F0h SYSEX START
00h NOVATION ID 1
20h NOVATION ID 2
29h NOVATION ID 3
01h DEVICE TYPE
42h KS SERIES
SyCh SYSEX CHANNEL 7Fh or current receive channel
41h MESSAGE TYPE program dump request
C CONTROL BYTE reply will copy this value for C
Vv SOFTWARE VERSION don't care
Vi VERSION INCREMENT don't care
B PROGRAM BANK 1-8 : don't care if C=0
P PROGRAM NUMBER 0-99 for program banks 1-4, 0-49 for drum banks 5-8
F7h END OF EXCLUSIVE

Devon

[lfm]

I've done a lot of that.

A hex editor HxD which is perfect for this kind of job
https://mh-nexus.de/en/

So if one packet only no problem.
The largest dumps I encountered was Hammond between 60kB-100kB and those were packets that needed to be unpacked before seeing the actuall sysex.

[DevonB]

After about 2 minutes to wrap my head around it, and another 5 to figure what to do the search and replace against, that was surprisingly easy. Thank you so much! Now, of course to actually test it, but considering it found and replaced 100 exactly, I think I got it.

Devon

[DevonB]

lfm - have you figured out a way to walk up numbers without having to manually do each one? Example - I have a bank that takes 1-128. I have patches that are 1-64. I want to place them in 65-128. If this was in an excel spreadsheet, it be a piece of cake! lol. You ever done that?

[lfm]

lfm - have you figured out a way to walk up numbers without having to manually do each one? Example - I have a bank that takes 1-128. I have patches that are 1-64. I want to place them in 65-128. If this was in an excel spreadsheet, it be a piece of cake! lol. You ever done that?

I have done similar things in Hammond sysex, replacing tonewheels etc. I moved tonewheels from XK3 series to XK1 etc - all in HxD editor.

When you identified the offsets in the files you can swap patches as you please shoving them around. They are usually logically built like an array - as you say cells in a row or column in Excel. But look out for 8-bit principles for the brand you are manipulating(more below on that). If you can read all bytes in each name of a patch correctly you are probably ok.

I clearly saw how name of patches occured, so looking at a dump specification you see if those name are first or where in each patch. Often they are the very first of a patch.

When dumping single patches you can also see how large each patch is in bytes. It's usually there in specs. And looking inside such a single patch dump - you can identify where data starts - and then compare where that is in the full sysex dump - is the name first or other bytes first etc.

If you clearly see these signs, beginning and end of each patch, you can go for it.

But doing stuff for Hammond, Korg and Roland I looked up specs how they are built to be sure. And some details follow the NRPN addresses within the patches to manipulate parts of a patch.

Also good to know if to go deeper into a patch is to know the 8-bit principles that each brand is using - since midi itself only is 7-bit. Korg have their way, Roland their - and Hammond packed sysex dumps like a protocol I had to make some software figuring that out.

So tread lightly if you go deeper.

But feel moving full patches are rather safe to do. But when you have 8-bit principles that 7-bytes are actually 7-bit only each, and possible 8th bit set is in the 8th byte - one 8th bit for each of those 7 - it can be tricky. So 7 consecutive bytes are stored as 7+1=8 bytes.

[DevonB]

I don't think I did a good job describing what I was trying to attempt, my bad. Let me try again.

So I know in this sysex dump where the patch # is within the structure; it's right after the bank number. So I search for a sufficiently long enough string to get a solid match and do a bulk replace to change the bank number. That works perfectly because I'm simply replacing one value with another value for all the bytes. However, what I'm asking is for the program number of the patch. I want to move Program 1 to Program 51. I want to move Program 2 to Program 52. I can manually do this obviously, but it's tedious. Like in Excel, I could easily do =A1+50, =B1+50 and just drag the cells and it'll do the heavy lift for you. Just wondering without coming up with something in a programming language like Perl or some such, is there an easy way to find and replace, but also add a value to those find and replaces. So like find 01, calculate 1+50, and replace the 2 bytes with 51. Find 2, calculate 2+50, and replace the 2 bytes with 52, etc.

Hopefully that's more clear.

Devon

[lfm]

I described the full process what you have to consider for this kind of work, probably in too many words as usuall. And doing all banks dump I assume this is then 4 x 100 programs.

Most probably there is no patch no, because the patch no is given to it's position in an array - one for each patch. But different brand may go about it differently and addressing each program arbitrary order within the total dump by bank+program no - and I stand corrected. I never saw that though, but only done Korg, Roland and Hammond.

But patch names, if such a synth that have a display with names, is really good guide where one patch is definately crossed.

If it's all a blur in what you see - no name, no structure to go on - best option is to make a patch dump - where you can disect the bytes in one patch, and thereby find the position within the total dump.

And in this way identify the position of what you want to swap.

And think manual work one by one is the way - unless you know the offsets to every program.
To hope for that only bank+program no is unique is not realistic, IMO.

But if what you said is true - that you can get solid match, that more bytes around bank+program really is unique and give one hit, and those extra bytes really are the same for every program - then you might be able to see if offset to each hit is the same for every hit. But we are talking about programming then.

If those identifiers you found that is in every program, you can search and see to that you get 4 x 100 hits for that - then it could be used. If not getting the same as number of programs in there - then no go.

Then I would go about it with finding one offset that is in the first program - and see if we call that ID+OFFSET to bank+program no and figure out offset to next program is SIZE bytes - then you could make a program in perl or something, that move SIZE bytes for each, finding each one in a loop.

You might be able to automate this in a good sysex manager - that request each program no and then send to another destination.

[DevonB]

Actually, it's quite clear where the patches are. I can find the bank number and patch number very easily on the sysex. Just didn't know if there was some 'trick' within the hex editor that I might have missed that made it easy to add a value to to a byte quickly and easily without resorting to a programming language to process the file. Like doing a find and replace, but you can manipulate values with the replace command with an operator. If you look at the top of the message, that format I clearly see in the hex editor. Oh well, manually it is then if I need to. Thanks for all the help man!

Devon

[pljones]

I wrote my own editor for my TrapKAT sysex dumps. (Sadly I need to rewrite it as I can't get it to compile any more... OK, maybe not sadly, as it means I get to write more code and pick another new language to learn. But sadly, I've other things that come first...)

[lfm]

Actually, it's quite clear where the patches are. I can find the bank number and patch number very easily on the sysex. Just didn't know if there was some 'trick' within the hex editor that I might have missed that made it easy to add a value to to a byte quickly and easily without resorting to a programming language to process the file. Like doing a find and replace, but you can manipulate values with the replace command with an operator. If you look at the top of the message, that format I clearly see in the hex editor. Oh well, manually it is then if I need to. Thanks for all the help man!

Devon

No problem, you're welcome.

So Novation do stack a series of sysex messages in full all the way through, I did not get that. That makes it a bit easier finding header of sysex for each.

It would be normal search feature to jump to next, close enough, and manually edit that proper location. Usually this is F3 function key to repeat search.