... it was bolded in the editor but not in the final message. These are the items that were to be bolded:
BC: 00 73 (115) TopAddr: [40,00,00,00]
[PART 01] AN-X PART COMMON
...
BC: 00 57 ( 87) TopAddr: [31,00,00,00]
[PART 02] FM-X PART COMMON
Current Yamaha Synthesizers: Montage Classic 7, Motif XF6, S90XS, MO6, EX5R
There is no need to check addresses 3 or 4 at all. As the table on p.217 shows:
1. If address 1 is '4p' it is an AN-X part record.
2. If address 1 is '3p' it is an FM-X part record.
You only need to check address 2 to distinguish between AWM2 and Drum records:
1. If a record with address 1 = '2p' but address 2 is NOT '10' it is an AWM2 part
2. If a record with address 1 ='2p' has address 2 = '10' it is a drum key record and the part is a drum part - there will be 73 of these records.
One CAVEAT with this, and all SysEx related things, is that an update can change things. Drum parts only have '10' for address 2 but the upcoming OS 3.0 update (or other future updates) could add additional records that might use a different address 2 value.
It's not clear to us why drum parts have the same address 1 value as awm2 parts or why the part type doesn't have it's own parameter in the part common record. But as Walter Cronkite is famous for saying "And that's the way it is".
I do this to further qualify so I only have one key that matches. It's a personal choice not to print out FM-X or AN-X or Normal every time there's a matching block that only belongs to one of the 4 Part types. I just wanted to flag it once per Part.
To follow that logic - I'd probably do well to even further qualify the Drum Part as you've mentioned. I didn't have a Perf with Drum Part(s) so I didn't run into that situation.
Drum Parts have the same record because they're sample-based AWM2 drums. "Normal" and "Drum" Parts are AWM2 and so addr1 means (I'm inferring) "AWM2".
Current Yamaha Synthesizers: Montage Classic 7, Motif XF6, S90XS, MO6, EX5R
I ran through all of the Montage M Sysex files and was able to find Drum parts in there. Modified the code to only spit out one identifier per Part and removed all of the other extra top address outputs.
:::::::::::::: MMONTAGEM-00112604-Greg_s_Vibe.syx :::::::::::::: BC: 00 30 ( 48) TopAddr: [20,00,00,00] [PART 01] NORMAL (AWM2) PART ELEMENT BC: 00 45 ( 69) TopAddr: [21,10,00,00] [PART 02] DRUM (AWM2) PART KEY BC: 00 73 (115) TopAddr: [42,00,00,00] [PART 03] AN-X PART COMMON BC: 00 73 (115) TopAddr: [43,00,00,00] [PART 04] AN-X PART COMMON
Modified (python) example code also correcting addr1 + 1 bug from before order of operations problem (find 0-based using mask then add 1 - not the other way around - fixes "[PART 16]" cases):
# Adding addr3==0, Element #1 only if ((addr1&0xf0==0x20) and (addr2==0x00) and (addr3==0x00) and (addr4==0x00)): print ("[PART %02d] NORMAL (AWM2) PART ELEMENT" % ((addr1 & 0x0f)+1)) # Adding addr3==0, Drum key #1 only if ((addr1&0xf0==0x20) and (addr2==0x10) and (addr3==0x00) and (addr4==0x00)): print ("[PART %02d] DRUM (AWM2) PART KEY" % ((addr1 & 0x0f)+1)) if ((addr1&0xf0==0x30) and (addr2==0x00) and (addr3==0x00) and (addr4==0x00)): print ("[PART %02d] FM-X PART COMMON" % ((addr1 & 0x0f)+1)) if ((addr1&0xf0==0x40) and (addr2==0x00) and (addr3==0x00) and (addr4==0x00)): print ("[PART %02d] AN-X PART COMMON" % ((addr1 & 0x0f)+1))
Current Yamaha Synthesizers: Montage Classic 7, Motif XF6, S90XS, MO6, EX5R
Thanks Toby and Jason for your insights so far. I have now got successful code that allows me to identify the part type for a specific part, although the code takes a few hundred ms to run when I request info for a specific part, which my application does on a relatively regular basis.
I'm wondering if a better approach would be to fetch all the part types at initialization time, rather than sending individual bulk requests for specific parts, but I'm struggling to get my head around the documentation...
I'm assuming that I want to use the Performance Bulk Header, which is noted as [04 II mm nn] but I can't seem to see where II, mm and nn are defined. Would I be right in thinking that I have to specify the address of the performance using these bytes somehow, rather than being able to query the currently-loaded performance?
'better approach'? Post info about what you are REALLY wanting to do?
Latency is unavoidable for bulk operations that fetch large amounts of data.
Performances with large numbers of parts have large amounts of data.
A performance that contains Smart Morph Data has large amounts of data - 3000 bytes each for the PNG data and map data
Bulk operations don't use MSB/LSB like PC operations do.
You need to know what bank a performance lives in to get to it: preset, library (which one?), user (which one?)
There is no easy, dynamic, way to know just where any given performance lives.
I'm assuming that I want to use the Performance Bulk Header, which is noted as [04 II mm nn] but I can't seem to see where II, mm and nn are defined. Would I be right in thinking that I have to specify the address of the performance using these bytes somehow, rather than being able to query the currently-loaded performance?
Yes - see the table on p.221 of the Data List doc. If you scan that table you will see things in the description column like:
1. Performance PRE 1 (nn = 0-127)
. . .
40. Performance PRE 40 (nn = 0-127)
41. Performance USER 1 (nn = 0-127)
'f0 43 20 7f 1c 0d 04 00 19 00 f7' -> note the '19' hex for address 3
The 'PRE' means 'preset' and 'PRE 19' means the 25th set of 128 performances. Multiply 25 * 128 = 3200
So that command returns performance #3201 'Ambient Fluid'. Use address 4 to get any of the 128 performances in block 19 hex (25 decimal).
Now for the HARD part - user and Library performances.
Did you notice that 'Performance PRE 40' entry in the table? But 40 * 128 = 5120 but there are currently only 3427 performances in OS 2.10.1 and only 60 or so coming in the OS 3.0 update due next month. So you can only use the entries that map up to performance 3427.
The p.221 table shows only 5 'Performance USER n' entries but there are 8 user banks available for live sets. Could just be that the doc hasn't yet added the other 3 but we haven't chased that down by trying to access beyond the 5.
Notice there are 80 'Performance LIBRARY nn' entries which would have 128 for each totalling 10240. The current OS supports 16 library files but OS 3.0 is expected to allow 32. We haven't tested the limits for the number of user performances but docs seem to say 640. That would be 5 sets of 128 so 16 libraries computes to 5 * 16 or 80 library entries in the table.
So it it likely that the first 5 library entries are 'Library 1' and so on but the performances you get will depend on which library you have in slot 1.
There's no real better way to do this that I am aware of.
The way all of this is read from the keyboard is through very inefficient MIDI-over-USB serial data that is inherently slow. I have asked for Yamaha to use USB data transfers instead of MIDI-over-USB so that you would be able to get much more data over the wire at near theoretical USB limits. I wanted this also not only for the configuration data (like SysEx registers) but also for the sample data so the USB-to-HOST connection could be used also for transfer of libraries, user files, and backup files instead of going through the USB port using a flash drive.
This hasn't seemed to be made available to users. So I'm not sure there's a better way. You'll need to rely on the slow MIDI-over-USB interface. And ... it's clumsy.
From the data list:
For information about “ll“,“mm” and “nn” shown in the following list, refer to the MIDI PARAMETER CHANGE TABLE (BULK CONTROL).
So using the table you see the 1st column is 04, 2nd column is "ll", 3rd column is "mm", 4th column is "nn".
Any preset Performance will have ll=0
Any user Performance will have ll=1
... and so on.
"mm" is equivalent to a page number. Every page has 128 entries (nn=0-127).
Preset Performance #1 would be ll=0, mm=0, nn=0
Preset Performance #2315 would be ll=0, mm=(2315-1)/128, nn=(2315-1)%128 (modulus, or remainder of the division by 128)
Performances in the datalist start with 1 but the underlying structures start with 0 which is why I subtract by one in the above. That is to zero-base the Performance number.
The user section has 5 pages so there's a total of 5*128 = 640 User Performances available at one time. Which should agree with the documentation somewhere -- and is the same limit as the 1st generation Montage Classic.
Current Yamaha Synthesizers: Montage Classic 7, Motif XF6, S90XS, MO6, EX5R
Thanks again, both. I'm looking forward to showing what I've been from working on.
We suggest you create a simple, offline dataset (e.g CSV file) to hold the data.
although the code takes a few hundred ms to run when I request info for a specific part, which my application does on a relatively regular basis.
Consider rethinking that 'relatively regular basis' approach. Unless you are exceptionally prolific performances don't change on a regular basis. The presets don't change at all except for new ones an OS update might provide. Although it is certainly possible for Yamaha to alter the existing presets when they provide an update we don't know of any instance. They could certainly update an existing waveform/samples - don't know if they do that.
I'm wondering if a better approach would be to fetch all the part types at initialization time, rather than sending individual bulk requests for specific parts, but I'm struggling to get my head around the documentation...
Without knowing your current, and potential long-term needs, it's hard to know what might be best for you but you might:
1. Bulk dump ALL of the performances and save them - you can do this offline when latency won't matter.
2. Use an app to access any desired info from the offline files from step #1.
3. For things like 'Part Type', and other static info save that info offline also.
4. Even a simple app can easily access a bunch of CSV files that have various info in them. Or you can go for a longer term approach (like we have) and create a local hierarchical database that gives access to any of the nested info dynamically.
In the past Yamaha actually provided data such as that in the data list in XML format but they seem to not do that anymore. See this SandSoftwareSound article - remove the leading hyphens (added to prevent the link from being removed)
-h-t-t-p-s-:-/-/-s-a-n-d-s-o-f-t-w-a-r-e-s-o-u-n-d-.-n-e-t-/genos-voice-editing-xml-notepad/
In my previous post about Yamaha Genos™ voice editing, I introduced the voice editing features provided by Yamaha Expansion Manager (YEM). This post describes a way to work around the shortcomings in YEM.
YEM stores low-level voice programming information in XML files with the “UVF” file name extension. In case you’re not familiar with XML, it’s a mark-up language that captures document formating and structure.
The files were hierarchical and had 'official' tag names for each level of the hierarchy. Third parties could write their own HTML or stylesheet software to extract/process any desired part of the document.
It would be rather easily, though timeconsuming grunt work, for Yamaha to provide an 'export' facility, at least for ESP to make the data more easily available for use.
... just to be clear, I'm not a part of the "we" in "We suggest you create a simple, offline dataset (e.g CSV file) to hold the data.". For whatever reason, Toby often uses "we" as if the account is part of a collective or group. So that's a self-declaration of a "we" and not including anyone else. In other words, I'm not a part of this self-declared collective.
I don't really care how you cache/shadow or store things or if you store anything at all. I understand the suggestion was essentially to cache (shadow) the data locally to avoid the pain of fetching data directly from the keyboard each time. A reasonable suggestion.
My focus here is to avoid confusion of the unclear pronoun reference "we" in the context of this thread. So ... there you go. Clarified.
Current Yamaha Synthesizers: Montage Classic 7, Motif XF6, S90XS, MO6, EX5R
Thanks both, and @jason I’m accustomed to @toby’s use of the plural first person, no confusion there.
Unfortunately my application can’t use pre-cached performance data, as it simply doesn’t work that way - it’s far more of an execution-time system rather than any sort of librarian that could make use of a large bank of stored data.
All will become clear in due course, but for now thank you both for your help in deciphering the documentation.
Looking forward to hearing more about your specific use case. You are the only one that can decide what trade-offs are best suited for your usage.
We (yes, 'we' are a corporate account and multiple persons are often involved in what we post) are just trying to shed some light on the various options - not trying to decide, and not in a position to decide, which of those options might be better than others.
You should get the least amount of latency when the performance you need info about is in the edit buffer. And assuming you know NOTHING about that performance but need to know what parts are being used and what 'Part Type' they have you could:
1. for each 'possible' active part issue a parameter request for the 'Part Switch' - 1st parm of table '1p 00 01' on p.237
All performances have 16 parts - but many of those parts will have a 'Part Switch' that is OFF
2. If the returned value is 00 that part is NOT active so go to the next of the 16 parts
3. Part is active so issue a parm request for the 'Element Switch' of element 1 - 1st parm of table '2p 00 00 00'
4. If there is no return data the part is NOT an AWM2 part because all AWM2 parts will have records for 1st 8 elements
5. Check for a drum part by requesting a parameter from the '2p 10 kk 00' table - No return data means it isn't a drum part
6. Check for FM-X and/or AN-X the same way request a parm from the '2p ...' or '3p ...' table - no return data means it isn't that type of part.
it’s far more of an execution-time system rather than any sort of librarian that could make use of a large bank of stored data.
Understood - but just for completeness let's look at the possibilities. There WILL BE 'stored data' and it has to be located somewhere. To get it from the instrument you need to request it a) a performance at a time, b) a part at a time or c) a parameter at a time.
If the data isn't in the edit buffer you either need to put it there so you can access it's parts or parameters or you need to get the performance and then parse it for what you need.
You could store all of the performance data locally and then access it from there.
Or you could create one or more 'mini stores' that your app could cache in memory.
There are only 4 part types so you only need 2 bits to store that. There are 16 possible parts so 16 * 2 is 32 bits or 4 bytes for each performance. You will need another 16 bits, one for each possible part, to indicate if a part is ACTIVE in the performance.
So if all you need to store/access is part type the minimum storage would be 6 bytes per performance. Four thousand performances would required 24 kbytes. You could store the data in a HashMap/HashTable/other to quickly access the info.
Small size - the trade-offs are more complicated code that others may not understand easily and code that isn't easily extensible when things change.
Not expressing ANY of the above as a 'solution' - just trying to illuminate the boundaries. Not knowing anything about your specific needs I'll just say that the usual key determinants, which you are probably well aware of, are
1. where is the data located?
2. how often does the data change?
3. how much data is involved?
4. how often are data requests presented?
5. what are the time constraints for responding to a request?
Considerations for those things will likely be the drivers behind approaches to consider.