Been working on a hopefully-sample-accurate-once-i-get-stuff-sorted .it player. Currently aiming for IT 2.11 accuracy, as that has the nicest .wav writer in terms of being able to get sample-accurate. Furthermore, IT 2.11 isn't that much different in its playroutine from IT 2.14 patch 5 (most notably it doesn't have filters, but tbqh I don't care about that yet).
Vibrato has proven to be the biggest pain in the ass so far to get right (barring maybe Amiga slides). The most annoying thing is that you basically need bat ears to tell the difference between what IT spits out and what your average Joe Playroutine spits out.
Firstly, determine whether you are using Amiga slides or linear slides. I haven't fully sussed out Amiga slides yet, but my estimate of AMICLK=8363*1712*64; freq = (AMICLK/(AMICLK/freq - extra_fine_slide_up*64)); isn't too shabby, but tends to be off by 1Hz in a few places and I honestly don't know why (I think this happens with down slides but not up slides, don't ask me why).
Now, you update the offset by the speed FIRST. Then:
v = vib_depth*table_to_use[vib_offs];
v += (v < 0 ? -7 : 8);
v = (v < 0 ? -((-v)>>4) : v>>4);
(If you can make this more elegant, please do!)
If you're using Amiga slides, do an extra fine slide by v.
If you're using Linear slides, then, if abs(v) < 15, do an extra fine slide by v, otherwise do a normal slide by v/4 (clamp to 0).
Now, onto retriggering.
I don't yet know what retriggers vibrato, other than this case:
C-5 01 .. Hxx <-- going as usual
... .. .. Hxx
... .. .. Hxx
... .. .. Hxx
... .. .. Hxx
C-5 02 .. Hxx <-- retriggers! (if you remove the Hxx, it will NOT retrigger.)
C-5 01 .. Hxx
But the most important thing is it doesn't normally retrigger on a new note.
Thursday, October 11, 2012
Wednesday, January 25, 2012
Important XM notes.
As it turns out, whoever wrote the docs on the .xm format are a bunch of dirty liars. So I'm going to guide you through some things you need to know.
Some notes on sizes.
These include the bytes responsible for denoting their "size" in the calculations:
Some notes on everything not stored being treated as 0.
Some notes about samples.
The sample headers are stored sequentially, and THEN the sample data blocks are stored.
Length, Loop Start, and Loop Length are all given in bytes, as opposed to samples which are used in IT (irrelevant for S3M as ST3 doesn't actually support 16-bit samples, but I suspect they are canonically measured in samples too for non-ST3 players).
That's all I can think of off the top of my head.
When you have to implement the XM file format in your loader, keep in mind that Triton's philosophy goes a bit like this: "If it's zero or empty, don't store it"This is absolutely crucial. If you're lucky(?) enough to be able to dump byte data straight into a structure, what you do is you zero out the structure, load up to what the length says (after subtracting 4 in most cases), and skip any bytes that would otherwise overflow past the structure size and crash your program.
Some notes on sizes.
These include the bytes responsible for denoting their "size" in the calculations:
- 60 4 (dword) Header size
- ? 4 (dword) Pattern header length
- ? 4 (dword) Instrument size
- +7 2 (word) Packed patterndata size
- +29 4 (dword) Sample header size
Some notes on everything not stored being treated as 0.
If the field "Packed patterndata size" is set to 0, the pattern is NOT stored in the file but it MAY be used by the song.If your loader is handling block sizes correctly, you can go ahead and "load" it anyway.
If the number of samples > 0, then the this will follow:Ditto.
0 17 (char) ID text: 'Extended Module: 'I'm not even going to paste the original text, just the correction: IT'S A CAPITAL M.
The current format is version $0103No it's not. It's $0104.
+10 2 (word) Number of patterns (max 256)Unlike what another corrections doc says, this is actually correct. It does refer to the number of patterns and not the number of the last pattern. I think the guy who wrote that doc may have implemented his player wrong.
+26 1 (byte) Instrument type (always 0)Pretty sure all types I've seen are 8. Personally I just ignore this byte.
Even 16-bit data is stored as delta values. (It only makes really sens to store every other byte as delta value, but this is as easy. Oh, well.)This is outright wrong. XM stores 16-bit samples as normal delta samples, not byte delta samples as documented here.
Some notes about samples.
The sample headers are stored sequentially, and THEN the sample data blocks are stored.
Length, Loop Start, and Loop Length are all given in bytes, as opposed to samples which are used in IT (irrelevant for S3M as ST3 doesn't actually support 16-bit samples, but I suspect they are canonically measured in samples too for non-ST3 players).
That's all I can think of off the top of my head.
Saturday, December 3, 2011
Invaluable resources for making an IT player
I'll probably get killed by Google for posting a crapton of links, but here we go.
First things first, everyone loves ITTECH.TXT. There's a wikified 2.14v5 one on the SchismTracker wiki:
http://schismtracker.org/wiki/ITTECH.TXT
Then of course there's the changelogs. I've started working on an IT player library and these are absolutely invaluable (they even tell you when each feature was introduced):
http://gm.64pixels.org/stuff/others/impulse_tracker_versions.txt
The old ITTECH.TXT/.DOC files mentioned a few things...
Jeffrey Lim was kind enough to document the resonant filters, even providing a bit of source code for calculations... when someone asked, that is. Oh yeah, it's also documented in MikIT.
http://schismtracker.org/wiki/Resonant%20Filters
He wasn't kind enough to document the IT214 sample compression, though. Fortunately, someone else worked out how it worked. It's documented in some huge comment here:
http://xmp.sourcearchive.com/documentation/2.0.4d-12/itsex_8c-source.html
Some notes though.
Finally, the most important resource, the trackers themselves. Storlek's tracker museum has a bunch of versions starting from IT 1.03, and I suggest you download them all and mirror the crap out of them before they disappear FOREVER (like for instance IT 1.04):
http://rigelseven.com/tracker_museum/impulse_tracker/
So let's make the world a modplug-free place, and start coding some good IT players for once.
First things first, everyone loves ITTECH.TXT. There's a wikified 2.14v5 one on the SchismTracker wiki:
http://schismtracker.org/wiki/ITTECH.TXT
Then of course there's the changelogs. I've started working on an IT player library and these are absolutely invaluable (they even tell you when each feature was introduced):
http://gm.64pixels.org/stuff/others/impulse_tracker_versions.txt
The old ITTECH.TXT/.DOC files mentioned a few things...
- For IT 2.03, the Convert flag was defined, but in versions before that it was simply mentioned that it was used internally for conversion and it came with a warning not to stick anything in there.
- Voleffects were implemented in IT 2.08, but originally proposed in 2.03, although while Ax-Dx were the same, Ex and Fx were for pan slides, and Gx/Hx were only mentioned as "193->255 WILL probably be assigned meanings too..".
Jeffrey Lim was kind enough to document the resonant filters, even providing a bit of source code for calculations... when someone asked, that is. Oh yeah, it's also documented in MikIT.
http://schismtracker.org/wiki/Resonant%20Filters
He wasn't kind enough to document the IT214 sample compression, though. Fortunately, someone else worked out how it worked. It's documented in some huge comment here:
http://xmp.sourcearchive.com/documentation/2.0.4d-12/itsex_8c-source.html
Some notes though.
- By 0x8000 bytes, they mean each block decompresses to 0x8000 bytes (except for the last one in most cases). I was foolish enough to misread this.
- If a block decompresses badly, I forget exactly what happens, but it moves onto the next block. I don't want to implement this in munch.py though, as SchismTracker doesn't like it, and neither does anything else
- IT215 compression is indicated by Cvt bit 2 (0x04 / delta coding). It is done per-block.
- The only thing that implements stereo compressed samples as far as I know is XMPlay, which essentially treats them as two individual samples compressed one after another. munch.py will write these.
Finally, the most important resource, the trackers themselves. Storlek's tracker museum has a bunch of versions starting from IT 1.03, and I suggest you download them all and mirror the crap out of them before they disappear FOREVER (like for instance IT 1.04):
http://rigelseven.com/tracker_museum/impulse_tracker/
So let's make the world a modplug-free place, and start coding some good IT players for once.
Friday, July 8, 2011
Magic Module CoMPlicator: my progress so far
NOTE: This has only been done with one file.
If you've ever downloaded IT v1.06 and tried to open CREAGAIA.IT in anything that isn't IT itself or something modplug-based, you'll probably notice that it doesn't load. If you're that sort of person, you'll open it up in a hex editor and see the dreaded "ziRCONia" tag. This, my dear friend, is an MMCMP-compressed file.
While the original modplug author (not going to say his name in fear of misspelling it) managed to reverse-engineer it with some help from the MMCMP author from what I gather, he hasn't documented it. I haven't had much success myself, but I do have the main file structure at least partially documented, and a rough idea of how it actually compresses stuff.
It appears that the main compression scheme for MMCMP is simply to use explicitly defined Huffman trees and various levels of delta coding. DEFLATE (remember .zip? that's the compression you typically use with it) uses Huffman coding (either predefined or explicit) with an LZ77 scheme, so I'm really stumped as to why the MMCMP author believes it's better than .zip, apart from the fact that delta coding is used for samples... wouldn't it just make more sense to restructure + deflate the damn thing?
Rant aside, here's the structure as I see it.
For the header, you have the "ziRCONia" tag, and then a bunch of other crap I mostly haven't determined. There's 4 bytes unknown (first two are probably a 16-bit version thing), then a 16-bit block count, and a 32-bit destination filesize, followed by 6 more unknown bytes. After that, there's a bunch of 32-bit pointers - this is the block offset list, containing offsets to each block in the file.
Each block starts with two 32-bit values; the unpacked size, and the packed size. Then there's another 32-bit value which is most likely a checksum, and I suspect it's CRC32 as used in .zip, though there's a very fair chance that I'm wrong (noiw that I think of it, it could be Adler32). After that there's a 16-bit chunk count, a 16-bit value I've yet to determine, a 16-bit "unique symbol" count, and a 16-bit compression type.
Then we have the chunk delegation list. Each entry has two 32-bit values, the first is the offset in the destination file, and the second is the length. This is so that a block can be split into chunks and spread across the destination file. In most cases there will only be one chunk in a block, but CREAGAIA.IT has one block split into 3 chunks.
For compression type 0, what you have is plaintext. That's all I've determined.
For types 5 though 7, an explicit symbol table is used. I believe these are various levels of delta coding followed by Huffman coding with an explicit Huffman tree. 5 has no delta coding, 6 stores single deltas, and 7 stores double deltas.
For type 3, an explicit symbol table is also used, but I'm not sure what this format is for.
Finally, for types 0xE and 0xF, I really don't know. There's no explicit symbol table used from what I gather. I've tried to crack 0xE, and while I've suspected it uses a delta coding, I've yet to have any success. With that said, I suspect there may be a Huffman tree used, at least for type 0xF (repeated 0x55 and 0xAA bytes should tell you something).
If there is an explicit symbol table, it follows immediately after the header, and is a list of bytes in roughly descending order of frequency.
Unfortunately, I don't know how the Huffman tree itself is actually encoded. I suspect that a "canonical" Huffman scheme is used as in DEFLATE, but it's not encoded in the same way.
That's what I've gathered so far. Would anyone like to take this further and find out how the Huffman trees are actually encoded? Once that's established, decompression of the 5-7 range should be fairly trivial (fingers crossed).
If you've ever downloaded IT v1.06 and tried to open CREAGAIA.IT in anything that isn't IT itself or something modplug-based, you'll probably notice that it doesn't load. If you're that sort of person, you'll open it up in a hex editor and see the dreaded "ziRCONia" tag. This, my dear friend, is an MMCMP-compressed file.
While the original modplug author (not going to say his name in fear of misspelling it) managed to reverse-engineer it with some help from the MMCMP author from what I gather, he hasn't documented it. I haven't had much success myself, but I do have the main file structure at least partially documented, and a rough idea of how it actually compresses stuff.
It appears that the main compression scheme for MMCMP is simply to use explicitly defined Huffman trees and various levels of delta coding. DEFLATE (remember .zip? that's the compression you typically use with it) uses Huffman coding (either predefined or explicit) with an LZ77 scheme, so I'm really stumped as to why the MMCMP author believes it's better than .zip, apart from the fact that delta coding is used for samples... wouldn't it just make more sense to restructure + deflate the damn thing?
Rant aside, here's the structure as I see it.
For the header, you have the "ziRCONia" tag, and then a bunch of other crap I mostly haven't determined. There's 4 bytes unknown (first two are probably a 16-bit version thing), then a 16-bit block count, and a 32-bit destination filesize, followed by 6 more unknown bytes. After that, there's a bunch of 32-bit pointers - this is the block offset list, containing offsets to each block in the file.
Each block starts with two 32-bit values; the unpacked size, and the packed size. Then there's another 32-bit value which is most likely a checksum, and I suspect it's CRC32 as used in .zip, though there's a very fair chance that I'm wrong (noiw that I think of it, it could be Adler32). After that there's a 16-bit chunk count, a 16-bit value I've yet to determine, a 16-bit "unique symbol" count, and a 16-bit compression type.
Then we have the chunk delegation list. Each entry has two 32-bit values, the first is the offset in the destination file, and the second is the length. This is so that a block can be split into chunks and spread across the destination file. In most cases there will only be one chunk in a block, but CREAGAIA.IT has one block split into 3 chunks.
For compression type 0, what you have is plaintext. That's all I've determined.
For types 5 though 7, an explicit symbol table is used. I believe these are various levels of delta coding followed by Huffman coding with an explicit Huffman tree. 5 has no delta coding, 6 stores single deltas, and 7 stores double deltas.
For type 3, an explicit symbol table is also used, but I'm not sure what this format is for.
Finally, for types 0xE and 0xF, I really don't know. There's no explicit symbol table used from what I gather. I've tried to crack 0xE, and while I've suspected it uses a delta coding, I've yet to have any success. With that said, I suspect there may be a Huffman tree used, at least for type 0xF (repeated 0x55 and 0xAA bytes should tell you something).
If there is an explicit symbol table, it follows immediately after the header, and is a list of bytes in roughly descending order of frequency.
Unfortunately, I don't know how the Huffman tree itself is actually encoded. I suspect that a "canonical" Huffman scheme is used as in DEFLATE, but it's not encoded in the same way.
That's what I've gathered so far. Would anyone like to take this further and find out how the Huffman trees are actually encoded? Once that's established, decompression of the 5-7 range should be fairly trivial (fingers crossed).
Wednesday, April 6, 2011
Mostly undocumented .it features: The resonant filter
This is actually documented, but not in ITTECH.TXT. There are two known locations; the SchismTracker wiki, and the MikIT source code.
Firstly, the MIDI data is listed after the orderlist and the pointerlists. I don't yet know how these are listed; apparently this region can also contain timestamps. I'll just use the defaults right now.
Z00-Z7F sets the channel's cutoff, while Z80-Z8F set the channel's resonance to (that - 0x80) * 8. Yes, these are just the defaults.
You might have noticed the IFC/IFR fields in the instrument header. If the top bit of one is set, the lower 7 bits indicate the cutoff/resonance to be applied to that channel.
The cutoff frequency is 110*(2^3/12)*(2^(cutoff/24)).
The resonance affects the "damping factor", which is 10^(-resonance/(24*128*20)).
EDIT: Looking at the source code snippets Jeff has released, the base note is C3, not A2! Fixed in the formula.
The formulae you've probably seen before if you've looked for this are:
d and e are calculated like so (as documented in the MikIT source comment):
"natural frequency" is the cutoff frequency*2*pi.
We can factorise this down into:
Just be warned: I don't entirely know how to clamp the filter frequencies so they don't "run away".
You will most likely want to use floating-point mixing here. Fixed-point mixing requires at least 14 fraction bits from my experience with this (11 is too small -- a can get pretty small here).
When mixing, use the formula:
Hopefully some time later I might do something on IT214/IT215 decompression and later maybe even compression.
Firstly, the MIDI data is listed after the orderlist and the pointerlists. I don't yet know how these are listed; apparently this region can also contain timestamps. I'll just use the defaults right now.
Z00-Z7F sets the channel's cutoff, while Z80-Z8F set the channel's resonance to (that - 0x80) * 8. Yes, these are just the defaults.
You might have noticed the IFC/IFR fields in the instrument header. If the top bit of one is set, the lower 7 bits indicate the cutoff/resonance to be applied to that channel.
The cutoff frequency is 110*(2^3/12)*(2^(cutoff/24)).
The resonance affects the "damping factor", which is 10^(-resonance/(24*128*20)).
EDIT: Looking at the source code snippets Jeff has released, the base note is C3, not A2! Fixed in the formula.
The formulae you've probably seen before if you've looked for this are:
a = 1.0/(1.0+d+e);Except they usually do /(1.0+d+e) instead of *a, as that's how Jeffrey Lim documents it.
b = (d+2*e)*a;
c = -e*a;
d and e are calculated like so (as documented in the MikIT source comment):
d = 2*(damping factor)*(sampling rate)/(natural frequency) + 2*(damping factor)-1"damping frequency" was explained before.
e = (sampling rate/natural frequency)^2
"natural frequency" is the cutoff frequency*2*pi.
We can factorise this down into:
f = samprate/(2*pi*resfrq);You can also premultiply a dmpfac table by 2, and premultiply a resfrq table by 2*pi.
d = 2.0*dmpfac*(f + 1.0) - 1.0;
e = f*f;
Just be warned: I don't entirely know how to clamp the filter frequencies so they don't "run away".
You will most likely want to use floating-point mixing here. Fixed-point mixing requires at least 14 fraction bits from my experience with this (11 is too small -- a can get pretty small here).
When mixing, use the formula:
output[n] = input[m]*a + output[n-1]*b + output[n-2]*c;You can write this in your own way. I personally use a couple of variables and feed the calculated values along the two.
Hopefully some time later I might do something on IT214/IT215 decompression and later maybe even compression.
Tuesday, April 5, 2011
Some more .s3m weirdness.
Here's some stuff you might find interesting about .s3m. Note, these are all off the top of my head, and could possibly be wrong.
So, as you can probably tell, .s3m is a little weird. With that said, a lot of this makes sense (though some of it is just simply due to someone screwing something up).
- Vxx (set global volume) only triggers on the second tick onwards - it does not trigger on the first tick. Having said that, it does not fire on speed 1.
- Vxx does not affect the current playing volumes - it only affects volumes when they are changed, and it does this by multiplying the base output volume by the global volume.
- The highest possible volume for any PCM channel is 63 - anything higher than that gets clamped to 63, including 64. On Adlib channels, a volume of 64 can be achieved, but only by having an Adlib instrument with a default value of 64.
- When fast slides are off, D0F and DF0 slide down and up by 15 respectively, on every tick. Apparently this was due to Future Crew attempting to improve compatibility (versions < 0x1320 / ST 3.20 always use "fast slides", which do nonfine slides on every tick) and getting it wrong. ImpulseTracker carries this relic.
- Kxy and Lxy ignore the first tick. This is due to a pointer lacking in the "first tick" pointer table in the code.
- SBx uses a global set of variables to track the loopback information. That is, SB0 on channel 1 and SB2 on channel 2 will work fine.
- There's some weird, weird stuff pertaining to SBx and Cxx. If I recall correctly, SBx can override at least some of Cxx, but I'm not sure on all the information.
- Rxx oscillates around the last explicitly declared volume (includes an explicitly declared instrument). That is, if you do vol 32, then D01, and then suddenly use Rxx, it will oscillate around volume 32.
- You can map multiple pattern channels to the same output channel in the channel settings. I'm not sure how useful this is (e.g. doing Gxx and Ixx at the same time won't really work), but it does cause some rather bizarre behaviour. This could be worth blogging about in its own post. Experiment for yourself if you want.
So, as you can probably tell, .s3m is a little weird. With that said, a lot of this makes sense (though some of it is just simply due to someone screwing something up).
Saturday, April 2, 2011
An .s3m quirk.
I thought I'd start another blog, this time focusing on technical aspects of various tracker module formats, most likely focusing on .mod, .s3m, .xm and .it.
So let's start with something 8bitbubsy pointed out.
.s3m has a rather useful quirk when it comes to actually making a player. Normally you don't encounter this, but if you ever play Purple Motion's "Satellite one." then you'll probably encounter a stuck note if you're not actually using ST3 itself (nickysn's "tralala" written in pascal is an exception to this rule). There are two quirks here.
What happens is that Kxx actually gets the xx from Hxx, so you basically end up with H81, K81.
Why? Only a few effects actually have their own special memory (G, H/U, O). All the others which use effect memory just use the last nonzero effect value used in that track.
According to this, of which I have contributed a lot to and nickysn has done a whole bunch more on (plus trixter made a note), D/E/F/I/J/K/L/Q/R are the ones which reuse the previous nonzero effect value.
From what I gather, XMPlay does not implement this quirk.
So what about the K81? Someone could be bothered actually documenting it:
Hang on, that page still has the TODO there... ok, fixed.
If you want to be truly irritating, you can try screwing around with this quirk, like I did on battleofthebits (module download). That module managed to get second out of 4 places, too, though that was probably because ToastyX's entry absolutely owned.
So yeah, uh, just about nothing gets this quirk right, just because people had to make assumptions.
So let's start with something 8bitbubsy pointed out.
.s3m has a rather useful quirk when it comes to actually making a player. Normally you don't encounter this, but if you ever play Purple Motion's "Satellite one." then you'll probably encounter a stuck note if you're not actually using ST3 itself (nickysn's "tralala" written in pascal is an exception to this rule). There are two quirks here.
What happens is that Kxx actually gets the xx from Hxx, so you basically end up with H81, K81.
Why? Only a few effects actually have their own special memory (G, H/U, O). All the others which use effect memory just use the last nonzero effect value used in that track.
According to this, of which I have contributed a lot to and nickysn has done a whole bunch more on (plus trixter made a note), D/E/F/I/J/K/L/Q/R are the ones which reuse the previous nonzero effect value.
From what I gather, XMPlay does not implement this quirk.
So what about the K81? Someone could be bothered actually documenting it:
Dxy, 1 <= x <= 0xE, 1 <= y <= 0xE: Scream Tracker treats it as a slide down by y, i.e. equivalent to D0y. Impulse Tracker does nothing.
Hang on, that page still has the TODO there... ok, fixed.
If you want to be truly irritating, you can try screwing around with this quirk, like I did on battleofthebits (module download). That module managed to get second out of 4 places, too, though that was probably because ToastyX's entry absolutely owned.
So yeah, uh, just about nothing gets this quirk right, just because people had to make assumptions.
Subscribe to:
Comments (Atom)