OSM - the Open Source Music format

<< Back

Introduction (for downloads, see the bottom of this page)

The name Open Source Music was chosen to encourage composers to not only publish their music, but also the raw tracks so that people who are interested can make their own mixes, if they want.

Tracker modules such as .xm, .it and .s3m provide a relatively well documented way to store multi-track music files. There are very few well-documented formats that support multi-track music, and that are at the same time capable of combining pattern-based music with recorded audio.

Problem

There is basically a single drawback in the .xm, .it and .s3m formats: they do not store the data very efficiently. (The IT format does have some sample compression built in, but quite primitive compared to many recent compression algorithms). A multi track file can easily grow to tens of megabytes in size, because of the sheer size of the audio. This currently makes .xm, .it and .s3m rather unsuitable for transporting multi-track files. Even a multi-track mp3 file would be fairly heavy; however if we can combine the benefits of mp3 with the benefits of module-stored music, we might achieve an overall better solution for transporting multi- track files.

Existing solutions

It would be possible to use a compression technology such as OGG or MP3 to compress the audio of these modules. A format that does this already exists: MO3 modules, by Ian Luck (www.un4seen.com). I was pleasantly surprised that the idea was already implemented and even used by one of my favorite mod composers (Skaven from Future Crew). However at the current time, the author does not seem to have intentions of opening up the source code of the format, and as he told me, it is fairly complicated. Time for a better solution ;)

The OSM Format:

OSM should be portable, easy to understand and easy to implement because like many programmers I don't have a lot of time on my hands. Also, I want to do this without getting into much detail about the module formats.

Features

The list of features, shamelessly stolen from Ian's MO3 format description:
MP3/OGG compression
Lossless compression
Structure compression
Sample specific tuning
Text removal
Decompressable
Playable

Implementation strategy

Suppose we rip the samples from a it/xm/s3m module to file and replace them by silence in the module. We then have a silent module that is a very good candidate for traditional compressors. We now compress the samples by some lossy or lossless algorithm, and write a little text file containing info about the original size, position and type of each sample in the silenced module.

When we compress the whole lot into a .tgz, we have a compressed module that would allow (except for the text removal, big deal) all features mentioned above. In addition:

- Easy to support compressong more module formats as long as samples can be accurately pinpointed in the original files. The rest of each module is preserved, so no module specific special features are lost in the compression process, nor do we need to know about them.

- Easy to decompress to rebuild 'original' modules: Possibly a rudimentary decompressor can be built with about 10 lines of shell script and 8 lines of C code.

Someone that has tar or winzip, mp3 decoder, a module editor and half a brain is probably able to reconstruct a module sounding very close to the original without ever needing a dedicated module decompressor.

- It would be possible to enhance the format with post processing info (mixing, equalization) for more serious studio work.

Building blocks

- To pinpoint samples in the modules: I'll leave this task to a very slightly modified DUMB library which supports S3M, IT and XM. Function it_read_sample_header (itread.c) is called with sample offset for each sample that is loaded. Thanks go to Ben Davis for this excellent library, for pointing me where to look and for support and tips.

- libtar for archiving. For overall compression, I can probably choose between libtar, zlib and the bzip2 library, credits and thanks go to their respective authors.

- steal code from DUMB for IT sample decompression (as it was stolen by Ben ;)) Simple delta decompression, I'll write the delta compressor for file restauration myself. Promise.

- use existing mp3/flac/ogg libraries for compression, once again credit where credit is due;

- Wrap the whole into a piece of bug-ridden code

- Inform people about the compressed format to gain support. Thanks already to Ben for showing intentions to add OSM support to his library.

- Fame and glory.

Project Scope

The scope of this project depends on the stage it is in.

Stage 1
- First target is having a compressor/decompressor for xm/s3m/it archives. In this stage, no additional information will be added to archives other than the strictly needed to recover the "original" modules (the recovered modules may sound very close to the original although not exactly equal due to lossy compression. Frequently recompressing and decompressing is not recommended, as it isn't in MP3 files.

Stage 2
- Allowing additional information, such as mixer info that is not contained in the module files.

Stage 3
- Migrating modules to a text-based, fully open module format.

Thanks go to Ben Davis for the DUMB library, which takes care of the most complex part of locating the samples inside the module file. This library was slightly adapted to allow it to output the sample location info.

Requirements:

- perl and tar available in the search path
- depending on (de)compression types used, the following utilities available in the search path: flac, oggenc, oggdec, lame, mpg321
- This application is only tested on linux so far but should run fine without modifications on most platforms supporting perl and gcc.

Downloads:

Beta release: osmzip-0.2.0beta.tgz

This is a beta release. The current feature set is:
- Compresses and decompresses .xm modules and .s3m modules
- .ogg, .mp3 and lossless .flac compression
- testmode

Last development snapshot: download here.

The development snapshot differs from the pre-release in the following ways:
- None. The current development snapshot *is* 0.2.0beta.

Known bugs:

- Lossless .flac compression isn't completely lossless yet.
- Program assumes all requirements are available, if some of them are not met the decompressed modules might be silent or sound wrong

Coming soon:

- Support for .mod files
- Support for .it files