Current Event File Format (MWK2)

Description

MWorks’ current event file format uses the file extension .mwk2.

Each event file is a standalone SQLite database. The database contains a single table (events) with three columns: code, time, and data, corresponding to the three elements of an MWorks event.

In a given row of the events table, the code and time values are always integers, while the type of the data value can be any of SQLite’s supported datatypes. A data value of type NULL, INTEGER, REAL, or TEXT represents the data for a single event. A data value of type BLOB contains MessagePack-encoded data for one or more events (all with the same code and time), in one of the following forms:

1. an uncompressed stream of MessagePack-encoded values, with each value of MessagePack type nil, boolean, integer, float, string, binary, array, or map

2. a single MessagePack extension type value with type code 1, whose data is a zlib-compressed, UTF-8-encoded string

3. a single MessagePack extension type value with type code 2, whose data is a zlib-compressed stream of MessagePack-encoded values (i.e. like (1) after decompression)

Example Code

The following Python code demonstrates the MWK2 format in detail by a implementing a simple reader for .mwk2 files. Apart from the msgpack package, it requires only the Python standard library:

import sqlite3
import zlib

import msgpack


class MWK2Reader:

    _compressed_text_type_code = 1
    _compressed_msgpack_stream_type_code = 2

    def __init__(self, filename):
        self._conn = sqlite3.connect(filename)

    def close(self):
        self._conn.close()

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        self.close()

    @staticmethod
    def _decompress(data):
        return zlib.decompress(data, -15)

    def __iter__(self):
        unpacker = msgpack.Unpacker(strict_map_key=False)
        for code, time, data in self._conn.execute('SELECT * FROM events'):
            if not isinstance(data, bytes):
                yield (code, time, data)
            else:
                unpacker.feed(data)
                for obj in unpacker:
                    if isinstance(obj, msgpack.ExtType):
                        if obj.code == self._compressed_text_type_code:
                            obj = self._decompress(obj.data).decode('utf-8')
                        elif (obj.code ==
                              self._compressed_msgpack_stream_type_code):
                            unpacker.feed(self._decompress(obj.data))
                            continue
                    yield (code, time, obj)

The MWK2Reader class defined above can be used as follows:

with MWK2Reader('my_data.mwk2') as event_file:
    for code, time, data in event_file:
        # Process the current event
        ...