Current Event File Format (MWK2)

Description

MWorks’ current event file format uses the file extension .mwk2.

Each event file is a standalone SQLite database. The database contains a single table (events) with three columns: code, time, and data, corresponding to the three elements of an MWorks event.

In a given row of the events table, the code and time values are always integers, while the type of the data value can be any of SQLite’s supported datatypes. A data value of type NULL, INTEGER, REAL, or TEXT represents the data for a single event. A data value of type BLOB contains MessagePack-encoded data for one or more events (all with the same code and time), in one of the following forms:

1. an uncompressed stream of MessagePack-encoded values, with each value of MessagePack type nil, boolean, integer, float, string, binary, array, or map

2. a single MessagePack extension type value with type code 1, whose data is a zlib-compressed, UTF-8-encoded string

3. a single MessagePack extension type value with type code 2, whose data is a zlib-compressed stream of MessagePack-encoded values (i.e. like (1) after decompression)

Example Code

The following Python code demonstrates the MWK2 format in detail by a implementing a simple reader for .mwk2 files. Apart from the msgpack package, it requires only the Python standard library:

from __future__ import division, print_function, unicode_literals
import sqlite3
import zlib

import msgpack

try:
    buffer
except NameError:
    # Python 3
    buffer = bytes


class MWK2Reader(object):

    _compressed_text_type_code = 1
    _compressed_msgpack_stream_type_code = 2

    def __init__(self, filename):
        self._conn = sqlite3.connect(filename)
        self._unpacker = msgpack.Unpacker(raw=False, strict_map_key=False)

    def close(self):
        self._conn.close()

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        self.close()

    @staticmethod
    def _decompress(data):
        return zlib.decompress(data, -15)

    def __iter__(self):
        for code, time, data in self._conn.execute('SELECT * FROM events'):
            if not isinstance(data, buffer):
                yield (code, time, data)
            else:
                try:
                    obj = msgpack.unpackb(data, raw=False)
                except msgpack.ExtraData:
                    # Multiple values, so not valid compressed data
                    pass
                else:
                    if isinstance(obj, msgpack.ExtType):
                        if obj.code == self._compressed_text_type_code:
                            yield (code,
                                   time,
                                   self._decompress(obj.data).decode('utf-8'))
                            continue
                        elif (obj.code ==
                              self._compressed_msgpack_stream_type_code):
                            data = self._decompress(obj.data)
                self._unpacker.feed(data)
                try:
                    while True:
                        yield (code, time, self._unpacker.unpack())
                except msgpack.OutOfData:
                    pass

The MWK2Reader class defined above can be used as follows:

with MWK2Reader('my_data.mwk2') as event_file:
    for code, time, data in event_file:
        # Process the current event
        ...