The Dirfile Standards

Introduction

The Dirfile Standards describe the dirfile database format, a filesystem based database for time-ordered binary data. Dirfiles are designed to be a fast, simple format for storing and reading binary time-ordered data. This document provides an unofficial overview of the Dirfile Standards. The official Dirfile Standards are distributed with GetData as three Unix man pages: dirfile(5), dirfile-format(5), and dirfile-encoding(5). Additionally, this document discusses some implementation-dependant behaviour specific to GetData not found in those documents. The latest release of the Dirfile Standards is Standards Version 8 (November 2010).

The dirfile database is centred around one or more time-ordered data streams (a time stream). Each time stream is written to disk in a separate file, in its native binary format. The name of these time stream files correspond to the time stream's field name, a descriptive textual tag.

Two time streams may have different constant sampling frequencies and mechanisms exist within the dirfile format to ensure these time streams remain properly sequenced in time. To do this, the time streams in the dirfile are subdivided into frames. Each frame contains an integer number of samples of each time stream. When synchronous retrieval of data from more than one time stream is required, position in the dirfile can be specified in frames, which will ensure synchronicity.

The time stream files are all located in a central directory, known as the dirfile directory. The dirfile as a whole is typically referred to by its dirfile directory. Included in the dirfile along with the time streams is the dirfile format specification, which is an ASCII text file called format located in the dirfile directory.

Version 3 of the Dirfile Standards introduced the large dirfile extension. This extension added the ability to distribute the dirfile format specification among multiple files (called fragments) in addition to the format file, as well as the ability to house portions of the database in subdirfiles. These subdirfiles may be fully-fledged dirfiles in their own right, but may also be contained within a larger, parent dirfile.

In addition to the raw fields on disk, the dirfile format specification may also specify derived fields which are calculated from one or more raw or derived time streams. Derived fields behave identically to raw fields when read via GetData. See below for a complete list of derived field types.

Dirfiles are designed to be written to and read simultaneously. The dirfile specification dictates that one particular raw field (specified either explicitly with the /REFERENCE directive or implicitly by the format file) is to be used as the reference field: all other vector fields are assumed to have at least as many frames as the reference field has, and the size (in frames) of the reference field is used as the size of the dirfile as a whole.

Version 6 of the Dirfile Standards added the ability to encode the binary files on disk. Each fragment may have its own encoding scheme. Notably this can be used to compress these files. See Dirfile Encodings for information on encoding schemes.

Dirfile Example

An example dirfile is presented as Figure 1 below. The dirfile as a whole is referenced by the name of the directory which forms its base, in this case "dirfile". The base directory contains a format file which contains the metadata for the dirfile database. This format file includes other files (called format file fragments) which provide additional database metadata, including a format file in a subdirectory (a subdirfile). A full description of the format file syntax is given below.

Figure 1: Graphical representation of a dirfile

This dirfile contains four time streams (indicated by RAW in the format file, and often called raw fields), one of which is located in the subdirfile. Five derived fields are also defined (indicated by LINCOM, MULTIPLY, and BIT in the format file—other derived field types also exist, see below).

Each time stream has a corresponding file of binary data containing the time stream itself. Each time stream may have a different sample rate, also indicated in the format file. In this example, for every sample of field1, there are four samples of field2, twelve samples of field3, and eight samples of bits. Derived fields inherit the sample rate of their first input field, so, for example, the derived field diff has the same sample rate as field2.

The binary file associated with the time stream bits is located in the subdirfile because it is defined in the format file fragment in that directory. This is a general rule: the binary file associated with a time stream must reside in the directory which contains the fragment that defines the field. (This is true even if the fragment isn't called format, i.e. binary files associated with raw fields defined in extra_format would have to reside in dirfile.) If this rule is not followed, GetData will be unable to locate the time stream on disk.

The subdirfile is a fully formed dirfile in its own right, since its metadata is fully specified by its format file, which does not refer to any fields defined by its parent. Note, however, that dirfile, is not a complete dirfile without the inclusion of the subdirfile, since the format fragment extra_format refers to a field defined in the subdirfile.

The Format File

The format file is a case-sensitive text file which contains the dirfile database metadata. The explicit text encoding is not specified by the Standards, but it must be 7-bit ASCII compatible. Examples of acceptable character encodings include all the ISO 8859 character sets (i.e. Latin-1 through Latin-10, among others), as well as the UTF-8 encoding of Unicode and UCS.

The format file is composed of directive lines and field specification lines, optionally separated by blank lines, or lines containing ony whitespace. Lines are separated by the line-feed character (0x0A). Unless escaped (see below), the hash mark (#) is the comment delimeter; the comment delimeter, and any text following it to the end of the line, is ignored.

Tokens

Both directive lines and field specification line consist of several tokens separated by whitespace. Whitespace consists of one or more whitespace characters. These are: space (0x20), horizontal tab (0x09), vertical tab (0x0B), form-feed (0x0C), and carriage return (0x0D). The first token of a directive line is always a reserved word, while a field specification line begins with a field name. As a result, no field may have the same name as a reserved word (although, as of Standards Version 8, all reserved words contain a forward slash character (/), which are prohibited in field names in any case).

Since tokens are separated by whitespace, to include a whitespace character in a token, it must either be escaped by preceding it by a backslash character (\), or replaced by a character escape sequence, see Table 1, below), or else the token must be enclosed in quotation marks ("). The quotation marks themselves will be stripped from the token. The null-token (that is, the token consisting of zero characters) may be specified by a pair of quotation marks with nothing between them (""). To include a literal quotation mark or backslash character in a token, it must be escaped (\" or \\). Similarly, a hash mark may be included in a token by including it in a quoted token or else by escaping it (\#), otherwise the hash mark will be understood as the comment delimiter.

It is a syntax error to have a line which contains unmatched quotation marks, or in which the last character is the backslash character.

Several characters when escaped by a preceding backslash character are interpreted as special characters in tokens. Some of these have already been mentioned. The full list of character escape sequences is presented in Table 1. Any other character which is escaped is interpreted as the character itself. (i.e. \c is interpreted as c).

No token may contain the NUL character (0x00). Furthermore, although support is present to create UTF-8 byte sequences, tokens are not required to be valid UTF-8 sequences. Any byte sequence not containing the NULL character forms a valid token. However, there may be further restrictions on allowed characters for a token in a particular situation, (for example, when used as a field name).

Standards Versions 5 and earlier do not recognise the character escape sequences, nor allow quoting of tokens. As a result, they prohibit both whitespace and the comment delimileter from being used in tokens.

Directives

There are eight directives, each specified by a different reserved word; all directives are optional. In Standards Version 8, all reserved words start with an initial forward slash (/), to distinguish them from field names. Standards Versions 5, 6 and 7 permit any reserved word to optionally omit its initial forward slash, without change in meaning. Reserved words in Standards Version 4 and earlier have no initial forward slash.

A number of the directives have fragment scope. A directive with fragment scope only applies to the fragment in which it is present, plus any sub-fragments indicated by the /INCLUDE directive, but only if those sub-fragments don't have their own corresponding directive. Directives which have fragment scope are: /ENCODING, /ENDIAN, /FRAMEOFFSET, and /PROTECT. Becase of these scoping rules, different portions of the dirfile may have different encodings, endiannesses, frame offsets, or protection levels.

If a directive with fragment scope appears more than once in a fragment, only the last such directive will be honoured, with the exception that the effect of a directive will not be propagated to sub-fragments if the directive line appears after the sub-fragment is included. The scoping rules of the remaining directives are discussed below.

• /ENCODING: The /ENCODING directive specifies the encoding scheme used to encode binary files in the dirfile. Syntax is:
  /ENCODING <scheme>

  The encoding scheme may be one of the pre-defined names listed below, which are described in more detail on the Dirfile Encodings page, or any other site-specific encoding scheme. The pre-defined schemes are:

  • none: The dirfile is unencoded.
  • bzip2: The dirfile is bzip2 encoded using the bzip2 compression library.
  • gzip: The dirfile is gzip encoded using the zlib compression library.
  • lzma: The dirfile is lzma encoded using the liblzma compression library.
  • slim: The dirfile is slim encoded using the slimlib compression library.
  • text: The dirfile is text encoded.

  GetData will accept dirfiles it encounters with an encoding scheme not listed here but, in this case, no binary file I/O (affecting getdata(), putdata(), get_nframes(), &c.) will be possible on the dirfile. If no encoding scheme is specified, GetData will try to determine the encoding scheme automatically, or in the case of a newly created dirfile, assume the dirfile is unencoded. The /ENCODING directive has fragment scope. Introduced in Standards Version 6.

• /ENDIAN: The /ENDIAN directive specifies the endianness of the raw data in the database. Syntax is:
  /ENDIAN ( big | little ) [ arm ]
  where big, and little specify the byte-ordering the data, and arm is specified when double-precision floating point data are in the middle-endian format used by older ARM processors.

  GetData will assume that any dirfile that omits this directive is in the native endianness of the architecture on which it is running (i.e. no endianness swapping will take place.) Otherwise, if the endianness of the dirfile is different than the endianness of the architecture, GetData will perform endianness conversion while reading from and writing to raw data on disk. The /ENDIAN directive has fragment scope. Introduced in Standards Version 5; Standards Version 8 added the optional second ("arm") token.

• /FRAMEOFFSET: The /FRAMEOFFSET directive specifies the frame number of the first frame in the database (the beginning-of-field marker). Syntax is:
  /FRAMEOFFSET <integer>
  Requests for data before the beginning-of-field marker will result in zero (for integer return types) or IEEE not-a-number (for floating point types). Attempts to write to the data before the beginning-of-field marker will result in an error, and no data written. The /FRAMEOFFSET directive has fragment scope. Introduced in Standards Version 1.
• /INCLUDE: The /INCLUDE directive specifies another file to parse for additional metadata for the dirfile. Syntax is:
  /INCLUDE <filename>
  The inclusion is treated as if the lines of the included file were pasted verbatim in place of the /INCLUDE directive line. The exception to this is that binary files associated with raw fields specified in the included file will be looked for in the directory containing the included file and not in the directory containing the parent format file. The included file may be specified either with an absolute path, or else a relative path from the current file. The /INCLUDE directive has no scope: it is processed immediately and has no long-term effect. Introduced in Standards Version 3.
• /META: The /META directive specifies a metafield attached to to a particular parent field. The field metadata may be of any allowed type except RAW. Metafields are retrieved in exactly the same way as regular field data, but the field code specified consists of the parent and metafield names joined with a forward slash:
  <parent-field>/<meta-field>
  /META field directives may not be specified before the parent field has been. Syntax is:
  /META <parent-field> {field specification line}
  As an illustration of this concept,
  /META parent meta CONST FLOAT64 3.291882
  provides a scalar metadatum called meta with value 3.291882 attached to the field parent. This particular metafield may be referred to by the field code "parent/meta". Note that different parent fields may have metafields with the same name, since all references to metafields must include the parent field name. Metafields may not themselves have further sub-metafields. The /META directive has no scope: it is processed immediately and has no long-term effect. /META directives are required to appear after their parent's specification, and in the same fragment.

  For simplicity, starting with Standards Version 7, the above metafield can also be specified as:

  parent/meta CONST FLOAT64 3.291882
  making it look like a regular field specification line. Introduced in Standards Version 6.
• /PROTECT: The /PROTECT directive specifies the advisory protection level of the current fragment and of the RAW fields defined therein. The protection level indicates whether writing to the format file fragment, or the binary data on disk is permitted. Syntax is:
  /PROTECT <level>
  Four advisory protection levels are defined:
  • none: No protection at all: data and metadata may be freely changed. This is the default, if no /PROTECT directive is present.
  • format: The dirfile metadata is protected from change, but RAW data on disk may be modified.
  • data: The RAW data on disk is protected from change, but metadata may be modified.
  • all: Both metadata and data on disk are protected from change.
  The /PROTECT directive has fragment scope. Introduced in Standards Version 6.
• /REFERENCE: The /REFERENCE directive specifies the name of the field to use as the dirfile's reference field. If no /REFERENCE directive is specified, the first RAW field encountered is used as the reference field. The /REFERENCE directive must specify a RAW field. Syntax is:
  /REFERENCE <field-code>
  The /REFERENCE directive has global scope: if multiple /REFERENCE directives appear in the dirfile metadata, only the last such will be honoured. Introduced in Standards Version 6.
• /VERSION: The /VERSION directive specifies the particular version of the Dirfile Standards to which the dirfile conforms. Syntax is:
  /VERSION <integer>
  When a /VERSION directive indicates a Standards Version greater than GetData is prepared to deal with, it triggers permissive mode. In permissive mode, unrecognised lines (which it assumes are valid syntax in a newer version of the Standards) are silently ignored.

  The /VERSION directive has immediate scope: its effect is immediate, and it applies only to metadata below it, including and propagating downwards to sub-fragments after the directive. Its effect will also propagate upwards back to the parent fragment, and affect subsequent metadata. Introduced in Standards Version 5.

Field Specifications

Any line which does not start with a reserved word is assumed to be a field specification line. A field specification line consists of at least two tokens. The first token is the field name. The second token is the field type. Subsequent tokens are field parameters. The meaning and number these parameters depends on the field type specified.

Field Names

A field name consists of one or more characters, excluding both ASCII control characters (bytes 0x00 through 0x1F) and the reserved characters listed in Table 2 according to Standards Version. Furthermore, the field name of a RAW field may only contain characters allowed in filenames^†. Although never allowed in a field name, a forward slash (/) can be used to define metafields; see above under the /META directive. Like the rest of the format file, field names are case sensitive.

The field name may not be INDEX, which is a special, implicit field which contains the integer frame index. Standards Version 5 and earlier also prohibit FILEFRAM as a field name; it was an alias for INDEX, (which arose in the prehistoric times of ReadData, GetData's spiritual predecessor).

Standards Version 3 and 4 restrict field names to 50 characters. Standards Version 2 and earlier restrict field names to 16 characters. Additionally, the filesystem will put restrictions on the length of a RAW field name, regardless of Standards Version.

Field Types

There are thirteen field types. Of these, ten are of vector type (BIT, DIVIDE, LINCOM, LINTERP, MULTIPLY, PHASE, POLYNOM, RAW, RECIP, and SBIT) and three are of scalar type (CARRAY, CONST, and STRING). The possible fields types are listed below with their field specification lines. For information on how to specify the parameters in the specification lines, see below under Field Parameters.

• BIT: The BIT field type extracts one or more bits out of an input vector field, treating the result as unsigned. Syntax is:
  <field-name> BIT <input> <first-bit> [ <bits> ]
  which specifies <field-name> to be the <bits>-bit long unsigned integer starting at bit <first-bit> (numbered from zero) of <input> (when <input> is converted to an endianness-corrected unsigned 64-bit integer). If <bits> is omitted, it is assumed to be 1. Standards Version 0 doesn't recognise the <bits> token.
• CARRAY: The ARRAY scalar field type is an list of constants fully specified in the format file metadata. Syntax is:
  <field-name> CARRAY <type> <value0> <value1> <value2> ...
  where type may be any supported native data type and value0, value1, &c. are the values of successive elements in the scalar list, interpreted as indicated by type. GetData is prepared to deal with at least 2²⁴ values. Note: despite being multivalued, this is not considered a vector field since the elements of the CARRAY are not indexed by frames. A CARRAY with a single element is identical to a CONST field. Introduced in Standards Version 8.
• CONST: The CONST scalar field type is a constant fully specified in the format file metadata. Syntax is:
  <field-name> CONST <type> <value>
  where type may be any supported native data type and value is the numerical value of the constant interpreted as indicated by type. Introduced in Standards Version 6.
• DIVIDE: The DIVIDE field type is the quotient of two vector fields. Syntax is:
  <field-name> DIVIDE <field₁> <field₂>
  where <field-name> will be computed as:
  field-name[n] = field₁[n] / field₂[n₂]
  with the index n₂ computed appropriately for the (potentially differing) sample rates of the input fields. The resultant field will have the same sample rate as <field1>. Introduced in Standards Version 8.
• LINCOM: The LINCOM field type is the linear combination of one, two or three input vector fields. Syntax is:
  <field-name> LINCOM [<n>] <field₁> <m₁> <b₁> [ <field₂> <m₂> <b₂> [ <field₃> <m₃> <b₃> ]]
  where:
  • <n> indicates the number of input fields (i.e. 1, 2, or 3). In Standards Version 7 and 8 it is optional; earlier Standards Versions require it. If omitted, the number of input fields will be determined by the number given.
  • <field-name> will be computed as:
    field-name[n] = (m₁ × field₁[n] + b₁) + (m₂ × field₂[n₂] + b₂) + (m₃ × field₃[n₃] + b₃)
    with <field₂>, <m₂>, <b₂>, and <field₃>, <m₃>, <b₃> included only if specified. The indices n₂ and n₃ are computed appropriately for the (potentially differing) sample rates of the input fields. The resultant field will have the same sample rate as <field₁>.
• LINTERP: The LINTERP field type specifies a table look up based on another vector field. Syntax is:
  <field-name> LINTERP <input> <table>
  where:
  • <input> is the input field for the table lookup
  • <table> is the path to the lookup table file for the field. The lookup table file is a text file with two whitespace separated columns of x and y values. Values are linearly interpolated between the points specified in the lookup table.
• MULTIPLY: The MULTIPLY field type is the product of two vector fields. Syntax is:
  <field-name> MULTIPLY <field₁> <field₂>
  where <field-name> will be computed as:
  field-name[n] = field₁[n] × field₂[n₂]
  with the index n₂ computed appropriately for the (potentially differing) sample rates of the input fields. The resultant field will have the same sample rate as <field1>. Introduced in Standards Version 2.
• PHASE: The PHASE field type shifts an input vector field by a specified number of samples. Syntax is:
  <field-name> PHASE <input> <shift>
  where:
  • <input> is the input field
  • <shift> is the shift, in frames. A positive shift indicates a shift forward in time (towards larger frame numbers).
  Introduced in Standards Version 4.
• POLYNOM: The POLYNOM field type specifies a polynomial function of a single input field. Synax is:
  <field-name> POLYNOM <input> <a₀> <a₁> [ <a₂> [ <a₃> [ <a₄> [ <a₅> ]]]]
  where:
  • <input> is the input field code
  • <field-name> will be computed as:
    field-name[n] = a₀ + a₁ × input[n] + a₂ × (input[n])² + a₃ × (input[n])³ + a₄ × (input[n])⁴ + a₅ × (input[n])⁵
    with the highter-order terms computed only if the corresponding co-efficients a_i are specified.
  Introduced in Standards Version 7.
• RAW: The RAW field type specifies time streams on disk. In this case, the field name must correspond to the name of the file containing the time stream. Syntax is:
  <field-name> RAW <type> <sample-rate>
  where:
  • <sample-rate> is the number of samples per dirfile frame for the time stream
  • <type> is a token specifying the native data type:
    • UINT8: unsigned 8-bit integer
    • INT8: signed 8-bit integer (Standards Version 5 and later)
    • UINT16: unsigned 16-bit integer
    • INT16: signed 16-bit integer
    • UINT32: unsigned 32-bit integer
    • INT32: signed 32-bit integer
    • UINT64: unsigned 64-bit integer (Standards Version 5 and later)
    • INT64: signed 64-bit integer (Standards Version 5 and later)
    • FLOAT32 or FLOAT: IEEE-754 standard 32-bit (single precision) floating point number
    • FLOAT64 or DOUBLE: IEEE-754 standard 64-bit (double precision) floating point number
    • COMPLEX64: C99-compatible single precision complex number (Standards Version 7 and later)
    • COMPLEX128: C99-compatible double precision complex number (Standards Version 7 and later)
    See the dirfile(5) manual page for full details on complex number storage formats.

    GetData will also recognise the following single character type aliases, which were allowed by Standards Versions 7 and earlier (note that not all supported types have a single character type alias):

    • c: UINT8
    • u: UINT16
    • s: INT16
    • U: UINT32
    • S or i: INT32
    • f: FLOAT32
    • d: FLOAT64
• RECIP: The RECIP field computes the reciprocal of an input field. Syntax is:
  <field-name> RECIP <input> <dividend>
  where:
  • <input> is the input field code
  • <field-name> will be computed as:
    field-name[n] = dividend / input[n]
    where <dividend> is a scalar.
  Introduced in Standards Version 8.
• SBIT: The SBIT field type extracts one or more bits out of an input vector field, treating the result as signed. Syntax is:
  <field-name> SBIT <input> <first-bit> [ <bits> ]
  which specifies <field-name> to be the <bits>-bit long signed integer starting at bit <first-bit> (numbered from zero) of <input> (when <input> is converted to an endianness-corrected signed 64-bit integer). If <bits> is omitted, it is assumed to be 1. Note: all signed integers are interpreted as two's complement numbers; so, a single-bit signed integer can take the values 0 or −1. Introduced in Standards Version 7.
• STRING: The STRING scalar field type is a character string fully specified in the format file metadata. Syntax is:
  <field-name> STRING <value>
  where <value> is the string value of the field. Note that <value> is a single token. To include whitespace in the string, enclose the value in quotation marks (""), or else escape the whitespace with the backslash character (\). Introduced in Standards Version 6.

Field Parameters

All input vector field parameters should be a field codes. Additionally, in Standards Version 6 and later, some of the numerical field parameters may be either literal numbers or else the field code of a CONST or CARRAY scalar field containing the value. In the case of a CARRAY, the field code may be immediately followed by an integer enclosed in angle brackets (< >) specifying which element (counting from zero) of the CARRAY to use (so: field_code<n>). If this is omitted, the first element is assumed. Parameters for which this is possible are:

• RAW: spf
• BIT, SBIT: bitnum, bits
• LINCOM: any of the m_i or b_i
• PHASE: shift
• POLYNOM: any of the a_i.
• RECIP: dividend

Since it is possible to create a field code which is identical to a literal number, a parameter is assumed to be the field code of a scalar field only if it doesn't look like a number.

In Standards Version 7 and later, a literal complex number is specified as two real (floating point) numbers separated by a semicolon (;) with no intervening whitespace. So, for example, the tokens:

represent, respectively, the real unit, the imaginary unit, the real number four, the imaginary number 5i, and the complex number 931.3+74.1i. Because the semicolon character cannot be used in field names, a complex valued literal can never be mistaken for a field code.

Complex literals allow, among other things, the composition of complex valued fields from purely real input fields. For example, a complex valued field, z, may be created from a real valued field re, representing the real part of the complex number, and the real valued field im, representing the imaginary part of the complex number, by specifying:

Field Codes

Both when specifying the inputs to a field (as a non-literal scalar parameter, or as an input vector field to a field), and when specifying a field to a GetData call, field codes are used. A field code is one of:

• a simple field name, indicating a vector or scalar field, ie. "field",
• starting with Standards Version 6, a parent field name, followed by a forward slash (/), followed by a metafield name, indicating a metafield, ie. "parent/field",
• starting with Standards Version 7, either of the above, followed by a period, followed by a "representation suffix, but only if the field or metafield specified is not a STRING type field; ie. "field.r", or "parent/field.m".

A representation suffix may be used used to extract a real number from a complex value. The available suffixes and their meanings are:

• .a: This representation indicates the angle (in radians) between the positive real axis and the value (ie. the complex argument). The argument is in the range [−π, π], and a branch cut exists along the negative real axis. At the branch cut, −π is returned if the imaginary part is −0, and π is returned if the branch cut is +0. If z=0, zero is returned.
• .i: This representation indicates the projection of the value onto the imaginary axis (ie. the imaginary part of the number).
• .m: This representation indicates the modulus of the value (ie. its absolute value).
• .r: This representation indicates the projection of the value onto the real axis (ie. the real part of the number).

If the specified field is purely real, the representations are calculated as if the imaginary part was equal to +0. For example, given a complex valued vector, z, a vector containing the real part of z, called re_z, could be produced with:

and similarly for the complex field's imaginary part, argument, and absolute value. (Although it should be pointed out this simplistic an example isn't strictly necessary, since z.r could be used wherever re_z would be.)

Standards Versions

The latest version of the Dirfile Standards is Version 8.