gd_getdata—retrieve data from a Dirfile database
size_t gd_getdata(DIRFILE *dirfile, const char *field_code, off_t first_frame, off_t first_sample, size_t num_frames, size_t num_samples, gd_type_t return_type, void *data_out);
The gd_getdata() function queries a dirfile(5) database specified by dirfile for the field field_code. It fetches num_frames frames plus num_samples samples from this field, starting first_sample samples past frame first_frame. The data is converted to the data type specified by return_type, and stored in the user-supplied buffer data_out.
The field_code may contain one of the representation suffixes listed in dirfile-format(5). If it does, gd_getdata() will compute the appropriate complex norm before returning the data.
The dirfile argument must point to a valid DIRFILE object previously created by a call to gd_open(3). The argument data_out must point to a valid memory location of sufficient size to hold all data requested.
Unless using GD_HERE (see below), the first sample returned will be
first_frame × samples_per_frame + first_sample
as measured from the start of the dirfile, where samples_per_frame is the number of samples per frame as returned by gd_spf(3). The number of samples fetched is, similarly,
num_frames × samples_per_frame + num_samples.
Although calling gd_getdata() using both samples and frames is possible, the function is typically called with either num_samples and first_sample, or num_frames and first_frames, equal to zero.
Instead of explicitly specifying the origin of the read, the caller may pass the special symbol GD_HERE as first_frame. This will result in the read occurring at the current position of the I/O pointer for the field (see GetData I/O Pointers below for a discussion of field I/O pointers). In this case, the value of first_sample is ignored.
When reading a SINDIR field, return_type must be GD_STRING. For all other field types, the return_type argument should be one of the following symbols, which indicates the desired return type of the data:
The return type of the data need not be the same as the type of the data stored in the database. Type conversion will be performed as necessary to return the requested type. If the field_code does not indicate a representation, but conversion from a complex value to a purely real one is required, only the real portion of the requested vector will be returned.
Upon successful completion, the I/O pointer of the field will be on the sample immediately following the last sample returned, if possible. On error, the position of the I/O pointer is not specified, and may not even be well defined.
GetData caches the value of the last sample from every MPLEX it reads so that a subsequent read of the field starting from the following sample (either through an explicit starting sample given by the caller or else implicitly using GD_HERE) will not need to scan the field backwards. This cache is invalidated if a different return type is used, or if an intervening operation moves the field's I/O pointer.
Backward-shifted PHASE fields do not suffer from this problem, since gd_getdata() pads reads past the beginning-of-field marker with NaN or zero as appropriate. Database creators who wish to use the PHASE field type with streaming data are encouraged to work around this limitation by only using backward-shifted PHASE fields, by writing RAW data at the maximal frame lag, and then back-shifting all data which should have been written earlier. Another possible work-around is to write systematically less data to the reference RAW field in proportion to the maximal forward phase shift. This method will work with applications which respect the database size reported by gd_nframes(3) resulting in these applications effectively ignoring all frames past the frame containing the maximally forward-shifted PHASE field's end-of-field marker.
In all cases, gd_getdata() returns the number of samples (not bytes) successfully read from the database. If the end-of-field is encountered before the requested number of samples have been read, a short count will result. this is not an error.
Requests for data before the beginning-of-field marker, which may have been shifted from frame zero by a PHASE field or /FRAMEOFFSET directive, will result in the the data being padded at the front by NaN or zero, depending on whether the return type is of floating point or integral type.
On error, this function returns zero and stores a negative-valued error code in the DIRFILE object which may be retrieved by a subsequent call to gd_error(3). Possible error codes are:
A descriptive error string for the error may be obtained by calling gd_error_string(3).
To save memory, gd_getdata() uses the memory pointed to by data_out as scratch space while computing derived fields. As a result, if an error is encountered during the computation, the contents of this memory buffer are unspecified, and may have been modified by this call, even though gd_getdata() will report zero samples returned on error.
Reading slim-compressed data (see defile-encoding(5)), may cause unexpected memory usage. This is because slimlib internally caches open decompressed files as they are read, and GetData doesn't close data files between gd_getdata() calls for efficiency's sake. Memory used by this internal slimlib buffer can be reclaimed by calling gd_raw_close(3) on fields when finished reading them.
When operating on a platform whose size_t is N-bytes wide, a single call of gd_getdata() will never return more than (2**(N-1) - 1) samples. The request will be truncated at (2**(N-M) - 1) samples, where M is the size, in bytes, of the largest data type used to calculate the returned field. If a larger request is specified, less data than requested will be returned, without raising an error. This limit is imposed even when return_type is GD_NULL or when reading from the INDEX field (i.e., even when no actual I/O or calculation occurs). In all cases, the actual amount of data is returned.
This is a general discussion of field I/O pointers in the GetData library, and contains information not directly applicable to gd_getdata().
Every RAW field in an open Dirfile has an I/O pointer which indicates the library's current read and write poisition in the field. These I/O pointers are useful when performing sequential reads or writes on Dirfile fields (see GD_HERE in the description above). The value of the I/O pointer of a field is reported by gd_tell(3).
Derived fields have virtual I/O pointers arising from the I/O pointers of their input fields. These virtual I/O pointers may be valid (when all input fields agree on their position in the dirfile) or invalid (when the input fields are not in agreement). The I/O pointer of some derived fields is always invalid. The usual reason for this is the derived field simultaneously reading from two different places in the same RAW field. For example, given the following Dirfile metadata specification:
the derived field c never has a valid I/O pointer, since any particular sample of c ultimately involves reading from more than one place in the RAW field a. Attempting to perform sequential reads or writes (with GD_HERE) on a derived field when its I/O pointer is invalid will result in an error (specifically, GD_E_DOMAIN).
The implicit INDEX field has an effective I/O pointer than mostly behaves like a true RAW field I/O pointer, except that it permits simultaneous reads from multiple locations. So, given the following metadata specification:
the I/O pointer of the derived field e will always be valid, unlike the similarly defined c above. The virtual I/O pointer of a derived field will change in response to movement of the RAW I/O pointers underlying the derived fields inputs, and vice versa: moving the I/O pointer of a derived field will move the I/O pointer of the RAW fields from which it ultimately derives. As a result, the I/O pointer of any particular field may move in unexpected ways if multiple fields are manipulated at the same time.
When a Dirfile is first opened, the I/O pointer of every RAW field is set to the beginning-of-frame (the value returned by gd_bof(3)), as is the I/O pointer of any newly-created RAW field.
The following library calls cause I/O pointers to move:
In general, when these calls fail, the I/O pointers of affected fields may be anything, even out-of-bounds or invalid. After an error, the caller should issue an explicit gd_seek(3) to repoisition I/O pointers before attempting further sequential operations.
The function getdata() appeared in GetData-0.3.0.
The GD_COMPLEX64 and GD_COMPLEX128 data types appeared in GetData-0.6.0.
In GetData-0.7.0, this function was renamed to gd_getdata().
The GD_HERE symbol used for sequential reads appeared in GetData-0.8.0.
The GD_STRING data type appeared in GetData-0.10.0.
GD_SIZE(3), gd_error(3), gd_error_string(3), gd_get_constant(3), gd_get_string(3), gd_mplex_lookback(3), gd_nframes(3), gd_open(3), gd_raw_close(3), gd_seek(3), gd_spf(3), gd_putdata(3), dirfile(5), dirfile-encoding(5)