Slocum glider data decoder

A Slocum file is factored. It names an eight-character CRC and carries no sensor list of its own, so the binary section cannot be interpreted without the matching <crc>.cac. A file whose cache is absent cannot be read partially or approximately, so the decoder reports the CRC it requires. Caches are written to the glider's cache directory and to the deployment directory the dockserver creates. Once a cache has been supplied it is retained in this browser and does not need to be supplied again.

The two kinds of file may be supplied in either order. A data file that arrives before its cache is retained, and it is decoded as soon as the cache arrives. Dropping the data, reading which cache it names, and then supplying that cache therefore works without dropping the data a second time. A whole folder may also be dropped or selected, which is the one action that supplies both: the file dialog selects within a single directory, and the data and the caches are in different ones.

Files are grouped into deployments before anything is written, and each deployment is exported as its own file. Two gliders, or one glider's spring and summer work, are therefore never written into a single file. The filenames and the sensor names are identical across such cases, so the two records cannot be separated once they have been combined.

A deployment is one glider over one continuous stretch of time. A different vehicle begins a new deployment, and so does a gap of three days or more. The gap is measured between segments and not between samples. A glider logs different sensors on very different schedules, and a slow channel that goes quiet over a weekend is not a break in the deployment. With a single deployment the page behaves as it does for a single pair of files.

Rows and the time base

A Slocum file has no single time base. Each sensor is written on its own subset of cycles: the flight computer logs depth every few seconds and position only at surfacings, and the science computer keeps a separate clock. Assembling a rectangular table is therefore a decision about which times become rows, and not a question of formatting.

The default assembles one row per time at which any sensor reported, and leaves a cell blank where that sensor did not report. No value is added and none is discarded, so the file can be reconstructed from the table. Most cells are blank, which is the shape of what a glider logs. The alternative interpolates every sensor onto one sensor's times. It is dense and immediately usable. Every value in it outside the base column is a number the glider did not record, and it is applied only when it is selected.

Column order

Position comes first, then depth, then the CTD and the quantities derived from it. The remaining columns follow in order of how many values they hold, which places the nearly empty engineering channels at the far right. The order the cache file gives is alphabetical over the glider's whole sensor namespace: on one test segment that placed a channel holding 3 values of 1,328 in the first column, and water temperature, holding 853, in the sixty-second.

Where the two rules disagree, the named quantity is placed first. Position is recorded only at surfacings and so holds few values, and it remains among the first columns. The CSV, the on-screen preview and both netCDF exports use one order.

One glider, two computers

A glider carries a flight computer and a science computer, and the sensor names record which of the two a sensor belongs to. sci_ is measured by the science computer. m_ is measured by the flight computer and c_ is commanded on it. u_ are parameters the user sets, f_ are factory values, and x_ and several others are derived channels.

Columns are grouped by glider first and by computer second. Two vehicles' m_depth are separate records, so they remain separate columns and each names its glider. Within one glider, a sensor written by both computers also remains two columns. sci_water_pressure is measured by the science computer and relayed to the flight computer at a much lower rate, and the sci_ prefix identifies which of the two holds the measurement.

Within one computer the files combine. sbd, mbd and dbd are three decimations of one flight record, and tbd, nbd and ebd are three decimations of one science record. The short files are transmitted over Iridium during the deployment and the long ones are recovered from the glider afterwards. Supplying any combination of them produces one column per sensor. Samples that appear in more than one file are merged by timestamp, and the number merged is reported. Where two files hold different values at one timestamp the value from the more complete file is retained, and the page reports that as well.

The merge is performed per sensor rather than per file, because the decimation lists are not nested. The lists are set independently with sbdlist.dat and mbdlist.dat: on one segment of the test deployment the sbd holds 64 sensors and the mbd 134, of which 58 are common and 6 are in the sbd alone. The merged table holds the union.

Seawater properties

A Slocum CTD writes conductivity, in-situ temperature and pressure. Selecting the seawater option adds practical salinity, Absolute Salinity, Conservative Temperature, density, σ₀ and sound speed, computed with the full-precision TEOS-10 implementation that the seawater calculator uses. These columns are marked as derived in the CSV heading and in the netCDF attributes. Without a position, or without the Absolute Salinity Anomaly atlas, the composition correction cannot be applied, and the column is named salinity_reference rather than salinity_absolute, which is the quantity it then holds.

OceanGliders OG1.0

The OG1.0 export writes the community trajectory format. OG1.0 requires metadata that a Slocum file does not carry, including the glider's WMO identifier, the people and institutions responsible, and the deployment time and position. The panel collects those fields, retains them in this browser, and writes them to a profile file. The segments of one deployment take the same values, so the profile is loaded rather than re-entered.

The variables, units and attributes are OG1.0. The encoding is netCDF-3 classic, because a browser cannot write netCDF-4 without a large HDF5 implementation, so the variables OG1.0 declares as strings are written as fixed-width character arrays. The CDL export declares them as string, and ncgen -4 file.cdl -o file.nc compiles it to netCDF-4. That path has been checked: ncgen compiles the CDL, and the values, the attributes and the string variables are identical across the two encodings. No OG1 validator has checked either file, and the community's own checkers describe themselves as experimental.

Four quantities that OG1.0 requires at every measurement are computed rather than recorded, and each records in its own attributes how it was obtained. Position between the glider's fixes is interpolated, and the fixes themselves are written to LATITUDE_GPS and LONGITUDE_GPS. Depth is computed from pressure and latitude. Practical salinity is computed from the CTD. The dive phase, with its segment and profile numbering, is taken from the glider's own behavior state where that state is logged, and inferred from the rate of change of pressure where it is not, which is the usual case for files transmitted over Iridium. The file records which of the two methods was used.

CSV and netCDF

The CSV holds data alone and carries no comment lines, so it opens in any reader. The netCDF is netCDF-3 classic, written directly rather than through a library, and it carries the provenance: the source files, the decoding history, and every note the table produced. Its units are the glider's own unit strings, degc, nodim and enum among them, carried verbatim, so the file does not claim CF conventions. The time variable is the exception and its units are udunits.

Processing not applied

The decoder reads files and does not process them. It applies no thermal-lag correction, no despiking, no quality control and no gridding, and it does not write the IOOS Glider DAC trajectory format. The OG1.0 quality flags are all zero, the value that records that no quality control has been applied. Position is converted from NMEA and is otherwise the glider's own dead-reckoned position.

Every sensor's times and values are compared with dbdreader by a SHA-256 over their raw bytes, for a matched flight and science pair from a real deployment. The decoder is a port of SlocumIO.jl, which was itself validated against dbdreader, so agreement here is agreement with both. The netCDF writer was checked separately by reading its output with scipy.io.netcdf_file.

That is one glider and one deployment, which is the extent of what has been tested.