Data

Data Format
Section titled “Data Format”LEAF publishes every measurement as a LeafPoint: a JSON object with a measurement, tags, fields and a timestamp.
These are the same four parts as the InfluxDB Line Protocol,
so a LeafPoint converts directly to line protocol and can be inserted into InfluxDB or any other timeseries database.
InfluxDB Line Protocol
Section titled “InfluxDB Line Protocol”Line protocol is a text based data model format that is designed for efficiently writing time-series data. This data model organizes timeseries data and data points into four components:
measurement: The identity of the measurement set and referring to the table that data is stored in. Note that later InfluxDB versions refer to this astable.tagsets: Optional key-value pair(s) that can be used as, or hold, the metadata to describe the data.fieldsets: Key-value pair(s) representing the data and its value (e.g.,temperature=10)timestamp: Optional unix timestamp
Format
Section titled “Format”These four components: timestamps;, measurement names; fields (key-value pairs representing the data); and tags (optional key-value pairs used to store metadata that describes the data) are arranged in a single line. A typical line in this format looks like this:
measurementName,tagKey=tagValue fieldKey="fieldValue" 1465839830100400200Where:
- measurementName: Identifies the
measurement. - tagKey=tagValue: Zero or more
tagsets separated by commas. Tags are optional but recommended for indexing. - fieldKey=“fieldValue”: At least one
fieldset, with multiple fields separated by commas. Fields are the actual data points. - 1465839830100400200: An optional
timestampfor the data point. If not specified, depending upon the recipient, the server’s current time is used.
measurementName,tagKey1=tagValue1,tagKey2=tagValue2 fieldKey1="fieldValue1",fieldKey2="fieldValue2" 1465839830100400200This format is highly optimized for time-series data, enabling quick parsing and writing as:
- everything before the first comma is a
measurement - everything between the first comma and the first white space is
tag, with distincttagbeing separated by a comma - everything between the first white space and the second white space is
field, with distinctfieldbeing separated by a comma - everything after the second white space is
timestamprepresented as an integer
Each line is then separated by \n.
Example
Section titled “Example”Example 1: Abundance
Section titled “Example 1: Abundance”As an example, say if we have a timeseries dataset where we count the occurrence (or abundance) of two species (G. ruber and G. bulloides) this would be represented tabularly as:
AbundanceCounts
| Time | Sample | Location | Analyst | Abundance: G. ruber | Abundance: G. bulloides |
|---|---|---|---|---|---|
| 1465839830100400200 | T86_5P#6_DIS_0_1cm | NAtl | BM | 50 | 76 |
| 1465839830100400800 | T86_5P#6_DIS_1_2cm | NAtl | JK | 47 | 81 |
| 1465839830100401200 | T86_5P#6_DIS_2_3cm | NAtl | JK | 31 | 99 |
With the data (Abundance: G. ruber and Abundance: G. bulloides) and metadata (Sample, Location, Analyst) being found on individual
rows with a timestamp index (Time). In line protocol this table would be:
AbundanceCounts,sample=T86_5P#6_DIS_0_1cm,location=NAtl,analyst=BM G_ruber=50,G_bulloides=76 1465839830100400200AbundanceCounts,sample=T86_5P#6_DIS_1_2cm,location=NAtl,analyst=JK G_ruber=47,G_bulloides=81 1465839830100400800AbundanceCounts,sample=T86_5P#6_DIS_2_3cm,location=NAtl,analyst=BM G_ruber=31,G_bulloides=99 1465839830100401200To make cleaner we could make it one message per analysis:
AbundanceCounts,sample=T86_5P#6_DIS_0_1cm,location=NAtl,analyst=BM G_ruber=50 1465839830100400200AbundanceCounts,sample=T86_5P#6_DIS_0_1cm,location=NAtl,analyst=BM G_bulloides=76 1465839830100400200AbundanceCounts,sample=T86_5P#6_DIS_1_2cm,location=NAtl,analyst=JK G_ruber=47 1465839830100400800AbundanceCounts,sample=T86_5P#6_DIS_1_2cm,location=NAtl,analyst=JK G_bulloides=81 1465839830100400800AbundanceCounts,sample=T86_5P#6_DIS_2_3cm,location=NAtl,analyst=BM G_ruber=31 1465839830100401200AbundanceCounts,sample=T86_5P#6_DIS_2_3cm,location=NAtl,analyst=BM G_bulloides=99 1465839830100401200To further simplify the line protocol and make it less dependent upon specific field keys we could add the species being counted as a separate field:
AbundanceCounts,sample=T86_5P#6_DIS_0_1cm,location=NAtl,analyst=BM counted=G_ruber,count=50 1465839830100400200AbundanceCounts,sample=T86_5P#6_DIS_0_1cm,location=NAtl,analyst=BM counted=G_bulloides,count=76 1465839830100400200AbundanceCounts,sample=T86_5P#6_DIS_1_2cm,location=NAtl,analyst=JK counted=G_ruber,count=47 1465839830100400800AbundanceCounts,sample=T86_5P#6_DIS_1_2cm,location=NAtl,analyst=JK counted=G_bulloides,count=81 1465839830100400800AbundanceCounts,sample=T86_5P#6_DIS_2_3cm,location=NAtl,analyst=BM counted=G_ruber,count=31 1465839830100401200AbundanceCounts,sample=T86_5P#6_DIS_2_3cm,location=NAtl,analyst=BM counted=G_bulloides,count=99 1465839830100401200By making the line protocol more generic additional datasets could be stored alongside this one from different locations, analysts, or even samples.
Example 2: Environmental variables
Section titled “Example 2: Environmental variables”A timeseries database of a sensor that measures rainfall and temperature would for example look like the following table:
EnvironmentalVariables
| Time | Location | Entity | Rainfall | Temperature |
|---|---|---|---|---|
| 1465839830100400200 | WUR Campus | Sensor1 | 0 | 22.4 |
| 1465839830100400800 | WUR Campus | Sensor1 | 0 | 21.7 |
| 1465839830100401200 | WUR Campus | Sensor1 | 0 | 21.9 |
This could be translated into:
EnvironmentalVariables,location=WUR\ Campus,entity=Sensor1 measurement=temperature,value=22.4 1465839830100400200EnvironmentalVariables,location=WUR\ Campus,entity=Sensor1 measurement=rainfall,value=0 1465839830100400200EnvironmentalVariables,location=WUR\ Campus,entity=Sensor1 measurement=temperature,value=21.7 1465839830100400800EnvironmentalVariables,location=WUR\ Campus,entity=Sensor1 measurement=rainfall,value=0 1465839830100400800EnvironmentalVariables,location=WUR\ Campus,entity=Sensor1 measurement=temperature,value=21.9 1465839830100401200EnvironmentalVariables,location=WUR\ Campus,entity=Sensor1 measurement=rainfall,value=0 1465839830100401200Python Package: leafobject
Section titled “Python Package: leafobject”LEAF builds its messages with the Python package leafobject and its class LeafPoint.
The package can be installed on its own:
pip install leafobjectThe package contains the following commands:
x = LeafPoint() x.set_measurement(...) x.set_tags(...) x.add_tag(..., ...) x.remove_tag(...) x.set_fields(...) x.add_field(..., ...) x.remove_field(...) x.set_timestamp(...) x.set_metric(...) # tag "metric" x.set_unit(...) # tag "unit" x.set_entity_tag(...) # tag "entity" x.set_organisation(...) # tag "organisation" x.set_department(...) # tag "department" x.set_value(...) # field "value" x.validate() x.to_json() x.from_json(...) x.to_line_protocol() x.parse_line_protocol(...)which can be used like so:
import datetimefrom leafobject import LeafPoint
point = LeafPoint()point.set_measurement("bioreactor")point.set_tags({"entity": "reactor1", "metric": "temperature", "unit": "C"})point.set_fields({"value": 37.2})point.set_timestamp(datetime.datetime(2021, 1, 1, tzinfo=datetime.timezone.utc))
# JSON (what LEAF publishes) and Line Protocol outputsprint(point.to_json())print(point.to_line_protocol())JSON representation (timestamp in seconds since epoch):
{ "measurement": "bioreactor", "tags": {"entity": "reactor1", "metric": "temperature", "unit": "C"}, "fields": {"value": 37.2}, "timestamp": 1609459200.0}Line Protocol representation (timestamp in whole seconds):
bioreactor,entity=reactor1,metric=temperature,unit=C value=37.2 1609459200A LeafPoint refuses data that does not fit this structure:
measurementmust be a string.- Tag values must be strings.
- Field values must be an
int,floatorstr. set_timestamptakes adatetime, or an integer epoch in seconds, milliseconds, microseconds or nanoseconds. Without it, the timestamp is the moment the point was created.to_json()andto_line_protocol()raise an error unless the measurement, the timestamp, at least one field and at least one tag are set.InfluxPoint, the name used by the older influxobject package, still works as a deprecated alias ofLeafPoint.
How LEAF publishes a LeafPoint
Section titled “How LEAF publishes a LeafPoint”A measurement is published on .../experiment/<experiment_id>/measurement/<metric>, where <metric> is the point’s metric
tag, or its measurement when there is no metric tag. When an adapter returns a list of points, LEAF groups them by that name
and publishes each group as a JSON array on its own topic. LEAF adds the organisation and department tags from the instance configuration to every point, unless the adapter set them itself. The metric tag with a value field is also what OAK expects.