1. Background

1.1 Short description

Data Acquisition services and in particular the preparation of data transfer (ENVRI RM: prepare data transfer) prior to data transmission are not yet sufficiently standardized. This hinders efficient, multi RI (Research Infrastructure) data processing routines such as Data Quality checking . This use case intends to promote standardization and move the standardization level close to the sensor. Objectives include:

• Standardized data transmission using OGC SWE Transactional SOS (Sensor Observation Service) as priority standard as well as using the Semantic Sensor Network (SSN) ontology . Both will be implemented and tested.
• Generic quality control (QC) routines suitable for multiple RIs (e.g. EMSO, EuroARGO, ANAEE, etc.) will be defined and implemented at own and/or EGI platforms.
• QC routines will be used to process these standardized data transmission streams to enable Near Real Time (NRT) quality control routines on raw data.

1.2 Contact

1.3 Use case type

The use case will be an implementation case.

1.4 Scientific domain and communities

Scientific domain

All

Communities

Data Acquisition, Data Service Provision

Behaviours

Relevant community behaviours: Instrument Configuration, Data Collection, Data Quality Checking, Semantic Harmonization

Relevant community roles: Sensor, Sensor Network, Measurement Model Designer, Data Acquisition Subsystem, Data Curator, Semantic Curator

2. Detailed description

Objective and Impact

The use case will move the standardization level close to the sensors of RIs, thus allow the implementation of common, generic data processing routines such as NRT QC .

...

Furthermore, it will implement generic quality control procedures such as those defined within WP 3.3, and make use of the standardized data transmission formats to perform generic, cross-RI NRT QC routines, tag the controlled data with appropriate data quality flags, again using the standard formats mentioned above.

Challenges

• Provide data as well as metadata on sensors and data in a standardized way
• Enable data transmission which is standardized, sufficiently described with metadata as well as resource friendly.
• Provide generic data quality routines that are relevant to most RIs

Detailed scenarios

The use case will test two scenarios. One will be based on the Sensor Web Enablement (SWE) suite of standards (e.g. SOS) while the other will use the Semantic Sensor Network (SSN) ontology.

...

The Semantic Sensor Network (SSN) ontology approach

Required services:

We have described an ENVRIplus Implementation Case that aims at embedding standards for the encoding and format of observation data into sensing devices. Of specific focus are standards by the OGC - in particular SensorML, Observation & Measurements, and Sensor Observation Service - and recommendations by the W3C - specifically the Semantic Sensor Network ontology. Embedding such standards into sensing devices enables the acquisition of observation data from sensing devices natively encoded and formatted following these standards. This will reduce the number of translations required during data acquisition. Given standardized streams of observation data, the Implementation Case investigates the execution of generic data processing routines on data streams. Of interest are routines for near real-time quality control (NRT QC).

...

We also intend to partner with EGI which could serve as platform for the deployment of Storm topologies on a distributed computer network.

[1] http://storm.apache.org/

Tasks:

• Define SSN templates for sensor data and metadata
• Implement the representation of SSN-conformant sensor metadata and data in sensors (dummy and/or at use case platform, in any case consistent with the sensor networks addressed in Task 1.4)
• Implement transmission of RDF stream data (stream of triples) to a triple store
• Provided access to SSN-conformant sensor metadata and data via a SPARQL endpoint

Near real-time quality control

Tasks:

• Define some basic NRT QC procedures (based on Zandvoort discussion, the WP3.3 evaluation (and e.g. https://ioos.noaa.gov/project/qartod/)
• Implement a SOS InsertObservation unpacking and transformation service
• Implement some basic QC services which use transformed SOS data as input
• Evaluate EGI services for applicability (e.g. virtual machines)
• Deploy both the transformation service and the QC service at MARUM and/or EGI
• Testing and analysis
• Based on the results from implementing QC services on SOS data, prototype examples for NRT QC on RDF streams

Technical status and requirements

The use case involves EGI and requires, e.g., the use of a scalable data processing environment via virtual machines.

Implementation plan and timetable

1.Month 5: Implementation of SOS based data transmission

...

5.Month 18: Deployment of NRT quality control routines

Expected output and evaluation of output

A generic NRT QC service capable of accepting standardized SOS data or SSN RDF data streams will be ready.

...