Ambition

The ICOS - eLTER Competence Centre groups two scientific communities: 

• The Integrated Carbon Observation Systems (ICOS) research infrastructure: https://www.icos-ri.eu
• Long-Term Ecosystem Research in Europe (eLTER) community: http://www.lter-europe.net/

The ICOS group integrates and tests generic services from the EOSC-hub portfolio with the ICOS Carbon portal to provide a scalable environment for researchers wishing to monitor and analyse carbon processes. The CC expects to have the first runs of the portal with actual near real-time data performed before the end of 2018 and to start full production or near real-time data in January 2019. The resulting data is to be automatically ingested and be published in the ICOS Carbon Portal and at the Thematic Center. At this stage the integration of the data flow from EAA can also start. The same data workflow adapted for final data quality data will be implemented in February 2019. The final production ready version of the workflow should be finished by M18.

The eLTER group ...<add as appropriate>

ICOS

Architecture & EOSC-hub technologies considered/assessed

Requirements for EOSC-hub

Technical and capacity requirements

None at the moment. Requirements are handled at the IT provider of the CC: CSC.

• Containerized data processing module, Ansible orchestration, OpenStack
• Community Linked Open Data repository (RDF, SparQL endpoint) publishes to global (meta)data systems (GEO, WMO, DataCite)
• FAIR, data license, identification, usage tracking essential

Validation plan

During the the first year of EOSC-hub the ICOS CC tested the new implementation of B2Safe at CSC and integrated this using the improved http API to the data object storage workflow in parallel with the production level iRODs data storage at PDC. Data transfer to the new B2Safe implementation has been very reliable except for 1 interruption and a few announced maintenance periods.
The CC worked on the automated receipt of the raw ecosystem flux data directly from the field data loggers and system into the Carbon Portal into the data staging area. From the staging area the data is transferred in daily packages to the ICOS CC object data store that is connected to B2Safe as its trusted repository.
The CC transferred the solution for ecosystem data processing developed in ENVRIplus based of the D4Science platform to the Cloud system, using the orchestration tools developed for the ICOS STILT footprint tool. Personnel from the ICOS Ecosystem thematic center have received training in the orchestration and integration with the ICOS semantic link framework that through persistent identifiers gives access to the input and output data through the B2SAFE repository. A preliminary test indicated that the processing of the daily raw data from 100 stations into the near real-time product could be completed within the hour, using only 20 cores.