Principal Investigators: Zhiming Zhao (University of Amsterdam, z.zhao@uva.nl)
Shepherds: Andrea Manzi (EGI Foundation, andrea.manzi@egi.eu)
Entry in the community requirement database: ENVRI-FAIR
About the pilot
Description of supported work
The project goal is to deploy a DevOps environment, with necessary capacity of Cloud Infrastructures and services for testing ENVRI-FAIR development. The project aims to automate the testing/integration of the FAIR data services developed by the teams in ENVRI-FAIR.
Objectives
- Automated Cloud execution for data workflow : Demonstrate it in the VREs or in ENVRI RIs (e.g., LifeWatch or others). It will help the ENVRI community to learn the EOSC services, and build practices for the other similar use cases
- Continuously testing and integration for ENVRI services: get familiar with the DevOps/Agile methodologies for software development, testing and operation
- Notebook based environment for FAIR data access and processing: : 1) provide the Jupyter service to users, with examples to access data sets and models, 2) users can perform customised experiments using the notebook services, access data, store the data, publish and share the results with the others.
Team
| Participant | Role | Name and Surname |
|---|
| ENVRI-FAIR | PI | Zhiming Zhao |
| EGI Foundation | Shepherd | Andrea Manzi |
| ENVRI-FAIR | Developers | Spiros Koulouzis, Yuandou Wang, |
|
|
|
Technical Plan
The full technical plan can be found here:
Work planned for Q1 | - Test services/tools developed in ENVRIplus in the test bed, (e.g., DRIP for OpenStack services, particle formation for OneData/Jupyter notebook, etc.)
- Get familiar with the services and test bed
- Test how to request resources based specific geo-locations
- Communicate with different sub domains/RIs to collect potential experiments on the test bed
- Make plan for the experiments
|
Work planned for Q2 | - Develop demonstrator for the automated VM planning, provisioning and service deployment. Extended it to a scientific workflow;
- Set up the DevOps pipeline, and test CI/CD for data management services (e.g., knowledge base content update, and experiments collected from Q1)
- Perform data experiments in the Jupyter hub, and perform some benchmark studies on the performance
|
Work planned for Q3 | - Develop demonstrator for a scientific workflow (e.g., from lifewatch or other sub-domains/RIs)
- Demonstrate the DevOps pipeline of Jelastic for data management services
- Demonstrate a number of Jupyter notebook examples
|
Work planned for Q4 | - Exploitation of the results to the ENVRI-FAIR communities
- Support ENVRI-FAIR community to take over the results and further develop them in their own context
- Support ENVRI-FAIR community to prepare for the further EOSC integration
|
EOSC services and providers
Providers
CESGA:
| Type | Resources | Resource Status | Service status |
|---|
| Jelastic Installation | - 1 VM 8 CPUs , 24 GB RAM , 1 TB storage via NFS
- 1 VM 12 CPUs, 24 GB RAM, 1.5 TB storage via NFS
| Provisioned | Provisioned https://jca.j.fedcloud.eu/ https://app.j.fedcloud.eu/ |
VM for Automated Cloud execution for data workflow | - 1 VM 12 CPUs, 16 GB RAM, 1.5 TB storage via NFS
| Provisioned | OK |
VM and Storage for Onezone installation | - 1 VM 8 CPUs, 32 GB RAM, 50 GB local storage
- 10 TB storage via NFS
| Provisioned | https://oneprovider-envri.datahub.egi.eu/ |
INFN-CATANIA:
| Type | Resources | Resource Status | Service status |
|---|
Extension of EGI Notebooks resources | - 4 VMs 8 CPUs, 16 GB RAM , 120GB local storage
| Provisioned | Provisioned https://envri-notebooks.fedcloud-tf.fedcloud.eu/ access to DataHub to be defined |
VM for Automated Cloud execution for data workflow | - 4 VMs 4 CPUs, 8GB RAM, 100 GB local storage
| Provisioned | OK |
Services
- EGI Cloud Compute
- EGI Notebooks
- Jelastic
- EGI DataHub