As a user I want to be able to run jobs and workflows on cloud-based resources

As a user I want my jobs/workflows to transparently burst onto external cloud resources, such as EGI FedCloud or a public cloud, when my local resources are at capacity

The Fusion CC wish to demonstrate making use of EOSC computational and storage resources for running containerised modelling applications (primarily HPC and HTC). This requirement derives from the fact that local resources are not scaled for peak demand and we wish to use the infrastructure provides by EOSC (and public cloud providers) as a scalable, non vendor specific resource.

At a high level, this is an opportunistic use case where we wish to make use of any spare resources at sites, and thus going through an ordering process would be non optimal since the user would not know local resources are exhausted until they submitted their job. It maybe that some sort of framework agreement would be needed between the community and the sites to allow this opportunistic use beyond the small number of cores already presented through EGI.

Different parts of the workflows may involve different computational requirements, from simple single core machines to many core/multi-node. In the first case we would request resources on a single site, but only instantiate the number of machines required for a specific element of the workflow. It is desirable to develop this further to allow the instantiation of machines for different parts of the workflow to meet the requirements of each step. We are also interested in using both using traditional workflows and workflows within the ITER Integrated Modelling and Analysis Suite (IMAS) which is anticipated to become the standard framework for both modelling and analysis work in the future.

In most cases the steps of the workflow communicate through files, with each stage producing it's own unique file which acts as an input to the next stage. While running at a single site, it would be possible to request storage at that site of appropriate size. However, in the desirable case of different stages of the workflow running at different sites, either storage system accessible to many sites will be required for these intermediate files, or they would need to be transferred between sites.

Final output data (and possibly intermediate data) should be accessible to the end user.

As a site data manager I need am looking at how we can improve user and computational access to experimental data, in addition to being driven to allow more open access to users beyond the fusion community by either national or international funding bodies (or both!). However, in common with most science disciplines my site places an embargo on experimental data to allow researchers to publish. In addition, some data will not be made public where it has no scientific value (engineering tests for example), or where work is done on behalf of industry. Significant analysis work is performed on the MARCONI/Fusion supercomputer based at CINECA and for data sets which will be accessible it would be beneficial to my users if data could be hosted there. In addition, we want to offload public data to partner sites for hosting and access to the wider science community and general public, so that data used by the fusion community is kept on site and only accessed by fusion users. This, combined with the restricted roadmap for tape technologies is pushing me towards replication as a means of bit preservation in the longer term. the community already has a data access mechanism (UDA) which it uses and must be usable at each site where the community will access data. General access will be via HTTP.

Thus as a site manager I would ideally like to put a full copy of my data on a trusted site which will prevent unauthorised access to the data (although not the metadata) during the embargo period but will make it accessible following that. That site should be able to provide me with data download statistics on an annual basis as part of my reporting to senior management and fundholders. Additionally, I would like that data to be copied from the trusted site to CINECA so it can be used optimally for analysis on the MARCONI computer and I would like a third copy to ensure there are four copies on my data availability for high availability (one local and three off site copies).

On submission of a containerised workflow, sufficient resources are provisioned on a remote site(s) to allow execution of that workflow. The users home credentials (or the services services credentials) must be accepted on the remote site(s)

Orchestrator/Kubernetes

EGI Fed Cloud/Other cloud

Suitable AAI mechanism (note the fusion community does not have a centralised IdP or AAI system, but each site provides its own authentication based on username/password)

PROMINENCE cloud execution service

During workflow execution, intermediate files should be written to a location which will be accessible to later stages of the workflow. ..

The final results should also be accessible at the users home institute