Introduction defining context and scope

System-level environmental science involves large quantities of data, often diverse and dispersed insofar as there are many different kinds of environmental data commonly held in small datasets. In addtion, the velocity of data gathered from detectors and other instruments can be very large. Data-driven experiments require not only access to distributed data sources, but also parallelisation of computing tasks for the processing of data. The performance of these applications determines the productivity of scientific research and some degree of optimisation of system-level performance is urgently needed by the RI projects in ENVRI+ as they enter production.

This topic focuses on how to improve many of the common services needed to perform data analysis and experiments on research infrastructure, with an emphasis on how data is delivered and processed by the underlying e-infrastructure. There needs to be consideration of the Service Level Agreements (SLAs) offered by e-infrastructure, and of the available mechanisms for controlling the system-level quality of service (QoS) offered to researchers. This topic should therefore focus on the mechanisms available for making decisions on resources, services, data sources and potential execution platforms, and on scheduling the execution of tasks. The semantic linking framework developed in Task 5.3 on linking data, infrastructure, and the underlying network can be used to embed the necessary intelligence to guide these decision procedures (semi-)autonomously.

Ultimately, based on the relevant task (7.2) of the ENVRI+ project, we will need to:

1. Provide an effective mapping between research-level quality attributes (ease-of-use, responsiveness, workflow support) to infrastructure-level quality attributes on computing, storage and network services provided by underlying e-infrastructures.
2. Define test-bed requirements for software and services, and identify conditions for operating final software and services inside each domain, and between multiple domains.
3. Extend and customise existing optimisation mechanisms for computing and storage resources, and provide an effective control model between processes of data analysis and the underlying e-infrastructure resources, making the application performance as easy as possible to control at runtime.

Thus the purpose of the technology review in ENVRI+ from the optimisation perspective is to determine two things:

1. What the RI projects already have at their disposal for effective data access, delivery and processing.
2. What facilities are offered by current e-infrastructures that can meet RI projects' processing and optimisation requirements.

The optimisation section of the ENVRI+ technology review focuses on the second point above; the first point should be addressed in other sections, particularly data processing.

Change history and amendment procedure

The review of this topic will be organised by . He will partition the exploration and gathering of information and collaborate on the analysis and formulation of the initial report. Record details of the major steps in the change history table below. For further details of the complete procedure see item 4 on the Getting Started page.

Note: Do not record editorial / typographical changes. Only record significant changes of content.

Date	Name	Institution	Nature of the information added / changed
3/1/2016		UvA	Provided introduction, context and scope for optimisation topic.
18/3/2016		UvA	Initial draft for technology review report.

Sources of information used

Places you have gone to for information e.g., RDA; standards bodies; GEOSS; existing RIs and projects and the technologies they use; technologies available from service providers (0.5 pages).

Two-to-five year analysis

Of state of the art and trends based on sources and experience. A distillation of surveyed information leading to your conclusions and recommendations about what should be done in ENVRIplus (0.5 - 1.5 pages) structured internally as appropriate for the topic but with at least the following headings.

State of the art

One paragraph describing the current state of the art.

Trends

Subsequent headings for each trend (if appropriate in this HL3 style)

Paragraph(s) describing a trend(s).

Problems to be overcome

Exploiting virtual (cloud) resources effectively

Conscripting elastic virtualised infrastructure services permits more ambitious data analysis and processing workflows, especially with regard to 'campaigns' where resources are enlisted only for a specific time period. Resources can be acquired, components installed, and processes executed with relatively little configuration time provided that the necessary tools and specifications are in place. These resources can then be released upon the completion of the immediate task. However in the research context, it is necessary to minimise the oversight and 'hands-on' requirement for researchers, and to automate as much as possible. This requires specialised software and intelligent support systems; such software either does not current exist, or operates still at too low a level to significantly reduce the technical burden imposed on researchers, who would presumably rather concentrate on research rather than programming.

Sub-headings as appropriate in HL3 style (one per problem)

A short paragraph describing a problem to be overcome or barrier in the way of progress. 

Details underpinning above analysis

Please supply here any additional information that can help to justify the previous section e.g., references to material that someone can look up for themselves.  

Sketch of a longer-term horizon

e.g., 5-10 years ahead. Your best judgement about the future direction of technology and research trends (0.5 - 1 page).

Relationships with requirements and use cases

Link your analysis of the topic with particular identified requirements and use cases, as this will increase the relevance and help others understand your insights. Consider using tables to do this (0.5 - 1 page).

Summary of analysis highlighting implications and issues

1. It is possible to automate large portions of research activity—however this is contingent on there being good formal descriptions of data and processes, and on there being good tool support for initiating and informing the automated procedures with regard specific experiments and applications.

Bibliography and references to sources

Insert numbered list of sources / references.