A two-year ELIXIR-funded implementation study has made it easier for scientific computing centres to adopt and deploy container protocols and services.

Containerisation is a technology that makes possible the reproducible execution of software and workflows across diverse computing environments.  Containers package applications with all necessary dependencies, including libraries and configuration files, meaning scientific workflows can be run on different systems.

The implementation study, which involved 12 ELIXIR national Nodes and the Compute and Tools Platforms, was made up of four packages of work: stakeholder engagement, user support, support for container service providers and sustainability planning. Here, we report on the work carried out to support container service providers, which aimed to establish services, processes and guidelines that align with community standards.

Training and standards alignment

The team created training materials targeted at ELIXIR Nodes and related infrastructure projects. The training covered use of containerisation in the development of scientific analysis workflows, and the execution of workflows in federated cloud environments in accordance with the Global Alliance for Genomics and Health (GA4GH) Cloud API standards.

A Galaxy service admin training programme was developed, comprising 48 modules combined in a learning path. Using this programme, three annual week-long hands-on workshops were held to train systems administrators. Support was also given to the deployment of seven national Galaxy instances, including Italy, Spain, the Czech Republic, France, Belgium, Germany and Canada.

Active alignment was maintained with the EOSC ecosystem through the EOSC Life project, and with the international standards organisation, GA4GH. In addition, the team established an GA4GH::ELIXIR Cloud landing page, a documentation hub and a tutorial repository with interactive demonstrations.

Wide ranging impact

As a result of the activities, systems administrators in scientific compute centres who deploy container-related technologies, particularly for scientific workflow execution in the cloud, now have access to ready-made solutions and comprehensive documentation. The adoption and documentation of community standards also benefits services developers wishing to integrate their solutions with container technologies.

In addition, users of scientific computing centres can more easily deploy container-related technologies for the execution of container-based scientific workflows, and they benefit from enhanced access to computing technology, a better user experience and increased reproducibility. This work means that organisations promoting community standards such as ELIXIR, EOSC and GA4GH see increased adoption of their services, standards and guidelines. Finally, the broader scientific community benefits from improved access to computing technology with better interoperability and reproducibility.

Maintaining momentum

The team have produced a list of technical recommendations which it is hoped will be widely adopted:

• Utilise software libraries such as CWL, Nextflow, Snakemake, Galaxy and WDL, for encoding scientific analysis workflows to improve the orchestration of container execution, especially in cloud environments.
• Deploy dedicated services for the execution of containers and containerized workflows at scientific compute centres.
• Embrace community standards, particularly the GA4GH Cloud APIs, the Workflow Execution Service (WES), and the GA4GH Task Execution Service (TES) API. ELIXIR was active in co-leading the development of the TES standard and continues to engage in related Cloud and Identity Access and Management standards. 
• Maintain an ongoing effort to align cloud computing strategies with other international initiatives, such as EOSC, the European Genomic Data Infrastructure project (GDI) and GA4GH, to foster greater collaboration and interoperability. ELIXIR coordinates the GDI project which is actively building products based on aligned GA4GH standards.
• Build on growing involvement of industry partners in the utilisation of such global standards and ELIXIR solutions.

The impact of providing support to container service providers is further enhanced by the other activities in the study (stakeholder engagement, user support and sustainability planning). Information on the output of these activities can be found below.

Find out more

For further information and queries please contact Jonathan Tedds jonathan.tedds [at] elixir-europe.org, Programme Manager for Scientific Computing.

View our webinar (recording to be added) - Making container services integratable, sustainable and widely used.

Supporting container service providers

• Galaxy service admin training: 48 modules combined in a learning path
• GA4GH::ELIXIR Cloud Landing page, Documentation hub and tutorial repository

Stakeholder engagement

• ELIXIR SIS-Containers - collection of training materials portfolio assembled in TeSS
• European Genomic Data Infrastructure project
• Documentation on RDMkit, Galaxy, ELIXIR Cloud and AAI

User support

• List of supported services: SIS Container User Documentation
• Formalisation of use cases: 2022 SIS Container User Documentation
• RDMkit data analysis page: Your tasks: Data analysis | RDMkit
• GA4GH::ELIXIR Cloud and AAI documentation: User guide - ELIXIR Cloud & AAI Documentation Hub

Sustainability

• Sustainability plan to follow

Nodes involved

• ELIXIR Belgium
• ELIXIR Switzerland
• ELIXIR Czech Republic
• ELIXIR Germany
• EMBL-EBI
• ELIXIR Spain
• ELIXIR Italy
• ELIXIR Finland
• ELIXIR France
• ELIXIR Greece
• ELIXIR Netherlands
• ELIXIR UK

Platforms involved

• Compute Platform
• Tools Platform