Microscopy Conference 2023

Session chairs

Prof.

Data processing workflows with nextflow and a ptychography use case

Workshop 1: Data management

IM 5: Quantitative image and diffraction data analysis

Abstract text (incl. figure legends and references) As electron microscopy experimental analysis has become more complex, involving multiple steps (each of which requiring its own analysis code) and large amounts of data, there is a need to organize and scale up this process beyond the personal computer. Ensuring reproducibility from raw data to end results is paramount. Due to the large amounts of data involved, there is a growing trend of moving from personal computers to a server (or cluster), involving interconnected machines. Traditionally, this meant writing specialized code that works on a specific cluster, i.e. the code could not easily be run on a personal computer for testing, or on a different cluster[1]. With the advent of workflow engines, some processing workflows can be implemented in a more hardware independent form[2], with greater ease and usability. This also facilitates the sharing of workflows. To demonstrate this software paradigm, we implement a ptychographic [3] reconstruction workflow with the workflow engine Nextflow [4], which is infrastructure independent. Within the workflow, processes are described and input/output relations mapped. Thus, possibilities for asynchronous execution are automatically identified and performed. Parallel execution of Nextflow processes is also possible, and done automatically when given multiple inputs that independently traverse the workflow steps. Parallelization within one dataset can also be done via explicit Nextflow constructs. One key advantage of Nextflow is the ability to mix different tools with each other. For example, the output of a Python script can be passed on to be processed by a compiled executable. Our implemented workflow has as input the raw data files from the microscope as well as a configuration yaml file which describes what settings should be applied for each step. This hierarchical file also controls which processing steps should be executed. As Nextflow does not use a deep hierarchy in its yaml configuration file, but this was desired for our workflow, custom functions were written to extend the functionality of Nextflow to accommodate this. The workflow itself has the task of preprocessing the raw microscope data and then performing a ptychographic reconstruction. Additional postprocessing steps within the workflow can then follow. The resulting workflow was developed and executed on a laptop and later executed on a local cluster consisting of one head node and two execution nodes managed by slurm [5]. A shared filesystem between the three machines was also required by Nextflow. Here, we successfully demonstrate the use of a workflows for electron microscopy data. The use of workflows can ease the transition of code from a personal computer to the cluster, while also emphasizing reproducibility and shareability, helping to manage the complexity in multistep data processing. [1] C Schiefer et al, (2020), https://doi.org/10.48550/arXiv.2006.03104 [2] F Lehmann, (2021), Proceedings of the CIKM 2021 Workshops http://ceur-ws.org/Vol-3052/short12.pdf [3] MJ Humphry et al., Nat. Comm. 3, (2012), p.1. https://doi.org/10.1038/ncomms1733 [4] P Di Tommaso et al., Nat. Biotechnol. 35, (2017), p. 316. https://doi.org/10.1038/nbt.3820 [5] Jette, M.; Grondona, M., Proceedings of ClusterWorld Conference and Expo, (June 2003). "SLURM: Simple Linux Utility for Resource Management" (PDF)

Poster

The EM glossary: a community effort to harmonize metadata in the electron microscopies

Abstract text (incl. figure legends and references) Research data has to fulfil a number of requirements for it to be fully exploitable - today and in the future. The FAIR principles [1] – findable, accessible, interoperable, and reusable – provide a conceptual framework to achieve this, but the pathway to implementation is often unclear. Here we focus on the challenge of implementing the "I" in FAIR. Interoperability is the potential for two independent agents to work on the same data in a coordinated fashion. It is especially relevant for researchers working in interdisciplinary fields, such as electron microscopy, where terminology in adjacent scientific disciplines is unknown, or where identical terms are used with differing meaning, for example, consider a mathematical or biological cell. The use and adoption of established vocabularies and semantic standards can provide solutions – however these may be infeasible to implement, non-existent, or scientifically unsuitable. This is why a number of initiatives are developing semantic artefacts that aim to describe experimental equipment, workflows, and analysis procedures in electron and ion microscopies. Harmonisation of these efforts to ensure interoperability across the community is required. To support the long-term semantic interoperability the Helmholtz Metadata Collaboration (HMC) is coordinating a community-wide effort, including over 45 scientists from more than 22 institutions across Switzerland, Austria, and Germany, and including representatives of several NFDI consortia [2], to create a joint resource to harmonize semantics in the field of electron and ion microscopies. This poster will report on this community effort - the EM glossary group [3]. We will describe how we achieve consensus on terms commonly used in electron and ion microscopies via a remote, collaborative workflow based on the platform GitLab. A text-mining approach has been used to evaluate EM image metadata and prioritise terms which are widely used in the community. We will show results, including some of our harmonized definitions, and provide details of our implementation strategy. The developed resource will provide harmonized and machine actionable semantics to support specific development efforts such as metadata schemas or ontologies in their respective fields. This will benefit experimental scientists by providing clarity when annotating and re-using research data; and by facilitating communication when conducting interdisciplinary studies. Do you work in electron and/or ion microscopy or related data analysis? Follow us on twitter @helmholtz_hmc or tweet #EMGlossary. You want to get involved in the development and/or join our meetings? Send an email to hmc@fz-juelich.de or hmc-matter@helmholtz-berlin.de 1. Wilkinson, M.D. et al. Scientific Data. http://dx.doi.org/10.1038/sdata.2016.18 (2016) 2. NFDI: Nationale Forschungsdateninfrastruktur. https://www.nfdi.de/ 3. EM Glossary GitLab repository: https://gitlab.hzdr.de/em_glossary

PS 18
Poster session WS 1: Data management

Appointment

Session chairs

Programme

PS 18Poster session WS 1: Data management

Appointment

Session chairs

Programme

PS 18
Poster session WS 1: Data management