Shared space for E-CAM related material

The Space team has made the following datasets and collections publicly available. You must be a logged-in member of the Space to access all the datasets and collections.


Viewing most recent datasets View All Datasets

Thumbnail of High Throughput Computing with Dask

High Throughput Computing with Dask

- Alan O'Cais (Forschungszentrum Juelich GmbH)
- David Swenson (École Normale Supérieure de Lyon)

High-throughput (task-based) computing is a flexible approach to parallelization. It involves splitting a problem into loosely-coupled tasks. A scheduler then orchestrates the parallel execution of those tasks, allowing programs to adaptively scale their resource usage. Individual tasks may themselves be parallelized using MPI or OpenMP, and the high-throughput approach can therefore enable new levels of scalability.

Dask is a powerful Python tool for task-based computing. The Dask library was originally developed to provide parallel and out-of-core versions of common data analysis routines from data analysis packages such as NumPy and Pandas. However, the flexibility and usefulness of the underlying scheduler has led to extensions that enable users to write custom task-based algorithms, and to execute those algorithms on high-performance computing (HPC) resources.

This workshop will be a series of virtual seminars/tutorials on tools in the Dask HPC ecosystem. The event will run online via Zoom for registered participants ("participate" tab) and it will be live streamed via YouTube at https://youtube.com/playlist?list=PLmhmpa4C4MzZ2_AUSg7Wod62uVwZdw4Rl.


21 January 2021, 3pm CET (2pm UTC):
Dask - a flexible library for parallel computing in Python (YouTube link: https://youtu.be/Tl8rO-baKuY )

4 February 2021, 3pm CET (2pm UTC):
Dask-Jobqueue - a library that integrates Dask with standard HPC queuing systems, such as SLURM or PBS (YouTube link: https://youtu.be/iNxhHXzmJ1w )

11 February 2021, 3pm CET (2pm UTC) :
Jobqueue-Features - a library that enables functionality aimed at enhancing scalability (YouTube link: https://youtu.be/FpMua8iJeTk )

Thumbnail of Dynamic Load-Balancing for Parallel Particle Simulations: from Motivation to Application

Dynamic Load-Balancing for Parallel Particle Simulations: from Motivation to Application

Scalability of parallel applications depends on a number of characteristics, among which is efficient communication, equal distribution of work or efficient data lay-out. Especially for methods based on domain decomposition, as it is standard for, e.g., molecular dynamics, dissipative particle dynamics or particle-in-cell methods, unequal load is to be expected for cases where particles are not distributed homogeneously, different costs of interaction calculations are present or heterogeneous architectures are invoked, to name a few. For these scenarios the code has to decide how to redistribute the work among processes according to a work sharing protocol or to dynamically adjust computational domains, to balance the workload.

The seminar will provide an overview about motivation, ideas for various methods and implementations on the level of tensor product decomposition, staggered grids, non-homogeneous mesh decomposition and a recently developed phase field approach. An implementation of several methods into the load balancing library ALL, which has been developed in the Centre of Excellence E-CAM, is presented. A use case is shown for the Materials Point Method (MPM), which is an Euler-Lagrange method for materials simulations on the macroscopic level, solving continuous materials equations.

The seminar is organised in three main parts:

- Overview of Load Balancing

- The ALL Load Balancing Library

- Balancing the Materials Point Method with ALL

Thumbnail of The E-CAM issue of Comics&Science

The E-CAM issue of Comics&Science

What are “simulations” in advanced research? Is High Performance Computing the Holy Grail of scientific simulations? Let’s find out together through this unique Comic story and book

Thumbnail of HPC Carpentry - Introduction to HPC

HPC Carpentry - Introduction to HPC

This workshop is an introduction to using high-performance computing systems effectively. We obviously can’t cover every case or give an exhaustive course on parallel programming in just two days’ teaching time. Instead, this workshop is intended to give students a good introduction and overview of the tools available and how to use them effectively.

By the end of this workshop, students will know how to:

Use the UNIX shell (also known as terminal or command line) to operate a computer, connect to a cluster, and write simple shell scripts.
Submit and manage jobs on a cluster using a scheduler, transfer files, and use software through environment modules.

Thumbnail of Introduction to EasyBuild Tutorial

Introduction to EasyBuild Tutorial

The tutorial will cover what we outlined in the tutorial proposal we made for ISC'20. Our proposal got accepted but since ISC'20 has been transformed into an online conference without tutorials, the tutorial has been postponed until ISC'21. We did not want to let this opportunity go to waste however...

If you are interested in learning more about the basics of EasyBuild, and if you are not afraid to make your hands dirty by following with the hands-on exercises, please try it out!

Thumbnail of Moving to online training

Moving to online training

This is a collection of (hopefully) useful information to help transition to online training. Guides to help with this are being rapidly created as the Covid-19 crises evolves, we try to keep the information here moderated to avoid overwhelming people.

If you know of something that could be of value in this list, please email Alan O'Cais (a.ocais@fz-juelich.de)


Viewing most recent collections View All Collections

Thumbnail of Integration of ESL modules into electronic-structure codes

Integration of ESL modules into electronic-structure codes

The evolutionary pressure on electronic structure software development is greatly increasing, due to the emergence of new paradigms, new kinds of users, new processes, and new tools. Electronic structure software complexity is consequently also increasing, requiring a larger effort on code maintenance. Developers of large electronic structure codes are trying to relieve some complexity by transitioning standardized algorithms into separate libraries [BigDFT-PSolver, ELPA, ELSI, LibXC, LibGridXC, etc.]. This paradigm shift requires library developers to have a hybrid developer profile where the scientific and computational skill set becomes equally important. These topics have been extensively and publicly discussed between developers of various projects including ABINIT, ASE, ATK, BigDFT, CASTEP, FHI-aims, GPAW, Octopus, Quantum Espresso, SIESTA, and SPR-KKR.

High-quality standardized libraries are not only a highly challenging effort lying at the hands of the library developers, they also open possibilities for codes to take advantage of a standard way to access commonly used algorithms. Integration of these libraries, however, requires a significant initial effort that is often sacrificed for new developments that often not even reach the mainstream branch of the code. Additionally, there are multiple challenges in adopting new libraries which have their roots in a variety of issues: installation, data structures, physical units and parallelism - all of which are code-dependent. On the other hand, adoption of common libraries ensures the immediate propagation of improvements within the respective library’s field of research and ensures codes are up-to-date with much less effort [LibXC]. Indeed, well-established libraries can have a huge impact on multiple scientific communities at once [PETSc].

In the Electronic Structure community, two issues are emerging. Libraries are being developed [esl, esl-gitlab] but require an ongoing commitment from the community with respect to sharing the maintenance and development effort. Secondly, existing codes will benefit from libraries by adopting their use. Both issues are mainly governed by the exposure of the libraries and the availability of library core developers, which are typically researchers pressured by publication deliverables and fund-raising burdens. They are thus not able to commit a large fraction of their time to software development.

An effort to allow code developers to make use of, and develop, shared components is needed. This requires an efficient coordination between various elements:

- A common and consistent code development infrastructure/education in terms of compilation, installation, testing and documentation.
- How to use and integrate already published libraries into existing projects.
- Creating long-lasting synergies between developers to reach a “critical mass” of component contributors.
- Relevant quality metrics ("TRLs" and “SRLs”), to provide businesses with useful information .

This is what the Electronic Structure Library (ESL)[esl, esl-gitlab] has been doing since 2014, with a wiki, a data-exchange standard, refactoring code of global interest into integrated modules, and regularly organizing workshops, within a wider movement lead by the European eXtreme Data and Computing Initiative [exdci].

[BigDFT-PSolver] http://bigdft.org/Wiki/index.php?title=The_Solver_Package
[ELPA] https://gitlab.mpcdf.mgp.de/elpa/elpa
[ELSI] http://elsi-interchange.org
[LibXC] http://www.tddft.org/programs/libxc/
[LibGridXC] https://launchpad.net/libgridxc
[PETSc] https://www.mcs.anl.gov/petsc/
[esl] http://esl.cecam.org/
[esl-gitlab] http://gitlab.e-cam2020.eu/esl
[exdci] https://exdci.eu/newsroom/press-releases/exdci-towards-common-hpc-strategy-europe

Thumbnail of ESDW in Quantum dynamics (2019)

ESDW in Quantum dynamics (2019)

Quantum molecular dynamics simulations are pivotal to understanding and predicting the microscopic details of molecules, and strongly rely on a combined theoretical and computational effort. When considering molecular systems, the complexity of the underlying equations is such that approximations have to be devised, and the resulting theories need to be translated into algorithms and computer programs for numerical simulations. In the last decades, the joint effort of theoretical physicists and quantum chemists around the challenges of quantum dynamics made it possible to investigate the quantum dynamics of complex molecular systems, with applications ranging from energy conversion, energy storage, organic electronics, light-emitting devices, biofluorescent molecules, or photocatalysis, to name a few.
Two different strategies have been successfully applied to perform quantum molecular dynamics: wavepacket propagation or trajectories. The first family of methods includes all quantum nuclear effects, but their computational cost hampers the simulation of systems with moderate number of more than 10-12 degrees of freedom. The method coined multi-configuration time-dependent Hartree (MCTDH) constitutes one of the most successful developments in this field and is often considered as a gold standard for quantum dynamics [1]. Other strategies for wavepacket propagation try to identify procedures to optimize the “space” where the wavefunction information is computed, such that Cartesian grids can be replaced with Smolyak grids [2]. The second family of methods introduces the idea of trajectories as a way to approximate the nuclear subsystem, either classically or semiclassically, and is exemplified by methods like the trajectory surface hopping and Ehrenfest schemes [3], or the more accurate methods coupled-trajectory mixed quantum-classical (CT-MQC) [4] and quantum-classical Liouville equation (QCLE) [5].
From a computational perspective, both families of methods require extensive electronic structure calculations, as the nuclei move under the effect of the electronic subsystem, either “statically” occupying its ground state or “dynamically” switching between excited states. Solving the quantum nuclear dynamics equations also becomes in itself very expensive in the case of wavepacket propagation methods. Contrary to other, more consolidated, areas of modeling, quantum dynamics simulations do not benefit from established community packages and most of the progress occurs based on in-house codes, difficult to maintain and with limits in optimization and portability. One of the core actions of E-CAM has been to seed a change in this situation, by promoting systematic developments of software, providing a repository to host and share code, and fostering collaborations on adding functionalities and improving the performance of common software scaffolds for wavepacket (Quantics) and trajectory-based (PaPIM) packages. Collaborations on developments on other codes have also been initiated. This workshop aims at continuing and extending these activities based on input from the community.

Extended Software Development Workshops

This gathers together all of the Extended Software Development Workshops (ESDW) that E-CAM holds. These workshops train scientists in the development of modular codes for high performance machines. Documentation and testing are key components of the workshops and the associated on-line manuals and test cases are made available through the E-CAM software repository.

Thumbnail of Software Development Best Practices

Software Development Best Practices

This is a collection of some material that is intended to support the software development aspect of an ESDW

The following datasets have been published through this Space and any affiliated Spaces.

Filter by:


Members: 71

Collections: 4

Datasets: 104

External Links

No External Links



Advanced Configuration

Expiration Enabled: empty
Time To Live (hours): empty