Release Notes for Salvus 0.11

Released: March 31st 2020

Salvus 0.11 contains a large number of internal improvements, bug fixes, and optimizations as well as already some preparations for the next release. This pages documents the major user facing changes of the Salvus software suite.

Headline Features

GPU support for CUDA-capable GPUs

GPU support for CUDA capable GPUs is now available for forward modelling applications! The following system requirements apply:

• CudaToolkit 10.x and the associated CUDA runtime libraries
• An NVIDIA GPU with a compute capability >= 6.0

If these requirements are satisfied, GPU support can be enabled by updating your site configuration file as follows:

Copy
[sites.site_name]
   ...
   use_cuda_capable_gpus = true
   ...
   [sites.site_name.site_specific]
   ...

With this new line in place, go ahead and re-run init-site for the Salvus Flow site in question. If this completes successfully, you're up and running with GPU support!

The relationship between the number of MPI ranks and the number of GPUs used is as follows. First, when the simulation is initialized, the number of CUDA-capable GPUs on each compute node is queried. Following this, the ranks which reside on each compute node are assigned in a round-robin fashion to the CUDA devices visible to the node. For example, if a simulation is run on two compute nodes, and each node has two CUDA-capable GPUs:

• Running with 1 rank per node will result in a total of two GPUs being used
• Running with 2 ranks per node will result in a total of four GPUs being used
• Running with 12 ranks per node will result in a total of four GPUs being used

For the best performance we recommend running Salvus with 1 rank per GPU. This recommendation is however expected to change as the GPU capability of Salvus evolves, so please either check back here, or contact the development team through the designated support channels, for updates to these best-practice guidelines.

Single Python Package

Up to version 0.10 Salvus came with several Python packages: SalvusMesh, SalvusFlow, SalvusFEM and SalvusToolbox

There is now a single Salvus Python package.

Before:

Copy
import salvus_mesh
import salvus_flow
import salvus_fem
import salvus_toolbox

Now:

Copy
import salvus.mesh
import salvus.flow
import salvus.fem
import salvus.toolbox

Quality of Life Improvements to Site Handling

We greatly improved all aspects of settings up and updating local as well as remote sites.

• salvus-flow init-site: Better checks and error messages for all aspects including SSH connections and site setup. Will now emit actionable suggestions of what to do when something breaks amongst other things.
• salvus-flow add-site: Can now automatically install Salvus on a remote site.
• salvus-flow upgrade: Command to upgrade the local Python installation for any future update.
• salvus-flow upgrade-site: Command to upgrade Salvus on a remote site for all future updates.

Ocean Loading

Salvus can now simulate the effect the weight of the ocean (or another load) has on the wave field.

Other User-Facing Improvements

SalvusCompute

• Parallel mesh loading powered by PETSc v3.13 and DMPlex.
• Consistent naming conventions in JSON schema.
• Anisotropic smoothing on spherical and cartesian domains.
• Fix error handling for invalid material coefficients
• Much faster TTI gradient computation.

SalvusFlow

• The environment variables on slurm are now explicitly passed in the sbatch file. Thus the environment variables set in the startup dotfiles no longer pollute the Salvus run.
• The module switching/(un)loading in the LSF site can now also happen inside the batch file.
• Ability to force update jobs and job arrays.
• New command to remove all Salvus managed files from a remote site.
• Added API and CLI function to cancel all running jobs on a given site.
• Allow sites with max_ranks set to 1.
• Ability to recover from ProxyCommand failures which for some reason also always seem to happen on KeyboardInterrupt exceptions.
• Sources and receivers are now immutable.
• Added simplified smoothing interface.
• Better error message when the JSON output parsing fails.
• Allow "SITE_NAME:..." remote file in the waveform simulation simple_config helper objects.
• Raising a proper error message if one tries to add side set sources/receivers to a waveform simulation object without a locally a available mesh.

SalvusMesh

• Auto-band is now the default in the mesher.
• Removing the deprecated salvus.mesh.simple_config.
• More flexible way to determine the data set names in the global topography files.
• Fix handling of fluid regions for absorbing BCs.

Varia

• Nicer logging interface for all of SalvusPy.
• Salvus version number included in all output files.