Version:

License Server

The various pieces of Salvus need to communicate with one of Mondaic's servers to ensure a valid license. Please make sure outgoing HTTPS connections from whichever machine Salvus runs on are allowed to this server:

  • URL: https://l.mondaic.com
  • Port: 443 (HTTPS)

License File

Your license credentials are stored in ~/.salvus-licenses.toml (please note the leading dot), which is a TOML file with the following group for each licensed product:

Copy
[[license]]
product = "SomeProduct"
product_license_version = "1.2.3"
server_url = "https://l.mondaic.com/licensing_server"
username = "SomeUser"
password = "SomePassword"
group = "SomeGroup"

You can create this file yourself, otherwise Mondaic's downloader will offer to create it for you and auto-fill it.

No Internet Connection (Dark Sites)

If you have no internet connection you can still run Salvus on dark machines using one of two solutions:

  • SalvusFlow can mint one-time use tokens to run SalvusCompute on dark sites. This requires SalvusFlow to run on machines with internet access (i.e. your laptop or workstation). This is a fully integrated and supported solution, and is the recommended approach. See "Tokenized Licensing" below.

  • Proxy the internet connection.

Both possibilities are described in the detail in the following.

Tokenized Licensing

In this scenario, SalvusFlow would communicate with the license server to generate a single-use token to run SalvusCompute on dark sites. This is fully automatic and to activate it just use the following setting in the site's configuration:

use_license_tokens = true

The only downside to this approach is the requirement to specify a wall-time for each run. Otherwise this works the same as normal Salvus runs.

Proxy the Connection

Clarify with your Admin

Please make sure this is acceptable at your local compute site by talking to your admin. This recipe will open a SOCKS5 proxy to the internet either on the login node or on a compute node which might violate the policy of some compute centers.

USE AT YOUR OWN RISK!

Most newer SSH implementations can open SOCKS5 (https://de.wikipedia.org/wiki/SOCKS) proxies over the SSH connection. Thus local internet access can be shared with remote resources that do not have internet access. All of Salvus as well as pip and conda do support it so if applicable it is very convenient.

Proxy internet to the login node

The following command will open a proxy on port 12345 so that remote_site can access the local internet. SSH can usually only open a proxy from remote to local so we first create a SOCKS5 proxy on localhost to localhost and then forward that port. It is a complicated command but the end result is that there it now a SOCKS5 proxy available on remote_site.

$ ssh -t -D 12345 localhost ssh -R 12345:localhost:12345 remote_site

One requirement for that command is that you can SSH into your own computer. On Linux this usually means installing the openssh-server package, on OSX it means enabling remote login in the sharing preferences.

The only thing left to do is to set the all_proxy environment variable on remote_site:

export all_proxy=socks5://localhost:12345

Proxy of login node to compute node

In the common environment where the login nodes have internet access but the compute nodes do not, the compute nodes can oftentimes SSH to the login nodes to create a SOCKS proxy. The following recipe should get you started.

#  Make sure there is an ssh keypair on the machine. Agent forwarding will
#  not work on the compute nodes.
#
#  Make one for example with:
#
#  $ ssh-keygen -t rsa -b 4096 -C "[email protected]"
#
#  Also copy the key and make sure there is a setting in ~/.ssh/config!
#
#  $ ssh-copy-id LOGIN_NODE_HOST_NAME

export LOGIN_NODE_HOST_NAME=...
export PORT=9123

# Socks proxy to log-in node.
ssh -fNM -D $PORT $LOGIN_NODE_HOST_NAME

# Store process id of ssh connection.
ps -x | grep "ssh -fNM -D $PORT $LOGIN_NODE_HOST_NAME" | \
    grep -v grep  | cut -d " " -f1 > pid.txt

# Use it.
export all_proxy=socks5://localhost:$PORT

# Launch some piece of Salvus.
python -c 'import salvus_mesh'

# Kill the ssh process again. Otherwise the job will run until it hits the wall
# time.
kill `cat pid.txt`

Troubleshooting

CA Store Issues

Salvus communicates with the server over HTTPS - thus some central authority (CA) deciding which certificates to trust is needed. Operating systems usually have this - Salvus tries its best to find the CA cert store of the systems it runs on - if it fails you might have to manually provide it via the SALVUS_CA_BUNDLE_PATH env variable:

# If your operating system provides no CA file, download for example the curl
# one. This is a converted version of Firefox' CA cert store. If you are
# security concious (which you should be) please clarify this with your local
# admin.
$ wget https://curl.haxx.se/ca/cacert.pem

# Set the environment variable to the filename.
$ export SALVUS_CA_BUNDLE_PATH=cacert.pem

# Salvus will now use this CA file.
$ ./salvus compute ...
PAGE CONTENTS