import os
import os, pathlib
from frequency_band import FrequencyBand
from salvus import namespace as sn
SIMULATION_TIME_IN_SECONDS = 1200.0
SALVUS_FLOW_SITE_NAME = os.environ.get("SITE_NAME", "local")
RANKS_PER_JOB = 4
PROJECT_DIR = "project_dir_central_europe"
fband_file = pathlib.Path("./frequency_band_70_120.pkl")
fband = FrequencyBand.load(fband_file)
fband
FrequencyBand(min_frequency_in_hertz=0.008333333333333333, max_frequency_in_hertz=0.014285714285714285)
SELECTED_EVENTS = [
"event_CRETE_GREECE_Mag_6.15_2015-04-16-18-07",
"event_ICELAND_REGION_Mag_5.75_2020-06-20-19-26",
"event_STRAIT_OF_GIBRALTAR_Mag_6.38_2016-01-25-04-22",
]
# Load the project with the data from the previous parts.
p = sn.Project(path=PROJECT_DIR)
events
)misfit_configuration
)prior_model
)mapping
)method
)preconditioner
)job_submission
)InverseProblemConfiguration
collects these parameters to setup the inverse problem.
Often enough, it is hard to choose the best setting a-prior, which is why the inverse problem is constructed as a tree that allows to branch off an ongoing inversion with changed parameters, or to run several scenarios concurrently.p += sn.InverseProblemConfiguration(
name=f"inv_{fband.period_band_name}",
events=SELECTED_EVENTS[:2],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
wavefield_compression=sn.WavefieldCompression(
forward_wavefield_sampling_interval=10
),
prior_model=f"initial_model_{fband.period_band_name}",
mapping=sn.Mapping(
scaling="relative_deviation_from_prior",
inversion_parameters=["VSH", "VSV", "VP"],
map_to_physical_parameters={
"VPV": "VP",
"VPH": "VP",
},
source_cutout_radius_in_meters=300000,
receiver_cutout_radius_in_meters=50000,
),
method=sn.TrustRegion(initial_trust_region_linf=0.02),
preconditioner=sn.ModelDependentSmoothing(
smoothing_lengths_in_wavelengths={
"VSH": [0.2, 0.5, 0.5],
"VSV": [0.2, 0.5, 0.5],
"VP": [0.2, 0.5, 0.5],
},
reference_frequency_in_hertz=fband.max_frequency_in_hertz,
reference_model="prior",
reference_velocities={"VSH": "VSH", "VSV": "VSH", "VP": "VSH"},
),
job_submission={
"forward": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"adjoint": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"preconditioner": sn.SiteConfig(
site_name="local", ranks_per_job=RANKS_PER_JOB
),
},
)
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
[2025-02-07 20:57:16,903] INFO: Adding new iteration #0.
True
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
Salvus
operates a task-based workflow.SalvusOpt
steps through an iteration, and automatically dispatches simulations whenever necessary. The function resume
will return whenever SalvusOpt
is waiting for other tasks to finish first. Calling it several time, will step through the iteration in sequence. The log messages inform about the current status and tasks.p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2025-02-07 20:57:17,735] INFO: Resuming iteration #0. Current stage: initialize [2025-02-07 20:57:17,735] INFO: 1 new tasks have been issued. [2025-02-07 20:57:17,735] INFO: Processing task `misfit_and_gradient` [2025-02-07 20:57:19,485] INFO: Submitting job array with 2 jobs ...
Uploading 1 files... Uploading 1 files... [2025-02-07 20:57:19,569] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2025-02-07 20:57:19,570] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2025-02-07 20:57:19,577] INFO: Resuming iteration #0. Current stage: check_convergence [2025-02-07 20:57:19,577] INFO: Processing task `misfit_and_gradient` [2025-02-07 20:57:19,726] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 20:57:19,794] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 20:57:19,795] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2025-02-07 20:57:19,802] INFO: Resuming iteration #0. Current stage: check_convergence [2025-02-07 20:57:19,802] INFO: Processing task `misfit_and_gradient` [2025-02-07 20:57:19,963] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 20:57:20,016] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 20:57:20,017] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
Salvus
to run an entire iteration at once. Note the parameter timeout_in_seconds
, which will force the cell to return even if the iteration has not been completed yet, and there might still be a few simulations running in the back.
Again, you can execute the cell several times or mix it with calls to the previous one until the iteration is complete.p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=1800,
ping_interval_in_seconds=100,
)
[2025-02-07 20:57:20,585] INFO: Resuming iteration #0. [2025-02-07 20:57:20,585] INFO: Processing task `misfit_and_gradient` [2025-02-07 20:57:20,731] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 20:57:20,778] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 20:57:20,778] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 20:59:00,780] INFO: Processing task `misfit_and_gradient` [2025-02-07 20:59:01,014] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 20:59:01,100] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 20:59:01,101] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 21:00:41,103] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:00:41,326] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:00:41,422] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:00:41,424] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 21:02:21,426] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:02:21,687] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:02:21,766] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:02:21,767] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 21:04:01,770] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:04:01,883] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:04:02,632] INFO: Iteration 0: Number of events: 2 chi = 309.9166552693356 ||g|| = 1.2260938629133952e-05 pred = --- ared = --- norm_update = --- tr_radius = --- [2025-02-07 21:04:02,635] INFO: 1 new tasks have been issued. [2025-02-07 21:04:02,636] INFO: Processing task `preconditioner` [2025-02-07 21:04:03,028] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 21:05:43,125] INFO: Processing task `preconditioner` [2025-02-07 21:05:43,600] INFO: 1 new tasks have been issued. [2025-02-07 21:05:43,600] INFO: Processing task `misfit` [2025-02-07 21:05:43,883] INFO: Submitting job array with 2 jobs ... [2025-02-07 21:05:43,984] INFO: Launched simulations for 2 events. Please check again to see if they are finished. [2025-02-07 21:05:43,985] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 21:07:24,168] INFO: Processing task `misfit` [2025-02-07 21:07:24,275] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-02-07 21:09:04,590] INFO: Processing task `misfit` [2025-02-07 21:09:05,572] INFO: old misfit control group: 309.9166552693356 new misfit control group: 223.07194915054765 predicted reduction control group: -43.80373405400671 actual reduction control group: -86.84470611878794 2 out of 2 event(s) improved the misfit. [2025-02-07 21:09:05,573] INFO: Model update accepted. [2025-02-07 21:09:05,574] INFO: 1 new tasks have been issued. [2025-02-07 21:09:05,574] INFO: Processing task `finalize_iteration` [2025-02-07 21:09:05,909] INFO: Succesfully completed iteration #0. [2025-02-07 21:09:05,911] INFO: Adding new iteration #1.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
scp
to transfer the data to your local machine.print(
p.inversions.get_iteration_directory(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=0,
).absolute()
)
/tmp/tmpukx0ujgs/project_dir_central_europe/INVERSIONS/inv_70.0_120.0_seconds/00000
p.inversions.add_events(
f"inv_{fband.period_band_name}",
SELECTED_EVENTS[-1],
)
previous_events = p.inversions.get_iteration(
f"inv_{fband.period_band_name}", 0
).events
new_events = [
SELECTED_EVENTS[-1],
]
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
parent_id=0,
events=previous_events + new_events,
)
[2025-02-07 21:09:07,086] INFO: Adding new iteration #2.
True
p.inversion.iterate
will always continue on the last added iteration marked by the green edge in the inversion tree.SalvusOpt
will automatically trigger the simulation once we continue with the inversion.p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
iterate
function and create a few more model updates.for i in range(2):
p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=3600,
ping_interval_in_seconds=120,
)
[2025-02-07 21:09:07,938] INFO: Resuming iteration #2. [2025-02-07 21:09:07,939] INFO: 2 new tasks have been issued. [2025-02-07 21:09:07,940] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:09:08,091] INFO: Submitting job ... [2025-02-07 21:09:08,180] INFO: Launched simulations for 1 events. Please check again to see if they are finished. [2025-02-07 21:09:08,181] INFO: Processing task `gradient` [2025-02-07 21:09:08,363] INFO: Submitting job array with 2 jobs ...
Uploading 1 files... Uploading 1 files... [2025-02-07 21:09:08,422] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2025-02-07 21:09:08,423] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:11:08,425] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:11:09,639] INFO: Submitting job ... Uploading 1 files... 🚀 Submitted job_2502072111645037_8cf2f0cbfb@local [2025-02-07 21:11:09,735] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished. [2025-02-07 21:11:09,737] INFO: Processing task `gradient` [2025-02-07 21:11:09,860] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:11:09,942] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:11:09,942] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:13:09,946] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:13:10,135] INFO: 1 events have already been submitted. They will not be submitted again. [2025-02-07 21:13:10,206] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:13:10,210] INFO: Processing task `gradient` [2025-02-07 21:13:11,355] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:13:11,473] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:13:11,477] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:15:11,483] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:15:11,732] INFO: 1 events have already been submitted. They will not be submitted again. [2025-02-07 21:15:11,811] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:15:11,814] INFO: Processing task `gradient` [2025-02-07 21:15:12,000] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:15:12,108] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:15:12,110] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:17:12,118] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:17:12,260] INFO: 1 events have already been submitted. They will not be submitted again. [2025-02-07 21:17:12,885] INFO: Processing task `gradient` [2025-02-07 21:17:12,991] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:17:13,063] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:17:13,064] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:19:13,067] INFO: Processing task `misfit_and_gradient` [2025-02-07 21:19:13,290] INFO: Processing task `gradient` [2025-02-07 21:19:13,348] INFO: 2 events have already been submitted. They will not be submitted again. [2025-02-07 21:19:14,248] INFO: Iteration 2: Number of events: 3 chi = 265.3314777012853 ||g|| = 1.2247869354238154e-05 pred = -43.80373405400671 ared = -86.84470611878794 norm_update = 9532185.789387032 tr_radius = 9532185.789387032 [2025-02-07 21:19:14,476] INFO: 1 new tasks have been issued. [2025-02-07 21:19:14,476] INFO: Processing task `preconditioner` [2025-02-07 21:19:14,962] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:21:15,143] INFO: Processing task `preconditioner` [2025-02-07 21:21:15,640] INFO: 1 new tasks have been issued. [2025-02-07 21:21:15,641] INFO: Processing task `misfit` [2025-02-07 21:21:15,897] INFO: Submitting job array with 3 jobs ... [2025-02-07 21:21:15,997] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2025-02-07 21:21:15,998] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:23:16,258] INFO: Processing task `misfit` [2025-02-07 21:23:16,349] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-02-07 21:25:16,734] INFO: Processing task `misfit` [2025-02-07 21:25:18,898] INFO: old misfit control group: 265.33147770128534 new misfit control group: 174.37595999793166 predicted reduction control group: -64.9565570597944 actual reduction control group: -90.95551770335368 3 out of 3 event(s) improved the misfit. [2025-02-07 21:25:18,899] INFO: Model update accepted. [2025-02-07 21:25:18,899] INFO: 1 new tasks have been issued. [2025-02-07 21:25:18,900] INFO: Processing task `finalize_iteration` [2025-02-07 21:25:19,614] INFO: Succesfully completed iteration #2. [2025-02-07 21:25:19,620] INFO: Adding new iteration #3. [2025-02-07 21:25:19,681] INFO: Resuming iteration #3. [2025-02-07 21:25:19,682] INFO: 1 new tasks have been issued. [2025-02-07 21:25:19,683] INFO: Processing task `gradient` [2025-02-07 21:25:19,984] INFO: Submitting job array with 3 jobs ...
Uploading 1 files... Uploading 1 files... Uploading 1 files... [2025-02-07 21:25:20,196] INFO: Launched adjoint simulations for 3 events. Please check again to see if they are finished. [2025-02-07 21:25:20,202] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:27:20,220] INFO: Processing task `gradient` [2025-02-07 21:27:20,407] INFO: 3 events have already been submitted. They will not be submitted again. [2025-02-07 21:27:20,532] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:27:20,535] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:29:20,539] INFO: Processing task `gradient` [2025-02-07 21:29:20,621] INFO: 3 events have already been submitted. They will not be submitted again. [2025-02-07 21:29:20,686] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:29:20,687] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:31:20,690] INFO: Processing task `gradient` [2025-02-07 21:31:20,824] INFO: 3 events have already been submitted. They will not be submitted again. [2025-02-07 21:31:20,906] INFO: Some simulations are still running. Please check again to see if they are finished. [2025-02-07 21:31:20,907] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:33:20,911] INFO: Processing task `gradient` [2025-02-07 21:33:20,978] INFO: 3 events have already been submitted. They will not be submitted again. [2025-02-07 21:33:21,786] INFO: Iteration 3: Number of events: 3 chi = 174.37595999793166 ||g|| = 9.652088809471465e-06 pred = -64.9565570597944 ared = -90.95551770335368 norm_update = 19064370.9745405 tr_radius = 19064371.578774065 [2025-02-07 21:33:22,054] INFO: 1 new tasks have been issued. [2025-02-07 21:33:22,054] INFO: Processing task `preconditioner` [2025-02-07 21:33:22,486] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:35:22,666] INFO: Processing task `preconditioner` [2025-02-07 21:35:23,185] INFO: 1 new tasks have been issued. [2025-02-07 21:35:23,185] INFO: Processing task `misfit` [2025-02-07 21:35:23,475] INFO: Submitting job array with 3 jobs ... [2025-02-07 21:35:23,837] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2025-02-07 21:35:23,838] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:37:25,088] INFO: Processing task `misfit` [2025-02-07 21:37:25,206] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-02-07 21:39:25,795] INFO: Processing task `misfit` [2025-02-07 21:39:28,385] INFO: old misfit control group: 174.37595999793166 new misfit control group: 145.66558267369612 predicted reduction control group: -20.26363726308793 actual reduction control group: -28.710377324235537 3 out of 3 event(s) improved the misfit. [2025-02-07 21:39:28,386] INFO: Model update accepted. [2025-02-07 21:39:28,390] INFO: 1 new tasks have been issued. [2025-02-07 21:39:28,391] INFO: Processing task `finalize_iteration` [2025-02-07 21:39:29,462] INFO: Succesfully completed iteration #3. [2025-02-07 21:39:29,470] INFO: Adding new iteration #4.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
p.viz.nb.misfit_comparison(
reference_data=f"initial_model_{fband.period_band_name}",
other_data=[
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
)
],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
event=SELECTED_EVENTS[-1],
)
ini... (reference) | inv... | Reduction inv... | |
---|---|---|---|
II.BORG.00 | 5.7347e+01 | 4.6661e+01 | 1.0686e+01 |
II.ESK.00 | 6.1326e+01 | 2.6348e+01 | 3.4979e+01 |
II.LVZ.00 | 3.6217e+01 | 2.1670e+01 | 1.4547e+01 |
II.OBN.00 | 1.7577e+01 | 8.5373e+00 | 9.0396e+00 |
IU.GRFO. | 5.9093e+01 | 2.4053e+01 | 3.5040e+01 |
IU.KEV.00 | 5.9060e+01 | 1.6920e+01 | 4.2140e+01 |
IU.KONO.00 | 3.5902e+01 | 2.4111e+01 | 1.1790e+01 |
IU.MACI. | 5.6436e+01 | 4.1375e+01 | 1.5061e+01 |
IU.PAB.00 | 2.5168e+00 | 1.9686e+00 | 5.4823e-01 |
<pandas.io.formats.style.Styler at 0x75499c6a5990>
p.viz.misfit_histogram(
simulation_configuration_a=f"initial_model_{fband.period_band_name}",
simulation_configuration_b=p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
events=SELECTED_EVENTS[-1],
merge_all_components=False,
)
STRAIT_OF_GIBRALTAR
and station GB.GAL1
for a good example.
Note that only the parts of the waveforms within the shaded windows informed the inverse problem about the model space.
Interestingly, however, as we approach better models some wiggles that were not selected also match the data better now then they did initially; see station RO.BUR31
for an example.
We could make use of this, re-pick windows and then continue with more iterations and a modified MisfitConfiguration
.p.viz.nb.waveforms(
[
f"PROCESSED_DATA:{fband.period_band_name}",
f"initial_model_{fband.period_band_name}",
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
],
receiver_field="displacement",
data_selection_configuration=f"initial_selection_{fband.period_band_name}",
)