import os
import os, pathlib
from frequency_band import FrequencyBand
from salvus import namespace as sn
SIMULATION_TIME_IN_SECONDS = 1200.0
SALVUS_FLOW_SITE_NAME = os.environ.get("SITE_NAME", "local")
RANKS_PER_JOB = 4
PROJECT_DIR = "project_dir_central_europe"
fband_file = pathlib.Path("./frequency_band_70_120.pkl")
fband = FrequencyBand.load(fband_file)
fband
FrequencyBand(min_frequency_in_hertz=0.008333333333333333, max_frequency_in_hertz=0.014285714285714285)
SELECTED_EVENTS = [
"event_CRETE_GREECE_Mag_6.15_2015-04-16-18-07",
"event_ICELAND_REGION_Mag_5.75_2020-06-20-19-26",
"event_STRAIT_OF_GIBRALTAR_Mag_6.38_2016-01-25-04-22",
]
# Load the project with the data from the previous parts.
p = sn.Project(path=PROJECT_DIR)
events
)misfit_configuration
)prior_model
)mapping
)method
)preconditioner
)job_submission
)InverseProblemConfiguration
collects these parameters to setup the inverse problem.
Often enough, it is hard to choose the best setting a-prior, which is why the inverse problem is constructed as a tree that allows to branch off an ongoing inversion with changed parameters, or to run several scenarios concurrently.p += sn.InverseProblemConfiguration(
name=f"inv_{fband.period_band_name}",
events=SELECTED_EVENTS[:2],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
wavefield_compression=sn.WavefieldCompression(
forward_wavefield_sampling_interval=10
),
prior_model=f"initial_model_{fband.period_band_name}",
mapping=sn.Mapping(
scaling="relative_deviation_from_prior",
inversion_parameters=["VSH", "VSV", "VP"],
map_to_physical_parameters={
"VPV": "VP",
"VPH": "VP",
},
source_cutout_radius_in_meters=300000,
receiver_cutout_radius_in_meters=50000,
),
method=sn.TrustRegion(initial_trust_region_linf=0.02),
preconditioner=sn.ModelDependentSmoothing(
smoothing_lengths_in_wavelengths={
"VSH": [0.2, 0.5, 0.5],
"VSV": [0.2, 0.5, 0.5],
"VP": [0.2, 0.5, 0.5],
},
reference_frequency_in_hertz=fband.max_frequency_in_hertz,
reference_model="prior",
reference_velocities={"VSH": "VSH", "VSV": "VSH", "VP": "VSH"},
),
job_submission={
"forward": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"adjoint": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"preconditioner": sn.SiteConfig(
site_name="local", ranks_per_job=RANKS_PER_JOB
),
},
)
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
[2024-11-15 14:03:43,544] INFO: Adding new iteration #0.
True
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
Salvus
operates a task-based workflow.SalvusOpt
steps through an iteration, and automatically dispatches simulations whenever necessary. The function resume
will return whenever SalvusOpt
is waiting for other tasks to finish first. Calling it several time, will step through the iteration in sequence. The log messages inform about the current status and tasks.p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2024-11-15 14:03:44,147] INFO: Resuming iteration #0. Current stage: initialize [2024-11-15 14:03:44,147] INFO: 1 new tasks have been issued. [2024-11-15 14:03:44,147] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:03:45,238] INFO: Submitting job array with 2 jobs ...
Uploading 1 files... Uploading 1 files... [2024-11-15 14:03:45,291] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2024-11-15 14:03:45,293] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2024-11-15 14:03:45,300] INFO: Resuming iteration #0. Current stage: check_convergence [2024-11-15 14:03:45,300] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:03:45,394] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:03:45,440] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:03:45,440] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2024-11-15 14:03:45,446] INFO: Resuming iteration #0. Current stage: check_convergence [2024-11-15 14:03:45,446] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:03:45,553] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:03:45,586] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:03:45,587] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
Salvus
to run an entire iteration at once. Note the parameter timeout_in_seconds
, which will force the cell to return even if the iteration has not been completed yet, and there might still be a few simulations running in the back.
Again, you can execute the cell several times or mix it with calls to the previous one until the iteration is complete.p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=1800,
ping_interval_in_seconds=100,
)
[2024-11-15 14:03:46,012] INFO: Resuming iteration #0. [2024-11-15 14:03:46,012] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:03:46,095] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:03:46,126] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:03:46,126] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-11-15 14:05:26,128] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:05:26,296] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:05:26,366] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:05:26,367] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-11-15 14:07:06,369] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:07:06,669] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:07:06,728] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:07:06,729] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-11-15 14:08:46,733] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:08:46,883] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:08:47,847] INFO: Iteration 0: Number of events: 2 chi = 309.88990540430245 ||g|| = 1.2262384848007887e-05 pred = --- ared = --- norm_update = --- tr_radius = --- [2024-11-15 14:08:47,851] INFO: 1 new tasks have been issued. [2024-11-15 14:08:47,852] INFO: Processing task `preconditioner` [2024-11-15 14:08:48,378] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-11-15 14:10:28,498] INFO: Processing task `preconditioner` [2024-11-15 14:10:29,199] INFO: 1 new tasks have been issued. [2024-11-15 14:10:29,201] INFO: Processing task `misfit` [2024-11-15 14:10:29,553] INFO: Submitting job array with 2 jobs ... [2024-11-15 14:10:29,715] INFO: Launched simulations for 2 events. Please check again to see if they are finished. [2024-11-15 14:10:29,716] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-11-15 14:12:10,089] INFO: Processing task `misfit` [2024-11-15 14:12:10,243] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-11-15 14:13:50,596] INFO: Processing task `misfit` [2024-11-15 14:13:52,220] INFO: old misfit control group: 309.88990540430245 new misfit control group: 223.05867201792984 predicted reduction control group: -43.794743188496945 actual reduction control group: -86.83123338637262 2 out of 2 event(s) improved the misfit. [2024-11-15 14:13:52,221] INFO: Model update accepted. [2024-11-15 14:13:52,224] INFO: 1 new tasks have been issued. [2024-11-15 14:13:52,225] INFO: Processing task `finalize_iteration` [2024-11-15 14:13:52,829] INFO: Succesfully completed iteration #0. [2024-11-15 14:13:52,835] INFO: Adding new iteration #1.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
scp
to transfer the data to your local machine.print(
p.inversions.get_iteration_directory(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=0,
).absolute()
)
/tmp/tmpd_8tn8y4/project_dir_central_europe/INVERSIONS/inv_70.0_120.0_seconds/00000
p.inversions.add_events(
f"inv_{fband.period_band_name}",
SELECTED_EVENTS[-1],
)
previous_events = p.inversions.get_iteration(
f"inv_{fband.period_band_name}", 0
).events
new_events = [
SELECTED_EVENTS[-1],
]
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
parent_id=0,
events=previous_events + new_events,
)
[2024-11-15 14:13:54,719] INFO: Adding new iteration #2.
True
p.inversion.iterate
will always continue on the last added iteration marked by the green edge in the inversion tree.SalvusOpt
will automatically trigger the simulation once we continue with the inversion.p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
iterate
function and create a few more model updates.for i in range(2):
p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=3600,
ping_interval_in_seconds=120,
)
[2024-11-15 14:13:56,048] INFO: Resuming iteration #2. [2024-11-15 14:13:56,055] INFO: 2 new tasks have been issued. [2024-11-15 14:13:56,056] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:13:56,225] INFO: Submitting job ... [2024-11-15 14:13:56,356] INFO: Launched simulations for 1 events. Please check again to see if they are finished. [2024-11-15 14:13:56,357] INFO: Processing task `gradient` [2024-11-15 14:13:56,771] INFO: Submitting job array with 2 jobs ...
Uploading 1 files... Uploading 1 files... [2024-11-15 14:13:56,931] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2024-11-15 14:13:56,932] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:15:56,948] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:15:57,239] INFO: Processing task `gradient` [2024-11-15 14:15:57,547] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:15:57,755] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:15:57,757] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:17:57,770] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:17:59,237] INFO: Submitting job ... Uploading 1 files... 🚀 Submitted job_2411151417240555_894ef9b394@local [2024-11-15 14:17:59,383] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished. [2024-11-15 14:17:59,388] INFO: Processing task `gradient` [2024-11-15 14:17:59,640] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:17:59,794] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:17:59,802] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:19:59,813] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:20:00,158] INFO: 1 events have already been submitted. They will not be submitted again. [2024-11-15 14:20:00,226] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:20:00,228] INFO: Processing task `gradient` [2024-11-15 14:20:00,450] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:20:00,660] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:20:00,664] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:22:00,674] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:22:00,957] INFO: 1 events have already been submitted. They will not be submitted again. [2024-11-15 14:22:01,107] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:22:01,111] INFO: Processing task `gradient` [2024-11-15 14:22:01,322] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:22:01,455] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:22:01,457] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:24:01,462] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:24:01,610] INFO: 1 events have already been submitted. They will not be submitted again. [2024-11-15 14:24:02,284] INFO: Processing task `gradient` [2024-11-15 14:24:02,396] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:24:02,486] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:24:02,487] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:26:02,491] INFO: Processing task `misfit_and_gradient` [2024-11-15 14:26:02,793] INFO: Processing task `gradient` [2024-11-15 14:26:02,875] INFO: 2 events have already been submitted. They will not be submitted again. [2024-11-15 14:26:03,780] INFO: Iteration 2: Number of events: 3 chi = 265.31456513164886 ||g|| = 1.2247350245637714e-05 pred = -43.794743188496945 ared = -86.83123338637262 norm_update = 9530613.729513792 tr_radius = 9530613.729513792 [2024-11-15 14:26:04,002] INFO: 1 new tasks have been issued. [2024-11-15 14:26:04,003] INFO: Processing task `preconditioner` [2024-11-15 14:26:04,538] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:28:04,781] INFO: Processing task `preconditioner` [2024-11-15 14:28:05,336] INFO: 1 new tasks have been issued. [2024-11-15 14:28:05,337] INFO: Processing task `misfit` [2024-11-15 14:28:05,651] INFO: Submitting job array with 3 jobs ... [2024-11-15 14:28:05,785] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2024-11-15 14:28:05,787] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:30:06,105] INFO: Processing task `misfit` [2024-11-15 14:30:06,251] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-11-15 14:32:06,758] INFO: Processing task `misfit` [2024-11-15 14:32:08,034] INFO: old misfit control group: 265.31456513164886 new misfit control group: 174.3756153686446 predicted reduction control group: -64.9405270119579 actual reduction control group: -90.93894976300427 3 out of 3 event(s) improved the misfit. [2024-11-15 14:32:08,035] INFO: Model update accepted. [2024-11-15 14:32:08,035] INFO: 1 new tasks have been issued. [2024-11-15 14:32:08,035] INFO: Processing task `finalize_iteration` [2024-11-15 14:32:08,413] INFO: Succesfully completed iteration #2. [2024-11-15 14:32:08,415] INFO: Adding new iteration #3. [2024-11-15 14:32:08,434] INFO: Resuming iteration #3. [2024-11-15 14:32:08,437] INFO: 1 new tasks have been issued. [2024-11-15 14:32:08,437] INFO: Processing task `gradient` [2024-11-15 14:32:08,575] INFO: Submitting job array with 3 jobs ...
Uploading 1 files... Uploading 1 files... Uploading 1 files... [2024-11-15 14:32:08,627] INFO: Launched adjoint simulations for 3 events. Please check again to see if they are finished. [2024-11-15 14:32:08,627] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:34:08,630] INFO: Processing task `gradient` [2024-11-15 14:34:08,857] INFO: 3 events have already been submitted. They will not be submitted again. [2024-11-15 14:34:08,969] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:34:08,971] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:36:08,977] INFO: Processing task `gradient` [2024-11-15 14:36:09,177] INFO: 3 events have already been submitted. They will not be submitted again. [2024-11-15 14:36:09,281] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:36:09,284] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:38:09,291] INFO: Processing task `gradient` [2024-11-15 14:38:09,519] INFO: 3 events have already been submitted. They will not be submitted again. [2024-11-15 14:38:09,646] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:38:09,647] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:40:09,667] INFO: Processing task `gradient` [2024-11-15 14:40:09,936] INFO: 3 events have already been submitted. They will not be submitted again. [2024-11-15 14:40:10,055] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-11-15 14:40:10,056] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:42:10,059] INFO: Processing task `gradient` [2024-11-15 14:42:10,203] INFO: 3 events have already been submitted. They will not be submitted again. [2024-11-15 14:42:11,420] INFO: Iteration 3: Number of events: 3 chi = 174.37561536864456 ||g|| = 9.651714176612556e-06 pred = -64.9405270119579 ared = -90.93894976300427 norm_update = 19061227.88864811 tr_radius = 19061227.459027585 [2024-11-15 14:42:11,798] INFO: 1 new tasks have been issued. [2024-11-15 14:42:11,799] INFO: Processing task `preconditioner` [2024-11-15 14:42:12,590] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:44:12,893] INFO: Processing task `preconditioner` [2024-11-15 14:44:13,567] INFO: 1 new tasks have been issued. [2024-11-15 14:44:13,569] INFO: Processing task `misfit` [2024-11-15 14:44:13,884] INFO: Submitting job array with 3 jobs ... [2024-11-15 14:44:14,040] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2024-11-15 14:44:14,040] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:46:14,609] INFO: Processing task `misfit` [2024-11-15 14:46:14,753] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-11-15 14:48:15,369] INFO: Processing task `misfit` [2024-11-15 14:48:17,301] INFO: old misfit control group: 174.3756153686446 new misfit control group: 145.6648893675632 predicted reduction control group: -20.268304475537672 actual reduction control group: -28.710726001081383 3 out of 3 event(s) improved the misfit. [2024-11-15 14:48:17,302] INFO: Model update accepted. [2024-11-15 14:48:17,302] INFO: 1 new tasks have been issued. [2024-11-15 14:48:17,303] INFO: Processing task `finalize_iteration` [2024-11-15 14:48:18,060] INFO: Succesfully completed iteration #3. [2024-11-15 14:48:18,063] INFO: Adding new iteration #4.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
p.viz.nb.misfit_comparison(
reference_data=f"initial_model_{fband.period_band_name}",
other_data=[
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
)
],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
event=SELECTED_EVENTS[-1],
)
ini... (reference) | inv... | Reduction inv... | |
---|---|---|---|
II.BORG.00 | 5.7342e+01 | 4.6662e+01 | 1.0681e+01 |
II.ESK.00 | 6.1319e+01 | 2.6345e+01 | 3.4974e+01 |
II.LVZ.00 | 3.6213e+01 | 2.1672e+01 | 1.4541e+01 |
II.OBN.00 | 1.7580e+01 | 8.5394e+00 | 9.0406e+00 |
IU.GRFO. | 5.9088e+01 | 2.4047e+01 | 3.5041e+01 |
IU.KEV.00 | 5.9051e+01 | 1.6921e+01 | 4.2130e+01 |
IU.KONO.00 | 3.5897e+01 | 2.4111e+01 | 1.1786e+01 |
IU.MACI. | 5.6432e+01 | 4.1373e+01 | 1.5059e+01 |
IU.PAB.00 | 2.5167e+00 | 1.9685e+00 | 5.4820e-01 |
<pandas.io.formats.style.Styler at 0x7db3aa14d690>
p.viz.misfit_histogram(
simulation_configuration_a=f"initial_model_{fband.period_band_name}",
simulation_configuration_b=p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
events=SELECTED_EVENTS[-1],
merge_all_components=False,
)
STRAIT_OF_GIBRALTAR
and station GB.GAL1
for a good example.
Note that only the parts of the waveforms within the shaded windows informed the inverse problem about the model space.
Interestingly, however, as we approach better models some wiggles that were not selected also match the data better now then they did initially; see station RO.BUR31
for an example.
We could make use of this, re-pick windows and then continue with more iterations and a modified MisfitConfiguration
.p.viz.nb.waveforms(
[
f"PROCESSED_DATA:{fband.period_band_name}",
f"initial_model_{fband.period_band_name}",
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
],
receiver_field="displacement",
data_selection_configuration=f"initial_selection_{fband.period_band_name}",
)