import os
import os, pathlib
from frequency_band import FrequencyBand
from salvus import namespace as sn
SIMULATION_TIME_IN_SECONDS = 1200.0
SALVUS_FLOW_SITE_NAME = os.environ.get("SITE_NAME", "local")
RANKS_PER_JOB = 4
PROJECT_DIR = "project_dir_central_europe"
fband_file = pathlib.Path("./frequency_band_70_120.pkl")
fband = FrequencyBand.load(fband_file)
fband
FrequencyBand(min_frequency_in_hertz=0.008333333333333333, max_frequency_in_hertz=0.014285714285714285)
SELECTED_EVENTS = [
"event_CRETE_GREECE_Mag_6.15_2015-04-16-18-07",
"event_ICELAND_REGION_Mag_5.75_2020-06-20-19-26",
"event_STRAIT_OF_GIBRALTAR_Mag_6.38_2016-01-25-04-22",
]
# Load the project with the data from the previous parts.
p = sn.Project(path=PROJECT_DIR)
events
)misfit_configuration
)prior_model
)mapping
)method
)preconditioner
)job_submission
)InverseProblemConfiguration
collects these parameters to setup the inverse problem.
Often enough, it is hard to choose the best setting a-prior, which is why the inverse problem is constructed as a tree that allows to branch off an ongoing inversion with changed parameters, or to run several scenarios concurrently.p += sn.InverseProblemConfiguration(
name=f"inv_{fband.period_band_name}",
events=SELECTED_EVENTS[:2],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
wavefield_compression=sn.WavefieldCompression(
forward_wavefield_sampling_interval=10
),
prior_model=f"initial_model_{fband.period_band_name}",
mapping=sn.Mapping(
scaling="relative_deviation_from_prior",
inversion_parameters=["VSH", "VSV", "VP"],
map_to_physical_parameters={
"VPV": "VP",
"VPH": "VP",
},
source_cutout_radius_in_meters=300000,
receiver_cutout_radius_in_meters=50000,
),
method=sn.TrustRegion(initial_trust_region_linf=0.02),
preconditioner=sn.ModelDependentSmoothing(
smoothing_lengths_in_wavelengths={
"VSH": [0.2, 0.5, 0.5],
"VSV": [0.2, 0.5, 0.5],
"VP": [0.2, 0.5, 0.5],
},
reference_frequency_in_hertz=fband.max_frequency_in_hertz,
reference_model="prior",
reference_velocities={"VSH": "VSH", "VSV": "VSH", "VP": "VSH"},
),
job_submission={
"forward": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"adjoint": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"preconditioner": sn.SiteConfig(
site_name="local", ranks_per_job=RANKS_PER_JOB
),
},
)
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
[2024-07-05 10:28:20,150] INFO: Adding new iteration #0.
True
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
Salvus
operates a task-based workflow.SalvusOpt
steps through an iteration, and automatically dispatches simulations whenever necessary. The function resume
will return whenever SalvusOpt
is waiting for other tasks to finish first. Calling it several time, will step through the iteration in sequence. The log messages inform about the current status and tasks.p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2024-07-05 10:28:20,824] INFO: Resuming iteration #0. Current stage: initialize [2024-07-05 10:28:20,824] INFO: 1 new tasks have been issued. [2024-07-05 10:28:20,824] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:28:22,019] INFO: Submitting job array with 2 jobs ...
Uploading 1 files... Uploading 1 files... [2024-07-05 10:28:22,076] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2024-07-05 10:28:22,077] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2024-07-05 10:28:22,082] INFO: Resuming iteration #0. Current stage: check_convergence [2024-07-05 10:28:22,082] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:28:22,193] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:28:22,234] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:28:22,234] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
[2024-07-05 10:28:22,240] INFO: Resuming iteration #0. Current stage: check_convergence [2024-07-05 10:28:22,240] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:28:22,352] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:28:22,388] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:28:22,389] INFO: Some tasks of iteration #0 are still running. Please check again later.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
Salvus
to run an entire iteration at once. Note the parameter timeout_in_seconds
, which will force the cell to return even if the iteration has not been completed yet, and there might still be a few simulations running in the back.
Again, you can execute the cell several times or mix it with calls to the previous one until the iteration is complete.p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=1800,
ping_interval_in_seconds=100,
)
[2024-07-05 10:28:22,839] INFO: Resuming iteration #0. [2024-07-05 10:28:22,839] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:28:22,956] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:28:22,988] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:28:22,989] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-07-05 10:30:02,990] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:30:03,124] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:30:03,169] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:30:03,172] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-07-05 10:31:43,175] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:31:43,311] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:31:43,374] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:31:43,375] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-07-05 10:33:23,378] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:33:23,483] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:33:23,968] INFO: Iteration 0: Number of events: 2 chi = 309.8899188526656 ||g|| = 1.226236317036059e-05 pred = --- ared = --- norm_update = --- tr_radius = --- [2024-07-05 10:33:23,969] INFO: 1 new tasks have been issued. [2024-07-05 10:33:23,970] INFO: Processing task `preconditioner` [2024-07-05 10:33:24,233] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-07-05 10:35:04,273] INFO: Processing task `preconditioner` [2024-07-05 10:35:04,577] INFO: 1 new tasks have been issued. [2024-07-05 10:35:04,577] INFO: Processing task `misfit` [2024-07-05 10:35:04,770] INFO: Submitting job array with 2 jobs ... [2024-07-05 10:35:04,854] INFO: Launched simulations for 2 events. Please check again to see if they are finished. [2024-07-05 10:35:04,854] INFO: Some tasks of iteration #0 are still running. Please check again later. [2024-07-05 10:36:44,943] INFO: Processing task `misfit` [2024-07-05 10:36:45,707] INFO: old misfit control group: 309.8899188526656 new misfit control group: 223.05853515353078 predicted reduction control group: -43.794787797175395 actual reduction control group: -86.83138369913479 2 out of 2 event(s) improved the misfit. [2024-07-05 10:36:45,707] INFO: Model update accepted. [2024-07-05 10:36:45,707] INFO: 1 new tasks have been issued. [2024-07-05 10:36:45,708] INFO: Processing task `finalize_iteration` [2024-07-05 10:36:45,870] INFO: Succesfully completed iteration #0. [2024-07-05 10:36:45,872] INFO: Adding new iteration #1.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
scp
to transfer the data to your local machine.print(
p.inversions.get_iteration_directory(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=0,
).absolute()
)
/tmp/tmpgwqm_r9c/project_dir_central_europe/INVERSIONS/inv_70.0_120.0_seconds/00000
p.inversions.add_events(
f"inv_{fband.period_band_name}",
SELECTED_EVENTS[-1],
)
previous_events = p.inversions.get_iteration(
f"inv_{fband.period_band_name}", 0
).events
new_events = [
SELECTED_EVENTS[-1],
]
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
parent_id=0,
events=previous_events + new_events,
)
[2024-07-05 10:36:46,993] INFO: Adding new iteration #2.
True
p.inversion.iterate
will always continue on the last added iteration marked by the green edge in the inversion tree.SalvusOpt
will automatically trigger the simulation once we continue with the inversion.p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
iterate
function and create a few more model updates.for i in range(2):
p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=3600,
ping_interval_in_seconds=120,
)
[2024-07-05 10:36:47,781] INFO: Resuming iteration #2. [2024-07-05 10:36:47,781] INFO: 2 new tasks have been issued. [2024-07-05 10:36:47,781] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:36:47,870] INFO: Submitting job ... [2024-07-05 10:36:47,929] INFO: Launched simulations for 1 events. Please check again to see if they are finished. [2024-07-05 10:36:47,929] INFO: Processing task `gradient` [2024-07-05 10:36:48,069] INFO: Submitting job array with 2 jobs ...
Uploading 1 files... Uploading 1 files... [2024-07-05 10:36:48,119] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2024-07-05 10:36:48,119] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-07-05 10:38:48,122] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:38:48,819] INFO: Submitting job ... Uploading 1 files... 🚀 Submitted job_2407051038822077_84d2acd5f1@local [2024-07-05 10:38:48,873] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished. [2024-07-05 10:38:48,874] INFO: Processing task `gradient` [2024-07-05 10:38:48,963] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:38:49,017] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:38:49,019] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-07-05 10:40:49,022] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:40:49,165] INFO: 1 events have already been submitted. They will not be submitted again. [2024-07-05 10:40:49,214] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:40:49,216] INFO: Processing task `gradient` [2024-07-05 10:40:49,337] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:40:49,426] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:40:49,427] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-07-05 10:42:49,431] INFO: Processing task `misfit_and_gradient` [2024-07-05 10:42:49,498] INFO: 1 events have already been submitted. They will not be submitted again. [2024-07-05 10:42:49,757] INFO: Processing task `gradient` [2024-07-05 10:42:49,811] INFO: 2 events have already been submitted. They will not be submitted again. [2024-07-05 10:42:50,343] INFO: Iteration 2: Number of events: 3 chi = 265.3144666469096 ||g|| = 1.2247339772033081e-05 pred = -43.794787797175395 ared = -86.83138369913479 norm_update = 9530625.409301922 tr_radius = 9530625.409301922 [2024-07-05 10:42:50,492] INFO: 1 new tasks have been issued. [2024-07-05 10:42:50,492] INFO: Processing task `preconditioner` [2024-07-05 10:42:50,835] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-07-05 10:44:50,919] INFO: Processing task `preconditioner` [2024-07-05 10:44:51,230] INFO: 1 new tasks have been issued. [2024-07-05 10:44:51,230] INFO: Processing task `misfit` [2024-07-05 10:44:51,431] INFO: Submitting job array with 3 jobs ... [2024-07-05 10:44:51,522] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2024-07-05 10:44:51,522] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-07-05 10:46:51,661] INFO: Processing task `misfit` [2024-07-05 10:46:51,713] INFO: Some tasks of iteration #2 are still running. Please check again later. [2024-07-05 10:48:51,922] INFO: Processing task `misfit` [2024-07-05 10:48:53,251] INFO: old misfit control group: 265.3144666469096 new misfit control group: 174.3754408561265 predicted reduction control group: -64.94062254906957 actual reduction control group: -90.93902579078309 3 out of 3 event(s) improved the misfit. [2024-07-05 10:48:53,251] INFO: Model update accepted. [2024-07-05 10:48:53,251] INFO: 1 new tasks have been issued. [2024-07-05 10:48:53,251] INFO: Processing task `finalize_iteration` [2024-07-05 10:48:53,518] INFO: Succesfully completed iteration #2. [2024-07-05 10:48:53,520] INFO: Adding new iteration #3. [2024-07-05 10:48:53,541] INFO: Resuming iteration #3. [2024-07-05 10:48:53,541] INFO: 1 new tasks have been issued. [2024-07-05 10:48:53,541] INFO: Processing task `gradient` [2024-07-05 10:48:53,697] INFO: Submitting job array with 3 jobs ...
Uploading 1 files... Uploading 1 files... Uploading 1 files... [2024-07-05 10:48:53,749] INFO: Launched adjoint simulations for 3 events. Please check again to see if they are finished. [2024-07-05 10:48:53,749] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-07-05 10:50:53,752] INFO: Processing task `gradient` [2024-07-05 10:50:53,862] INFO: 3 events have already been submitted. They will not be submitted again. [2024-07-05 10:50:53,917] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:50:53,918] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-07-05 10:52:53,921] INFO: Processing task `gradient` [2024-07-05 10:52:54,015] INFO: 3 events have already been submitted. They will not be submitted again. [2024-07-05 10:52:54,080] INFO: Some simulations are still running. Please check again to see if they are finished. [2024-07-05 10:52:54,081] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-07-05 10:54:54,085] INFO: Processing task `gradient` [2024-07-05 10:54:54,161] INFO: 3 events have already been submitted. They will not be submitted again. [2024-07-05 10:54:54,827] INFO: Iteration 3: Number of events: 3 chi = 174.3754408561265 ||g|| = 9.651699183590254e-06 pred = -64.94062254906957 ared = -90.93902579078309 norm_update = 19061250.35559088 tr_radius = 19061250.818603843 [2024-07-05 10:54:55,062] INFO: 1 new tasks have been issued. [2024-07-05 10:54:55,062] INFO: Processing task `preconditioner` [2024-07-05 10:54:55,427] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-07-05 10:56:55,563] INFO: Processing task `preconditioner` [2024-07-05 10:56:55,936] INFO: 1 new tasks have been issued. [2024-07-05 10:56:55,936] INFO: Processing task `misfit` [2024-07-05 10:56:56,137] INFO: Submitting job array with 3 jobs ... [2024-07-05 10:56:56,228] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2024-07-05 10:56:56,229] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-07-05 10:58:56,419] INFO: Processing task `misfit` [2024-07-05 10:58:56,474] INFO: Some tasks of iteration #3 are still running. Please check again later. [2024-07-05 11:00:56,743] INFO: Processing task `misfit` [2024-07-05 11:00:59,689] INFO: old misfit control group: 174.3754408561265 new misfit control group: 145.66482741880066 predicted reduction control group: -20.268206745593197 actual reduction control group: -28.710613437325833 3 out of 3 event(s) improved the misfit. [2024-07-05 11:00:59,690] INFO: Model update accepted. [2024-07-05 11:00:59,691] INFO: 1 new tasks have been issued. [2024-07-05 11:00:59,692] INFO: Processing task `finalize_iteration` [2024-07-05 11:01:00,369] INFO: Succesfully completed iteration #3. [2024-07-05 11:01:00,375] INFO: Adding new iteration #4.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
p.viz.nb.misfit_comparison(
reference_data=f"initial_model_{fband.period_band_name}",
other_data=[
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
)
],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
event=SELECTED_EVENTS[-1],
)
ini... (reference) | inv... | Reduction inv... | |
---|---|---|---|
II.BORG.00 | 5.7342e+01 | 4.6662e+01 | 1.0681e+01 |
II.ESK.00 | 6.1319e+01 | 2.6345e+01 | 3.4974e+01 |
II.LVZ.00 | 3.6213e+01 | 2.1672e+01 | 1.4541e+01 |
II.OBN.00 | 1.7580e+01 | 8.5394e+00 | 9.0406e+00 |
IU.GRFO. | 5.9088e+01 | 2.4047e+01 | 3.5041e+01 |
IU.KEV.00 | 5.9051e+01 | 1.6920e+01 | 4.2130e+01 |
IU.KONO.00 | 3.5897e+01 | 2.4111e+01 | 1.1786e+01 |
IU.MACI. | 5.6432e+01 | 4.1373e+01 | 1.5059e+01 |
IU.PAB.00 | 2.5167e+00 | 1.9685e+00 | 5.4820e-01 |
<pandas.io.formats.style.Styler at 0x7f52fa474210>
p.viz.misfit_histogram(
simulation_configuration_a=f"initial_model_{fband.period_band_name}",
simulation_configuration_b=p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
events=SELECTED_EVENTS[-1],
merge_all_components=False,
)
STRAIT_OF_GIBRALTAR
and station GB.GAL1
for a good example.
Note that only the parts of the waveforms within the shaded windows informed the inverse problem about the model space.
Interestingly, however, as we approach better models some wiggles that were not selected also match the data better now then they did initially; see station RO.BUR31
for an example.
We could make use of this, re-pick windows and then continue with more iterations and a modified MisfitConfiguration
.p.viz.nb.waveforms(
[
f"PROCESSED_DATA:{fband.period_band_name}",
f"initial_model_{fband.period_band_name}",
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
],
receiver_field="displacement",
data_selection_configuration=f"initial_selection_{fband.period_band_name}",
)