[2024-11-20 13:01:22,816] INFO: Adding new iteration #0.
[2024-11-20 13:01:22,840] INFO: Resuming iteration #0.
[2024-11-20 13:01:22,844] INFO: 1 new tasks have been issued.
[2024-11-20 13:01:22,846] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:22,980] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2024-11-20 13:01:23,036] INFO: Submitting job array with 5 jobs ...
[2024-11-20 13:01:23,261] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-20 13:01:23,263] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:25,265] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:25,469] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:27,471] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:27,664] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:29,666] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:32,804] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-11-20 13:01:32,946] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-11-20 13:01:32,949] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:34,956] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:35,511] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:01:35,668] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-20 13:01:35,674] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:37,677] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:38,462] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:01:38,596] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-20 13:01:38,598] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:40,603] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:41,079] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:01:41,195] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-20 13:01:41,199] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:43,203] INFO: Processing task `misfit_and_gradient`
[2024-11-20 13:01:43,722] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:01:44,598] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2024-11-20 13:01:44,602] INFO: 1 new tasks have been issued.
[2024-11-20 13:01:44,603] INFO: Processing task `preconditioner`
[2024-11-20 13:01:44,849] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:46,917] INFO: Processing task `preconditioner`
[2024-11-20 13:01:47,140] INFO: 1 new tasks have been issued.
[2024-11-20 13:01:47,141] INFO: Processing task `misfit`
[2024-11-20 13:01:47,242] INFO: Submitting job array with 5 jobs ...
[2024-11-20 13:01:47,558] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-20 13:01:47,560] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:49,636] INFO: Processing task `misfit`
[2024-11-20 13:01:49,824] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:51,876] INFO: Processing task `misfit`
[2024-11-20 13:01:52,049] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-20 13:01:54,123] INFO: Processing task `misfit`
[2024-11-20 13:01:57,897] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2024-11-20 13:01:57,897] INFO:
Model update accepted.
[2024-11-20 13:01:57,898] INFO: 1 new tasks have been issued.
[2024-11-20 13:01:57,899] INFO: Processing task `finalize_iteration`
[2024-11-20 13:01:57,988] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2024-11-20 13:01:58,015] INFO: Freed up 1.2 MB of space.
[2024-11-20 13:01:58,016] INFO: Succesfully completed iteration #0.
[2024-11-20 13:01:58,018] INFO: Adding new iteration #1.
[2024-11-20 13:01:58,025] INFO: Resuming iteration #1.
[2024-11-20 13:01:58,026] INFO: 1 new tasks have been issued.
[2024-11-20 13:01:58,026] INFO: Processing task `gradient`
[2024-11-20 13:01:58,518] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-11-20 13:01:58,699] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-11-20 13:01:58,702] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:00,708] INFO: Processing task `gradient`
[2024-11-20 13:02:00,994] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:02:01,142] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-20 13:02:01,147] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:03,156] INFO: Processing task `gradient`
[2024-11-20 13:02:03,479] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:02:03,643] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-20 13:02:03,648] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:05,651] INFO: Processing task `gradient`
[2024-11-20 13:02:05,930] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:02:06,088] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-20 13:02:06,092] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:08,098] INFO: Processing task `gradient`
[2024-11-20 13:02:08,400] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-20 13:02:09,379] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2024-11-20 13:02:09,403] INFO: 1 new tasks have been issued.
[2024-11-20 13:02:09,407] INFO: Processing task `preconditioner`
[2024-11-20 13:02:09,620] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:11,668] INFO: Processing task `preconditioner`
[2024-11-20 13:02:11,773] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:13,810] INFO: Processing task `preconditioner`
[2024-11-20 13:02:14,068] INFO: 1 new tasks have been issued.
[2024-11-20 13:02:14,069] INFO: Processing task `misfit`
[2024-11-20 13:02:14,149] INFO: Submitting job array with 5 jobs ...
[2024-11-20 13:02:14,351] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-20 13:02:14,353] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:16,403] INFO: Processing task `misfit`
[2024-11-20 13:02:17,019] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:19,070] INFO: Processing task `misfit`
[2024-11-20 13:02:19,171] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:21,227] INFO: Processing task `misfit`
[2024-11-20 13:02:21,383] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-20 13:02:23,478] INFO: Processing task `misfit`
[2024-11-20 13:02:25,849] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2024-11-20 13:02:25,850] INFO:
Model update accepted.
[2024-11-20 13:02:25,856] INFO: 1 new tasks have been issued.
[2024-11-20 13:02:25,856] INFO: Processing task `finalize_iteration`
[2024-11-20 13:02:25,978] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2024-11-20 13:02:26,219] INFO: Freed up 4.8 MB of space.
[2024-11-20 13:02:26,220] INFO: Succesfully completed iteration #1.
[2024-11-20 13:02:26,223] INFO: Adding new iteration #2.