[2024-11-15 13:03:16,092] INFO: Adding new iteration #0.
[2024-11-15 13:03:16,099] INFO: Resuming iteration #0.
[2024-11-15 13:03:16,099] INFO: 1 new tasks have been issued.
[2024-11-15 13:03:16,100] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:16,162] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2024-11-15 13:03:16,185] INFO: Submitting job array with 5 jobs ...
[2024-11-15 13:03:16,279] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-15 13:03:16,280] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:18,282] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:18,387] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:20,389] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:20,477] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:22,478] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:24,441] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-11-15 13:03:24,504] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-11-15 13:03:24,506] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:26,507] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:26,732] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-15 13:03:26,795] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-15 13:03:26,796] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:28,797] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:29,243] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-15 13:03:29,307] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-15 13:03:29,308] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:31,309] INFO: Processing task `misfit_and_gradient`
[2024-11-15 13:03:31,550] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-15 13:03:32,034] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2024-11-15 13:03:32,035] INFO: 1 new tasks have been issued.
[2024-11-15 13:03:32,035] INFO: Processing task `preconditioner`
[2024-11-15 13:03:32,110] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:34,124] INFO: Processing task `preconditioner`
[2024-11-15 13:03:34,163] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:36,179] INFO: Processing task `preconditioner`
[2024-11-15 13:03:36,357] INFO: 1 new tasks have been issued.
[2024-11-15 13:03:36,358] INFO: Processing task `misfit`
[2024-11-15 13:03:36,445] INFO: Submitting job array with 5 jobs ...
[2024-11-15 13:03:36,698] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-15 13:03:36,705] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:38,762] INFO: Processing task `misfit`
[2024-11-15 13:03:38,836] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:40,875] INFO: Processing task `misfit`
[2024-11-15 13:03:40,962] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:43,009] INFO: Processing task `misfit`
[2024-11-15 13:03:43,082] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-15 13:03:45,118] INFO: Processing task `misfit`
[2024-11-15 13:03:48,176] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2024-11-15 13:03:48,176] INFO:
Model update accepted.
[2024-11-15 13:03:48,181] INFO: 1 new tasks have been issued.
[2024-11-15 13:03:48,182] INFO: Processing task `finalize_iteration`
[2024-11-15 13:03:48,265] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2024-11-15 13:03:48,289] INFO: Freed up 1.2 MB of space.
[2024-11-15 13:03:48,290] INFO: Succesfully completed iteration #0.
[2024-11-15 13:03:48,292] INFO: Adding new iteration #1.
[2024-11-15 13:03:48,301] INFO: Resuming iteration #1.
[2024-11-15 13:03:48,303] INFO: 1 new tasks have been issued.
[2024-11-15 13:03:48,304] INFO: Processing task `gradient`
[2024-11-15 13:03:48,655] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-11-15 13:03:48,770] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-11-15 13:03:48,771] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:03:50,777] INFO: Processing task `gradient`
[2024-11-15 13:03:51,047] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-15 13:03:51,154] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-15 13:03:51,156] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:03:53,161] INFO: Processing task `gradient`
[2024-11-15 13:03:53,395] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-15 13:03:53,499] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-15 13:03:53,501] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:03:55,504] INFO: Processing task `gradient`
[2024-11-15 13:03:55,750] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-15 13:03:56,457] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2024-11-15 13:03:56,476] INFO: 1 new tasks have been issued.
[2024-11-15 13:03:56,477] INFO: Processing task `preconditioner`
[2024-11-15 13:03:56,636] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:03:58,682] INFO: Processing task `preconditioner`
[2024-11-15 13:03:58,772] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:04:00,810] INFO: Processing task `preconditioner`
[2024-11-15 13:04:00,999] INFO: 1 new tasks have been issued.
[2024-11-15 13:04:01,000] INFO: Processing task `misfit`
[2024-11-15 13:04:01,082] INFO: Submitting job array with 5 jobs ...
[2024-11-15 13:04:01,234] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-15 13:04:01,236] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:04:03,290] INFO: Processing task `misfit`
[2024-11-15 13:04:03,364] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:04:05,429] INFO: Processing task `misfit`
[2024-11-15 13:04:05,523] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-15 13:04:07,579] INFO: Processing task `misfit`
[2024-11-15 13:04:10,241] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2024-11-15 13:04:10,242] INFO:
Model update accepted.
[2024-11-15 13:04:10,242] INFO: 1 new tasks have been issued.
[2024-11-15 13:04:10,243] INFO: Processing task `finalize_iteration`
[2024-11-15 13:04:10,350] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2024-11-15 13:04:10,557] INFO: Freed up 4.8 MB of space.
[2024-11-15 13:04:10,557] INFO: Succesfully completed iteration #1.
[2024-11-15 13:04:10,560] INFO: Adding new iteration #2.