[2025-01-09 21:39:35,080] INFO: Adding new iteration #0.
[2025-01-09 21:39:35,086] INFO: Resuming iteration #0.
[2025-01-09 21:39:35,087] INFO: 1 new tasks have been issued.
[2025-01-09 21:39:35,087] INFO: Processing task `misfit_and_gradient`
[2025-01-09 21:39:35,150] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2025-01-09 21:39:35,170] INFO: Submitting job array with 5 jobs ...
[2025-01-09 21:39:35,258] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-01-09 21:39:35,259] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:37,260] INFO: Processing task `misfit_and_gradient`
[2025-01-09 21:39:37,346] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:39,348] INFO: Processing task `misfit_and_gradient`
[2025-01-09 21:39:41,070] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2025-01-09 21:39:41,122] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-01-09 21:39:41,123] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:43,124] INFO: Processing task `misfit_and_gradient`
[2025-01-09 21:39:43,355] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-01-09 21:39:43,417] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 21:39:43,418] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:45,419] INFO: Processing task `misfit_and_gradient`
[2025-01-09 21:39:45,641] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-01-09 21:39:46,228] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2025-01-09 21:39:46,229] INFO: 1 new tasks have been issued.
[2025-01-09 21:39:46,229] INFO: Processing task `preconditioner`
[2025-01-09 21:39:46,295] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:48,305] INFO: Processing task `preconditioner`
[2025-01-09 21:39:48,422] INFO: 1 new tasks have been issued.
[2025-01-09 21:39:48,422] INFO: Processing task `misfit`
[2025-01-09 21:39:48,464] INFO: Submitting job array with 5 jobs ...
[2025-01-09 21:39:48,551] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-01-09 21:39:48,552] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:50,574] INFO: Processing task `misfit`
[2025-01-09 21:39:50,621] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 21:39:52,643] INFO: Processing task `misfit`
[2025-01-09 21:39:54,109] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2025-01-09 21:39:54,110] INFO:
Model update accepted.
[2025-01-09 21:39:54,110] INFO: 1 new tasks have been issued.
[2025-01-09 21:39:54,110] INFO: Processing task `finalize_iteration`
[2025-01-09 21:39:54,152] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2025-01-09 21:39:54,162] INFO: Freed up 1.2 MB of space.
[2025-01-09 21:39:54,162] INFO: Succesfully completed iteration #0.
[2025-01-09 21:39:54,164] INFO: Adding new iteration #1.
[2025-01-09 21:39:54,168] INFO: Resuming iteration #1.
[2025-01-09 21:39:54,169] INFO: 1 new tasks have been issued.
[2025-01-09 21:39:54,169] INFO: Processing task `gradient`
[2025-01-09 21:39:54,350] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2025-01-09 21:39:54,407] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-01-09 21:39:54,408] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 21:39:56,410] INFO: Processing task `gradient`
[2025-01-09 21:39:56,535] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-01-09 21:39:56,598] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 21:39:56,600] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 21:39:58,601] INFO: Processing task `gradient`
[2025-01-09 21:39:58,728] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-01-09 21:39:59,119] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2025-01-09 21:39:59,129] INFO: 1 new tasks have been issued.
[2025-01-09 21:39:59,129] INFO: Processing task `preconditioner`
[2025-01-09 21:39:59,204] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 21:40:01,224] INFO: Processing task `preconditioner`
[2025-01-09 21:40:01,528] INFO: 1 new tasks have been issued.
[2025-01-09 21:40:01,528] INFO: Processing task `misfit`
[2025-01-09 21:40:01,567] INFO: Submitting job array with 5 jobs ...
[2025-01-09 21:40:01,657] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-01-09 21:40:01,658] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 21:40:03,691] INFO: Processing task `misfit`
[2025-01-09 21:40:03,738] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 21:40:05,773] INFO: Processing task `misfit`
[2025-01-09 21:40:07,243] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2025-01-09 21:40:07,243] INFO:
Model update accepted.
[2025-01-09 21:40:07,243] INFO: 1 new tasks have been issued.
[2025-01-09 21:40:07,244] INFO: Processing task `finalize_iteration`
[2025-01-09 21:40:07,305] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2025-01-09 21:40:07,416] INFO: Freed up 4.8 MB of space.
[2025-01-09 21:40:07,416] INFO: Succesfully completed iteration #1.
[2025-01-09 21:40:07,418] INFO: Adding new iteration #2.