[2024-07-02 13:45:04,532] INFO: Adding new iteration #0.
[2024-07-02 13:45:04,554] INFO: Resuming iteration #0.
[2024-07-02 13:45:04,562] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:04,566] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:04,790] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2024-07-02 13:45:04,839] INFO: Submitting job array with 5 jobs ...
[2024-07-02 13:45:05,195] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-07-02 13:45:05,199] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:07,201] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:07,531] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:09,534] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:09,854] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:11,864] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:14,385] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-07-02 13:45:14,601] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-07-02 13:45:14,608] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:16,611] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:17,393] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-02 13:45:17,632] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-02 13:45:17,635] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:19,637] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:20,355] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-02 13:45:20,526] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-02 13:45:20,532] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:22,534] INFO: Processing task `misfit_and_gradient`
[2024-07-02 13:45:23,226] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-02 13:45:24,110] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2024-07-02 13:45:24,114] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:24,115] INFO: Processing task `preconditioner`
[2024-07-02 13:45:24,269] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:26,282] INFO: Processing task `preconditioner`
[2024-07-02 13:45:26,327] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:28,342] INFO: Processing task `preconditioner`
[2024-07-02 13:45:28,540] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:28,540] INFO: Processing task `misfit`
[2024-07-02 13:45:28,640] INFO: Submitting job array with 5 jobs ...
[2024-07-02 13:45:28,858] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-07-02 13:45:28,860] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:30,900] INFO: Processing task `misfit`
[2024-07-02 13:45:31,024] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:33,055] INFO: Processing task `misfit`
[2024-07-02 13:45:33,628] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-02 13:45:35,666] INFO: Processing task `misfit`
[2024-07-02 13:45:36,782] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2024-07-02 13:45:36,784] INFO:
Model update accepted.
[2024-07-02 13:45:36,790] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:36,791] INFO: Processing task `finalize_iteration`
[2024-07-02 13:45:36,871] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2024-07-02 13:45:36,886] INFO: Freed up 1.2 MB of space.
[2024-07-02 13:45:36,887] INFO: Succesfully completed iteration #0.
[2024-07-02 13:45:36,889] INFO: Adding new iteration #1.
[2024-07-02 13:45:36,898] INFO: Resuming iteration #1.
[2024-07-02 13:45:36,906] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:36,907] INFO: Processing task `gradient`
[2024-07-02 13:45:37,554] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-07-02 13:45:37,739] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-07-02 13:45:37,742] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:39,749] INFO: Processing task `gradient`
[2024-07-02 13:45:40,178] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-02 13:45:40,355] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-02 13:45:40,358] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:42,362] INFO: Processing task `gradient`
[2024-07-02 13:45:42,806] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-02 13:45:43,023] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-02 13:45:43,030] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:45,033] INFO: Processing task `gradient`
[2024-07-02 13:45:45,469] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-02 13:45:46,334] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2024-07-02 13:45:46,373] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:46,375] INFO: Processing task `preconditioner`
[2024-07-02 13:45:46,587] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:48,640] INFO: Processing task `preconditioner`
[2024-07-02 13:45:48,825] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:48,830] INFO: Processing task `misfit`
[2024-07-02 13:45:49,023] INFO: Submitting job array with 5 jobs ...
[2024-07-02 13:45:49,283] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-07-02 13:45:49,288] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:51,341] INFO: Processing task `misfit`
[2024-07-02 13:45:51,899] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:53,957] INFO: Processing task `misfit`
[2024-07-02 13:45:54,089] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-02 13:45:56,139] INFO: Processing task `misfit`
[2024-07-02 13:45:57,272] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2024-07-02 13:45:57,273] INFO:
Model update accepted.
[2024-07-02 13:45:57,274] INFO: 1 new tasks have been issued.
[2024-07-02 13:45:57,274] INFO: Processing task `finalize_iteration`
[2024-07-02 13:45:57,375] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2024-07-02 13:45:57,637] INFO: Freed up 4.8 MB of space.
[2024-07-02 13:45:57,638] INFO: Succesfully completed iteration #1.
[2024-07-02 13:45:57,641] INFO: Adding new iteration #2.