[2024-07-05 09:50:41,783] INFO: Adding new iteration #0.
[2024-07-05 09:50:41,797] INFO: Resuming iteration #0.
[2024-07-05 09:50:41,798] INFO: 1 new tasks have been issued.
[2024-07-05 09:50:41,798] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:41,943] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2024-07-05 09:50:41,991] INFO: Submitting job array with 5 jobs ...
[2024-07-05 09:50:42,207] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-07-05 09:50:42,209] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:44,211] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:44,370] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:46,371] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:46,518] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:48,520] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:48,721] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:50,723] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:52,522] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-07-05 09:50:52,659] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-07-05 09:50:52,662] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:54,664] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:55,187] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:50:55,306] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:50:55,308] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:57,309] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:50:57,819] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:50:57,927] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:50:57,929] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:50:59,931] INFO: Processing task `misfit_and_gradient`
[2024-07-05 09:51:00,456] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:01,093] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2024-07-05 09:51:01,094] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:01,094] INFO: Processing task `preconditioner`
[2024-07-05 09:51:01,203] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:03,211] INFO: Processing task `preconditioner`
[2024-07-05 09:51:03,256] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:05,266] INFO: Processing task `preconditioner`
[2024-07-05 09:51:05,396] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:05,397] INFO: Processing task `misfit`
[2024-07-05 09:51:05,479] INFO: Submitting job array with 5 jobs ...
[2024-07-05 09:51:05,634] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-07-05 09:51:05,636] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:07,664] INFO: Processing task `misfit`
[2024-07-05 09:51:07,743] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:09,768] INFO: Processing task `misfit`
[2024-07-05 09:51:10,198] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:12,227] INFO: Processing task `misfit`
[2024-07-05 09:51:12,307] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:14,336] INFO: Processing task `misfit`
[2024-07-05 09:51:14,409] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:16,434] INFO: Processing task `misfit`
[2024-07-05 09:51:16,522] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:18,555] INFO: Processing task `misfit`
[2024-07-05 09:51:18,625] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:20,650] INFO: Processing task `misfit`
[2024-07-05 09:51:20,714] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-07-05 09:51:22,736] INFO: Processing task `misfit`
[2024-07-05 09:51:23,653] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2024-07-05 09:51:23,654] INFO:
Model update accepted.
[2024-07-05 09:51:23,655] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:23,655] INFO: Processing task `finalize_iteration`
[2024-07-05 09:51:23,708] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2024-07-05 09:51:23,717] INFO: Freed up 1.2 MB of space.
[2024-07-05 09:51:23,718] INFO: Succesfully completed iteration #0.
[2024-07-05 09:51:23,719] INFO: Adding new iteration #1.
[2024-07-05 09:51:23,725] INFO: Resuming iteration #1.
[2024-07-05 09:51:23,725] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:23,726] INFO: Processing task `gradient`
[2024-07-05 09:51:24,067] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-07-05 09:51:24,168] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-07-05 09:51:24,170] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:26,172] INFO: Processing task `gradient`
[2024-07-05 09:51:26,464] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:26,583] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:51:26,585] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:28,587] INFO: Processing task `gradient`
[2024-07-05 09:51:28,856] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:28,970] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:51:28,971] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:30,974] INFO: Processing task `gradient`
[2024-07-05 09:51:31,278] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:31,397] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:51:31,399] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:33,401] INFO: Processing task `gradient`
[2024-07-05 09:51:34,005] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:34,123] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:51:34,125] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:36,127] INFO: Processing task `gradient`
[2024-07-05 09:51:36,393] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:36,489] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:51:36,491] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:38,493] INFO: Processing task `gradient`
[2024-07-05 09:51:38,790] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:38,911] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-07-05 09:51:38,912] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:40,915] INFO: Processing task `gradient`
[2024-07-05 09:51:41,231] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-07-05 09:51:41,931] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2024-07-05 09:51:41,966] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:41,968] INFO: Processing task `preconditioner`
[2024-07-05 09:51:42,118] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:44,150] INFO: Processing task `preconditioner`
[2024-07-05 09:51:44,217] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:46,242] INFO: Processing task `preconditioner`
[2024-07-05 09:51:46,401] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:46,402] INFO: Processing task `misfit`
[2024-07-05 09:51:46,478] INFO: Submitting job array with 5 jobs ...
[2024-07-05 09:51:46,623] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-07-05 09:51:46,625] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:48,681] INFO: Processing task `misfit`
[2024-07-05 09:51:48,757] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:50,794] INFO: Processing task `misfit`
[2024-07-05 09:51:50,865] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:52,906] INFO: Processing task `misfit`
[2024-07-05 09:51:52,987] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-07-05 09:51:55,025] INFO: Processing task `misfit`
[2024-07-05 09:51:56,160] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2024-07-05 09:51:56,161] INFO:
Model update accepted.
[2024-07-05 09:51:56,161] INFO: 1 new tasks have been issued.
[2024-07-05 09:51:56,162] INFO: Processing task `finalize_iteration`
[2024-07-05 09:51:56,234] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2024-07-05 09:51:56,420] INFO: Freed up 4.8 MB of space.
[2024-07-05 09:51:56,421] INFO: Succesfully completed iteration #1.
[2024-07-05 09:51:56,423] INFO: Adding new iteration #2.