[2024-11-06 22:06:38,105] INFO: Adding new iteration #0.
[2024-11-06 22:06:38,130] INFO: Resuming iteration #0.
[2024-11-06 22:06:38,131] INFO: 1 new tasks have been issued.
[2024-11-06 22:06:38,135] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:38,266] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2024-11-06 22:06:38,320] INFO: Submitting job array with 5 jobs ...
[2024-11-06 22:06:38,538] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-06 22:06:38,540] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:40,542] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:40,727] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:42,728] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:42,875] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:44,877] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:45,070] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:47,074] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:51,820] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-11-06 22:06:51,957] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-11-06 22:06:51,960] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:53,963] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:54,547] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:06:54,667] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:06:54,671] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:56,675] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:57,134] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:06:57,257] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:06:57,260] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:06:59,262] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:06:59,677] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:06:59,790] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:06:59,792] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:01,793] INFO: Processing task `misfit_and_gradient`
[2024-11-06 22:07:02,224] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:07:03,020] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2024-11-06 22:07:03,021] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:03,024] INFO: Processing task `preconditioner`
[2024-11-06 22:07:03,195] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:05,261] INFO: Processing task `preconditioner`
[2024-11-06 22:07:05,386] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:07,438] INFO: Processing task `preconditioner`
[2024-11-06 22:07:07,717] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:07,720] INFO: Processing task `misfit`
[2024-11-06 22:07:07,808] INFO: Submitting job array with 5 jobs ...
[2024-11-06 22:07:08,617] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-06 22:07:08,619] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:10,660] INFO: Processing task `misfit`
[2024-11-06 22:07:10,750] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:12,809] INFO: Processing task `misfit`
[2024-11-06 22:07:12,912] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:14,950] INFO: Processing task `misfit`
[2024-11-06 22:07:15,043] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2024-11-06 22:07:17,083] INFO: Processing task `misfit`
[2024-11-06 22:07:19,704] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2024-11-06 22:07:19,704] INFO:
Model update accepted.
[2024-11-06 22:07:19,705] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:19,705] INFO: Processing task `finalize_iteration`
[2024-11-06 22:07:19,787] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2024-11-06 22:07:19,811] INFO: Freed up 1.2 MB of space.
[2024-11-06 22:07:19,812] INFO: Succesfully completed iteration #0.
[2024-11-06 22:07:19,815] INFO: Adding new iteration #1.
[2024-11-06 22:07:19,823] INFO: Resuming iteration #1.
[2024-11-06 22:07:19,824] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:19,825] INFO: Processing task `gradient`
[2024-11-06 22:07:20,127] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2024-11-06 22:07:20,218] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2024-11-06 22:07:20,220] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:22,223] INFO: Processing task `gradient`
[2024-11-06 22:07:22,451] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:07:22,536] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:07:22,539] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:24,542] INFO: Processing task `gradient`
[2024-11-06 22:07:24,814] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:07:24,930] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:07:24,935] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:26,940] INFO: Processing task `gradient`
[2024-11-06 22:07:27,282] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:07:27,459] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:07:27,462] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:29,464] INFO: Processing task `gradient`
[2024-11-06 22:07:29,727] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:07:29,852] INFO: Some simulations are still running. Please check again to see if they are finished.
[2024-11-06 22:07:29,857] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:31,859] INFO: Processing task `gradient`
[2024-11-06 22:07:32,412] INFO: 5 events have already been submitted. They will not be submitted again.
[2024-11-06 22:07:33,024] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2024-11-06 22:07:33,041] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:33,043] INFO: Processing task `preconditioner`
[2024-11-06 22:07:33,223] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:35,270] INFO: Processing task `preconditioner`
[2024-11-06 22:07:35,359] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:37,405] INFO: Processing task `preconditioner`
[2024-11-06 22:07:37,617] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:37,621] INFO: Processing task `misfit`
[2024-11-06 22:07:37,705] INFO: Submitting job array with 5 jobs ...
[2024-11-06 22:07:37,949] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2024-11-06 22:07:37,952] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:40,016] INFO: Processing task `misfit`
[2024-11-06 22:07:40,130] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:42,193] INFO: Processing task `misfit`
[2024-11-06 22:07:42,284] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2024-11-06 22:07:44,359] INFO: Processing task `misfit`
[2024-11-06 22:07:46,923] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2024-11-06 22:07:46,927] INFO:
Model update accepted.
[2024-11-06 22:07:46,928] INFO: 1 new tasks have been issued.
[2024-11-06 22:07:46,929] INFO: Processing task `finalize_iteration`
[2024-11-06 22:07:47,060] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2024-11-06 22:07:47,342] INFO: Freed up 4.8 MB of space.
[2024-11-06 22:07:47,344] INFO: Succesfully completed iteration #1.
[2024-11-06 22:07:47,347] INFO: Adding new iteration #2.