[2025-02-07 19:42:53,060] INFO: Adding new iteration #0.
[2025-02-07 19:42:53,074] INFO: Resuming iteration #0.
[2025-02-07 19:42:53,076] INFO: 1 new tasks have been issued.
[2025-02-07 19:42:53,076] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:42:53,199] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2025-02-07 19:42:53,238] INFO: Submitting job array with 5 jobs ...
[2025-02-07 19:42:53,998] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-02-07 19:42:54,001] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:42:56,003] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:42:56,168] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:42:58,170] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:42:58,332] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:00,334] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:00,501] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:02,503] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:02,663] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:04,665] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:04,867] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:06,871] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:07,041] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:09,044] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:11,836] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2025-02-07 19:43:11,933] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-02-07 19:43:11,935] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:13,937] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:14,836] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:43:14,957] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:43:14,959] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:16,961] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:17,471] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:43:17,608] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:43:17,611] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:19,613] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:20,036] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:43:20,145] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:43:20,148] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:22,150] INFO: Processing task `misfit_and_gradient`
[2025-02-07 19:43:22,679] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:43:23,488] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2025-02-07 19:43:23,489] INFO: 1 new tasks have been issued.
[2025-02-07 19:43:23,490] INFO: Processing task `preconditioner`
[2025-02-07 19:43:23,632] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:25,652] INFO: Processing task `preconditioner`
[2025-02-07 19:43:25,727] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:27,771] INFO: Processing task `preconditioner`
[2025-02-07 19:43:27,845] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:29,866] INFO: Processing task `preconditioner`
[2025-02-07 19:43:29,939] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:31,960] INFO: Processing task `preconditioner`
[2025-02-07 19:43:32,063] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:34,090] INFO: Processing task `preconditioner`
[2025-02-07 19:43:34,386] INFO: 1 new tasks have been issued.
[2025-02-07 19:43:34,387] INFO: Processing task `misfit`
[2025-02-07 19:43:34,507] INFO: Submitting job array with 5 jobs ...
[2025-02-07 19:43:34,818] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-02-07 19:43:34,826] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:36,881] INFO: Processing task `misfit`
[2025-02-07 19:43:36,980] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:39,015] INFO: Processing task `misfit`
[2025-02-07 19:43:39,180] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:41,223] INFO: Processing task `misfit`
[2025-02-07 19:43:41,742] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:43,780] INFO: Processing task `misfit`
[2025-02-07 19:43:43,868] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:45,908] INFO: Processing task `misfit`
[2025-02-07 19:43:46,012] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:48,060] INFO: Processing task `misfit`
[2025-02-07 19:43:48,164] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:50,206] INFO: Processing task `misfit`
[2025-02-07 19:43:50,363] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:52,426] INFO: Processing task `misfit`
[2025-02-07 19:43:52,534] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:54,591] INFO: Processing task `misfit`
[2025-02-07 19:43:54,716] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:56,791] INFO: Processing task `misfit`
[2025-02-07 19:43:56,924] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:43:58,958] INFO: Processing task `misfit`
[2025-02-07 19:43:59,065] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:44:01,116] INFO: Processing task `misfit`
[2025-02-07 19:44:01,225] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:44:03,281] INFO: Processing task `misfit`
[2025-02-07 19:44:03,395] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:44:05,437] INFO: Processing task `misfit`
[2025-02-07 19:44:05,885] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-02-07 19:44:07,921] INFO: Processing task `misfit`
[2025-02-07 19:44:10,294] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2025-02-07 19:44:10,294] INFO:
Model update accepted.
[2025-02-07 19:44:10,302] INFO: 1 new tasks have been issued.
[2025-02-07 19:44:10,303] INFO: Processing task `finalize_iteration`
[2025-02-07 19:44:10,452] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2025-02-07 19:44:10,518] INFO: Freed up 1.2 MB of space.
[2025-02-07 19:44:10,519] INFO: Succesfully completed iteration #0.
[2025-02-07 19:44:10,521] INFO: Adding new iteration #1.
[2025-02-07 19:44:10,528] INFO: Resuming iteration #1.
[2025-02-07 19:44:10,528] INFO: 1 new tasks have been issued.
[2025-02-07 19:44:10,529] INFO: Processing task `gradient`
[2025-02-07 19:44:10,904] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2025-02-07 19:44:11,025] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-02-07 19:44:11,027] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:13,037] INFO: Processing task `gradient`
[2025-02-07 19:44:13,439] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:13,593] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:44:13,596] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:15,599] INFO: Processing task `gradient`
[2025-02-07 19:44:15,912] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:16,052] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:44:16,061] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:18,070] INFO: Processing task `gradient`
[2025-02-07 19:44:18,379] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:18,497] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:44:18,503] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:20,506] INFO: Processing task `gradient`
[2025-02-07 19:44:20,732] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:20,851] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:44:20,856] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:22,858] INFO: Processing task `gradient`
[2025-02-07 19:44:23,195] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:23,339] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:44:23,342] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:25,346] INFO: Processing task `gradient`
[2025-02-07 19:44:25,663] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:25,858] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-02-07 19:44:25,863] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:27,866] INFO: Processing task `gradient`
[2025-02-07 19:44:28,780] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-02-07 19:44:29,591] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2025-02-07 19:44:29,611] INFO: 1 new tasks have been issued.
[2025-02-07 19:44:29,612] INFO: Processing task `preconditioner`
[2025-02-07 19:44:29,796] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:31,839] INFO: Processing task `preconditioner`
[2025-02-07 19:44:31,933] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:33,968] INFO: Processing task `preconditioner`
[2025-02-07 19:44:34,041] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:36,082] INFO: Processing task `preconditioner`
[2025-02-07 19:44:36,204] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:38,244] INFO: Processing task `preconditioner`
[2025-02-07 19:44:38,304] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:40,339] INFO: Processing task `preconditioner`
[2025-02-07 19:44:40,451] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:42,483] INFO: Processing task `preconditioner`
[2025-02-07 19:44:42,617] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:44,665] INFO: Processing task `preconditioner`
[2025-02-07 19:44:44,868] INFO: 1 new tasks have been issued.
[2025-02-07 19:44:44,868] INFO: Processing task `misfit`
[2025-02-07 19:44:44,943] INFO: Submitting job array with 5 jobs ...
[2025-02-07 19:44:45,172] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-02-07 19:44:45,181] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:47,256] INFO: Processing task `misfit`
[2025-02-07 19:44:47,359] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:49,430] INFO: Processing task `misfit`
[2025-02-07 19:44:49,520] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:51,584] INFO: Processing task `misfit`
[2025-02-07 19:44:51,679] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:53,761] INFO: Processing task `misfit`
[2025-02-07 19:44:53,881] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:55,958] INFO: Processing task `misfit`
[2025-02-07 19:44:56,067] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:44:58,131] INFO: Processing task `misfit`
[2025-02-07 19:44:58,221] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:00,302] INFO: Processing task `misfit`
[2025-02-07 19:45:00,397] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:02,461] INFO: Processing task `misfit`
[2025-02-07 19:45:02,913] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:04,964] INFO: Processing task `misfit`
[2025-02-07 19:45:05,054] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:07,143] INFO: Processing task `misfit`
[2025-02-07 19:45:07,262] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:09,327] INFO: Processing task `misfit`
[2025-02-07 19:45:09,409] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:11,485] INFO: Processing task `misfit`
[2025-02-07 19:45:11,587] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-02-07 19:45:13,675] INFO: Processing task `misfit`
[2025-02-07 19:45:16,424] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2025-02-07 19:45:16,426] INFO:
Model update accepted.
[2025-02-07 19:45:16,427] INFO: 1 new tasks have been issued.
[2025-02-07 19:45:16,429] INFO: Processing task `finalize_iteration`
[2025-02-07 19:45:16,544] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2025-02-07 19:45:16,830] INFO: Freed up 4.8 MB of space.
[2025-02-07 19:45:16,833] INFO: Succesfully completed iteration #1.
[2025-02-07 19:45:16,837] INFO: Adding new iteration #2.