[2022-08-03 09:14:55,226] INFO: Adding new iteration #0.
[2022-08-03 09:14:55,241] INFO: Resuming iteration #0.
[2022-08-03 09:14:55,242] INFO: 1 new tasks have been issued.
[2022-08-03 09:14:55,243] INFO: Processing task `misfit_and_gradient`
[2022-08-03 09:14:55,282] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event']
[2022-08-03 09:14:55,303] INFO: Submitting job ...
[2022-08-03 09:14:55,402] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:14:56,404] INFO: Processing task `misfit_and_gradient`
[2022-08-03 09:14:56,488] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2022-08-03 09:14:57,492] INFO: Processing task `misfit_and_gradient`
[2022-08-03 09:14:57,626] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2022-08-03 09:14:58,630] INFO: Processing task `misfit_and_gradient`
[2022-08-03 09:14:59,206] INFO: Submitting job ...
[2022-08-03 09:14:59,279] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:14:59,280] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2022-08-03 09:15:00,284] INFO: Processing task `misfit_and_gradient`
[2022-08-03 09:15:00,483] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:00,609] INFO: Some simulations are still running. Please check again to see if they are finished.
[2022-08-03 09:15:00,610] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2022-08-03 09:15:01,615] INFO: Processing task `misfit_and_gradient`
[2022-08-03 09:15:01,798] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:01,994] INFO:
Iteration 0: Number of events: 1 chi = 3.7156538945162333e-10 ||g|| = 3.3910352108304556e-12
pred = --- ared = --- norm_update = --- tr_radius = ---
[2022-08-03 09:15:02,015] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:02,016] INFO: Processing task `misfit`
[2022-08-03 09:15:02,064] INFO: Submitting job ...
[2022-08-03 09:15:02,141] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:03,160] INFO: Processing task `misfit`
[2022-08-03 09:15:03,315] INFO:
old misfit control group: 3.7156538945162333e-10
new misfit control group: 1.073889926771491e-10
predicted reduction control group: -2.1653938171027937e-10
actual reduction control group: -2.6417639677447424e-10
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:03,316] INFO:
Model update accepted.
[2022-08-03 09:15:03,329] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:03,330] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:03,357] INFO: Succesfully completed iteration #0.
[2022-08-03 09:15:03,360] INFO: Adding new iteration #1.
[2022-08-03 09:15:03,369] INFO: Resuming iteration #1.
[2022-08-03 09:15:03,370] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:03,371] INFO: Processing task `gradient`
[2022-08-03 09:15:03,502] INFO: Submitting job ...
[2022-08-03 09:15:03,585] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:04,591] INFO: Processing task `gradient`
[2022-08-03 09:15:04,708] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:04,794] INFO: Some simulations are still running. Please check again to see if they are finished.
[2022-08-03 09:15:04,795] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2022-08-03 09:15:05,800] INFO: Processing task `gradient`
[2022-08-03 09:15:05,887] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:06,249] INFO:
Iteration 1: Number of events: 1 chi = 1.073889926771491e-10 ||g|| = 1.0372298369038717e-12
pred = -2.1653938171027937e-10 ared = -2.6417639677447424e-10 norm_update = 127.71284769835779 tr_radius = 127.71284769835776
[2022-08-03 09:15:06,283] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:06,284] INFO: Processing task `misfit`
[2022-08-03 09:15:06,335] INFO: Submitting job ...
[2022-08-03 09:15:06,419] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:07,440] INFO: Processing task `misfit`
[2022-08-03 09:15:07,488] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2022-08-03 09:15:08,507] INFO: Processing task `misfit`
[2022-08-03 09:15:08,699] INFO:
old misfit control group: 1.073889926771491e-10
new misfit control group: 7.29671608458056e-11
predicted reduction control group: -1.886437562563871e-11
actual reduction control group: -3.44218318313435e-11
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:08,700] INFO:
Model update accepted.
[2022-08-03 09:15:08,719] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:08,720] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:08,755] INFO: Succesfully completed iteration #1.
[2022-08-03 09:15:08,760] INFO: Adding new iteration #2.
[2022-08-03 09:15:08,773] INFO: Resuming iteration #2.
[2022-08-03 09:15:08,774] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:08,775] INFO: Processing task `gradient`
[2022-08-03 09:15:08,932] INFO: Submitting job ...
[2022-08-03 09:15:09,015] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:10,020] INFO: Processing task `gradient`
[2022-08-03 09:15:10,145] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:10,342] INFO:
Iteration 2: Number of events: 1 chi = 7.29671608458056e-11 ||g|| = 8.841261519831265e-13
pred = -1.886437562563871e-11 ared = -3.44218318313435e-11 norm_update = 38.0551693560016 tr_radius = 255.42569539671553
[2022-08-03 09:15:10,383] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:10,384] INFO: Processing task `misfit`
[2022-08-03 09:15:10,428] INFO: Submitting job ...
[2022-08-03 09:15:10,514] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:11,536] INFO: Processing task `misfit`
[2022-08-03 09:15:11,646] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2022-08-03 09:15:12,672] INFO: Processing task `misfit`
[2022-08-03 09:15:12,775] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2022-08-03 09:15:13,792] INFO: Processing task `misfit`
[2022-08-03 09:15:13,956] INFO:
old misfit control group: 7.29671608458056e-11
new misfit control group: 7.299256037681704e-12
predicted reduction control group: -5.394786815026022e-11
actual reduction control group: -6.566790480812389e-11
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:13,957] INFO:
Model update accepted.
[2022-08-03 09:15:13,973] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:13,973] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:14,005] INFO: Succesfully completed iteration #2.
[2022-08-03 09:15:14,009] INFO: Adding new iteration #3.
[2022-08-03 09:15:14,023] INFO: Resuming iteration #3.
[2022-08-03 09:15:14,024] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:14,025] INFO: Processing task `gradient`
[2022-08-03 09:15:14,154] INFO: Submitting job ...
[2022-08-03 09:15:14,237] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:15,243] INFO: Processing task `gradient`
[2022-08-03 09:15:15,347] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:15,552] INFO:
Iteration 3: Number of events: 1 chi = 7.299256037681704e-12 ||g|| = 2.1081376205856607e-13
pred = -5.394786815026022e-11 ared = -6.566790480812389e-11 norm_update = 127.7128476983578 tr_radius = 127.71284769835776
[2022-08-03 09:15:15,603] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:15,604] INFO: Processing task `misfit`
[2022-08-03 09:15:15,663] INFO: Submitting job ...
[2022-08-03 09:15:15,762] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:16,788] INFO: Processing task `misfit`
[2022-08-03 09:15:16,886] INFO: Some tasks of iteration #3 are still running. Please check again later.
[2022-08-03 09:15:17,918] INFO: Processing task `misfit`
[2022-08-03 09:15:18,101] INFO:
old misfit control group: 7.299256037681704e-12
new misfit control group: 3.854014700886898e-12
predicted reduction control group: -3.625596729119787e-12
actual reduction control group: -3.445241336794806e-12
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:18,102] INFO:
Model update accepted.
[2022-08-03 09:15:18,117] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:18,118] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:18,158] INFO: Succesfully completed iteration #3.
[2022-08-03 09:15:18,165] INFO: Adding new iteration #4.
[2022-08-03 09:15:18,189] INFO: Resuming iteration #4.
[2022-08-03 09:15:18,190] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:18,190] INFO: Processing task `gradient`
[2022-08-03 09:15:18,327] INFO: Submitting job ...
[2022-08-03 09:15:18,405] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:19,412] INFO: Processing task `gradient`
[2022-08-03 09:15:19,533] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:19,744] INFO:
Iteration 4: Number of events: 1 chi = 3.854014700886898e-12 ||g|| = 5.49268495605808e-14
pred = -3.625596729119787e-12 ared = -3.445241336794806e-12 norm_update = 37.63481392457643 tr_radius = 255.42569539671553
[2022-08-03 09:15:19,806] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:19,807] INFO: Processing task `misfit`
[2022-08-03 09:15:19,850] INFO: Submitting job ...
[2022-08-03 09:15:19,954] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:20,972] INFO: Processing task `misfit`
[2022-08-03 09:15:21,020] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2022-08-03 09:15:22,043] INFO: Processing task `misfit`
[2022-08-03 09:15:22,193] INFO:
old misfit control group: 3.854014700886898e-12
new misfit control group: 3.660829188352088e-12
predicted reduction control group: -1.1396020048819344e-13
actual reduction control group: -1.9318551253481e-13
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:22,194] INFO:
Model update accepted.
[2022-08-03 09:15:22,209] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:22,210] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:22,242] INFO: Succesfully completed iteration #4.
[2022-08-03 09:15:22,251] INFO: Adding new iteration #5.
[2022-08-03 09:15:22,272] INFO: Resuming iteration #5.
[2022-08-03 09:15:22,273] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:22,273] INFO: Processing task `gradient`
[2022-08-03 09:15:22,436] INFO: Submitting job ...
[2022-08-03 09:15:22,520] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:23,530] INFO: Processing task `gradient`
[2022-08-03 09:15:23,614] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:23,699] INFO: Some simulations are still running. Please check again to see if they are finished.
[2022-08-03 09:15:23,700] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2022-08-03 09:15:24,707] INFO: Processing task `gradient`
[2022-08-03 09:15:24,820] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:25,041] INFO:
Iteration 5: Number of events: 1 chi = 3.660829188352088e-12 ||g|| = 3.682497961048612e-14
pred = -1.1396020048819344e-13 ared = -1.9318551253481e-13 norm_update = 4.998486563033788 tr_radius = 127.71284769835776
[2022-08-03 09:15:25,121] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:25,122] INFO: Processing task `misfit`
[2022-08-03 09:15:25,171] INFO: Submitting job ...
[2022-08-03 09:15:25,252] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:26,272] INFO: Processing task `misfit`
[2022-08-03 09:15:26,338] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2022-08-03 09:15:27,356] INFO: Processing task `misfit`
[2022-08-03 09:15:27,566] INFO:
old misfit control group: 3.660829188352088e-12
new misfit control group: 2.9816817056581784e-12
predicted reduction control group: -4.4282031537730207e-13
actual reduction control group: -6.791474826939099e-13
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:27,567] INFO:
Model update accepted.
[2022-08-03 09:15:27,588] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:27,589] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:27,630] INFO: Succesfully completed iteration #5.
[2022-08-03 09:15:27,636] INFO: Adding new iteration #6.
[2022-08-03 09:15:27,660] INFO: Resuming iteration #6.
[2022-08-03 09:15:27,661] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:27,662] INFO: Processing task `gradient`
[2022-08-03 09:15:27,788] INFO: Submitting job ...
[2022-08-03 09:15:27,874] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:28,882] INFO: Processing task `gradient`
[2022-08-03 09:15:29,017] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:29,220] INFO:
Iteration 6: Number of events: 1 chi = 2.9816817056581784e-12 ||g|| = 1.1331505473068221e-13
pred = -4.4282031537730207e-13 ared = -6.791474826939099e-13 norm_update = 25.921529889997185 tr_radius = 63.85642384917888
[2022-08-03 09:15:29,311] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:29,312] INFO: Processing task `misfit`
[2022-08-03 09:15:29,358] INFO: Submitting job ...
[2022-08-03 09:15:29,442] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:30,468] INFO: Processing task `misfit`
[2022-08-03 09:15:30,592] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2022-08-03 09:15:31,615] INFO: Processing task `misfit`
[2022-08-03 09:15:31,772] INFO:
old misfit control group: 2.9816817056581784e-12
new misfit control group: 1.9569397963199763e-12
predicted reduction control group: -6.667460761684826e-13
actual reduction control group: -1.024741909338202e-12
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:31,773] INFO:
Model update accepted.
[2022-08-03 09:15:31,791] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:31,792] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:31,824] INFO: Succesfully completed iteration #6.
[2022-08-03 09:15:31,834] INFO: Adding new iteration #7.
[2022-08-03 09:15:31,862] INFO: Resuming iteration #7.
[2022-08-03 09:15:31,863] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:31,863] INFO: Processing task `gradient`
[2022-08-03 09:15:32,030] INFO: Submitting job ...
[2022-08-03 09:15:32,129] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:33,139] INFO: Processing task `gradient`
[2022-08-03 09:15:33,229] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:33,306] INFO: Some simulations are still running. Please check again to see if they are finished.
[2022-08-03 09:15:33,307] INFO: Some tasks of iteration #7 are still running. Please check again later.
[2022-08-03 09:15:34,320] INFO: Processing task `gradient`
[2022-08-03 09:15:34,402] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:34,598] INFO:
Iteration 7: Number of events: 1 chi = 1.9569397963199763e-12 ||g|| = 1.8079729535611823e-13
pred = -6.667460761684826e-13 ared = -1.024741909338202e-12 norm_update = 51.843059779994626 tr_radius = 51.84305977999437
[2022-08-03 09:15:34,701] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:34,702] INFO: Processing task `misfit`
[2022-08-03 09:15:34,746] INFO: Submitting job ...
[2022-08-03 09:15:34,830] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:35,850] INFO: Processing task `misfit`
[2022-08-03 09:15:36,016] INFO:
old misfit control group: 1.9569397963199763e-12
new misfit control group: 7.204028930773391e-13
predicted reduction control group: -8.890227313369757e-13
actual reduction control group: -1.2365369032426372e-12
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:36,017] INFO:
Model update accepted.
[2022-08-03 09:15:36,034] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:36,035] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:36,068] INFO: Succesfully completed iteration #7.
[2022-08-03 09:15:36,079] INFO: Adding new iteration #8.
[2022-08-03 09:15:36,112] INFO: Resuming iteration #8.
[2022-08-03 09:15:36,113] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:36,114] INFO: Processing task `gradient`
[2022-08-03 09:15:36,245] INFO: Submitting job ...
[2022-08-03 09:15:36,359] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:37,372] INFO: Processing task `gradient`
[2022-08-03 09:15:37,479] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:37,537] INFO: Some simulations are still running. Please check again to see if they are finished.
[2022-08-03 09:15:37,538] INFO: Some tasks of iteration #8 are still running. Please check again later.
[2022-08-03 09:15:38,547] INFO: Processing task `gradient`
[2022-08-03 09:15:38,639] INFO: 1 events have already been submitted. They will not be submitted again.
[2022-08-03 09:15:38,834] INFO:
Iteration 8: Number of events: 1 chi = 7.204028930773391e-13 ||g|| = 1.669056263732925e-13
pred = -8.890227313369757e-13 ared = -1.2365369032426372e-12 norm_update = 83.05575807376013 tr_radius = 103.68611955998874
[2022-08-03 09:15:38,986] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:38,987] INFO: Processing task `misfit`
[2022-08-03 09:15:39,042] INFO: Submitting job ...
[2022-08-03 09:15:39,131] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2022-08-03 09:15:40,156] INFO: Processing task `misfit`
[2022-08-03 09:15:40,319] INFO:
old misfit control group: 7.204028930773391e-13
new misfit control group: 3.398970799518259e-14
predicted reduction control group: -5.173392667836802e-13
actual reduction control group: -6.864131850821564e-13
1 out of 1 event(s) improved the misfit.
[2022-08-03 09:15:40,320] INFO:
Model update accepted.
[2022-08-03 09:15:40,342] INFO: 1 new tasks have been issued.
[2022-08-03 09:15:40,343] INFO: Processing task `finalize_iteration`
[2022-08-03 09:15:40,386] INFO: Succesfully completed iteration #8.
[2022-08-03 09:15:40,401] INFO: Adding new iteration #9.
[2022-08-03 09:15:40,440] INFO: Resuming iteration #9.
[2022-08-03 09:15:40,440] INFO: The total number of iterations in this inversion tree has been reached and no new iteration will be added. If you want to continue with this inversion, you need to adjust `max_iterations_global` using `set_stopping_criteria`.