[2025-01-09 22:10:49,237] INFO: Adding new iteration #0.
[2025-01-09 22:10:49,246] INFO: Resuming iteration #0.
[2025-01-09 22:10:49,247] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:49,247] INFO: Processing task `misfit_and_gradient`
[2025-01-09 22:10:49,267] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event']
[2025-01-09 22:10:49,281] INFO: Submitting job ...
[2025-01-09 22:10:49,346] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:49,347] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 22:10:50,348] INFO: Processing task `misfit_and_gradient`
[2025-01-09 22:10:50,613] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092210615809_7c3f10c2c1@local
[2025-01-09 22:10:50,682] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:50,682] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 22:10:51,684] INFO: Processing task `misfit_and_gradient`
[2025-01-09 22:10:51,757] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:10:51,916] INFO:
Iteration 0: Number of events: 1 chi = 3.7156536966723097e-10 ||g|| = 3.3910359372571976e-12
pred = --- ared = --- norm_update = --- tr_radius = ---
[2025-01-09 22:10:51,921] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:51,922] INFO: Processing task `misfit`
[2025-01-09 22:10:51,957] INFO: Submitting job ...
[2025-01-09 22:10:52,027] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:52,028] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-01-09 22:10:53,042] INFO: Processing task `misfit`
[2025-01-09 22:10:53,172] INFO:
old misfit control group: 3.7156536966723097e-10
new misfit control group: 1.073889989952079e-10
predicted reduction control group: -2.1653942143243586e-10
actual reduction control group: -2.6417637067202306e-10
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:10:53,172] INFO:
Model update accepted.
[2025-01-09 22:10:53,173] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:53,173] INFO: Processing task `finalize_iteration`
[2025-01-09 22:10:53,204] INFO: Succesfully completed iteration #0.
[2025-01-09 22:10:53,207] INFO: Adding new iteration #1.
[2025-01-09 22:10:53,213] INFO: Resuming iteration #1.
[2025-01-09 22:10:53,213] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:53,214] INFO: Processing task `gradient`
[2025-01-09 22:10:53,286] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092210289051_8643b8a01b@local
[2025-01-09 22:10:53,353] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:53,353] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 22:10:54,355] INFO: Processing task `gradient`
[2025-01-09 22:10:54,393] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:10:54,537] INFO:
Iteration 1: Number of events: 1 chi = 1.073889989952079e-10 ||g|| = 1.037229670389857e-12
pred = -2.1653942143243586e-10 ared = -2.6417637067202306e-10 norm_update = 127.71284376748982 tr_radius = 127.71284376748997
[2025-01-09 22:10:54,558] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:54,558] INFO: Processing task `misfit`
[2025-01-09 22:10:54,591] INFO: Submitting job ...
[2025-01-09 22:10:54,661] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:54,662] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-01-09 22:10:55,680] INFO: Processing task `misfit`
[2025-01-09 22:10:55,830] INFO:
old misfit control group: 1.073889989952079e-10
new misfit control group: 7.296709075757565e-11
predicted reduction control group: -1.886436638060351e-11
actual reduction control group: -3.442190823763225e-11
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:10:55,830] INFO:
Model update accepted.
[2025-01-09 22:10:55,831] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:55,831] INFO: Processing task `finalize_iteration`
[2025-01-09 22:10:55,926] INFO: Succesfully completed iteration #1.
[2025-01-09 22:10:55,928] INFO: Adding new iteration #2.
[2025-01-09 22:10:55,966] INFO: Resuming iteration #2.
[2025-01-09 22:10:55,967] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:55,967] INFO: Processing task `gradient`
[2025-01-09 22:10:56,041] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092210043625_494b961277@local
[2025-01-09 22:10:56,146] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:56,150] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-01-09 22:10:57,155] INFO: Processing task `gradient`
[2025-01-09 22:10:57,209] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:10:57,262] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 22:10:57,263] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-01-09 22:10:58,283] INFO: Processing task `gradient`
[2025-01-09 22:10:58,321] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:10:58,621] INFO:
Iteration 2: Number of events: 1 chi = 7.296709075757565e-11 ||g|| = 8.841255720378636e-13
pred = -1.886436638060351e-11 ared = -3.442190823763225e-11 norm_update = 38.055155588990125 tr_radius = 255.42568753497994
[2025-01-09 22:10:58,668] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:58,669] INFO: Processing task `misfit`
[2025-01-09 22:10:58,701] INFO: Submitting job ...
[2025-01-09 22:10:58,830] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:10:58,830] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-01-09 22:10:59,868] INFO: Processing task `misfit`
[2025-01-09 22:10:59,971] INFO:
old misfit control group: 7.296709075757565e-11
new misfit control group: 7.29921706188182e-12
predicted reduction control group: -5.3947833613705984e-11
actual reduction control group: -6.566787369569383e-11
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:10:59,972] INFO:
Model update accepted.
[2025-01-09 22:10:59,973] INFO: 1 new tasks have been issued.
[2025-01-09 22:10:59,973] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:00,018] INFO: Succesfully completed iteration #2.
[2025-01-09 22:11:00,024] INFO: Adding new iteration #3.
[2025-01-09 22:11:00,037] INFO: Resuming iteration #3.
[2025-01-09 22:11:00,038] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:00,039] INFO: Processing task `gradient`
[2025-01-09 22:11:00,114] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092211117607_cd0ed343d8@local
[2025-01-09 22:11:00,196] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:00,197] INFO: Some tasks of iteration #3 are still running. Please check again later.
[2025-01-09 22:11:01,200] INFO: Processing task `gradient`
[2025-01-09 22:11:01,257] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:01,401] INFO:
Iteration 3: Number of events: 1 chi = 7.29921706188182e-12 ||g|| = 2.1080709923504721e-13
pred = -5.3947833613705984e-11 ared = -6.566787369569383e-11 norm_update = 127.71284376748984 tr_radius = 127.71284376748997
[2025-01-09 22:11:01,430] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:01,430] INFO: Processing task `misfit`
[2025-01-09 22:11:01,473] INFO: Submitting job ...
[2025-01-09 22:11:01,542] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:01,544] INFO: Some tasks of iteration #3 are still running. Please check again later.
[2025-01-09 22:11:02,575] INFO: Processing task `misfit`
[2025-01-09 22:11:02,615] INFO: Some tasks of iteration #3 are still running. Please check again later.
[2025-01-09 22:11:03,637] INFO: Processing task `misfit`
[2025-01-09 22:11:03,961] INFO:
old misfit control group: 7.29921706188182e-12
new misfit control group: 3.853990185690102e-12
predicted reduction control group: -3.625527296947774e-12
actual reduction control group: -3.4452268761917177e-12
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:11:03,961] INFO:
Model update accepted.
[2025-01-09 22:11:03,963] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:03,963] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:04,344] INFO: Succesfully completed iteration #3.
[2025-01-09 22:11:04,346] INFO: Adding new iteration #4.
[2025-01-09 22:11:04,359] INFO: Resuming iteration #4.
[2025-01-09 22:11:04,360] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:04,360] INFO: Processing task `gradient`
[2025-01-09 22:11:04,424] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092211426836_70ba54e5d9@local
[2025-01-09 22:11:04,614] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:04,615] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:05,619] INFO: Processing task `gradient`
[2025-01-09 22:11:05,660] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:05,711] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 22:11:05,712] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:06,726] INFO: Processing task `gradient`
[2025-01-09 22:11:06,764] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:06,899] INFO:
Iteration 4: Number of events: 1 chi = 3.853990185690102e-12 ||g|| = 5.491251362114438e-14
pred = -3.625527296947774e-12 ared = -3.4452268761917177e-12 norm_update = 37.634593986576924 tr_radius = 255.42568753497994
[2025-01-09 22:11:06,930] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:06,931] INFO: Processing task `misfit`
[2025-01-09 22:11:06,960] INFO: Submitting job ...
[2025-01-09 22:11:07,023] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:07,024] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:08,040] INFO: Processing task `misfit`
[2025-01-09 22:11:08,085] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:09,101] INFO: Processing task `misfit`
[2025-01-09 22:11:09,158] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:10,295] INFO: Processing task `misfit`
[2025-01-09 22:11:10,386] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:11,650] INFO: Processing task `misfit`
[2025-01-09 22:11:11,835] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-01-09 22:11:12,995] INFO: Processing task `misfit`
[2025-01-09 22:11:13,429] INFO:
old misfit control group: 3.853990185690102e-12
new misfit control group: 3.660798440136839e-12
predicted reduction control group: -1.1394903979292904e-13
actual reduction control group: -1.9319174555326333e-13
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:11:13,430] INFO:
Model update accepted.
[2025-01-09 22:11:13,430] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:13,430] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:13,626] INFO: Succesfully completed iteration #4.
[2025-01-09 22:11:13,629] INFO: Adding new iteration #5.
[2025-01-09 22:11:13,850] INFO: Resuming iteration #5.
[2025-01-09 22:11:13,851] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:13,851] INFO: Processing task `gradient`
[2025-01-09 22:11:13,918] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092211921396_0cb6143a6d@local
[2025-01-09 22:11:14,104] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:14,105] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2025-01-09 22:11:15,140] INFO: Processing task `gradient`
[2025-01-09 22:11:15,176] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:15,547] INFO:
Iteration 5: Number of events: 1 chi = 3.660798440136839e-12 ||g|| = 3.682525298193585e-14
pred = -1.1394903979292904e-13 ared = -1.9319174555326333e-13 norm_update = 4.998974676234866 tr_radius = 127.71284376748997
[2025-01-09 22:11:15,606] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:15,606] INFO: Processing task `misfit`
[2025-01-09 22:11:15,634] INFO: Submitting job ...
[2025-01-09 22:11:15,721] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:15,721] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2025-01-09 22:11:16,760] INFO: Processing task `misfit`
[2025-01-09 22:11:16,941] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2025-01-09 22:11:18,006] INFO: Processing task `misfit`
[2025-01-09 22:11:18,770] INFO:
old misfit control group: 3.660798440136839e-12
new misfit control group: 2.9810284223434555e-12
predicted reduction control group: -4.4323683818356934e-13
actual reduction control group: -6.797700177933832e-13
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:11:18,771] INFO:
Model update accepted.
[2025-01-09 22:11:18,771] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:18,771] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:18,894] INFO: Succesfully completed iteration #5.
[2025-01-09 22:11:18,897] INFO: Adding new iteration #6.
[2025-01-09 22:11:19,001] INFO: Resuming iteration #6.
[2025-01-09 22:11:19,002] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:19,002] INFO: Processing task `gradient`
[2025-01-09 22:11:19,061] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092211063854_c0b797c9c1@local
[2025-01-09 22:11:19,364] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:19,365] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-01-09 22:11:20,381] INFO: Processing task `gradient`
[2025-01-09 22:11:20,413] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:20,573] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 22:11:20,573] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-01-09 22:11:21,576] INFO: Processing task `gradient`
[2025-01-09 22:11:21,615] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:21,716] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 22:11:21,717] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-01-09 22:11:22,732] INFO: Processing task `gradient`
[2025-01-09 22:11:22,774] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:23,086] INFO:
Iteration 6: Number of events: 1 chi = 2.9810284223434555e-12 ||g|| = 1.1334072418867954e-13
pred = -4.4323683818356934e-13 ared = -6.797700177933832e-13 norm_update = 25.945609247270596 tr_radius = 63.856421883744986
[2025-01-09 22:11:23,123] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:23,123] INFO: Processing task `misfit`
[2025-01-09 22:11:23,149] INFO: Submitting job ...
[2025-01-09 22:11:23,278] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:23,278] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-01-09 22:11:24,365] INFO: Processing task `misfit`
[2025-01-09 22:11:24,637] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-01-09 22:11:25,852] INFO: Processing task `misfit`
[2025-01-09 22:11:26,146] INFO:
old misfit control group: 2.9810284223434555e-12
new misfit control group: 1.9555860418200896e-12
predicted reduction control group: -6.673080610254194e-13
actual reduction control group: -1.025442380523366e-12
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:11:26,146] INFO:
Model update accepted.
[2025-01-09 22:11:26,147] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:26,147] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:26,230] INFO: Succesfully completed iteration #6.
[2025-01-09 22:11:26,234] INFO: Adding new iteration #7.
[2025-01-09 22:11:26,260] INFO: Resuming iteration #7.
[2025-01-09 22:11:26,261] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:26,264] INFO: Processing task `gradient`
[2025-01-09 22:11:26,318] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092211320009_352bb0636c@local
[2025-01-09 22:11:26,416] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:26,416] INFO: Some tasks of iteration #7 are still running. Please check again later.
[2025-01-09 22:11:27,466] INFO: Processing task `gradient`
[2025-01-09 22:11:27,497] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:27,537] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-01-09 22:11:27,537] INFO: Some tasks of iteration #7 are still running. Please check again later.
[2025-01-09 22:11:28,540] INFO: Processing task `gradient`
[2025-01-09 22:11:28,570] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:28,776] INFO:
Iteration 7: Number of events: 1 chi = 1.9555860418200896e-12 ||g|| = 1.8082278362551017e-13
pred = -6.673080610254194e-13 ared = -1.025442380523366e-12 norm_update = 51.891218494541064 tr_radius = 51.89121849454119
[2025-01-09 22:11:28,815] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:28,816] INFO: Processing task `misfit`
[2025-01-09 22:11:28,838] INFO: Submitting job ...
[2025-01-09 22:11:28,880] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:28,880] INFO: Some tasks of iteration #7 are still running. Please check again later.
[2025-01-09 22:11:29,892] INFO: Processing task `misfit`
[2025-01-09 22:11:29,983] INFO:
old misfit control group: 1.9555860418200896e-12
new misfit control group: 7.195492692978477e-13
predicted reduction control group: -8.887018361461577e-13
actual reduction control group: -1.2360367725222419e-12
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:11:29,984] INFO:
Model update accepted.
[2025-01-09 22:11:29,984] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:29,984] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:30,036] INFO: Succesfully completed iteration #7.
[2025-01-09 22:11:30,039] INFO: Adding new iteration #8.
[2025-01-09 22:11:30,051] INFO: Resuming iteration #8.
[2025-01-09 22:11:30,055] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:30,055] INFO: Processing task `gradient`
[2025-01-09 22:11:30,106] INFO: Submitting job ...
Uploading 1 files...
🚀 Submitted job_2501092211107756_e77d19c7c3@local
[2025-01-09 22:11:30,147] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:30,147] INFO: Some tasks of iteration #8 are still running. Please check again later.
[2025-01-09 22:11:31,151] INFO: Processing task `gradient`
[2025-01-09 22:11:31,181] INFO: 1 events have already been submitted. They will not be submitted again.
[2025-01-09 22:11:31,301] INFO:
Iteration 8: Number of events: 1 chi = 7.195492692978477e-13 ||g|| = 1.668305656509431e-13
pred = -8.887018361461577e-13 ared = -1.2360367725222419e-12 norm_update = 83.04605740553336 tr_radius = 103.78243698908238
[2025-01-09 22:11:31,357] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:31,358] INFO: Processing task `misfit`
[2025-01-09 22:11:31,384] INFO: Submitting job ...
[2025-01-09 22:11:31,429] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-01-09 22:11:31,430] INFO: Some tasks of iteration #8 are still running. Please check again later.
[2025-01-09 22:11:32,444] INFO: Processing task `misfit`
[2025-01-09 22:11:32,602] INFO:
old misfit control group: 7.195492692978477e-13
new misfit control group: 3.428681080907987e-14
predicted reduction control group: -5.160693460554207e-13
actual reduction control group: -6.852624584887678e-13
1 out of 1 event(s) improved the misfit.
[2025-01-09 22:11:32,602] INFO:
Model update accepted.
[2025-01-09 22:11:32,602] INFO: 1 new tasks have been issued.
[2025-01-09 22:11:32,603] INFO: Processing task `finalize_iteration`
[2025-01-09 22:11:32,630] INFO: Succesfully completed iteration #8.
[2025-01-09 22:11:32,634] INFO: Adding new iteration #9.
[2025-01-09 22:11:32,649] INFO: Resuming iteration #9.
[2025-01-09 22:11:32,649] INFO: The total number of iterations in this inversion tree has been reached and no new iteration will be added. If you want to continue with this inversion, you need to adjust `max_iterations_global` using `set_stopping_criteria`.