Note
Click here to download the full example code
ELBoxEmbeddings
This example is based on the paper Description Logic EL++ Embeddings with Intersectional Closure. This paper is based on the idea of EL Embeddings, but in this work the main point is to solve the intersectional closure problem.
In the case of EL Embeddings, the geometric objects representing ontology classes are \(n\)-dimensional balls. One of the normal forms in EL is:
\[C_1 \sqcap C_2 \sqsubseteq D\]
As we can see, there is an intersection operation \(C_1 \sqcap C_2\). Computing this intersection using balls is not a closed operations because the region contained in the intersection of two balls is not a ball. To solve that issue, this paper proposes the idea of changing the geometric objects to boxes, for which the intersection operation has the closure property.
This example is quite similar to the one found in EL Embeddings. There might be slight changes in the training part but the most important changes are in the definition of loss functions definition of the loss functions for each normal form.
import mowl
mowl.init_jvm("10g")
import torch as th
ELBoxEmbeddings (PyTorch) module
ELBoxEmbeddings defines a geometric modelling for all the GCIs in the EL language.
The implementation of ELEmbeddings module can be found at mowl.nn.el.elem.module.ELBoxModule
ELBoxEmbeddings model
The module mowl.nn.el.elem.module.ELBoxModule is used in the mowl.models.elboxembeddings.model.ELBoxEmbeddings.
In the use case of this example, we will test over a biological problem, which is
protein-protein interactions. Given two proteins \(p_1,p_2\), the phenomenon
“\(p_1\) interacts with \(p_2\)” is encoded using GCI 2 as:
\[p_1 \sqsubseteq interacts\_with. p_2\]
For that, we can use the class mowl.models.elembeddings.examples.model_ppi.ELBoxPPI mode, which uses the mowl.datasets.builtin.PPIYeastSlimDataset dataset.
Training the model
from mowl.datasets.builtin import PPIYeastSlimDataset
from mowl.models.elboxembeddings.examples.model_ppi import ELBoxPPI
dataset = PPIYeastSlimDataset()
model = ELBoxPPI(dataset,
embed_dim=30,
margin=-0.05,
reg_norm=1,
learning_rate=0.001,
epochs=20,
batch_size=4096,
model_filepath=None,
device='cpu')
model.train()
0%| | 0/20 [00:00<?, ?it/s]Saving model..
Epoch 0: Train loss: 7.491465091705322 Valid loss: 2.7105441093444824
5%|5 | 1/20 [00:04<01:31, 4.82s/it]Saving model..
Epoch 1: Train loss: 7.3488688468933105 Valid loss: 2.697273015975952
10%|# | 2/20 [00:09<01:26, 4.83s/it]Saving model..
Epoch 2: Train loss: 7.181153774261475 Valid loss: 2.6833877563476562
15%|#5 | 3/20 [00:14<01:24, 4.97s/it]Saving model..
Epoch 3: Train loss: 7.073298931121826 Valid loss: 2.669271230697632
20%|## | 4/20 [00:19<01:19, 4.99s/it]Saving model..
Epoch 4: Train loss: 6.910976409912109 Valid loss: 2.654946804046631
25%|##5 | 5/20 [00:24<01:14, 4.99s/it]Saving model..
Epoch 5: Train loss: 6.83297872543335 Valid loss: 2.640538454055786
30%|### | 6/20 [00:29<01:09, 5.00s/it]Saving model..
Epoch 6: Train loss: 6.676943302154541 Valid loss: 2.6260926723480225
35%|###5 | 7/20 [00:34<01:05, 5.03s/it]Saving model..
Epoch 7: Train loss: 6.582958698272705 Valid loss: 2.611572504043579
40%|#### | 8/20 [00:39<01:00, 5.01s/it]Saving model..
Epoch 8: Train loss: 6.47788667678833 Valid loss: 2.596987009048462
45%|####5 | 9/20 [00:44<00:55, 5.03s/it]Saving model..
Epoch 9: Train loss: 6.351139545440674 Valid loss: 2.5824296474456787
50%|##### | 10/20 [00:49<00:50, 5.03s/it]Saving model..
Epoch 10: Train loss: 6.247873306274414 Valid loss: 2.5679264068603516
55%|#####5 | 11/20 [00:55<00:45, 5.05s/it]Saving model..
Epoch 11: Train loss: 6.137197017669678 Valid loss: 2.5534563064575195
60%|###### | 12/20 [01:00<00:40, 5.02s/it]Saving model..
Epoch 12: Train loss: 6.036550521850586 Valid loss: 2.539074659347534
65%|######5 | 13/20 [01:05<00:35, 5.04s/it]Saving model..
Epoch 13: Train loss: 5.933111667633057 Valid loss: 2.524772882461548
70%|####### | 14/20 [01:10<00:30, 5.08s/it]Saving model..
Epoch 14: Train loss: 5.8326029777526855 Valid loss: 2.5105109214782715
75%|#######5 | 15/20 [01:15<00:25, 5.10s/it]Saving model..
Epoch 15: Train loss: 5.745277404785156 Valid loss: 2.4963250160217285
80%|######## | 16/20 [01:20<00:20, 5.12s/it]Saving model..
Epoch 16: Train loss: 5.640264987945557 Valid loss: 2.48223876953125
85%|########5 | 17/20 [01:25<00:15, 5.13s/it]Saving model..
Epoch 17: Train loss: 5.557387351989746 Valid loss: 2.4682109355926514
90%|######### | 18/20 [01:30<00:10, 5.09s/it]Saving model..
Epoch 18: Train loss: 5.429860591888428 Valid loss: 2.4542572498321533
95%|#########5| 19/20 [01:35<00:05, 5.09s/it]Saving model..
Epoch 19: Train loss: 5.366162300109863 Valid loss: 2.440342426300049
100%|##########| 20/20 [01:41<00:00, 5.18s/it]
100%|##########| 20/20 [01:41<00:00, 5.06s/it]
1
Evaluating the model
Now, it is time to evaluate embeddings. For this, we use the
ModelRankBasedEvaluator class.
from mowl.evaluation.rank_based import ModelRankBasedEvaluator
with th.no_grad():
model.load_best_model()
evaluator = ModelRankBasedEvaluator(
model,
device = "cpu",
eval_method = model.eval_method,
)
evaluator.evaluate(show=True)
0%| | 0/12040 [00:00<?, ?it/s]
0%| | 15/12040 [00:00<01:22, 145.96it/s]
0%| | 34/12040 [00:00<01:10, 170.09it/s]
0%| | 54/12040 [00:00<01:06, 179.89it/s]
1%| | 72/12040 [00:00<01:39, 120.04it/s]
1%| | 92/12040 [00:00<01:25, 139.52it/s]
1%| | 111/12040 [00:00<01:18, 152.80it/s]
1%|1 | 128/12040 [00:00<01:42, 116.62it/s]
1%|1 | 146/12040 [00:01<01:31, 130.58it/s]
1%|1 | 165/12040 [00:01<01:22, 143.54it/s]
2%|1 | 184/12040 [00:01<01:16, 155.23it/s]
2%|1 | 203/12040 [00:01<01:12, 164.39it/s]
2%|1 | 221/12040 [00:01<01:35, 123.39it/s]
2%|2 | 241/12040 [00:01<01:24, 139.29it/s]
2%|2 | 261/12040 [00:01<01:17, 152.52it/s]
2%|2 | 281/12040 [00:01<01:12, 162.37it/s]
2%|2 | 299/12040 [00:02<01:34, 124.03it/s]
3%|2 | 318/12040 [00:02<01:24, 138.37it/s]
3%|2 | 337/12040 [00:02<01:17, 150.41it/s]
3%|2 | 355/12040 [00:02<01:14, 157.70it/s]
3%|3 | 373/12040 [00:02<01:36, 120.79it/s]
3%|3 | 393/12040 [00:02<01:24, 137.02it/s]
3%|3 | 411/12040 [00:03<01:43, 112.83it/s]
4%|3 | 429/12040 [00:03<01:32, 126.16it/s]
4%|3 | 448/12040 [00:03<01:23, 139.42it/s]
4%|3 | 468/12040 [00:03<01:15, 152.43it/s]
4%|4 | 488/12040 [00:03<01:11, 162.68it/s]
4%|4 | 507/12040 [00:03<01:07, 169.91it/s]
4%|4 | 525/12040 [00:03<01:30, 127.14it/s]
5%|4 | 544/12040 [00:03<01:22, 139.82it/s]
5%|4 | 563/12040 [00:03<01:16, 150.92it/s]
5%|4 | 580/12040 [00:04<01:14, 154.74it/s]
5%|4 | 597/12040 [00:04<01:39, 114.49it/s]
5%|5 | 614/12040 [00:04<01:30, 125.82it/s]
5%|5 | 630/12040 [00:04<01:25, 133.44it/s]
5%|5 | 649/12040 [00:04<01:17, 146.76it/s]
6%|5 | 666/12040 [00:04<01:39, 114.38it/s]
6%|5 | 685/12040 [00:04<01:27, 130.17it/s]
6%|5 | 703/12040 [00:05<01:20, 140.36it/s]
6%|5 | 719/12040 [00:05<01:41, 111.75it/s]
6%|6 | 739/12040 [00:05<01:27, 129.80it/s]
6%|6 | 758/12040 [00:05<01:19, 142.34it/s]
6%|6 | 776/12040 [00:05<01:14, 151.67it/s]
7%|6 | 794/12040 [00:05<01:11, 158.16it/s]
7%|6 | 812/12040 [00:05<01:08, 163.97it/s]
7%|6 | 830/12040 [00:06<01:32, 120.63it/s]
7%|7 | 848/12040 [00:06<01:24, 133.13it/s]
7%|7 | 866/12040 [00:06<01:17, 144.03it/s]
7%|7 | 886/12040 [00:06<01:11, 156.47it/s]
8%|7 | 903/12040 [00:06<01:32, 120.19it/s]
8%|7 | 922/12040 [00:06<01:22, 135.34it/s]
8%|7 | 941/12040 [00:06<01:15, 147.97it/s]
8%|7 | 958/12040 [00:06<01:35, 116.42it/s]
8%|8 | 977/12040 [00:07<01:24, 131.47it/s]
8%|8 | 996/12040 [00:07<01:16, 143.78it/s]
8%|8 | 1013/12040 [00:07<01:36, 114.46it/s]
9%|8 | 1033/12040 [00:07<01:23, 131.88it/s]
9%|8 | 1052/12040 [00:07<01:15, 145.10it/s]
9%|8 | 1071/12040 [00:07<01:10, 155.65it/s]
9%|9 | 1090/12040 [00:07<01:06, 163.79it/s]
9%|9 | 1110/12040 [00:07<01:03, 171.61it/s]
9%|9 | 1129/12040 [00:08<01:26, 125.77it/s]
10%|9 | 1146/12040 [00:08<01:21, 134.16it/s]
10%|9 | 1165/12040 [00:08<01:14, 146.65it/s]
10%|9 | 1184/12040 [00:08<01:09, 156.57it/s]
10%|9 | 1201/12040 [00:08<01:30, 120.19it/s]
10%|# | 1221/12040 [00:08<01:19, 136.36it/s]
10%|# | 1240/12040 [00:08<01:12, 148.21it/s]
10%|# | 1257/12040 [00:09<01:32, 116.18it/s]
11%|# | 1276/12040 [00:09<01:21, 131.30it/s]
11%|# | 1295/12040 [00:09<01:14, 144.72it/s]
11%|# | 1312/12040 [00:09<01:33, 115.12it/s]
11%|#1 | 1331/12040 [00:09<01:22, 130.47it/s]
11%|#1 | 1350/12040 [00:09<01:14, 142.57it/s]
11%|#1 | 1368/12040 [00:09<01:10, 151.68it/s]
12%|#1 | 1387/12040 [00:09<01:06, 160.87it/s]
12%|#1 | 1406/12040 [00:10<01:03, 166.80it/s]
12%|#1 | 1424/12040 [00:10<01:25, 124.81it/s]
12%|#1 | 1443/12040 [00:10<01:16, 139.03it/s]
12%|#2 | 1463/12040 [00:10<01:09, 151.67it/s]
12%|#2 | 1482/12040 [00:10<01:05, 160.89it/s]
12%|#2 | 1500/12040 [00:10<01:25, 122.72it/s]
13%|#2 | 1518/12040 [00:10<01:18, 134.74it/s]
13%|#2 | 1537/12040 [00:11<01:11, 146.53it/s]
13%|#2 | 1554/12040 [00:11<01:32, 113.41it/s]
13%|#3 | 1574/12040 [00:11<01:20, 130.44it/s]
13%|#3 | 1593/12040 [00:11<01:13, 142.50it/s]
13%|#3 | 1610/12040 [00:11<01:34, 110.14it/s]
14%|#3 | 1629/12040 [00:11<01:22, 125.75it/s]
14%|#3 | 1649/12040 [00:11<01:13, 141.09it/s]
14%|#3 | 1668/12040 [00:12<01:07, 152.87it/s]
14%|#4 | 1688/12040 [00:12<01:03, 163.33it/s]
14%|#4 | 1708/12040 [00:12<01:00, 171.62it/s]
14%|#4 | 1727/12040 [00:12<01:20, 128.23it/s]
15%|#4 | 1746/12040 [00:12<01:13, 140.69it/s]
15%|#4 | 1766/12040 [00:12<01:07, 152.67it/s]
15%|#4 | 1784/12040 [00:12<01:04, 158.48it/s]
15%|#4 | 1802/12040 [00:13<01:24, 121.49it/s]
15%|#5 | 1821/12040 [00:13<01:15, 135.06it/s]
15%|#5 | 1840/12040 [00:13<01:09, 147.20it/s]
15%|#5 | 1857/12040 [00:13<01:27, 116.33it/s]
16%|#5 | 1876/12040 [00:13<01:17, 131.09it/s]
16%|#5 | 1895/12040 [00:13<01:10, 143.41it/s]
16%|#5 | 1912/12040 [00:13<01:28, 114.56it/s]
16%|#6 | 1931/12040 [00:14<01:17, 130.49it/s]
16%|#6 | 1951/12040 [00:14<01:09, 145.21it/s]
16%|#6 | 1971/12040 [00:14<01:04, 156.96it/s]
17%|#6 | 1991/12040 [00:14<01:00, 166.45it/s]
17%|#6 | 2009/12040 [00:14<01:20, 124.01it/s]
17%|#6 | 2028/12040 [00:14<01:12, 137.99it/s]
17%|#7 | 2048/12040 [00:14<01:06, 151.17it/s]
17%|#7 | 2067/12040 [00:14<01:02, 159.79it/s]
17%|#7 | 2086/12040 [00:14<00:59, 167.10it/s]
17%|#7 | 2104/12040 [00:15<01:18, 126.17it/s]
18%|#7 | 2124/12040 [00:15<01:10, 141.26it/s]
18%|#7 | 2143/12040 [00:15<01:04, 152.83it/s]
18%|#7 | 2160/12040 [00:15<01:23, 118.64it/s]
18%|#8 | 2180/12040 [00:15<01:12, 135.10it/s]
18%|#8 | 2199/12040 [00:15<01:06, 147.38it/s]
18%|#8 | 2216/12040 [00:16<01:24, 116.91it/s]
19%|#8 | 2236/12040 [00:16<01:13, 133.48it/s]
19%|#8 | 2255/12040 [00:16<01:06, 146.60it/s]
19%|#8 | 2275/12040 [00:16<01:01, 158.99it/s]
19%|#9 | 2295/12040 [00:16<00:58, 167.92it/s]
19%|#9 | 2313/12040 [00:16<01:16, 126.79it/s]
19%|#9 | 2333/12040 [00:16<01:08, 141.79it/s]
20%|#9 | 2353/12040 [00:16<01:02, 153.81it/s]
20%|#9 | 2373/12040 [00:17<00:59, 163.33it/s]
20%|#9 | 2391/12040 [00:17<01:17, 124.88it/s]
20%|## | 2410/12040 [00:17<01:09, 138.82it/s]
20%|## | 2429/12040 [00:17<01:04, 150.06it/s]
20%|## | 2448/12040 [00:17<01:00, 159.72it/s]
20%|## | 2466/12040 [00:17<01:18, 122.55it/s]
21%|## | 2485/12040 [00:17<01:09, 137.10it/s]
21%|## | 2504/12040 [00:18<01:24, 112.92it/s]
21%|## | 2524/12040 [00:18<01:13, 129.79it/s]
21%|##1 | 2543/12040 [00:18<01:06, 142.91it/s]
21%|##1 | 2563/12040 [00:18<01:00, 155.88it/s]
21%|##1 | 2582/12040 [00:18<00:57, 163.87it/s]
22%|##1 | 2601/12040 [00:18<00:55, 169.93it/s]
22%|##1 | 2619/12040 [00:18<01:14, 127.27it/s]
22%|##1 | 2638/12040 [00:18<01:06, 141.07it/s]
22%|##2 | 2658/12040 [00:19<01:00, 154.05it/s]
22%|##2 | 2677/12040 [00:19<00:57, 163.18it/s]
22%|##2 | 2695/12040 [00:19<01:14, 124.70it/s]
23%|##2 | 2714/12040 [00:19<01:07, 138.99it/s]
23%|##2 | 2733/12040 [00:19<01:01, 150.66it/s]
23%|##2 | 2750/12040 [00:19<01:18, 118.15it/s]
23%|##2 | 2769/12040 [00:19<01:09, 132.94it/s]
23%|##3 | 2788/12040 [00:20<01:03, 146.21it/s]
23%|##3 | 2805/12040 [00:20<01:19, 115.85it/s]
23%|##3 | 2824/12040 [00:20<01:10, 130.85it/s]
24%|##3 | 2844/12040 [00:20<01:02, 146.60it/s]
24%|##3 | 2864/12040 [00:20<00:57, 159.40it/s]
24%|##3 | 2884/12040 [00:20<00:54, 168.07it/s]
24%|##4 | 2903/12040 [00:20<00:52, 173.69it/s]
24%|##4 | 2922/12040 [00:21<01:10, 130.25it/s]
24%|##4 | 2941/12040 [00:21<01:03, 142.37it/s]
25%|##4 | 2960/12040 [00:21<00:59, 152.93it/s]
25%|##4 | 2979/12040 [00:21<00:55, 162.33it/s]
25%|##4 | 2997/12040 [00:21<01:12, 124.55it/s]
25%|##5 | 3016/12040 [00:21<01:05, 138.66it/s]
25%|##5 | 3035/12040 [00:21<00:59, 150.61it/s]
25%|##5 | 3052/12040 [00:21<01:16, 118.01it/s]
26%|##5 | 3071/12040 [00:22<01:07, 133.57it/s]
26%|##5 | 3090/12040 [00:22<01:01, 146.68it/s]
26%|##5 | 3107/12040 [00:22<01:17, 115.40it/s]
26%|##5 | 3126/12040 [00:22<01:08, 131.05it/s]
26%|##6 | 3144/12040 [00:22<01:02, 142.17it/s]
26%|##6 | 3163/12040 [00:22<00:57, 153.97it/s]
26%|##6 | 3183/12040 [00:22<00:53, 164.24it/s]
27%|##6 | 3202/12040 [00:22<00:51, 171.07it/s]
27%|##6 | 3220/12040 [00:23<01:09, 126.04it/s]
27%|##6 | 3240/12040 [00:23<01:02, 140.87it/s]
27%|##7 | 3259/12040 [00:23<00:57, 152.05it/s]
27%|##7 | 3279/12040 [00:23<00:53, 162.37it/s]
27%|##7 | 3297/12040 [00:23<01:10, 124.52it/s]
28%|##7 | 3316/12040 [00:23<01:03, 137.72it/s]
28%|##7 | 3335/12040 [00:23<00:58, 149.01it/s]
28%|##7 | 3352/12040 [00:24<01:14, 117.37it/s]
28%|##7 | 3371/12040 [00:24<01:05, 132.93it/s]
28%|##8 | 3391/12040 [00:24<00:58, 147.48it/s]
28%|##8 | 3408/12040 [00:24<01:14, 115.77it/s]
28%|##8 | 3427/12040 [00:24<01:05, 131.10it/s]
29%|##8 | 3445/12040 [00:24<01:00, 141.10it/s]
29%|##8 | 3464/12040 [00:24<00:56, 152.15it/s]
29%|##8 | 3484/12040 [00:24<00:52, 163.51it/s]
29%|##9 | 3503/12040 [00:25<00:50, 169.08it/s]
29%|##9 | 3521/12040 [00:25<01:09, 122.34it/s]
29%|##9 | 3540/12040 [00:25<01:02, 136.73it/s]
30%|##9 | 3560/12040 [00:25<00:56, 150.57it/s]
30%|##9 | 3578/12040 [00:25<00:53, 157.75it/s]
30%|##9 | 3596/12040 [00:25<01:09, 121.71it/s]
30%|### | 3616/12040 [00:25<01:00, 138.20it/s]
30%|### | 3635/12040 [00:26<00:56, 149.83it/s]
30%|### | 3652/12040 [00:26<01:11, 117.83it/s]
30%|### | 3671/12040 [00:26<01:03, 132.69it/s]
31%|### | 3691/12040 [00:26<00:56, 147.65it/s]
31%|### | 3708/12040 [00:26<01:11, 117.15it/s]
31%|### | 3728/12040 [00:26<01:02, 133.53it/s]
31%|###1 | 3748/12040 [00:26<00:56, 147.59it/s]
31%|###1 | 3767/12040 [00:27<00:52, 157.70it/s]
31%|###1 | 3787/12040 [00:27<00:49, 167.18it/s]
32%|###1 | 3805/12040 [00:27<01:05, 126.27it/s]
32%|###1 | 3825/12040 [00:27<00:57, 141.68it/s]
32%|###1 | 3844/12040 [00:27<00:53, 153.08it/s]
32%|###2 | 3864/12040 [00:27<00:50, 163.45it/s]
32%|###2 | 3883/12040 [00:27<01:04, 125.59it/s]
32%|###2 | 3902/12040 [00:27<00:58, 138.81it/s]
33%|###2 | 3921/12040 [00:28<00:54, 150.28it/s]
33%|###2 | 3940/12040 [00:28<00:50, 160.17it/s]
33%|###2 | 3958/12040 [00:28<01:09, 116.41it/s]
33%|###3 | 3975/12040 [00:28<01:03, 126.64it/s]
33%|###3 | 3994/12040 [00:28<00:57, 140.23it/s]
33%|###3 | 4010/12040 [00:28<01:11, 112.67it/s]
33%|###3 | 4027/12040 [00:28<01:04, 123.36it/s]
34%|###3 | 4044/12040 [00:29<01:00, 132.33it/s]
34%|###3 | 4061/12040 [00:29<00:56, 140.54it/s]
34%|###3 | 4081/12040 [00:29<00:51, 154.81it/s]
34%|###4 | 4101/12040 [00:29<00:47, 165.49it/s]
34%|###4 | 4119/12040 [00:29<01:03, 125.21it/s]
34%|###4 | 4139/12040 [00:29<00:56, 140.95it/s]
35%|###4 | 4159/12040 [00:29<00:51, 153.81it/s]
35%|###4 | 4179/12040 [00:29<00:47, 163.97it/s]
35%|###4 | 4197/12040 [00:30<01:05, 120.05it/s]
35%|###5 | 4216/12040 [00:30<00:58, 134.25it/s]
35%|###5 | 4236/12040 [00:30<00:52, 148.31it/s]
35%|###5 | 4253/12040 [00:30<01:06, 117.79it/s]
35%|###5 | 4273/12040 [00:30<00:57, 134.23it/s]
36%|###5 | 4293/12040 [00:30<00:52, 148.22it/s]
36%|###5 | 4310/12040 [00:31<01:05, 117.81it/s]
36%|###5 | 4330/12040 [00:31<00:57, 134.16it/s]
36%|###6 | 4349/12040 [00:31<00:52, 147.08it/s]
36%|###6 | 4366/12040 [00:31<00:51, 150.34it/s]
36%|###6 | 4383/12040 [00:31<00:55, 137.68it/s]
37%|###6 | 4400/12040 [00:31<00:52, 144.72it/s]
37%|###6 | 4416/12040 [00:31<01:06, 113.93it/s]
37%|###6 | 4436/12040 [00:31<00:57, 131.77it/s]
37%|###7 | 4456/12040 [00:32<00:51, 146.50it/s]
37%|###7 | 4475/12040 [00:32<00:48, 156.29it/s]
37%|###7 | 4492/12040 [00:32<01:02, 120.87it/s]
37%|###7 | 4511/12040 [00:32<00:55, 134.64it/s]
38%|###7 | 4527/12040 [00:32<00:55, 134.46it/s]
38%|###7 | 4544/12040 [00:32<01:10, 106.89it/s]
38%|###7 | 4562/12040 [00:32<01:01, 122.04it/s]
38%|###8 | 4578/12040 [00:33<00:57, 130.22it/s]
38%|###8 | 4596/12040 [00:33<00:52, 141.51it/s]
38%|###8 | 4612/12040 [00:33<01:09, 107.25it/s]
38%|###8 | 4630/12040 [00:33<01:00, 122.15it/s]
39%|###8 | 4650/12040 [00:33<00:53, 138.94it/s]
39%|###8 | 4667/12040 [00:33<00:50, 146.07it/s]
39%|###8 | 4684/12040 [00:33<00:52, 139.20it/s]
39%|###9 | 4700/12040 [00:34<01:07, 108.91it/s]
39%|###9 | 4718/12040 [00:34<00:59, 123.01it/s]
39%|###9 | 4734/12040 [00:34<00:55, 131.30it/s]
39%|###9 | 4751/12040 [00:34<00:52, 139.87it/s]
40%|###9 | 4771/12040 [00:34<00:47, 153.80it/s]
40%|###9 | 4788/12040 [00:34<01:02, 116.41it/s]
40%|###9 | 4808/12040 [00:34<00:54, 133.62it/s]
40%|#### | 4828/12040 [00:34<00:48, 148.36it/s]
40%|#### | 4845/12040 [00:35<01:01, 117.24it/s]
40%|#### | 4865/12040 [00:35<00:53, 133.93it/s]
41%|#### | 4885/12040 [00:35<00:48, 147.97it/s]
41%|#### | 4902/12040 [00:35<01:00, 117.53it/s]
41%|#### | 4922/12040 [00:35<00:53, 134.26it/s]
41%|####1 | 4942/12040 [00:35<00:47, 148.63it/s]
41%|####1 | 4962/12040 [00:35<00:44, 160.30it/s]
41%|####1 | 4982/12040 [00:35<00:41, 169.08it/s]
42%|####1 | 5001/12040 [00:36<00:54, 128.19it/s]
42%|####1 | 5021/12040 [00:36<00:49, 142.77it/s]
42%|####1 | 5041/12040 [00:36<00:45, 155.06it/s]
42%|####2 | 5061/12040 [00:36<00:42, 165.06it/s]
42%|####2 | 5079/12040 [00:36<00:54, 126.65it/s]
42%|####2 | 5098/12040 [00:36<00:49, 140.51it/s]
43%|####2 | 5118/12040 [00:36<00:45, 153.48it/s]
43%|####2 | 5138/12040 [00:37<00:42, 163.42it/s]
43%|####2 | 5156/12040 [00:37<00:54, 125.28it/s]
43%|####2 | 5176/12040 [00:37<00:48, 140.25it/s]
43%|####3 | 5195/12040 [00:37<00:59, 115.71it/s]
43%|####3 | 5215/12040 [00:37<00:51, 132.08it/s]
43%|####3 | 5235/12040 [00:37<00:46, 146.36it/s]
44%|####3 | 5255/12040 [00:37<00:43, 157.75it/s]
44%|####3 | 5275/12040 [00:38<00:40, 166.43it/s]
44%|####3 | 5295/12040 [00:38<00:38, 173.35it/s]
44%|####4 | 5314/12040 [00:38<00:51, 130.59it/s]
44%|####4 | 5334/12040 [00:38<00:46, 144.85it/s]
44%|####4 | 5353/12040 [00:38<00:43, 155.18it/s]
45%|####4 | 5373/12040 [00:38<00:40, 164.88it/s]
45%|####4 | 5391/12040 [00:38<00:52, 125.46it/s]
45%|####4 | 5411/12040 [00:39<00:47, 140.48it/s]
45%|####5 | 5431/12040 [00:39<00:43, 152.85it/s]
45%|####5 | 5448/12040 [00:39<00:54, 119.95it/s]
45%|####5 | 5468/12040 [00:39<00:48, 135.99it/s]
46%|####5 | 5488/12040 [00:39<00:43, 149.20it/s]
46%|####5 | 5505/12040 [00:39<00:55, 118.03it/s]
46%|####5 | 5525/12040 [00:39<00:48, 134.37it/s]
46%|####6 | 5545/12040 [00:39<00:43, 147.90it/s]
46%|####6 | 5564/12040 [00:40<00:40, 158.18it/s]
46%|####6 | 5584/12040 [00:40<00:38, 167.51it/s]
47%|####6 | 5602/12040 [00:40<00:50, 127.00it/s]
47%|####6 | 5621/12040 [00:40<00:45, 140.91it/s]
47%|####6 | 5641/12040 [00:40<00:41, 154.00it/s]
47%|####7 | 5661/12040 [00:40<00:38, 164.01it/s]
47%|####7 | 5679/12040 [00:40<00:50, 124.99it/s]
47%|####7 | 5698/12040 [00:41<00:45, 139.01it/s]
47%|####7 | 5714/12040 [00:41<00:44, 142.91it/s]
48%|####7 | 5733/12040 [00:41<00:40, 153.96it/s]
48%|####7 | 5750/12040 [00:41<00:52, 119.25it/s]
48%|####7 | 5769/12040 [00:41<00:46, 134.78it/s]
48%|####8 | 5789/12040 [00:41<00:41, 149.01it/s]
48%|####8 | 5806/12040 [00:41<00:52, 117.85it/s]
48%|####8 | 5826/12040 [00:42<00:46, 134.30it/s]
49%|####8 | 5846/12040 [00:42<00:41, 148.07it/s]
49%|####8 | 5866/12040 [00:42<00:38, 159.75it/s]
49%|####8 | 5886/12040 [00:42<00:36, 168.84it/s]
49%|####9 | 5904/12040 [00:42<00:48, 127.33it/s]
49%|####9 | 5923/12040 [00:42<00:43, 141.07it/s]
49%|####9 | 5943/12040 [00:42<00:39, 153.66it/s]
50%|####9 | 5963/12040 [00:42<00:37, 163.96it/s]
50%|####9 | 5981/12040 [00:43<00:48, 125.30it/s]
50%|####9 | 6000/12040 [00:43<00:43, 139.24it/s]
50%|####9 | 6019/12040 [00:43<00:39, 151.21it/s]
50%|##### | 6038/12040 [00:43<00:37, 161.01it/s]
50%|##### | 6056/12040 [00:43<00:48, 122.27it/s]
50%|##### | 6072/12040 [00:43<00:46, 128.81it/s]
51%|##### | 6088/12040 [00:43<00:44, 134.44it/s]
51%|##### | 6103/12040 [00:44<00:56, 105.63it/s]
51%|##### | 6119/12040 [00:44<00:50, 116.54it/s]
51%|##### | 6136/12040 [00:44<00:46, 127.91it/s]
51%|#####1 | 6154/12040 [00:44<00:41, 140.66it/s]
51%|#####1 | 6173/12040 [00:44<00:38, 152.01it/s]
51%|#####1 | 6190/12040 [00:44<00:45, 129.61it/s]
52%|#####1 | 6205/12040 [00:44<00:57, 101.02it/s]
52%|#####1 | 6225/12040 [00:44<00:48, 120.21it/s]
52%|#####1 | 6244/12040 [00:45<00:43, 134.51it/s]
52%|#####1 | 6260/12040 [00:45<00:42, 137.46it/s]
52%|#####2 | 6276/12040 [00:45<00:55, 103.06it/s]
52%|#####2 | 6295/12040 [00:45<00:47, 120.20it/s]
52%|#####2 | 6314/12040 [00:45<00:42, 135.95it/s]
53%|#####2 | 6334/12040 [00:45<00:38, 149.94it/s]
53%|#####2 | 6351/12040 [00:46<00:48, 117.37it/s]
53%|#####2 | 6370/12040 [00:46<00:42, 132.14it/s]
53%|#####3 | 6390/12040 [00:46<00:38, 146.53it/s]
53%|#####3 | 6407/12040 [00:46<00:48, 116.55it/s]
53%|#####3 | 6427/12040 [00:46<00:42, 133.40it/s]
54%|#####3 | 6447/12040 [00:46<00:37, 147.61it/s]
54%|#####3 | 6467/12040 [00:46<00:35, 158.94it/s]
54%|#####3 | 6487/12040 [00:46<00:33, 168.23it/s]
54%|#####4 | 6505/12040 [00:47<00:43, 126.31it/s]
54%|#####4 | 6525/12040 [00:47<00:39, 141.23it/s]
54%|#####4 | 6542/12040 [00:47<00:37, 146.11it/s]
54%|#####4 | 6559/12040 [00:47<00:39, 140.28it/s]
55%|#####4 | 6575/12040 [00:47<00:48, 111.91it/s]
55%|#####4 | 6595/12040 [00:47<00:42, 129.43it/s]
55%|#####4 | 6615/12040 [00:47<00:37, 144.58it/s]
55%|#####5 | 6635/12040 [00:47<00:34, 156.63it/s]
55%|#####5 | 6652/12040 [00:48<00:44, 120.25it/s]
55%|#####5 | 6671/12040 [00:48<00:39, 134.87it/s]
56%|#####5 | 6688/12040 [00:48<00:37, 141.50it/s]
56%|#####5 | 6704/12040 [00:48<00:48, 109.46it/s]
56%|#####5 | 6723/12040 [00:48<00:42, 126.43it/s]
56%|#####6 | 6743/12040 [00:48<00:37, 142.29it/s]
56%|#####6 | 6763/12040 [00:48<00:34, 154.82it/s]
56%|#####6 | 6782/12040 [00:49<00:32, 163.89it/s]
56%|#####6 | 6800/12040 [00:49<00:42, 123.52it/s]
57%|#####6 | 6819/12040 [00:49<00:37, 138.02it/s]
57%|#####6 | 6838/12040 [00:49<00:34, 150.17it/s]
57%|#####6 | 6857/12040 [00:49<00:32, 159.06it/s]
57%|#####7 | 6875/12040 [00:49<00:44, 115.47it/s]
57%|#####7 | 6893/12040 [00:49<00:40, 128.66it/s]
57%|#####7 | 6912/12040 [00:50<00:35, 142.46it/s]
58%|#####7 | 6931/12040 [00:50<00:33, 152.44it/s]
58%|#####7 | 6948/12040 [00:50<00:42, 118.80it/s]
58%|#####7 | 6967/12040 [00:50<00:37, 134.10it/s]
58%|#####8 | 6987/12040 [00:50<00:34, 148.48it/s]
58%|#####8 | 7004/12040 [00:50<00:45, 111.26it/s]
58%|#####8 | 7022/12040 [00:50<00:40, 125.13it/s]
58%|#####8 | 7042/12040 [00:51<00:35, 141.20it/s]
59%|#####8 | 7061/12040 [00:51<00:32, 151.61it/s]
59%|#####8 | 7080/12040 [00:51<00:30, 160.11it/s]
59%|#####8 | 7098/12040 [00:51<00:40, 123.05it/s]
59%|#####9 | 7118/12040 [00:51<00:35, 138.52it/s]
59%|#####9 | 7137/12040 [00:51<00:32, 149.65it/s]
59%|#####9 | 7156/12040 [00:51<00:30, 158.75it/s]
60%|#####9 | 7174/12040 [00:52<00:39, 121.82it/s]
60%|#####9 | 7192/12040 [00:52<00:36, 133.82it/s]
60%|#####9 | 7212/12040 [00:52<00:32, 148.20it/s]
60%|###### | 7232/12040 [00:52<00:30, 159.23it/s]
60%|###### | 7250/12040 [00:52<00:38, 122.84it/s]
60%|###### | 7269/12040 [00:52<00:34, 137.43it/s]
61%|###### | 7288/12040 [00:52<00:41, 113.61it/s]
61%|###### | 7308/12040 [00:52<00:36, 130.46it/s]
61%|###### | 7327/12040 [00:53<00:33, 142.44it/s]
61%|######1 | 7347/12040 [00:53<00:30, 154.86it/s]
61%|######1 | 7366/12040 [00:53<00:28, 163.25it/s]
61%|######1 | 7385/12040 [00:53<00:27, 169.77it/s]
61%|######1 | 7403/12040 [00:53<00:36, 127.10it/s]
62%|######1 | 7422/12040 [00:53<00:32, 140.77it/s]
62%|######1 | 7441/12040 [00:53<00:30, 152.55it/s]
62%|######1 | 7460/12040 [00:53<00:28, 161.90it/s]
62%|######2 | 7478/12040 [00:54<00:37, 123.29it/s]
62%|######2 | 7498/12040 [00:54<00:32, 138.94it/s]
62%|######2 | 7517/12040 [00:54<00:30, 150.57it/s]
63%|######2 | 7534/12040 [00:54<00:39, 115.39it/s]
63%|######2 | 7553/12040 [00:54<00:34, 131.04it/s]
63%|######2 | 7570/12040 [00:54<00:32, 138.31it/s]
63%|######3 | 7586/12040 [00:54<00:31, 142.19it/s]
63%|######3 | 7602/12040 [00:55<00:39, 111.17it/s]
63%|######3 | 7621/12040 [00:55<00:34, 128.27it/s]
63%|######3 | 7640/12040 [00:55<00:30, 142.66it/s]
64%|######3 | 7660/12040 [00:55<00:28, 155.82it/s]
64%|######3 | 7680/12040 [00:55<00:26, 166.40it/s]
64%|######3 | 7698/12040 [00:55<00:34, 125.60it/s]
64%|######4 | 7718/12040 [00:55<00:30, 140.74it/s]
64%|######4 | 7738/12040 [00:55<00:28, 153.32it/s]
64%|######4 | 7758/12040 [00:56<00:26, 163.79it/s]
65%|######4 | 7776/12040 [00:56<00:35, 121.55it/s]
65%|######4 | 7792/12040 [00:56<00:33, 128.71it/s]
65%|######4 | 7811/12040 [00:56<00:29, 141.34it/s]
65%|######5 | 7831/12040 [00:56<00:27, 154.41it/s]
65%|######5 | 7848/12040 [00:56<00:36, 114.40it/s]
65%|######5 | 7863/12040 [00:56<00:34, 121.65it/s]
65%|######5 | 7878/12040 [00:57<00:33, 124.45it/s]
66%|######5 | 7892/12040 [00:57<00:44, 92.52it/s]
66%|######5 | 7904/12040 [00:57<00:45, 91.70it/s]
66%|######5 | 7920/12040 [00:57<00:39, 105.38it/s]
66%|######5 | 7939/12040 [00:57<00:32, 124.58it/s]
66%|######6 | 7959/12040 [00:57<00:28, 142.19it/s]
66%|######6 | 7979/12040 [00:57<00:26, 155.70it/s]
66%|######6 | 7996/12040 [00:58<00:33, 119.04it/s]
67%|######6 | 8016/12040 [00:58<00:29, 135.85it/s]
67%|######6 | 8036/12040 [00:58<00:26, 150.01it/s]
67%|######6 | 8056/12040 [00:58<00:24, 161.58it/s]
67%|######7 | 8074/12040 [00:58<00:32, 123.64it/s]
67%|######7 | 8089/12040 [00:58<00:33, 119.03it/s]
67%|######7 | 8103/12040 [00:58<00:33, 118.84it/s]
67%|######7 | 8120/12040 [00:59<00:30, 130.39it/s]
68%|######7 | 8135/12040 [00:59<00:38, 101.22it/s]
68%|######7 | 8151/12040 [00:59<00:34, 112.64it/s]
68%|######7 | 8170/12040 [00:59<00:29, 129.02it/s]
68%|######7 | 8185/12040 [00:59<00:36, 104.77it/s]
68%|######8 | 8204/12040 [00:59<00:31, 122.56it/s]
68%|######8 | 8224/12040 [00:59<00:27, 139.64it/s]
68%|######8 | 8240/12040 [00:59<00:26, 144.64it/s]
69%|######8 | 8256/12040 [01:00<00:25, 148.00it/s]
69%|######8 | 8272/12040 [01:00<00:24, 151.23it/s]
69%|######8 | 8288/12040 [01:00<00:32, 115.92it/s]
69%|######8 | 8307/12040 [01:00<00:28, 132.85it/s]
69%|######9 | 8327/12040 [01:00<00:25, 148.08it/s]
69%|######9 | 8347/12040 [01:00<00:23, 159.99it/s]
69%|######9 | 8367/12040 [01:00<00:21, 168.56it/s]
70%|######9 | 8385/12040 [01:01<00:28, 126.46it/s]
70%|######9 | 8404/12040 [01:01<00:26, 139.83it/s]
70%|######9 | 8422/12040 [01:01<00:24, 147.48it/s]
70%|####### | 8439/12040 [01:01<00:32, 109.51it/s]
70%|####### | 8458/12040 [01:01<00:28, 125.63it/s]
70%|####### | 8477/12040 [01:01<00:25, 140.03it/s]
71%|####### | 8494/12040 [01:01<00:31, 113.49it/s]
71%|####### | 8513/12040 [01:02<00:27, 128.89it/s]
71%|####### | 8532/12040 [01:02<00:24, 142.35it/s]
71%|#######1 | 8551/12040 [01:02<00:22, 153.96it/s]
71%|#######1 | 8571/12040 [01:02<00:21, 164.40it/s]
71%|#######1 | 8589/12040 [01:02<00:27, 124.90it/s]
72%|#######1 | 8609/12040 [01:02<00:24, 139.86it/s]
72%|#######1 | 8628/12040 [01:02<00:22, 151.42it/s]
72%|#######1 | 8647/12040 [01:02<00:21, 160.84it/s]
72%|#######1 | 8666/12040 [01:02<00:20, 168.18it/s]
72%|#######2 | 8684/12040 [01:03<00:28, 116.90it/s]
72%|#######2 | 8704/12040 [01:03<00:25, 133.00it/s]
72%|#######2 | 8723/12040 [01:03<00:22, 145.88it/s]
73%|#######2 | 8740/12040 [01:03<00:28, 115.98it/s]
73%|#######2 | 8759/12040 [01:03<00:25, 130.72it/s]
73%|#######2 | 8778/12040 [01:03<00:22, 143.65it/s]
73%|#######3 | 8795/12040 [01:04<00:30, 107.00it/s]
73%|#######3 | 8809/12040 [01:04<00:28, 111.79it/s]
73%|#######3 | 8828/12040 [01:04<00:25, 127.65it/s]
73%|#######3 | 8847/12040 [01:04<00:22, 142.17it/s]
74%|#######3 | 8864/12040 [01:04<00:21, 147.89it/s]
74%|#######3 | 8884/12040 [01:04<00:19, 159.56it/s]
74%|#######3 | 8901/12040 [01:04<00:27, 113.70it/s]
74%|#######4 | 8920/12040 [01:05<00:24, 128.92it/s]
74%|#######4 | 8939/12040 [01:05<00:21, 142.86it/s]
74%|#######4 | 8958/12040 [01:05<00:20, 154.07it/s]
75%|#######4 | 8976/12040 [01:05<00:25, 120.40it/s]
75%|#######4 | 8995/12040 [01:05<00:22, 135.37it/s]
75%|#######4 | 9015/12040 [01:05<00:20, 149.49it/s]
75%|#######5 | 9032/12040 [01:05<00:25, 118.07it/s]
75%|#######5 | 9051/12040 [01:06<00:22, 133.54it/s]
75%|#######5 | 9068/12040 [01:06<00:21, 141.04it/s]
75%|#######5 | 9084/12040 [01:06<00:27, 106.34it/s]
76%|#######5 | 9103/12040 [01:06<00:23, 123.56it/s]
76%|#######5 | 9123/12040 [01:06<00:20, 139.50it/s]
76%|#######5 | 9142/12040 [01:06<00:19, 151.78it/s]
76%|#######6 | 9161/12040 [01:06<00:17, 160.34it/s]
76%|#######6 | 9181/12040 [01:06<00:16, 168.63it/s]
76%|#######6 | 9199/12040 [01:07<00:22, 126.38it/s]
77%|#######6 | 9218/12040 [01:07<00:20, 139.67it/s]
77%|#######6 | 9237/12040 [01:07<00:18, 151.63it/s]
77%|#######6 | 9257/12040 [01:07<00:17, 162.22it/s]
77%|#######7 | 9275/12040 [01:07<00:22, 124.08it/s]
77%|#######7 | 9295/12040 [01:07<00:19, 139.29it/s]
77%|#######7 | 9315/12040 [01:07<00:17, 152.54it/s]
78%|#######7 | 9332/12040 [01:08<00:22, 119.47it/s]
78%|#######7 | 9351/12040 [01:08<00:20, 134.36it/s]
78%|#######7 | 9370/12040 [01:08<00:18, 147.06it/s]
78%|#######7 | 9387/12040 [01:08<00:22, 116.65it/s]
78%|#######8 | 9406/12040 [01:08<00:20, 131.63it/s]
78%|#######8 | 9425/12040 [01:08<00:18, 144.95it/s]
78%|#######8 | 9445/12040 [01:08<00:16, 156.99it/s]
79%|#######8 | 9464/12040 [01:08<00:15, 164.07it/s]
79%|#######8 | 9484/12040 [01:09<00:20, 124.44it/s]
79%|#######8 | 9500/12040 [01:09<00:19, 131.64it/s]
79%|#######9 | 9516/12040 [01:09<00:18, 136.12it/s]
79%|#######9 | 9535/12040 [01:09<00:16, 147.67it/s]
79%|#######9 | 9554/12040 [01:09<00:15, 156.70it/s]
79%|#######9 | 9571/12040 [01:09<00:20, 119.60it/s]
80%|#######9 | 9590/12040 [01:09<00:18, 134.91it/s]
80%|#######9 | 9609/12040 [01:10<00:16, 148.15it/s]
80%|#######9 | 9626/12040 [01:10<00:15, 151.24it/s]
80%|######## | 9643/12040 [01:10<00:20, 114.85it/s]
80%|######## | 9663/12040 [01:10<00:17, 132.93it/s]
80%|######## | 9680/12040 [01:10<00:21, 109.69it/s]
81%|######## | 9700/12040 [01:10<00:18, 127.21it/s]
81%|######## | 9719/12040 [01:10<00:16, 141.24it/s]
81%|######## | 9739/12040 [01:10<00:14, 153.86it/s]
81%|########1 | 9759/12040 [01:11<00:13, 164.36it/s]
81%|########1 | 9777/12040 [01:11<00:13, 167.49it/s]
81%|########1 | 9795/12040 [01:11<00:17, 125.07it/s]
82%|########1 | 9814/12040 [01:11<00:15, 139.31it/s]
82%|########1 | 9834/12040 [01:11<00:14, 151.80it/s]
82%|########1 | 9854/12040 [01:11<00:13, 161.89it/s]
82%|########1 | 9872/12040 [01:11<00:17, 123.75it/s]
82%|########2 | 9891/12040 [01:12<00:15, 137.62it/s]
82%|########2 | 9910/12040 [01:12<00:14, 149.72it/s]
82%|########2 | 9927/12040 [01:12<00:17, 117.75it/s]
83%|########2 | 9946/12040 [01:12<00:15, 132.50it/s]
83%|########2 | 9966/12040 [01:12<00:14, 146.98it/s]
83%|########2 | 9983/12040 [01:12<00:17, 115.67it/s]
83%|########3 | 10003/12040 [01:12<00:15, 132.89it/s]
83%|########3 | 10022/12040 [01:13<00:13, 145.62it/s]
83%|########3 | 10040/12040 [01:13<00:12, 153.94it/s]
84%|########3 | 10059/12040 [01:13<00:12, 162.09it/s]
84%|########3 | 10079/12040 [01:13<00:11, 170.78it/s]
84%|########3 | 10097/12040 [01:13<00:15, 127.85it/s]
84%|########4 | 10116/12040 [01:13<00:13, 141.37it/s]
84%|########4 | 10136/12040 [01:13<00:12, 153.58it/s]
84%|########4 | 10155/12040 [01:13<00:11, 162.01it/s]
84%|########4 | 10173/12040 [01:14<00:15, 123.57it/s]
85%|########4 | 10192/12040 [01:14<00:13, 137.85it/s]
85%|########4 | 10212/12040 [01:14<00:12, 151.08it/s]
85%|########4 | 10229/12040 [01:14<00:15, 115.12it/s]
85%|########5 | 10245/12040 [01:14<00:14, 124.41it/s]
85%|########5 | 10264/12040 [01:14<00:12, 138.22it/s]
85%|########5 | 10280/12040 [01:14<00:15, 110.77it/s]
86%|########5 | 10299/12040 [01:15<00:13, 126.81it/s]
86%|########5 | 10319/12040 [01:15<00:12, 142.09it/s]
86%|########5 | 10339/12040 [01:15<00:11, 154.43it/s]
86%|########6 | 10359/12040 [01:15<00:10, 164.10it/s]
86%|########6 | 10379/12040 [01:15<00:09, 171.57it/s]
86%|########6 | 10398/12040 [01:15<00:13, 118.72it/s]
87%|########6 | 10415/12040 [01:15<00:12, 129.28it/s]
87%|########6 | 10434/12040 [01:15<00:11, 142.90it/s]
87%|########6 | 10452/12040 [01:16<00:10, 151.66it/s]
87%|########6 | 10469/12040 [01:16<00:13, 117.43it/s]
87%|########7 | 10489/12040 [01:16<00:11, 134.42it/s]
87%|########7 | 10508/12040 [01:16<00:10, 145.39it/s]
87%|########7 | 10525/12040 [01:16<00:14, 108.08it/s]
88%|########7 | 10544/12040 [01:16<00:12, 123.94it/s]
88%|########7 | 10563/12040 [01:16<00:10, 138.23it/s]
88%|########7 | 10580/12040 [01:17<00:13, 111.90it/s]
88%|########8 | 10600/12040 [01:17<00:11, 128.93it/s]
88%|########8 | 10620/12040 [01:17<00:09, 144.13it/s]
88%|########8 | 10639/12040 [01:17<00:09, 155.11it/s]
89%|########8 | 10658/12040 [01:17<00:08, 162.66it/s]
89%|########8 | 10677/12040 [01:17<00:08, 169.97it/s]
89%|########8 | 10695/12040 [01:17<00:10, 126.24it/s]
89%|########8 | 10713/12040 [01:18<00:09, 137.33it/s]
89%|########9 | 10730/12040 [01:18<00:09, 144.08it/s]
89%|########9 | 10747/12040 [01:18<00:08, 150.65it/s]
89%|########9 | 10764/12040 [01:18<00:10, 117.22it/s]
90%|########9 | 10783/12040 [01:18<00:09, 132.27it/s]
90%|########9 | 10802/12040 [01:18<00:08, 145.69it/s]
90%|########9 | 10821/12040 [01:18<00:07, 155.82it/s]
90%|######### | 10838/12040 [01:19<00:10, 119.53it/s]
90%|######### | 10857/12040 [01:19<00:08, 134.19it/s]
90%|######### | 10875/12040 [01:19<00:08, 144.41it/s]
90%|######### | 10892/12040 [01:19<00:10, 107.60it/s]
91%|######### | 10911/12040 [01:19<00:09, 124.27it/s]
91%|######### | 10930/12040 [01:19<00:08, 138.61it/s]
91%|######### | 10949/12040 [01:19<00:07, 150.73it/s]
91%|#########1| 10968/12040 [01:19<00:06, 160.48it/s]
91%|#########1| 10986/12040 [01:20<00:08, 120.29it/s]
91%|#########1| 11002/12040 [01:20<00:08, 126.44it/s]
92%|#########1| 11017/12040 [01:20<00:07, 130.07it/s]
92%|#########1| 11036/12040 [01:20<00:06, 144.00it/s]
92%|#########1| 11055/12040 [01:20<00:06, 155.53it/s]
92%|#########1| 11072/12040 [01:20<00:08, 118.11it/s]
92%|#########2| 11091/12040 [01:20<00:07, 133.33it/s]
92%|#########2| 11110/12040 [01:20<00:06, 146.34it/s]
92%|#########2| 11127/12040 [01:21<00:07, 114.94it/s]
93%|#########2| 11146/12040 [01:21<00:06, 130.06it/s]
93%|#########2| 11165/12040 [01:21<00:06, 143.57it/s]
93%|#########2| 11182/12040 [01:21<00:07, 113.72it/s]
93%|#########3| 11201/12040 [01:21<00:06, 129.73it/s]
93%|#########3| 11219/12040 [01:21<00:05, 139.59it/s]
93%|#########3| 11235/12040 [01:21<00:05, 143.89it/s]
93%|#########3| 11251/12040 [01:22<00:05, 142.20it/s]
94%|#########3| 11270/12040 [01:22<00:05, 153.50it/s]
94%|#########3| 11287/12040 [01:22<00:06, 117.67it/s]
94%|#########3| 11306/12040 [01:22<00:05, 133.60it/s]
94%|#########4| 11325/12040 [01:22<00:04, 146.33it/s]
94%|#########4| 11344/12040 [01:22<00:04, 156.66it/s]
94%|#########4| 11361/12040 [01:23<00:06, 102.83it/s]
94%|#########4| 11377/12040 [01:23<00:05, 113.93it/s]
95%|#########4| 11396/12040 [01:23<00:04, 130.11it/s]
95%|#########4| 11415/12040 [01:23<00:04, 143.83it/s]
95%|#########4| 11432/12040 [01:23<00:05, 113.83it/s]
95%|#########5| 11451/12040 [01:23<00:04, 129.74it/s]
95%|#########5| 11470/12040 [01:23<00:03, 143.21it/s]
95%|#########5| 11487/12040 [01:23<00:04, 113.43it/s]
96%|#########5| 11506/12040 [01:24<00:04, 129.18it/s]
96%|#########5| 11524/12040 [01:24<00:03, 140.94it/s]
96%|#########5| 11543/12040 [01:24<00:03, 152.40it/s]
96%|#########6| 11562/12040 [01:24<00:02, 161.84it/s]
96%|#########6| 11580/12040 [01:24<00:03, 121.87it/s]
96%|#########6| 11596/12040 [01:24<00:03, 129.38it/s]
96%|#########6| 11612/12040 [01:24<00:03, 135.00it/s]
97%|#########6| 11628/12040 [01:24<00:02, 140.47it/s]
97%|#########6| 11646/12040 [01:25<00:02, 148.59it/s]
97%|#########6| 11662/12040 [01:25<00:03, 113.15it/s]
97%|#########7| 11681/12040 [01:25<00:02, 129.57it/s]
97%|#########7| 11700/12040 [01:25<00:02, 141.69it/s]
97%|#########7| 11716/12040 [01:25<00:02, 124.86it/s]
97%|#########7| 11730/12040 [01:25<00:03, 98.17it/s]
98%|#########7| 11746/12040 [01:25<00:02, 110.06it/s]
98%|#########7| 11760/12040 [01:26<00:02, 115.36it/s]
98%|#########7| 11773/12040 [01:26<00:02, 89.44it/s]
98%|#########7| 11792/12040 [01:26<00:02, 109.56it/s]
98%|#########8| 11811/12040 [01:26<00:01, 127.09it/s]
98%|#########8| 11830/12040 [01:26<00:01, 141.35it/s]
98%|#########8| 11849/12040 [01:26<00:01, 152.86it/s]
99%|#########8| 11866/12040 [01:26<00:01, 145.61it/s]
99%|#########8| 11882/12040 [01:27<00:01, 110.04it/s]
99%|#########8| 11900/12040 [01:27<00:01, 124.94it/s]
99%|#########8| 11919/12040 [01:27<00:00, 139.72it/s]
99%|#########9| 11938/12040 [01:27<00:00, 150.52it/s]
99%|#########9| 11956/12040 [01:27<00:00, 118.03it/s]
99%|#########9| 11975/12040 [01:27<00:00, 132.99it/s]
100%|#########9| 11994/12040 [01:27<00:00, 145.33it/s]
100%|#########9| 12013/12040 [01:27<00:00, 155.83it/s]
100%|#########9| 12030/12040 [01:28<00:00, 119.15it/s]
100%|##########| 12040/12040 [01:28<00:00, 136.46it/s]
Hits@1: 0.00 Filtered: 0.00
Hits@10: 0.00 Filtered: 0.00
Hits@100: 0.02 Filtered: 0.02
MR: 2899.35 Filtered: 2843.59
AUC: 0.52 Filtered: 0.53
Evaluation finished. Access the results using the "metrics" attribute.
Total running time of the script: ( 3 minutes 38.765 seconds)
Estimated memory usage: 2996 MB