Skip to content

Instantly share code, notes, and snippets.

@colesbury

colesbury/gist:7976640c82af042fd34eeb5e21322652

Created August 6, 2020 20:14
Show Gist options
  • Save colesbury/7976640c82af042fd34eeb5e21322652 to your computer and use it in GitHub Desktop.
Save colesbury/7976640c82af042fd34eeb5e21322652 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
INFO:root:World size is : 8
INFO:root:Running without GIL
INFO:root:Set start method of multiprocessing to spawn
INFO:root:Using the config
{"seed": 0, "data": {"dataset": "iamdb", "data_path": "/datasets01/iamdb/060820/", "img_height": 64}, "criterion_type": "transducer", "criterion": {"blank": true, "allow_repeats": false, "ngram": 0}, "model_type": "tds2d", "model": {"depth": 4, "tds_groups": [{"channels": 4, "num_blocks": 3, "stride": [2, 2]}, {"channels": 16, "num_blocks": 3, "stride": [2, 2]}, {"channels": 32, "num_blocks": 3, "stride": [2, 1]}, {"channels": 64, "num_blocks": 3, "stride": [2, 1]}], "kernel_size": [5, 7], "dropout": 0.1}, "optim": {"batch_size": 32, "epochs": 400, "learning_rate": 0.1, "crit_learning_rate": 0.1, "step_size": 100, "max_grad_norm": 5}}
INFO:root:Loading dataset ...
INFO:root:Loading model ...
INFO:root:Training tds2d model with 2,644,859 parameters.
INFO:root:Starting training ...
INFO:root:Epoch 1 started.
INFO:root:Epoch 1 complete. nUpdates 192, Loss 4.470, CER 0.931, Time 47.532 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 3.305, CER 0.817, Best Loss 3.305, Best CER 0.817
INFO:root:Timing Info: ds_fetch : 38.27ms, model_fwd : 40.80ms, crit_fwd : 28.69ms, bwd : 129.65ms, optim : 2.83ms, metrics : 7.03ms, train_total : 47532.01ms, test_total : 3674.34ms
INFO:root:Epoch 2 started.
INFO:root:Epoch 2 complete. nUpdates 384, Loss 3.234, CER 0.845, Time 46.452 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 3.677, CER 0.926, Best Loss 3.305, Best CER 0.817
INFO:root:Timing Info: ds_fetch : 36.60ms, model_fwd : 37.67ms, crit_fwd : 29.17ms, bwd : 128.92ms, optim : 2.18ms, metrics : 7.11ms, train_total : 46451.94ms, test_total : 3593.01ms
INFO:root:Epoch 3 started.
INFO:root:Epoch 3 complete. nUpdates 576, Loss 2.842, CER 0.777, Time 46.395 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 2.408, CER 0.690, Best Loss 2.408, Best CER 0.690
INFO:root:Timing Info: ds_fetch : 38.73ms, model_fwd : 37.46ms, crit_fwd : 28.41ms, bwd : 127.46ms, optim : 2.18ms, metrics : 7.11ms, train_total : 46395.50ms, test_total : 3605.94ms
INFO:root:Epoch 4 started.
INFO:root:Epoch 4 complete. nUpdates 768, Loss 2.528, CER 0.723, Time 46.785 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 2.285, CER 0.728, Best Loss 2.285, Best CER 0.690
INFO:root:Timing Info: ds_fetch : 38.58ms, model_fwd : 37.61ms, crit_fwd : 28.34ms, bwd : 129.35ms, optim : 2.18ms, metrics : 7.31ms, train_total : 46784.92ms, test_total : 3661.85ms
INFO:root:Epoch 5 started.
INFO:root:Epoch 5 complete. nUpdates 960, Loss 2.204, CER 0.646, Time 46.128 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 1.657, CER 0.484, Best Loss 1.657, Best CER 0.484
INFO:root:Timing Info: ds_fetch : 38.19ms, model_fwd : 37.40ms, crit_fwd : 28.03ms, bwd : 126.87ms, optim : 2.10ms, metrics : 7.35ms, train_total : 46127.70ms, test_total : 3684.60ms
INFO:root:Epoch 6 started.
INFO:root:Epoch 6 complete. nUpdates 1152, Loss 1.961, CER 0.566, Time 46.694 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 1.365, CER 0.419, Best Loss 1.365, Best CER 0.419
INFO:root:Timing Info: ds_fetch : 38.16ms, model_fwd : 37.75ms, crit_fwd : 29.33ms, bwd : 128.13ms, optim : 2.10ms, metrics : 7.44ms, train_total : 46694.23ms, test_total : 3687.88ms
INFO:root:Epoch 7 started.
INFO:root:Epoch 7 complete. nUpdates 1344, Loss 1.694, CER 0.492, Time 46.903 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 1.174, CER 0.337, Best Loss 1.174, Best CER 0.337
INFO:root:Timing Info: ds_fetch : 36.00ms, model_fwd : 38.27ms, crit_fwd : 29.77ms, bwd : 130.32ms, optim : 2.06ms, metrics : 7.57ms, train_total : 46902.69ms, test_total : 3694.65ms
INFO:root:Epoch 8 started.
INFO:root:Epoch 8 complete. nUpdates 1536, Loss 1.509, CER 0.434, Time 47.187 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 1.113, CER 0.342, Best Loss 1.113, Best CER 0.337
INFO:root:Timing Info: ds_fetch : 37.48ms, model_fwd : 38.01ms, crit_fwd : 29.63ms, bwd : 130.72ms, optim : 2.08ms, metrics : 7.55ms, train_total : 47187.17ms, test_total : 3727.71ms
INFO:root:Epoch 9 started.
INFO:root:Epoch 9 complete. nUpdates 1728, Loss 1.394, CER 0.399, Time 45.951 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.889, CER 0.260, Best Loss 0.889, Best CER 0.260
INFO:root:Timing Info: ds_fetch : 36.12ms, model_fwd : 36.75ms, crit_fwd : 28.28ms, bwd : 128.22ms, optim : 2.08ms, metrics : 7.60ms, train_total : 45950.68ms, test_total : 3739.38ms
INFO:root:Epoch 10 started.
INFO:root:Epoch 10 complete. nUpdates 1920, Loss 1.274, CER 0.366, Time 46.207 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.821, CER 0.226, Best Loss 0.821, Best CER 0.226
INFO:root:Timing Info: ds_fetch : 38.00ms, model_fwd : 37.60ms, crit_fwd : 29.95ms, bwd : 124.97ms, optim : 2.18ms, metrics : 7.65ms, train_total : 46207.30ms, test_total : 3801.52ms
INFO:root:Epoch 11 started.
INFO:root:Epoch 11 complete. nUpdates 2112, Loss 1.166, CER 0.335, Time 47.115 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.724, CER 0.211, Best Loss 0.724, Best CER 0.211
INFO:root:Timing Info: ds_fetch : 36.81ms, model_fwd : 37.81ms, crit_fwd : 29.82ms, bwd : 130.78ms, optim : 2.09ms, metrics : 7.80ms, train_total : 47114.80ms, test_total : 3688.74ms
INFO:root:Epoch 12 started.
INFO:root:Epoch 12 complete. nUpdates 2304, Loss 1.078, CER 0.312, Time 47.372 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.669, CER 0.185, Best Loss 0.669, Best CER 0.185
INFO:root:Timing Info: ds_fetch : 40.33ms, model_fwd : 37.88ms, crit_fwd : 29.01ms, bwd : 129.33ms, optim : 2.13ms, metrics : 7.74ms, train_total : 47372.05ms, test_total : 3736.71ms
INFO:root:Epoch 13 started.
INFO:root:Epoch 13 complete. nUpdates 2496, Loss 1.007, CER 0.291, Time 46.739 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.654, CER 0.180, Best Loss 0.654, Best CER 0.180
INFO:root:Timing Info: ds_fetch : 37.88ms, model_fwd : 37.58ms, crit_fwd : 28.75ms, bwd : 129.20ms, optim : 2.16ms, metrics : 7.57ms, train_total : 46739.39ms, test_total : 3756.95ms
INFO:root:Epoch 14 started.
INFO:root:Epoch 14 complete. nUpdates 2688, Loss 0.954, CER 0.275, Time 47.836 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.598, CER 0.168, Best Loss 0.598, Best CER 0.168
INFO:root:Timing Info: ds_fetch : 38.95ms, model_fwd : 37.85ms, crit_fwd : 29.20ms, bwd : 132.90ms, optim : 2.12ms, metrics : 7.83ms, train_total : 47836.05ms, test_total : 3800.32ms
INFO:root:Epoch 15 started.
INFO:root:Epoch 15 complete. nUpdates 2880, Loss 0.905, CER 0.261, Time 47.234 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.601, CER 0.173, Best Loss 0.598, Best CER 0.168
INFO:root:Timing Info: ds_fetch : 37.74ms, model_fwd : 37.31ms, crit_fwd : 28.82ms, bwd : 131.90ms, optim : 2.12ms, metrics : 7.83ms, train_total : 47234.32ms, test_total : 3751.95ms
INFO:root:Epoch 16 started.
INFO:root:Epoch 16 complete. nUpdates 3072, Loss 0.866, CER 0.249, Time 47.105 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.550, CER 0.151, Best Loss 0.550, Best CER 0.151
INFO:root:Timing Info: ds_fetch : 37.64ms, model_fwd : 37.41ms, crit_fwd : 28.88ms, bwd : 131.21ms, optim : 2.09ms, metrics : 7.83ms, train_total : 47105.51ms, test_total : 3838.30ms
INFO:root:Epoch 17 started.
INFO:root:Epoch 17 complete. nUpdates 3264, Loss 0.843, CER 0.244, Time 47.156 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.530, CER 0.149, Best Loss 0.530, Best CER 0.149
INFO:root:Timing Info: ds_fetch : 37.47ms, model_fwd : 37.54ms, crit_fwd : 29.32ms, bwd : 131.03ms, optim : 2.11ms, metrics : 7.85ms, train_total : 47156.50ms, test_total : 3807.74ms
INFO:root:Epoch 18 started.
INFO:root:Epoch 18 complete. nUpdates 3456, Loss 0.835, CER 0.239, Time 48.185 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.503, CER 0.138, Best Loss 0.503, Best CER 0.138
INFO:root:Timing Info: ds_fetch : 38.06ms, model_fwd : 37.89ms, crit_fwd : 29.70ms, bwd : 135.00ms, optim : 2.09ms, metrics : 7.95ms, train_total : 48185.15ms, test_total : 3895.67ms
INFO:root:Epoch 19 started.
INFO:root:Epoch 19 complete. nUpdates 3648, Loss 0.770, CER 0.224, Time 47.848 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.501, CER 0.139, Best Loss 0.501, Best CER 0.138
INFO:root:Timing Info: ds_fetch : 38.90ms, model_fwd : 37.76ms, crit_fwd : 29.74ms, bwd : 132.57ms, optim : 2.05ms, metrics : 7.90ms, train_total : 47848.57ms, test_total : 3789.76ms
INFO:root:Epoch 20 started.
INFO:root:Epoch 20 complete. nUpdates 3840, Loss 0.740, CER 0.215, Time 47.471 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.484, CER 0.137, Best Loss 0.484, Best CER 0.137
INFO:root:Timing Info: ds_fetch : 37.24ms, model_fwd : 37.88ms, crit_fwd : 30.16ms, bwd : 131.64ms, optim : 2.13ms, metrics : 7.88ms, train_total : 47471.44ms, test_total : 3813.85ms
INFO:root:Epoch 21 started.
INFO:root:Epoch 21 complete. nUpdates 4032, Loss 0.734, CER 0.212, Time 47.454 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.455, CER 0.125, Best Loss 0.455, Best CER 0.125
INFO:root:Timing Info: ds_fetch : 38.51ms, model_fwd : 37.84ms, crit_fwd : 31.24ms, bwd : 129.13ms, optim : 2.07ms, metrics : 8.07ms, train_total : 47453.77ms, test_total : 3727.41ms
INFO:root:Epoch 22 started.
INFO:root:Epoch 22 complete. nUpdates 4224, Loss 0.708, CER 0.206, Time 47.658 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.474, CER 0.131, Best Loss 0.455, Best CER 0.125
INFO:root:Timing Info: ds_fetch : 36.97ms, model_fwd : 37.61ms, crit_fwd : 29.65ms, bwd : 133.70ms, optim : 2.08ms, metrics : 7.93ms, train_total : 47658.36ms, test_total : 3768.50ms
INFO:root:Epoch 23 started.
INFO:root:Epoch 23 complete. nUpdates 4416, Loss 0.696, CER 0.202, Time 46.635 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.436, CER 0.121, Best Loss 0.436, Best CER 0.121
INFO:root:Timing Info: ds_fetch : 36.78ms, model_fwd : 37.37ms, crit_fwd : 28.48ms, bwd : 129.92ms, optim : 2.08ms, metrics : 7.97ms, train_total : 46635.25ms, test_total : 3926.95ms
INFO:root:Epoch 24 started.
INFO:root:Epoch 24 complete. nUpdates 4608, Loss 0.668, CER 0.193, Time 46.862 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.432, CER 0.117, Best Loss 0.432, Best CER 0.117
INFO:root:Timing Info: ds_fetch : 37.79ms, model_fwd : 37.75ms, crit_fwd : 29.62ms, bwd : 128.54ms, optim : 2.15ms, metrics : 7.94ms, train_total : 46862.00ms, test_total : 3945.65ms
INFO:root:Epoch 25 started.
INFO:root:Epoch 25 complete. nUpdates 4800, Loss 0.655, CER 0.190, Time 47.151 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.449, CER 0.124, Best Loss 0.432, Best CER 0.117
INFO:root:Timing Info: ds_fetch : 37.86ms, model_fwd : 37.21ms, crit_fwd : 29.12ms, bwd : 131.05ms, optim : 2.08ms, metrics : 7.96ms, train_total : 47151.55ms, test_total : 3813.09ms
INFO:root:Epoch 26 started.
INFO:root:Epoch 26 complete. nUpdates 4992, Loss 0.638, CER 0.185, Time 47.770 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.424, CER 0.116, Best Loss 0.424, Best CER 0.116
INFO:root:Timing Info: ds_fetch : 37.66ms, model_fwd : 37.78ms, crit_fwd : 28.94ms, bwd : 134.21ms, optim : 2.03ms, metrics : 7.89ms, train_total : 47769.78ms, test_total : 3981.84ms
INFO:root:Epoch 27 started.
INFO:root:Epoch 27 complete. nUpdates 5184, Loss 0.614, CER 0.179, Time 47.183 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.414, CER 0.113, Best Loss 0.414, Best CER 0.113
INFO:root:Timing Info: ds_fetch : 39.56ms, model_fwd : 37.35ms, crit_fwd : 29.56ms, bwd : 128.98ms, optim : 2.15ms, metrics : 7.84ms, train_total : 47182.71ms, test_total : 3850.78ms
INFO:root:Epoch 28 started.
INFO:root:Epoch 28 complete. nUpdates 5376, Loss 0.620, CER 0.181, Time 47.921 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.431, CER 0.114, Best Loss 0.414, Best CER 0.113
INFO:root:Timing Info: ds_fetch : 38.32ms, model_fwd : 38.26ms, crit_fwd : 30.96ms, bwd : 131.41ms, optim : 2.15ms, metrics : 8.19ms, train_total : 47921.20ms, test_total : 3751.84ms
INFO:root:Epoch 29 started.
INFO:root:Epoch 29 complete. nUpdates 5568, Loss 0.598, CER 0.174, Time 47.869 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.406, CER 0.110, Best Loss 0.406, Best CER 0.110
INFO:root:Timing Info: ds_fetch : 37.72ms, model_fwd : 38.06ms, crit_fwd : 30.93ms, bwd : 132.30ms, optim : 2.07ms, metrics : 7.93ms, train_total : 47869.22ms, test_total : 3763.76ms
INFO:root:Epoch 30 started.
INFO:root:Epoch 30 complete. nUpdates 5760, Loss 0.600, CER 0.175, Time 46.575 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.397, CER 0.106, Best Loss 0.397, Best CER 0.106
INFO:root:Timing Info: ds_fetch : 36.84ms, model_fwd : 37.07ms, crit_fwd : 29.23ms, bwd : 129.23ms, optim : 2.03ms, metrics : 7.89ms, train_total : 46574.90ms, test_total : 3851.98ms
INFO:root:Epoch 31 started.
INFO:root:Epoch 31 complete. nUpdates 5952, Loss 0.601, CER 0.174, Time 46.679 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.394, CER 0.106, Best Loss 0.394, Best CER 0.106
INFO:root:Timing Info: ds_fetch : 36.58ms, model_fwd : 36.92ms, crit_fwd : 29.56ms, bwd : 129.77ms, optim : 2.04ms, metrics : 7.97ms, train_total : 46679.52ms, test_total : 3743.29ms
INFO:root:Epoch 32 started.
INFO:root:Epoch 32 complete. nUpdates 6144, Loss 0.569, CER 0.167, Time 47.416 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.388, CER 0.105, Best Loss 0.388, Best CER 0.105
INFO:root:Timing Info: ds_fetch : 39.58ms, model_fwd : 37.23ms, crit_fwd : 30.18ms, bwd : 129.45ms, optim : 2.09ms, metrics : 8.13ms, train_total : 47416.08ms, test_total : 3906.03ms
INFO:root:Epoch 33 started.
INFO:root:Epoch 33 complete. nUpdates 6336, Loss 0.577, CER 0.168, Time 48.301 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.393, CER 0.107, Best Loss 0.388, Best CER 0.105
INFO:root:Timing Info: ds_fetch : 40.68ms, model_fwd : 38.06ms, crit_fwd : 31.43ms, bwd : 130.72ms, optim : 2.17ms, metrics : 8.20ms, train_total : 48301.39ms, test_total : 3779.26ms
INFO:root:Epoch 34 started.
INFO:root:Epoch 34 complete. nUpdates 6528, Loss 0.558, CER 0.164, Time 47.779 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.380, CER 0.101, Best Loss 0.380, Best CER 0.101
INFO:root:Timing Info: ds_fetch : 39.90ms, model_fwd : 37.80ms, crit_fwd : 30.28ms, bwd : 130.35ms, optim : 2.20ms, metrics : 8.02ms, train_total : 47779.37ms, test_total : 4032.06ms
INFO:root:Epoch 35 started.
INFO:root:Epoch 35 complete. nUpdates 6720, Loss 0.549, CER 0.160, Time 48.209 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.381, CER 0.102, Best Loss 0.380, Best CER 0.101
INFO:root:Timing Info: ds_fetch : 38.46ms, model_fwd : 37.91ms, crit_fwd : 29.47ms, bwd : 134.92ms, optim : 2.06ms, metrics : 7.98ms, train_total : 48209.58ms, test_total : 3874.04ms
INFO:root:Epoch 36 started.
INFO:root:Epoch 36 complete. nUpdates 6912, Loss 0.531, CER 0.155, Time 47.826 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.378, CER 0.101, Best Loss 0.378, Best CER 0.101
INFO:root:Timing Info: ds_fetch : 39.04ms, model_fwd : 37.97ms, crit_fwd : 30.85ms, bwd : 130.67ms, optim : 2.15ms, metrics : 8.11ms, train_total : 47826.28ms, test_total : 4049.82ms
INFO:root:Epoch 37 started.
INFO:root:Epoch 37 complete. nUpdates 7104, Loss 0.537, CER 0.156, Time 47.824 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.363, CER 0.098, Best Loss 0.363, Best CER 0.098
INFO:root:Timing Info: ds_fetch : 36.07ms, model_fwd : 37.85ms, crit_fwd : 31.46ms, bwd : 133.13ms, optim : 2.05ms, metrics : 8.25ms, train_total : 47824.25ms, test_total : 3807.32ms
INFO:root:Epoch 38 started.
INFO:root:Epoch 38 complete. nUpdates 7296, Loss 0.511, CER 0.150, Time 47.042 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.353, CER 0.096, Best Loss 0.353, Best CER 0.096
INFO:root:Timing Info: ds_fetch : 36.44ms, model_fwd : 37.84ms, crit_fwd : 30.46ms, bwd : 129.57ms, optim : 2.17ms, metrics : 8.24ms, train_total : 47041.84ms, test_total : 3933.40ms
INFO:root:Epoch 39 started.
INFO:root:Epoch 39 complete. nUpdates 7488, Loss 0.519, CER 0.152, Time 48.108 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.356, CER 0.095, Best Loss 0.353, Best CER 0.095
INFO:root:Timing Info: ds_fetch : 39.33ms, model_fwd : 37.89ms, crit_fwd : 29.88ms, bwd : 132.69ms, optim : 2.09ms, metrics : 8.38ms, train_total : 48108.26ms, test_total : 3964.27ms
INFO:root:Epoch 40 started.
INFO:root:Epoch 40 complete. nUpdates 7680, Loss 0.515, CER 0.151, Time 48.245 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.360, CER 0.097, Best Loss 0.353, Best CER 0.095
INFO:root:Timing Info: ds_fetch : 38.00ms, model_fwd : 38.14ms, crit_fwd : 30.62ms, bwd : 133.90ms, optim : 2.09ms, metrics : 8.25ms, train_total : 48244.73ms, test_total : 3914.11ms
INFO:root:Epoch 41 started.
INFO:root:Epoch 41 complete. nUpdates 7872, Loss 0.507, CER 0.148, Time 47.743 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.350, CER 0.094, Best Loss 0.350, Best CER 0.094
INFO:root:Timing Info: ds_fetch : 40.88ms, model_fwd : 37.31ms, crit_fwd : 30.08ms, bwd : 130.03ms, optim : 2.04ms, metrics : 8.03ms, train_total : 47743.03ms, test_total : 3965.88ms
INFO:root:Epoch 42 started.
INFO:root:Epoch 42 complete. nUpdates 8064, Loss 0.492, CER 0.145, Time 47.866 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.352, CER 0.095, Best Loss 0.350, Best CER 0.094
INFO:root:Timing Info: ds_fetch : 36.84ms, model_fwd : 38.04ms, crit_fwd : 30.79ms, bwd : 133.19ms, optim : 2.09ms, metrics : 8.05ms, train_total : 47866.50ms, test_total : 3817.52ms
INFO:root:Epoch 43 started.
INFO:root:Epoch 43 complete. nUpdates 8256, Loss 0.501, CER 0.146, Time 47.405 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.346, CER 0.093, Best Loss 0.346, Best CER 0.093
INFO:root:Timing Info: ds_fetch : 37.64ms, model_fwd : 38.48ms, crit_fwd : 30.74ms, bwd : 129.44ms, optim : 2.14ms, metrics : 8.18ms, train_total : 47404.87ms, test_total : 4004.43ms
INFO:root:Epoch 44 started.
INFO:root:Epoch 44 complete. nUpdates 8448, Loss 0.503, CER 0.147, Time 47.743 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.358, CER 0.096, Best Loss 0.346, Best CER 0.093
INFO:root:Timing Info: ds_fetch : 39.31ms, model_fwd : 37.52ms, crit_fwd : 29.92ms, bwd : 131.42ms, optim : 2.11ms, metrics : 8.09ms, train_total : 47742.64ms, test_total : 3941.07ms
INFO:root:Epoch 45 started.
INFO:root:Epoch 45 complete. nUpdates 8640, Loss 0.471, CER 0.138, Time 47.982 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.342, CER 0.092, Best Loss 0.342, Best CER 0.092
INFO:root:Timing Info: ds_fetch : 40.80ms, model_fwd : 37.72ms, crit_fwd : 30.03ms, bwd : 130.74ms, optim : 2.06ms, metrics : 8.25ms, train_total : 47981.91ms, test_total : 3861.01ms
INFO:root:Epoch 46 started.
INFO:root:Epoch 46 complete. nUpdates 8832, Loss 0.494, CER 0.144, Time 47.786 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.337, CER 0.091, Best Loss 0.337, Best CER 0.091
INFO:root:Timing Info: ds_fetch : 39.22ms, model_fwd : 38.03ms, crit_fwd : 32.29ms, bwd : 128.46ms, optim : 2.08ms, metrics : 8.50ms, train_total : 47786.61ms, test_total : 3895.54ms
INFO:root:Epoch 47 started.
INFO:root:Epoch 47 complete. nUpdates 9024, Loss 0.489, CER 0.144, Time 47.350 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.339, CER 0.090, Best Loss 0.337, Best CER 0.090
INFO:root:Timing Info: ds_fetch : 38.02ms, model_fwd : 37.68ms, crit_fwd : 29.73ms, bwd : 130.70ms, optim : 2.12ms, metrics : 8.07ms, train_total : 47349.66ms, test_total : 3978.92ms
INFO:root:Epoch 48 started.
INFO:root:Epoch 48 complete. nUpdates 9216, Loss 0.467, CER 0.137, Time 48.605 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.346, CER 0.095, Best Loss 0.337, Best CER 0.090
INFO:root:Timing Info: ds_fetch : 38.16ms, model_fwd : 38.56ms, crit_fwd : 33.00ms, bwd : 132.30ms, optim : 2.19ms, metrics : 8.64ms, train_total : 48604.90ms, test_total : 3920.85ms
INFO:root:Epoch 49 started.
INFO:root:Epoch 49 complete. nUpdates 9408, Loss 0.478, CER 0.141, Time 48.011 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.337, CER 0.091, Best Loss 0.337, Best CER 0.090
INFO:root:Timing Info: ds_fetch : 38.52ms, model_fwd : 38.13ms, crit_fwd : 31.02ms, bwd : 131.64ms, optim : 2.13ms, metrics : 8.32ms, train_total : 48011.18ms, test_total : 3873.95ms
INFO:root:Epoch 50 started.
INFO:root:Epoch 50 complete. nUpdates 9600, Loss 0.472, CER 0.139, Time 47.491 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.345, CER 0.093, Best Loss 0.337, Best CER 0.090
INFO:root:Timing Info: ds_fetch : 36.77ms, model_fwd : 36.89ms, crit_fwd : 30.75ms, bwd : 132.48ms, optim : 2.06ms, metrics : 8.11ms, train_total : 47490.77ms, test_total : 3825.25ms
INFO:root:Epoch 51 started.
INFO:root:Epoch 51 complete. nUpdates 9792, Loss 0.459, CER 0.134, Time 48.187 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.334, CER 0.089, Best Loss 0.334, Best CER 0.089
INFO:root:Timing Info: ds_fetch : 39.44ms, model_fwd : 37.60ms, crit_fwd : 30.26ms, bwd : 133.13ms, optim : 2.06ms, metrics : 8.17ms, train_total : 48186.70ms, test_total : 3958.90ms
INFO:root:Epoch 52 started.
INFO:root:Epoch 52 complete. nUpdates 9984, Loss 0.451, CER 0.132, Time 47.840 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.336, CER 0.091, Best Loss 0.334, Best CER 0.089
INFO:root:Timing Info: ds_fetch : 38.41ms, model_fwd : 38.17ms, crit_fwd : 31.53ms, bwd : 130.36ms, optim : 2.20ms, metrics : 8.19ms, train_total : 47839.91ms, test_total : 4020.39ms
INFO:root:Epoch 53 started.
INFO:root:Epoch 53 complete. nUpdates 10176, Loss 0.451, CER 0.133, Time 47.741 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.333, CER 0.089, Best Loss 0.333, Best CER 0.089
INFO:root:Timing Info: ds_fetch : 37.92ms, model_fwd : 38.43ms, crit_fwd : 32.59ms, bwd : 128.90ms, optim : 2.08ms, metrics : 8.43ms, train_total : 47741.46ms, test_total : 3950.42ms
INFO:root:Epoch 54 started.
INFO:root:Epoch 54 complete. nUpdates 10368, Loss 0.433, CER 0.128, Time 48.118 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.332, CER 0.089, Best Loss 0.332, Best CER 0.089
INFO:root:Timing Info: ds_fetch : 37.11ms, model_fwd : 37.93ms, crit_fwd : 31.60ms, bwd : 133.19ms, optim : 2.05ms, metrics : 8.45ms, train_total : 48118.32ms, test_total : 3844.73ms
INFO:root:Epoch 55 started.
INFO:root:Epoch 55 complete. nUpdates 10560, Loss 0.456, CER 0.134, Time 47.926 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.331, CER 0.090, Best Loss 0.331, Best CER 0.089
INFO:root:Timing Info: ds_fetch : 39.02ms, model_fwd : 37.68ms, crit_fwd : 29.99ms, bwd : 132.39ms, optim : 2.09ms, metrics : 8.15ms, train_total : 47925.80ms, test_total : 3933.52ms
INFO:root:Epoch 56 started.
INFO:root:Epoch 56 complete. nUpdates 10752, Loss 0.439, CER 0.130, Time 47.311 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.327, CER 0.087, Best Loss 0.327, Best CER 0.087
INFO:root:Timing Info: ds_fetch : 37.68ms, model_fwd : 37.39ms, crit_fwd : 31.59ms, bwd : 128.98ms, optim : 2.11ms, metrics : 8.36ms, train_total : 47311.08ms, test_total : 4023.32ms
INFO:root:Epoch 57 started.
INFO:root:Epoch 57 complete. nUpdates 10944, Loss 0.429, CER 0.127, Time 48.314 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.323, CER 0.088, Best Loss 0.323, Best CER 0.087
INFO:root:Timing Info: ds_fetch : 37.52ms, model_fwd : 37.75ms, crit_fwd : 30.26ms, bwd : 135.51ms, optim : 2.08ms, metrics : 8.24ms, train_total : 48314.39ms, test_total : 4099.41ms
INFO:root:Epoch 58 started.
INFO:root:Epoch 58 complete. nUpdates 11136, Loss 0.454, CER 0.133, Time 48.573 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.325, CER 0.088, Best Loss 0.323, Best CER 0.087
INFO:root:Timing Info: ds_fetch : 38.18ms, model_fwd : 38.57ms, crit_fwd : 32.82ms, bwd : 132.43ms, optim : 2.09ms, metrics : 8.60ms, train_total : 48572.68ms, test_total : 3962.61ms
INFO:root:Epoch 59 started.
INFO:root:Epoch 59 complete. nUpdates 11328, Loss 0.417, CER 0.123, Time 47.815 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.333, CER 0.088, Best Loss 0.323, Best CER 0.087
INFO:root:Timing Info: ds_fetch : 37.28ms, model_fwd : 37.60ms, crit_fwd : 31.00ms, bwd : 132.45ms, optim : 2.13ms, metrics : 8.28ms, train_total : 47814.84ms, test_total : 4077.95ms
INFO:root:Epoch 60 started.
INFO:root:Epoch 60 complete. nUpdates 11520, Loss 0.424, CER 0.125, Time 47.770 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.320, CER 0.085, Best Loss 0.320, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 39.56ms, model_fwd : 37.59ms, crit_fwd : 29.86ms, bwd : 131.19ms, optim : 2.15ms, metrics : 8.12ms, train_total : 47769.93ms, test_total : 4006.16ms
INFO:root:Epoch 61 started.
INFO:root:Epoch 61 complete. nUpdates 11712, Loss 0.428, CER 0.127, Time 47.970 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.324, CER 0.086, Best Loss 0.320, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 41.22ms, model_fwd : 37.45ms, crit_fwd : 30.12ms, bwd : 130.55ms, optim : 2.04ms, metrics : 8.16ms, train_total : 47970.11ms, test_total : 3896.87ms
INFO:root:Epoch 62 started.
INFO:root:Epoch 62 complete. nUpdates 11904, Loss 0.428, CER 0.127, Time 48.509 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.323, CER 0.087, Best Loss 0.320, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 37.22ms, model_fwd : 38.02ms, crit_fwd : 31.12ms, bwd : 135.62ms, optim : 2.02ms, metrics : 8.36ms, train_total : 48508.77ms, test_total : 4138.91ms
INFO:root:Epoch 63 started.
INFO:root:Epoch 63 complete. nUpdates 12096, Loss 0.416, CER 0.123, Time 48.369 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.323, CER 0.086, Best Loss 0.320, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 39.39ms, model_fwd : 38.00ms, crit_fwd : 30.93ms, bwd : 133.02ms, optim : 2.06ms, metrics : 8.22ms, train_total : 48369.12ms, test_total : 4021.88ms
INFO:root:Epoch 64 started.
INFO:root:Epoch 64 complete. nUpdates 12288, Loss 0.407, CER 0.121, Time 48.392 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.317, CER 0.085, Best Loss 0.317, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 41.86ms, model_fwd : 37.79ms, crit_fwd : 30.95ms, bwd : 130.95ms, optim : 2.07ms, metrics : 8.10ms, train_total : 48392.35ms, test_total : 3932.58ms
INFO:root:Epoch 65 started.
INFO:root:Epoch 65 complete. nUpdates 12480, Loss 0.407, CER 0.121, Time 47.736 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.324, CER 0.086, Best Loss 0.317, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 38.86ms, model_fwd : 37.83ms, crit_fwd : 32.11ms, bwd : 129.16ms, optim : 2.06ms, metrics : 8.31ms, train_total : 47736.03ms, test_total : 4069.11ms
INFO:root:Epoch 66 started.
INFO:root:Epoch 66 complete. nUpdates 12672, Loss 0.412, CER 0.121, Time 47.684 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.314, CER 0.085, Best Loss 0.314, Best CER 0.085
INFO:root:Timing Info: ds_fetch : 38.86ms, model_fwd : 37.25ms, crit_fwd : 31.21ms, bwd : 130.12ms, optim : 2.12ms, metrics : 8.50ms, train_total : 47683.70ms, test_total : 3989.37ms
INFO:root:Epoch 67 started.
INFO:root:Epoch 67 complete. nUpdates 12864, Loss 0.399, CER 0.118, Time 48.281 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.315, CER 0.084, Best Loss 0.314, Best CER 0.084
INFO:root:Timing Info: ds_fetch : 39.37ms, model_fwd : 38.02ms, crit_fwd : 30.35ms, bwd : 133.14ms, optim : 2.09ms, metrics : 8.20ms, train_total : 48281.59ms, test_total : 4005.26ms
INFO:root:Epoch 68 started.
INFO:root:Epoch 68 complete. nUpdates 13056, Loss 0.416, CER 0.122, Time 47.720 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.318, CER 0.085, Best Loss 0.314, Best CER 0.084
INFO:root:Timing Info: ds_fetch : 36.90ms, model_fwd : 38.16ms, crit_fwd : 31.76ms, bwd : 131.17ms, optim : 2.09ms, metrics : 8.16ms, train_total : 47720.18ms, test_total : 3962.35ms
INFO:root:Epoch 69 started.
INFO:root:Epoch 69 complete. nUpdates 13248, Loss 0.396, CER 0.118, Time 48.670 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.311, CER 0.082, Best Loss 0.311, Best CER 0.082
INFO:root:Timing Info: ds_fetch : 37.61ms, model_fwd : 38.65ms, crit_fwd : 33.01ms, bwd : 133.38ms, optim : 2.13ms, metrics : 8.41ms, train_total : 48670.07ms, test_total : 4122.49ms
INFO:root:Epoch 70 started.
INFO:root:Epoch 70 complete. nUpdates 13440, Loss 0.392, CER 0.116, Time 47.159 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.316, CER 0.083, Best Loss 0.311, Best CER 0.082
INFO:root:Timing Info: ds_fetch : 37.42ms, model_fwd : 36.99ms, crit_fwd : 29.68ms, bwd : 131.03ms, optim : 2.08ms, metrics : 8.14ms, train_total : 47159.60ms, test_total : 4157.84ms
INFO:root:Epoch 71 started.
INFO:root:Epoch 71 complete. nUpdates 13632, Loss 0.390, CER 0.116, Time 47.837 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.315, CER 0.084, Best Loss 0.311, Best CER 0.082
INFO:root:Timing Info: ds_fetch : 35.58ms, model_fwd : 37.60ms, crit_fwd : 31.62ms, bwd : 133.65ms, optim : 2.11ms, metrics : 8.32ms, train_total : 47837.57ms, test_total : 4215.69ms
INFO:root:Epoch 72 started.
INFO:root:Epoch 72 complete. nUpdates 13824, Loss 0.414, CER 0.122, Time 48.073 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.323, CER 0.086, Best Loss 0.311, Best CER 0.082
INFO:root:Timing Info: ds_fetch : 37.72ms, model_fwd : 37.35ms, crit_fwd : 31.09ms, bwd : 133.38ms, optim : 2.12ms, metrics : 8.43ms, train_total : 48073.05ms, test_total : 4055.53ms
INFO:root:Epoch 73 started.
INFO:root:Epoch 73 complete. nUpdates 14016, Loss 0.385, CER 0.115, Time 48.490 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.315, CER 0.083, Best Loss 0.311, Best CER 0.082
INFO:root:Timing Info: ds_fetch : 40.50ms, model_fwd : 37.75ms, crit_fwd : 31.53ms, bwd : 132.10ms, optim : 2.14ms, metrics : 8.22ms, train_total : 48490.06ms, test_total : 3940.73ms
INFO:root:Epoch 74 started.
INFO:root:Epoch 74 complete. nUpdates 14208, Loss 0.401, CER 0.119, Time 47.558 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.310, CER 0.081, Best Loss 0.310, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 37.25ms, model_fwd : 37.39ms, crit_fwd : 29.95ms, bwd : 132.63ms, optim : 2.10ms, metrics : 8.09ms, train_total : 47558.28ms, test_total : 4083.81ms
INFO:root:Epoch 75 started.
INFO:root:Epoch 75 complete. nUpdates 14400, Loss 0.391, CER 0.116, Time 48.411 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.315, CER 0.083, Best Loss 0.310, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 39.58ms, model_fwd : 37.53ms, crit_fwd : 31.32ms, bwd : 132.98ms, optim : 2.11ms, metrics : 8.31ms, train_total : 48411.38ms, test_total : 3917.95ms
INFO:root:Epoch 76 started.
INFO:root:Epoch 76 complete. nUpdates 14592, Loss 0.393, CER 0.117, Time 48.030 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.318, CER 0.084, Best Loss 0.310, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 37.91ms, model_fwd : 37.82ms, crit_fwd : 30.67ms, bwd : 133.10ms, optim : 2.06ms, metrics : 8.30ms, train_total : 48029.77ms, test_total : 4055.03ms
INFO:root:Epoch 77 started.
INFO:root:Epoch 77 complete. nUpdates 14784, Loss 0.378, CER 0.113, Time 47.312 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.305, CER 0.081, Best Loss 0.305, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 37.99ms, model_fwd : 37.22ms, crit_fwd : 29.87ms, bwd : 130.65ms, optim : 2.16ms, metrics : 8.23ms, train_total : 47312.50ms, test_total : 4112.51ms
INFO:root:Epoch 78 started.
INFO:root:Epoch 78 complete. nUpdates 14976, Loss 0.387, CER 0.115, Time 48.391 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.310, CER 0.081, Best Loss 0.305, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 37.82ms, model_fwd : 38.07ms, crit_fwd : 31.54ms, bwd : 133.91ms, optim : 2.08ms, metrics : 8.33ms, train_total : 48390.71ms, test_total : 4049.45ms
INFO:root:Epoch 79 started.
INFO:root:Epoch 79 complete. nUpdates 15168, Loss 0.377, CER 0.112, Time 48.702 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.310, CER 0.083, Best Loss 0.305, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 39.36ms, model_fwd : 38.03ms, crit_fwd : 31.51ms, bwd : 134.03ms, optim : 2.08ms, metrics : 8.35ms, train_total : 48702.22ms, test_total : 3923.78ms
INFO:root:Epoch 80 started.
INFO:root:Epoch 80 complete. nUpdates 15360, Loss 0.384, CER 0.115, Time 48.539 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.306, CER 0.081, Best Loss 0.305, Best CER 0.081
INFO:root:Timing Info: ds_fetch : 37.89ms, model_fwd : 38.06ms, crit_fwd : 31.41ms, bwd : 134.80ms, optim : 2.05ms, metrics : 8.31ms, train_total : 48539.51ms, test_total : 4094.73ms
INFO:root:Epoch 81 started.
INFO:root:Epoch 81 complete. nUpdates 15552, Loss 0.375, CER 0.111, Time 48.401 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.300, CER 0.080, Best Loss 0.300, Best CER 0.080
INFO:root:Timing Info: ds_fetch : 38.92ms, model_fwd : 37.42ms, crit_fwd : 31.14ms, bwd : 133.86ms, optim : 2.13ms, metrics : 8.33ms, train_total : 48400.69ms, test_total : 3993.04ms
INFO:root:Epoch 82 started.
INFO:root:Epoch 82 complete. nUpdates 15744, Loss 0.379, CER 0.112, Time 49.149 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.305, CER 0.080, Best Loss 0.300, Best CER 0.080
INFO:root:Timing Info: ds_fetch : 42.90ms, model_fwd : 37.99ms, crit_fwd : 31.47ms, bwd : 132.85ms, optim : 2.07ms, metrics : 8.38ms, train_total : 49149.14ms, test_total : 4015.95ms
INFO:root:Epoch 83 started.
INFO:root:Epoch 83 complete. nUpdates 15936, Loss 0.365, CER 0.109, Time 48.848 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.331, CER 0.087, Best Loss 0.300, Best CER 0.080
INFO:root:Timing Info: ds_fetch : 39.26ms, model_fwd : 37.80ms, crit_fwd : 31.94ms, bwd : 134.77ms, optim : 2.10ms, metrics : 8.25ms, train_total : 48847.65ms, test_total : 4143.01ms
INFO:root:Epoch 84 started.
INFO:root:Epoch 84 complete. nUpdates 16128, Loss 0.367, CER 0.109, Time 48.223 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.302, CER 0.080, Best Loss 0.300, Best CER 0.080
INFO:root:Timing Info: ds_fetch : 37.63ms, model_fwd : 37.48ms, crit_fwd : 30.78ms, bwd : 134.49ms, optim : 2.14ms, metrics : 8.35ms, train_total : 48223.02ms, test_total : 3976.46ms
INFO:root:Epoch 85 started.
INFO:root:Epoch 85 complete. nUpdates 16320, Loss 0.373, CER 0.111, Time 48.125 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.300, CER 0.078, Best Loss 0.300, Best CER 0.078
INFO:root:Timing Info: ds_fetch : 36.82ms, model_fwd : 37.92ms, crit_fwd : 31.14ms, bwd : 133.93ms, optim : 2.21ms, metrics : 8.34ms, train_total : 48125.02ms, test_total : 4280.53ms
INFO:root:Epoch 86 started.
INFO:root:Epoch 86 complete. nUpdates 16512, Loss 0.366, CER 0.109, Time 48.096 (s)
INFO:root:Evaluating validation set..
INFO:root:Validation Set: Loss 0.301, CER 0.079, Best Loss 0.300, Best CER 0.078
INFO:root:Timing Info: ds_fetch : 38.60ms, model_fwd : 37.71ms, crit_fwd : 31.53ms, bwd : 131.95ms, optim : 2.10ms, metrics : 8.32ms, train_total : 48096.61ms, test_total : 4084.42ms
INFO:root:Epoch 87 started.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/rightrec/train.py", line 326, in <module>
main()
File "/private/home/sgross/rightrec/train.py", line 318, in main
torch.multiprocessing.spawn(
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
while not context.join():
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 119, in join
raise Exception(msg)
Exception:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait
ready = selector.select(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/private/home/sgross/rightrec/train.py", line 236, in train
for inputs, targets in train_loader:
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
data = self._next_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
success, data = self._try_get_data()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers
finalizer()
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup
sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 104 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-i_c5vf76': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-cydyvl4w': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-kgqqwupb': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8m1lxtn0': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ee27nftv': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-4klj5ndv': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-4s0h5plm': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wud9al_o': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-d5ciwdux': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-h7wiji_u': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-aakfp4fv': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8o1s8943': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-niwqbj9c': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-9nf85d7d': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-2syt7fks': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-dez9oywf': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7i2wfzt2': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-2akdnrvz': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-fqgjol_b': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-o2yu23nk': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-57l1cx8d': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-qtiruoar': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-48bmkcx3': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-qhrgp40u': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-lv80t6jf': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-bwck76gf': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-9147pm_n': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-mi3twqq2': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-w5wsm_sk': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-a4ieflsa': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-n9xerarz': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-60eg5_w1': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wftu14uc': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-_2f3rcz8': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-5ofcq7zr': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ddwa1f7m': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-qn_uyj12': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wvayrt7s': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-mauo9sh0': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-cazf74b6': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-28fvr5jt': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vyrgki7_': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-psou2nwd': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ar99ec7n': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ob01ml2k': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-722bdo0f': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-4d68tw6t': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-0deki0f_': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wa7ury_1': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-3jcglm8l': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-yjbl54po': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7wpvr7te': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ckxyvg1o': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-pimwrhbj': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-o4ghmld9': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-68jic65t': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vnuzek0_': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-z8e51wc4': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-phg45jly': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-9unovp60': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-n8o2tu5u': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-60f8ul67': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-q87tm4w9': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-httzg6f2': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-grgfqo6b': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-hl9semjg': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-xb4t_awz': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-3vw9s34e': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-sevp09om': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-djieof_h': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7r_omihx': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vymcigu5': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7fbhwsfa': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vlilcolw': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-d45nvsl3': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-trfycjnv': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-6fl4r_8u': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gatw9887': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-big61gcq': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-201pgnus': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-yc2lq0td': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-u_6m0ens': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-c99wplxu': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-dat9gvb9': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-p9h8922r': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-kg0v6e4f': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ccus0tiy': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-sck2v2eg': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gs79yyny': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-23tcxlpt': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-al3u5w1e': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-kbc67ww6': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-z83tfu60': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gmesc_q3': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-u3tghhso': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-57pvcmvf': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-zfv74g_i': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-meod0g4u': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-orxf67mp': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-dyp3sn4r': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vyorvbhp': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-nvro2pp_': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gfcsw21g': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-_m_a88vc': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
srun: error: learnfair5199: task 0: Exited with exit code 1
srun: Terminating job step 28820487.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment