Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 02:50:22.192674: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:50:52.742418: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 02:50:52.743580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 02:50:52.767771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:ca:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 02:50:52.767820: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:50:52.786199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 02:50:52.786264: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 02:50:52.794846: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 02:50:52.809449: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 02:50:52.819532: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 02:50:52.827141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 02:50:52.833942: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 02:50:52.839817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 02:50:52.840229: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 02:50:52.840417: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 02:50:52.841460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:ca:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 02:50:52.841483: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:50:52.841496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 02:50:52.841506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 02:50:52.841516: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 02:50:52.841525: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 02:50:52.841533: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 02:50:52.841542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 02:50:52.841567: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 02:50:52.841844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 02:50:52.841864: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:50:53.261790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 02:50:53.261855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 02:50:53.261864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 02:50:53.262485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:ca:00.0, compute capability: 8.9)
2026-05-11 02:50:56.209425: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 02:50:56.209945: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 02:50:59.649895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 02:51:01.596322: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 02:51:01.634205: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 02:51:05.487915: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 02:51:05.593723: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 02:51:48.842690: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:04:46.470915: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:06:14.520837: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:12:07.581774: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:12:07.589885: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:12:07.641829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:ca:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:12:07.641914: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:12:07.668196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:12:07.668317: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:12:07.681470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:12:07.694231: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:12:07.709916: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:12:07.722785: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:12:07.735386: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:12:07.735850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:12:07.736177: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:12:07.736330: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:12:07.736547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:ca:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:12:07.736568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:12:07.736583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:12:07.736593: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:12:07.736604: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:12:07.736614: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:12:07.736624: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:12:07.736634: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:12:07.736644: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:12:07.736918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:12:07.736940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:12:08.165227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:12:08.165308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:12:08.165318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:12:08.165990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:ca:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/104 [00:00<?, ?it/s]2026-05-11 03:12:10.014348: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:12:10.014875: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:12:10.312887: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:12:10.856678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:12:10.858554: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:12:12.686426: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:12:12.771543: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/104 [01:34<2:42:45, 94.81s/it]
batch:   2%|▏         | 2/104 [01:35<1:06:36, 39.18s/it]
batch:   3%|▎         | 3/104 [01:35<36:01, 21.40s/it]  
batch:   4%|▍         | 4/104 [01:35<21:44, 13.05s/it]
batch:   5%|▍         | 5/104 [01:35<13:54,  8.43s/it]
batch:   6%|▌         | 6/104 [01:36<09:13,  5.64s/it]
batch:   7%|▋         | 7/104 [01:36<06:16,  3.88s/it]
batch:   8%|▊         | 8/104 [01:36<04:21,  2.72s/it]
batch:   9%|▊         | 9/104 [01:36<03:04,  1.94s/it]
batch:  10%|▉         | 10/104 [01:36<02:13,  1.42s/it]
batch:  11%|█         | 11/104 [01:37<01:38,  1.06s/it]
batch:  12%|█▏        | 12/104 [01:37<01:14,  1.24it/s]
batch:  12%|█▎        | 13/104 [01:37<00:57,  1.57it/s]
batch:  13%|█▎        | 14/104 [01:37<00:46,  1.94it/s]
batch:  14%|█▍        | 15/104 [01:38<00:38,  2.31it/s]
batch:  15%|█▌        | 16/104 [01:38<00:32,  2.67it/s]
batch:  16%|█▋        | 17/104 [01:38<00:29,  2.99it/s]
batch:  17%|█▋        | 18/104 [01:38<00:26,  3.28it/s]
batch:  18%|█▊        | 19/104 [01:39<00:24,  3.50it/s]
batch:  19%|█▉        | 20/104 [01:39<00:22,  3.68it/s]
batch:  20%|██        | 21/104 [01:39<00:21,  3.82it/s]
batch:  21%|██        | 22/104 [01:39<00:20,  3.92it/s]
batch:  22%|██▏       | 23/104 [01:40<00:20,  4.00it/s]
batch:  23%|██▎       | 24/104 [01:40<00:19,  4.05it/s]
batch:  24%|██▍       | 25/104 [01:40<00:19,  4.08it/s]
batch:  25%|██▌       | 26/104 [01:40<00:18,  4.11it/s]
batch:  26%|██▌       | 27/104 [01:41<00:18,  4.13it/s]
batch:  27%|██▋       | 28/104 [01:41<00:18,  4.15it/s]
batch:  28%|██▊       | 29/104 [01:41<00:18,  4.16it/s]
batch:  29%|██▉       | 30/104 [01:41<00:17,  4.16it/s]
batch:  30%|██▉       | 31/104 [01:41<00:17,  4.18it/s]
batch:  31%|███       | 32/104 [01:42<00:17,  4.16it/s]
batch:  32%|███▏      | 33/104 [01:42<00:17,  4.17it/s]
batch:  33%|███▎      | 34/104 [01:42<00:16,  4.17it/s]
batch:  34%|███▎      | 35/104 [01:42<00:16,  4.18it/s]
batch:  35%|███▍      | 36/104 [01:43<00:16,  4.19it/s]
batch:  36%|███▌      | 37/104 [01:43<00:16,  4.16it/s]
batch:  37%|███▋      | 38/104 [01:43<00:15,  4.18it/s]
batch:  38%|███▊      | 39/104 [01:43<00:15,  4.18it/s]
batch:  38%|███▊      | 40/104 [01:44<00:15,  4.19it/s]
batch:  39%|███▉      | 41/104 [01:44<00:15,  4.20it/s]
batch:  40%|████      | 42/104 [01:44<00:14,  4.18it/s]
batch:  41%|████▏     | 43/104 [01:44<00:14,  4.19it/s]
batch:  42%|████▏     | 44/104 [01:45<00:14,  4.19it/s]
batch:  43%|████▎     | 45/104 [01:45<00:14,  4.19it/s]
batch:  44%|████▍     | 46/104 [01:45<00:13,  4.19it/s]
batch:  45%|████▌     | 47/104 [01:45<00:13,  4.19it/s]
batch:  46%|████▌     | 48/104 [01:46<00:13,  4.19it/s]
batch:  47%|████▋     | 49/104 [01:46<00:13,  4.17it/s]
batch:  48%|████▊     | 50/104 [01:46<00:12,  4.17it/s]
batch:  49%|████▉     | 51/104 [01:46<00:12,  4.18it/s]
batch:  50%|█████     | 52/104 [01:47<00:12,  4.17it/s]
batch:  51%|█████     | 53/104 [01:47<00:12,  4.18it/s]
batch:  52%|█████▏    | 54/104 [01:47<00:11,  4.18it/s]
batch:  53%|█████▎    | 55/104 [01:47<00:11,  4.18it/s]
batch:  54%|█████▍    | 56/104 [01:47<00:11,  4.18it/s]
batch:  55%|█████▍    | 57/104 [01:48<00:11,  4.18it/s]
batch:  56%|█████▌    | 58/104 [01:48<00:11,  4.18it/s]
batch:  57%|█████▋    | 59/104 [01:48<00:10,  4.16it/s]
batch:  58%|█████▊    | 60/104 [01:48<00:10,  4.17it/s]
batch:  59%|█████▊    | 61/104 [01:49<00:10,  4.17it/s]
batch:  60%|█████▉    | 62/104 [01:49<00:10,  4.18it/s]
batch:  61%|██████    | 63/104 [01:49<00:09,  4.18it/s]
batch:  62%|██████▏   | 64/104 [01:49<00:09,  4.19it/s]
batch:  62%|██████▎   | 65/104 [01:50<00:09,  4.19it/s]
batch:  63%|██████▎   | 66/104 [01:50<00:10,  3.62it/s]
batch:  64%|██████▍   | 67/104 [01:50<00:09,  3.76it/s]
batch:  65%|██████▌   | 68/104 [01:50<00:09,  3.88it/s]
batch:  66%|██████▋   | 69/104 [01:51<00:08,  3.97it/s]
batch:  67%|██████▋   | 70/104 [01:51<00:08,  4.03it/s]
batch:  68%|██████▊   | 71/104 [01:51<00:08,  4.07it/s]
batch:  69%|██████▉   | 72/104 [01:51<00:07,  4.11it/s]
batch:  70%|███████   | 73/104 [01:52<00:07,  4.13it/s]
batch:  71%|███████   | 74/104 [01:52<00:07,  4.14it/s]
batch:  72%|███████▏  | 75/104 [01:52<00:06,  4.16it/s]
batch:  73%|███████▎  | 76/104 [01:52<00:06,  4.16it/s]
batch:  74%|███████▍  | 77/104 [01:53<00:06,  4.17it/s]
batch:  75%|███████▌  | 78/104 [01:53<00:06,  4.17it/s]
batch:  76%|███████▌  | 79/104 [01:53<00:05,  4.18it/s]
batch:  77%|███████▋  | 80/104 [01:53<00:05,  4.18it/s]
batch:  78%|███████▊  | 81/104 [01:54<00:05,  4.18it/s]
batch:  79%|███████▉  | 82/104 [01:54<00:05,  4.19it/s]
batch:  80%|███████▉  | 83/104 [01:54<00:05,  4.18it/s]
batch:  81%|████████  | 84/104 [01:54<00:04,  4.19it/s]
batch:  82%|████████▏ | 85/104 [01:55<00:04,  4.18it/s]
batch:  83%|████████▎ | 86/104 [01:55<00:04,  4.18it/s]
batch:  84%|████████▎ | 87/104 [01:55<00:04,  4.19it/s]
batch:  85%|████████▍ | 88/104 [01:55<00:03,  4.19it/s]
batch:  86%|████████▌ | 89/104 [01:55<00:03,  4.19it/s]
batch:  87%|████████▋ | 90/104 [01:56<00:03,  4.19it/s]
batch:  88%|████████▊ | 91/104 [01:56<00:03,  4.19it/s]
batch:  88%|████████▊ | 92/104 [01:56<00:02,  4.19it/s]
batch:  89%|████████▉ | 93/104 [01:56<00:02,  4.18it/s]
batch:  90%|█████████ | 94/104 [01:57<00:02,  4.19it/s]
batch:  91%|█████████▏| 95/104 [01:57<00:02,  4.19it/s]
batch:  92%|█████████▏| 96/104 [01:57<00:01,  4.20it/s]
batch:  93%|█████████▎| 97/104 [01:57<00:01,  4.19it/s]
batch:  94%|█████████▍| 98/104 [01:58<00:01,  4.18it/s]
batch:  95%|█████████▌| 99/104 [01:58<00:01,  4.19it/s]
batch:  96%|█████████▌| 100/104 [01:58<00:00,  4.19it/s]
batch:  97%|█████████▋| 101/104 [01:58<00:00,  4.20it/s]
batch:  98%|█████████▊| 102/104 [01:59<00:00,  4.19it/s]
batch:  99%|█████████▉| 103/104 [01:59<00:00,  4.74it/s]
batch: 100%|██████████| 104/104 [01:59<00:00,  1.15s/it]

  0%|          | 0/6562 [00:00<?, ?it/s]
  2%|▏         | 122/6562 [00:00<00:05, 1200.99it/s]
  4%|▎         | 245/6562 [00:00<00:05, 1208.41it/s]
  6%|▌         | 369/6562 [00:00<00:05, 1217.92it/s]
  7%|▋         | 491/6562 [00:00<00:05, 1211.61it/s]
  9%|▉         | 617/6562 [00:00<00:04, 1221.62it/s]
 11%|█▏        | 740/6562 [00:00<00:05, 1015.04it/s]
 13%|█▎        | 861/6562 [00:00<00:05, 1064.88it/s]
 15%|█▍        | 983/6562 [00:00<00:05, 1109.23it/s]
 17%|█▋        | 1102/6562 [00:00<00:04, 1132.43it/s]
 19%|█▊        | 1221/6562 [00:01<00:04, 1148.84it/s]
 20%|██        | 1340/6562 [00:01<00:04, 1156.06it/s]
 22%|██▏       | 1459/6562 [00:01<00:04, 1161.62it/s]
 24%|██▍       | 1579/6562 [00:01<00:04, 1169.85it/s]
 26%|██▌       | 1698/6562 [00:01<00:04, 1169.76it/s]
 28%|██▊       | 1817/6562 [00:01<00:04, 1170.08it/s]
 29%|██▉       | 1935/6562 [00:01<00:03, 1171.17it/s]
 31%|███▏      | 2053/6562 [00:01<00:03, 1170.66it/s]
 33%|███▎      | 2171/6562 [00:01<00:03, 1165.85it/s]
 35%|███▍      | 2289/6562 [00:01<00:03, 1164.42it/s]
 37%|███▋      | 2406/6562 [00:02<00:03, 1161.95it/s]
 38%|███▊      | 2523/6562 [00:02<00:03, 1156.15it/s]
 40%|████      | 2639/6562 [00:02<00:03, 1151.84it/s]
 42%|████▏     | 2756/6562 [00:02<00:03, 1151.47it/s]
 44%|████▍     | 2874/6562 [00:02<00:03, 1158.40it/s]
 46%|████▌     | 2991/6562 [00:02<00:03, 1157.18it/s]
 47%|████▋     | 3107/6562 [00:02<00:03, 1145.25it/s]
 49%|████▉     | 3222/6562 [00:02<00:02, 1134.94it/s]
 51%|█████     | 3336/6562 [00:02<00:02, 1133.17it/s]
 53%|█████▎    | 3450/6562 [00:03<00:02, 1130.87it/s]
 54%|█████▍    | 3565/6562 [00:03<00:02, 1132.94it/s]
 56%|█████▌    | 3679/6562 [00:03<00:02, 1126.57it/s]
 58%|█████▊    | 3792/6562 [00:03<00:02, 1126.62it/s]
 60%|█████▉    | 3905/6562 [00:03<00:02, 1126.21it/s]
 61%|██████    | 4018/6562 [00:03<00:02, 1114.71it/s]
 63%|██████▎   | 4131/6562 [00:03<00:02, 1114.41it/s]
 65%|██████▍   | 4243/6562 [00:03<00:02, 1111.13it/s]
 66%|██████▋   | 4355/6562 [00:03<00:01, 1109.97it/s]
 68%|██████▊   | 4467/6562 [00:03<00:01, 1107.57it/s]
 70%|██████▉   | 4579/6562 [00:04<00:01, 1105.61it/s]
 71%|███████▏  | 4690/6562 [00:04<00:01, 1106.60it/s]
 73%|███████▎  | 4801/6562 [00:04<00:01, 1104.40it/s]
 75%|███████▍  | 4912/6562 [00:04<00:01, 1099.53it/s]
 77%|███████▋  | 5022/6562 [00:04<00:01, 1098.66it/s]
 78%|███████▊  | 5132/6562 [00:04<00:01, 1090.84it/s]
 80%|███████▉  | 5242/6562 [00:04<00:01, 1088.46it/s]
 82%|████████▏ | 5351/6562 [00:04<00:01, 1085.68it/s]
 83%|████████▎ | 5460/6562 [00:04<00:01, 1084.25it/s]
 85%|████████▍ | 5569/6562 [00:04<00:00, 1084.48it/s]
 87%|████████▋ | 5678/6562 [00:05<00:00, 1079.17it/s]
 88%|████████▊ | 5786/6562 [00:05<00:00, 1077.52it/s]
 90%|████████▉ | 5895/6562 [00:05<00:00, 1076.87it/s]
 91%|█████████▏| 6003/6562 [00:05<00:00, 1076.05it/s]
 93%|█████████▎| 6113/6562 [00:05<00:00, 1077.43it/s]
 95%|█████████▍| 6221/6562 [00:05<00:00, 1076.77it/s]
 96%|█████████▋| 6330/6562 [00:05<00:00, 1080.00it/s]
 98%|█████████▊| 6439/6562 [00:05<00:00, 1075.09it/s]
100%|█████████▉| 6547/6562 [00:05<00:00, 1059.45it/s]
100%|██████████| 6562/6562 [00:05<00:00, 1120.51it/s]

  0%|          | 0/6562 [00:00<?, ?it/s]
  2%|▏         | 122/6562 [00:00<00:05, 1201.41it/s]
  4%|▎         | 245/6562 [00:00<00:05, 1209.15it/s]
  6%|▌         | 369/6562 [00:00<00:05, 1218.24it/s]
  7%|▋         | 491/6562 [00:00<00:05, 1212.31it/s]
  9%|▉         | 617/6562 [00:00<00:04, 1222.82it/s]
 11%|█▏        | 740/6562 [00:00<00:04, 1223.71it/s]
 13%|█▎        | 863/6562 [00:00<00:04, 1216.72it/s]
 15%|█▌        | 985/6562 [00:00<00:04, 1211.21it/s]
 17%|█▋        | 1107/6562 [00:00<00:04, 1204.61it/s]
 19%|█▊        | 1228/6562 [00:01<00:04, 1197.49it/s]
 21%|██        | 1348/6562 [00:01<00:04, 1192.45it/s]
 22%|██▏       | 1468/6562 [00:01<00:04, 1189.29it/s]
 24%|██▍       | 1587/6562 [00:01<00:04, 1186.08it/s]
 26%|██▌       | 1706/6562 [00:01<00:04, 1186.50it/s]
 28%|██▊       | 1825/6562 [00:01<00:04, 1175.85it/s]
 30%|██▉       | 1943/6562 [00:01<00:03, 1176.26it/s]
 31%|███▏      | 2061/6562 [00:01<00:03, 1175.72it/s]
 33%|███▎      | 2179/6562 [00:01<00:03, 1176.07it/s]
 35%|███▌      | 2297/6562 [00:01<00:03, 1165.64it/s]
 37%|███▋      | 2414/6562 [00:02<00:03, 1161.11it/s]
 39%|███▊      | 2531/6562 [00:02<00:03, 1155.83it/s]
 40%|████      | 2647/6562 [00:02<00:03, 1151.57it/s]
 42%|████▏     | 2764/6562 [00:02<00:03, 1152.13it/s]
 44%|████▍     | 2882/6562 [00:02<00:03, 1159.69it/s]
 46%|████▌     | 2999/6562 [00:02<00:03, 1159.34it/s]
 47%|████▋     | 3115/6562 [00:02<00:03, 1146.84it/s]
 49%|████▉     | 3230/6562 [00:02<00:02, 1135.07it/s]
 51%|█████     | 3344/6562 [00:02<00:02, 1135.28it/s]
 53%|█████▎    | 3459/6562 [00:02<00:02, 1134.65it/s]
 54%|█████▍    | 3574/6562 [00:03<00:02, 1135.86it/s]
 56%|█████▌    | 3688/6562 [00:03<00:02, 1133.69it/s]
 58%|█████▊    | 3802/6562 [00:03<00:02, 1127.35it/s]
 60%|█████▉    | 3915/6562 [00:03<00:02, 1127.85it/s]
 61%|██████▏   | 4028/6562 [00:03<00:02, 1122.62it/s]
 63%|██████▎   | 4141/6562 [00:03<00:02, 1115.09it/s]
 65%|██████▍   | 4254/6562 [00:03<00:02, 1114.58it/s]
 67%|██████▋   | 4366/6562 [00:03<00:01, 1112.46it/s]
 68%|██████▊   | 4478/6562 [00:03<00:01, 1109.90it/s]
 70%|██████▉   | 4589/6562 [00:03<00:01, 1109.73it/s]
 72%|███████▏  | 4700/6562 [00:04<00:01, 1104.72it/s]
 73%|███████▎  | 4811/6562 [00:04<00:01, 1103.60it/s]
 75%|███████▌  | 4922/6562 [00:04<00:01, 1105.07it/s]
 77%|███████▋  | 5033/6562 [00:04<00:01, 1097.09it/s]
 78%|███████▊  | 5143/6562 [00:04<00:01, 1089.88it/s]
 80%|████████  | 5253/6562 [00:04<00:01, 1087.43it/s]
 82%|████████▏ | 5362/6562 [00:04<00:01, 1084.90it/s]
 83%|████████▎ | 5471/6562 [00:04<00:01, 1084.66it/s]
 85%|████████▌ | 5580/6562 [00:04<00:00, 1085.30it/s]
 87%|████████▋ | 5689/6562 [00:04<00:00, 1079.48it/s]
 88%|████████▊ | 5798/6562 [00:05<00:00, 1079.78it/s]
 90%|█████████ | 5906/6562 [00:05<00:00, 1078.00it/s]
 92%|█████████▏| 6014/6562 [00:05<00:00, 1077.30it/s]
 93%|█████████▎| 6125/6562 [00:05<00:00, 1081.61it/s]
 95%|█████████▌| 6234/6562 [00:05<00:00, 1075.21it/s]
 97%|█████████▋| 6344/6562 [00:05<00:00, 1082.45it/s]
 98%|█████████▊| 6453/6562 [00:05<00:00, 1075.85it/s]
100%|█████████▉| 6561/6562 [00:05<00:00, 1063.93it/s]
100%|██████████| 6562/6562 [00:05<00:00, 1133.73it/s]
2026-05-11 03:17:07.255361: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:11.112682: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:20:11.120404: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:20:11.177162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:ca:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:20:11.177221: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:11.211756: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:20:11.211846: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:20:11.229256: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:20:11.245820: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:20:11.264458: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:20:11.281238: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:20:11.298158: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:20:11.298558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:20:11.298908: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:20:11.299048: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:20:11.299282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:ca:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:20:11.299311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:11.299326: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:20:11.299339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:20:11.299352: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:20:11.299365: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:20:11.299378: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:20:11.299390: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:20:11.299403: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:20:11.299695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:20:11.299721: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:11.719332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:20:11.719420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:20:11.719446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:20:11.720164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:ca:00.0, compute capability: 8.9)
2026-05-11 03:20:11.759188: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:20:11.772781: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:20:16.841764: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:20:17.327632: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:20:17.331393: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:20:19.579028: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:20:19.662708: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:21:43.068531: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)