Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:23:09.712051: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:36.892521: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:28:36.902391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:28:36.935441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:28:36.935536: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:36.953737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:28:36.953815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:28:36.962953: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:28:36.971903: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:28:36.982486: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:28:36.991406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:28:36.999758: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:28:37.000224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:28:37.000657: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:28:37.000823: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:28:37.001038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:28:37.001057: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:37.001070: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:28:37.001080: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:28:37.001089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:28:37.001098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:28:37.001107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:28:37.001116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:28:37.001140: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:28:37.001408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:28:37.001428: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:37.418481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:28:37.418574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:28:37.418585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:28:37.419252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 03:28:39.267664: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:28:39.268180: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:28:43.970367: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:28:44.458874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:28:44.463639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:28:46.033891: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:28:46.123969: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:30:12.407511: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:44:07.928470: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:45:44.575577: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:08.718210: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:08.719259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:51:08.747380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:51:08.747478: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:08.771531: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:08.771640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:08.783623: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:08.795164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:08.808608: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:08.820738: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:08.832514: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:08.832996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:08.833357: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:51:08.833508: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:08.833724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:51:08.833745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:08.833759: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:08.833770: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:08.833781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:08.833791: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:08.833801: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:08.833811: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:08.833821: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:08.834109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:08.834131: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:09.263972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:51:09.264073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:51:09.264084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:51:09.264774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/68 [00:00<?, ?it/s]2026-05-11 03:51:11.049847: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:51:11.050366: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:51:11.352944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:11.883710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:11.885926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:13.677220: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:51:13.786116: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|▏         | 1/68 [02:09<2:24:33, 129.45s/it]
batch:   3%|▎         | 2/68 [02:09<58:47, 53.45s/it]   
batch:   4%|▍         | 3/68 [02:09<31:34, 29.15s/it]
batch:   6%|▌         | 4/68 [02:10<18:55, 17.74s/it]
batch:   7%|▋         | 5/68 [02:10<12:00, 11.43s/it]
batch:   9%|▉         | 6/68 [02:10<07:52,  7.63s/it]
batch:  10%|█         | 7/68 [02:10<05:17,  5.21s/it]
batch:  12%|█▏        | 8/68 [02:11<03:37,  3.63s/it]
batch:  13%|█▎        | 9/68 [02:11<02:31,  2.57s/it]
batch:  15%|█▍        | 10/68 [02:11<01:47,  1.85s/it]
batch:  16%|█▌        | 11/68 [02:11<01:17,  1.36s/it]
batch:  18%|█▊        | 12/68 [02:12<00:56,  1.02s/it]
batch:  19%|█▉        | 13/68 [02:12<00:43,  1.28it/s]
batch:  21%|██        | 14/68 [02:12<00:33,  1.62it/s]
batch:  22%|██▏       | 15/68 [02:12<00:26,  1.98it/s]
batch:  24%|██▎       | 16/68 [02:13<00:22,  2.35it/s]
batch:  25%|██▌       | 17/68 [02:13<00:18,  2.70it/s]
batch:  26%|██▋       | 18/68 [02:13<00:16,  3.03it/s]
batch:  28%|██▊       | 19/68 [02:13<00:14,  3.30it/s]
batch:  29%|██▉       | 20/68 [02:14<00:13,  3.51it/s]
batch:  31%|███       | 21/68 [02:14<00:12,  3.69it/s]
batch:  32%|███▏      | 22/68 [02:14<00:12,  3.81it/s]
batch:  34%|███▍      | 23/68 [02:14<00:11,  3.91it/s]
batch:  35%|███▌      | 24/68 [02:14<00:11,  3.99it/s]
batch:  37%|███▋      | 25/68 [02:15<00:10,  4.03it/s]
batch:  38%|███▊      | 26/68 [02:15<00:10,  4.07it/s]
batch:  40%|███▉      | 27/68 [02:15<00:10,  4.09it/s]
batch:  41%|████      | 28/68 [02:15<00:09,  4.11it/s]
batch:  43%|████▎     | 29/68 [02:16<00:09,  4.13it/s]
batch:  44%|████▍     | 30/68 [02:16<00:09,  4.14it/s]
batch:  46%|████▌     | 31/68 [02:16<00:08,  4.14it/s]
batch:  47%|████▋     | 32/68 [02:16<00:08,  4.13it/s]
batch:  49%|████▊     | 33/68 [02:17<00:08,  4.15it/s]
batch:  50%|█████     | 34/68 [02:17<00:08,  4.13it/s]
batch:  51%|█████▏    | 35/68 [02:17<00:07,  4.14it/s]
batch:  53%|█████▎    | 36/68 [02:17<00:07,  4.15it/s]
batch:  54%|█████▍    | 37/68 [02:18<00:07,  4.12it/s]
batch:  56%|█████▌    | 38/68 [02:18<00:07,  4.14it/s]
batch:  57%|█████▋    | 39/68 [02:18<00:07,  4.14it/s]
batch:  59%|█████▉    | 40/68 [02:18<00:06,  4.14it/s]
batch:  60%|██████    | 41/68 [02:19<00:06,  4.15it/s]
batch:  62%|██████▏   | 42/68 [02:19<00:06,  4.14it/s]
batch:  63%|██████▎   | 43/68 [02:19<00:06,  4.15it/s]
batch:  65%|██████▍   | 44/68 [02:19<00:05,  4.15it/s]
batch:  66%|██████▌   | 45/68 [02:20<00:05,  4.15it/s]
batch:  68%|██████▊   | 46/68 [02:20<00:05,  4.15it/s]
batch:  69%|██████▉   | 47/68 [02:20<00:05,  4.16it/s]
batch:  71%|███████   | 48/68 [02:20<00:04,  4.17it/s]
batch:  72%|███████▏  | 49/68 [02:21<00:04,  4.15it/s]
batch:  74%|███████▎  | 50/68 [02:21<00:04,  4.15it/s]
batch:  75%|███████▌  | 51/68 [02:21<00:04,  4.16it/s]
batch:  76%|███████▋  | 52/68 [02:21<00:03,  4.17it/s]
batch:  78%|███████▊  | 53/68 [02:21<00:03,  4.18it/s]
batch:  79%|███████▉  | 54/68 [02:22<00:03,  4.17it/s]
batch:  81%|████████  | 55/68 [02:22<00:03,  4.17it/s]
batch:  82%|████████▏ | 56/68 [02:22<00:02,  4.16it/s]
batch:  84%|████████▍ | 57/68 [02:22<00:02,  4.17it/s]
batch:  85%|████████▌ | 58/68 [02:23<00:02,  4.16it/s]
batch:  87%|████████▋ | 59/68 [02:23<00:02,  4.15it/s]
batch:  88%|████████▊ | 60/68 [02:23<00:01,  4.16it/s]
batch:  90%|████████▉ | 61/68 [02:23<00:01,  4.16it/s]
batch:  91%|█████████ | 62/68 [02:24<00:01,  4.16it/s]
batch:  93%|█████████▎| 63/68 [02:24<00:01,  4.17it/s]
batch:  94%|█████████▍| 64/68 [02:24<00:00,  4.18it/s]
batch:  96%|█████████▌| 65/68 [02:24<00:00,  4.14it/s]
batch:  97%|█████████▋| 66/68 [02:25<00:00,  3.57it/s]
batch:  99%|█████████▊| 67/68 [02:25<00:00,  3.74it/s]
batch: 100%|██████████| 68/68 [02:25<00:00,  4.50it/s]
batch: 100%|██████████| 68/68 [02:25<00:00,  2.14s/it]

  0%|          | 0/4312 [00:00<?, ?it/s]
  3%|▎         | 126/4312 [00:00<00:03, 1241.95it/s]
  6%|▌         | 251/4312 [00:00<00:03, 1237.02it/s]
  9%|▊         | 375/4312 [00:00<00:03, 1234.25it/s]
 12%|█▏        | 501/4312 [00:00<00:03, 1236.44it/s]
 14%|█▍        | 625/4312 [00:00<00:02, 1237.27it/s]
 17%|█▋        | 749/4312 [00:00<00:02, 1223.49it/s]
 20%|██        | 872/4312 [00:00<00:02, 1220.25it/s]
 23%|██▎       | 995/4312 [00:00<00:02, 1212.42it/s]
 26%|██▌       | 1118/4312 [00:00<00:02, 1212.33it/s]
 29%|██▉       | 1240/4312 [00:01<00:02, 1213.92it/s]
 32%|███▏      | 1363/4312 [00:01<00:02, 1213.89it/s]
 34%|███▍      | 1485/4312 [00:01<00:02, 1214.17it/s]
 37%|███▋      | 1607/4312 [00:01<00:02, 1200.29it/s]
 40%|████      | 1728/4312 [00:01<00:02, 1190.15it/s]
 43%|████▎     | 1848/4312 [00:01<00:02, 1188.80it/s]
 46%|████▌     | 1970/4312 [00:01<00:01, 1194.76it/s]
 48%|████▊     | 2090/4312 [00:01<00:01, 1180.16it/s]
 51%|█████     | 2209/4312 [00:01<00:01, 1172.06it/s]
 54%|█████▍    | 2327/4312 [00:01<00:01, 1173.72it/s]
 57%|█████▋    | 2445/4312 [00:02<00:01, 1174.99it/s]
 59%|█████▉    | 2563/4312 [00:02<00:01, 1164.88it/s]
 62%|██████▏   | 2680/4312 [00:02<00:01, 1158.20it/s]
 65%|██████▍   | 2796/4312 [00:02<00:01, 1145.36it/s]
 68%|██████▊   | 2911/4312 [00:02<00:01, 1138.94it/s]
 70%|███████   | 3025/4312 [00:02<00:01, 1130.14it/s]
 73%|███████▎  | 3140/4312 [00:02<00:01, 1135.00it/s]
 76%|███████▌  | 3256/4312 [00:02<00:00, 1136.73it/s]
 78%|███████▊  | 3371/4312 [00:02<00:00, 1138.33it/s]
 81%|████████  | 3485/4312 [00:02<00:00, 1134.52it/s]
 83%|████████▎ | 3600/4312 [00:03<00:00, 1135.63it/s]
 86%|████████▌ | 3716/4312 [00:03<00:00, 1135.94it/s]
 89%|████████▉ | 3830/4312 [00:03<00:00, 1126.04it/s]
 91%|█████████▏| 3943/4312 [00:03<00:00, 1111.66it/s]
 94%|█████████▍| 4061/4312 [00:03<00:00, 1126.13it/s]
 97%|█████████▋| 4174/4312 [00:03<00:00, 1117.10it/s]
 99%|█████████▉| 4286/4312 [00:03<00:00, 1103.64it/s]
100%|██████████| 4312/4312 [00:03<00:00, 1166.14it/s]

  0%|          | 0/4312 [00:00<?, ?it/s]
  3%|▎         | 126/4312 [00:00<00:03, 1241.20it/s]
  6%|▌         | 251/4312 [00:00<00:03, 1236.42it/s]
  9%|▊         | 375/4312 [00:00<00:03, 1233.64it/s]
 12%|█▏        | 501/4312 [00:00<00:03, 1236.09it/s]
 14%|█▍        | 625/4312 [00:00<00:02, 1236.33it/s]
 17%|█▋        | 749/4312 [00:00<00:02, 1223.62it/s]
 20%|██        | 872/4312 [00:00<00:02, 1198.22it/s]
 23%|██▎       | 994/4312 [00:00<00:02, 1201.22it/s]
 26%|██▌       | 1118/4312 [00:00<00:02, 1208.35it/s]
 29%|██▉       | 1241/4312 [00:01<00:02, 1214.57it/s]
 32%|███▏      | 1363/4312 [00:01<00:02, 1213.66it/s]
 34%|███▍      | 1485/4312 [00:01<00:02, 1215.07it/s]
 37%|███▋      | 1607/4312 [00:01<00:02, 1202.43it/s]
 40%|████      | 1728/4312 [00:01<00:02, 1192.74it/s]
 43%|████▎     | 1848/4312 [00:01<00:02, 1191.44it/s]
 46%|████▌     | 1970/4312 [00:01<00:01, 1198.60it/s]
 48%|████▊     | 2090/4312 [00:01<00:01, 1183.92it/s]
 51%|█████     | 2209/4312 [00:01<00:01, 1176.24it/s]
 54%|█████▍    | 2328/4312 [00:01<00:01, 1180.20it/s]
 57%|█████▋    | 2448/4312 [00:02<00:01, 1179.71it/s]
 60%|█████▉    | 2566/4312 [00:02<00:01, 1169.11it/s]
 62%|██████▏   | 2683/4312 [00:02<00:01, 1163.06it/s]
 65%|██████▍   | 2800/4312 [00:02<00:01, 1147.41it/s]
 68%|██████▊   | 2915/4312 [00:02<00:01, 1141.32it/s]
 70%|███████   | 3030/4312 [00:02<00:01, 1135.86it/s]
 73%|███████▎  | 3147/4312 [00:02<00:01, 1145.25it/s]
 76%|███████▌  | 3262/4312 [00:02<00:00, 1142.68it/s]
 78%|███████▊  | 3377/4312 [00:02<00:00, 1143.71it/s]
 81%|████████  | 3492/4312 [00:02<00:00, 1141.75it/s]
 84%|████████▎ | 3607/4312 [00:03<00:00, 1141.83it/s]
 86%|████████▋ | 3722/4312 [00:03<00:00, 1139.10it/s]
 89%|████████▉ | 3836/4312 [00:03<00:00, 1133.76it/s]
 92%|█████████▏| 3950/4312 [00:03<00:00, 1115.60it/s]
 94%|█████████▍| 4066/4312 [00:03<00:00, 1125.36it/s]
 97%|█████████▋| 4179/4312 [00:03<00:00, 1118.73it/s]
100%|█████████▉| 4291/4312 [00:03<00:00, 1104.89it/s]
100%|██████████| 4312/4312 [00:03<00:00, 1168.37it/s]
2026-05-11 03:59:46.902009: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:08:11.335799: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:08:11.370513: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:08:11.418266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:08:11.418376: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:08:11.450924: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:08:11.451060: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:08:11.467128: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:08:11.482793: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:08:11.500638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:08:11.516825: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:08:11.533146: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:08:11.533655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:08:11.534160: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:08:11.534322: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:08:11.534567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:08:11.534598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:08:11.534613: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:08:11.534627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:08:11.534640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:08:11.534653: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:08:11.534665: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:08:11.534678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:08:11.534691: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:08:11.534980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:08:11.535007: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:08:11.953895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:08:11.953987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:08:11.953997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:08:11.954718: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 04:08:11.994581: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:08:12.009217: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 04:08:15.733888: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:08:16.227439: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:08:16.231878: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:08:17.972445: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:08:18.063976: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:10:15.441736: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)