Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 21:11:52.864663: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:12:05.271486: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:12:05.272789: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:12:05.299407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:12:05.299464: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:12:05.304285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:12:05.304347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:12:05.306505: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:12:05.307952: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:12:05.311510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:12:05.313251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:12:05.314561: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:12:05.314935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:12:05.315428: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:12:05.317350: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:12:05.317606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:12:05.317627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:12:05.317640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:12:05.317650: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:12:05.317659: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:12:05.317668: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:12:05.317677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:12:05.317686: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:12:05.317705: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:12:05.317989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:12:05.318009: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:12:05.740010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:12:05.740113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:12:05.740125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:12:05.740857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 21:12:26.339703: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:12:26.340393: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:12:27.435892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:12:27.952119: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:12:27.956920: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:12:29.525694: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:12:29.654230: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:12:55.822085: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 21:20:25.836423: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 21:20:29.845094: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:20:36.633220: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:20:36.634111: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:20:36.660982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:20:36.661031: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:20:36.665828: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:20:36.665914: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:20:36.668169: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:20:36.669628: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:20:36.673271: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:20:36.675136: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:20:36.676479: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:20:36.676868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:20:36.677109: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:20:36.678673: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:20:36.678914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:20:36.678936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:20:36.678949: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:20:36.678959: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:20:36.678969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:20:36.678979: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:20:36.678988: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:20:36.678998: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:20:36.679008: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:20:36.679296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:20:36.679318: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:20:37.096842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:20:37.096934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:20:37.096944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:20:37.097648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/56 [00:00<?, ?it/s]2025-12-29 21:20:38.696672: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:20:38.697329: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:20:38.911656: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:20:39.427388: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:20:39.429341: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:20:41.070949: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:20:41.198605: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/56 [00:32<29:35, 32.28s/it]
batch:   4%|▎         | 2/56 [00:32<12:05, 13.44s/it]
batch:   5%|▌         | 3/56 [00:32<06:32,  7.41s/it]
batch:   7%|▋         | 4/56 [00:33<03:58,  4.58s/it]
batch:   9%|▉         | 5/56 [00:33<02:33,  3.01s/it]
batch:  11%|█         | 6/56 [00:33<01:43,  2.07s/it]
batch:  12%|█▎        | 7/56 [00:33<01:12,  1.47s/it]
batch:  14%|█▍        | 8/56 [00:33<00:51,  1.08s/it]
batch:  16%|█▌        | 9/56 [00:34<00:38,  1.23it/s]
batch:  18%|█▊        | 10/56 [00:34<00:29,  1.57it/s]
batch:  20%|█▉        | 11/56 [00:34<00:23,  1.93it/s]
batch:  21%|██▏       | 12/56 [00:34<00:19,  2.31it/s]
batch:  23%|██▎       | 13/56 [00:35<00:16,  2.67it/s]
batch:  25%|██▌       | 14/56 [00:35<00:14,  2.98it/s]
batch:  27%|██▋       | 15/56 [00:35<00:12,  3.24it/s]
batch:  29%|██▊       | 16/56 [00:35<00:11,  3.46it/s]
batch:  30%|███       | 17/56 [00:36<00:10,  3.64it/s]
batch:  32%|███▏      | 18/56 [00:36<00:10,  3.78it/s]
batch:  34%|███▍      | 19/56 [00:36<00:09,  3.85it/s]
batch:  36%|███▌      | 20/56 [00:36<00:09,  3.93it/s]
batch:  38%|███▊      | 21/56 [00:37<00:08,  4.00it/s]
batch:  39%|███▉      | 22/56 [00:37<00:08,  4.04it/s]
batch:  41%|████      | 23/56 [00:37<00:08,  4.06it/s]
batch:  43%|████▎     | 24/56 [00:37<00:07,  4.14it/s]
batch:  45%|████▍     | 25/56 [00:38<00:07,  4.17it/s]
batch:  46%|████▋     | 26/56 [00:38<00:07,  4.17it/s]
batch:  48%|████▊     | 27/56 [00:38<00:06,  4.18it/s]
batch:  50%|█████     | 28/56 [00:38<00:06,  4.19it/s]
batch:  52%|█████▏    | 29/56 [00:39<00:06,  4.19it/s]
batch:  54%|█████▎    | 30/56 [00:39<00:06,  4.20it/s]
batch:  55%|█████▌    | 31/56 [00:39<00:05,  4.19it/s]
batch:  57%|█████▋    | 32/56 [00:39<00:05,  4.18it/s]
batch:  59%|█████▉    | 33/56 [00:39<00:05,  4.20it/s]
batch:  61%|██████    | 34/56 [00:40<00:05,  4.18it/s]
batch:  62%|██████▎   | 35/56 [00:40<00:04,  4.20it/s]
batch:  64%|██████▍   | 36/56 [00:40<00:04,  4.18it/s]
batch:  66%|██████▌   | 37/56 [00:40<00:04,  4.16it/s]
batch:  68%|██████▊   | 38/56 [00:41<00:04,  4.18it/s]
batch:  70%|██████▉   | 39/56 [00:41<00:04,  4.19it/s]
batch:  71%|███████▏  | 40/56 [00:41<00:03,  4.19it/s]
batch:  73%|███████▎  | 41/56 [00:41<00:03,  4.19it/s]
batch:  75%|███████▌  | 42/56 [00:42<00:03,  4.20it/s]
batch:  77%|███████▋  | 43/56 [00:42<00:03,  4.22it/s]
batch:  79%|███████▊  | 44/56 [00:42<00:02,  4.21it/s]
batch:  80%|████████  | 45/56 [00:42<00:02,  4.21it/s]
batch:  82%|████████▏ | 46/56 [00:43<00:02,  4.21it/s]
batch:  84%|████████▍ | 47/56 [00:43<00:02,  4.23it/s]
batch:  86%|████████▌ | 48/56 [00:43<00:01,  4.21it/s]
batch:  88%|████████▊ | 49/56 [00:43<00:01,  4.21it/s]
batch:  89%|████████▉ | 50/56 [00:44<00:01,  4.23it/s]
batch:  91%|█████████ | 51/56 [00:44<00:01,  4.23it/s]
batch:  93%|█████████▎| 52/56 [00:44<00:00,  4.23it/s]
batch:  95%|█████████▍| 53/56 [00:44<00:00,  4.24it/s]
batch:  96%|█████████▋| 54/56 [00:44<00:00,  4.19it/s]
batch:  98%|█████████▊| 55/56 [00:45<00:00,  4.22it/s]
batch: 100%|██████████| 56/56 [00:45<00:00,  4.52it/s]
batch: 100%|██████████| 56/56 [00:45<00:00,  1.23it/s]

  0%|          | 0/3568 [00:00<?, ?it/s]
  3%|▎         | 122/3568 [00:00<00:02, 1197.74it/s]
  7%|▋         | 242/3568 [00:00<00:02, 1176.66it/s]
 10%|█         | 362/3568 [00:00<00:02, 1186.73it/s]
 13%|█▎        | 481/3568 [00:00<00:02, 1181.36it/s]
 17%|█▋        | 600/3568 [00:00<00:02, 1174.91it/s]
 20%|██        | 718/3568 [00:00<00:02, 1167.67it/s]
 23%|██▎       | 835/3568 [00:00<00:02, 1164.99it/s]
 27%|██▋       | 952/3568 [00:00<00:02, 1165.35it/s]
 30%|██▉       | 1069/3568 [00:00<00:02, 1149.23it/s]
 33%|███▎      | 1184/3568 [00:01<00:02, 1148.20it/s]
 36%|███▋      | 1299/3568 [00:01<00:01, 1136.58it/s]
 40%|███▉      | 1413/3568 [00:01<00:01, 1133.81it/s]
 43%|████▎     | 1527/3568 [00:01<00:01, 1124.40it/s]
 46%|████▌     | 1640/3568 [00:01<00:01, 1124.27it/s]
 49%|████▉     | 1753/3568 [00:01<00:01, 1122.03it/s]
 52%|█████▏    | 1866/3568 [00:01<00:01, 1105.70it/s]
 55%|█████▌    | 1979/3568 [00:01<00:01, 1108.96it/s]
 59%|█████▊    | 2090/3568 [00:01<00:01, 1100.39it/s]
 62%|██████▏   | 2201/3568 [00:01<00:01, 1098.25it/s]
 65%|██████▍   | 2311/3568 [00:02<00:01, 1068.62it/s]
 68%|██████▊   | 2420/3568 [00:02<00:01, 1070.37it/s]
 71%|███████   | 2528/3568 [00:02<00:00, 1072.70it/s]
 74%|███████▍  | 2636/3568 [00:02<00:00, 1072.76it/s]
 77%|███████▋  | 2744/3568 [00:02<00:00, 1070.38it/s]
 80%|███████▉  | 2852/3568 [00:02<00:00, 1068.45it/s]
 83%|████████▎ | 2959/3568 [00:02<00:00, 1057.14it/s]
 86%|████████▌ | 3065/3568 [00:02<00:00, 1057.92it/s]
 89%|████████▉ | 3171/3568 [00:02<00:00, 1052.87it/s]
 92%|█████████▏| 3277/3568 [00:02<00:00, 1047.28it/s]
 95%|█████████▍| 3382/3568 [00:03<00:00, 1042.09it/s]
 98%|█████████▊| 3487/3568 [00:03<00:00, 1040.67it/s]
100%|██████████| 3568/3568 [00:03<00:00, 1098.89it/s]

  0%|          | 0/3568 [00:00<?, ?it/s]
  3%|▎         | 121/3568 [00:00<00:02, 1208.48it/s]
  7%|▋         | 242/3568 [00:00<00:02, 1186.60it/s]
 10%|█         | 362/3568 [00:00<00:02, 1191.92it/s]
 14%|█▎        | 482/3568 [00:00<00:02, 1187.11it/s]
 17%|█▋        | 601/3568 [00:00<00:02, 1130.89it/s]
 20%|██        | 718/3568 [00:00<00:02, 1138.36it/s]
 23%|██▎       | 834/3568 [00:00<00:02, 1144.03it/s]
 27%|██▋       | 950/3568 [00:00<00:02, 1144.32it/s]
 30%|██▉       | 1065/3568 [00:00<00:02, 1142.67it/s]
 33%|███▎      | 1180/3568 [00:01<00:02, 1132.96it/s]
 36%|███▋      | 1294/3568 [00:01<00:02, 1124.12it/s]
 39%|███▉      | 1407/3568 [00:01<00:01, 1119.02it/s]
 43%|████▎     | 1519/3568 [00:01<00:01, 1115.27it/s]
 46%|████▌     | 1631/3568 [00:01<00:01, 1111.10it/s]
 49%|████▉     | 1743/3568 [00:01<00:01, 1108.49it/s]
 52%|█████▏    | 1854/3568 [00:01<00:01, 1103.28it/s]
 55%|█████▌    | 1965/3568 [00:01<00:01, 1092.35it/s]
 58%|█████▊    | 2076/3568 [00:01<00:01, 1096.77it/s]
 61%|██████▏   | 2186/3568 [00:01<00:01, 1087.61it/s]
 64%|██████▍   | 2295/3568 [00:02<00:01, 1082.19it/s]
 67%|██████▋   | 2404/3568 [00:02<00:01, 1064.08it/s]
 70%|███████   | 2513/3568 [00:02<00:00, 1070.95it/s]
 73%|███████▎  | 2621/3568 [00:02<00:00, 1068.51it/s]
 76%|███████▋  | 2728/3568 [00:02<00:00, 1034.63it/s]
 79%|███████▉  | 2834/3568 [00:02<00:00, 1041.31it/s]
 82%|████████▏ | 2940/3568 [00:02<00:00, 1044.26it/s]
 85%|████████▌ | 3047/3568 [00:02<00:00, 1046.09it/s]
 88%|████████▊ | 3154/3568 [00:02<00:00, 1047.56it/s]
 91%|█████████▏| 3259/3568 [00:02<00:00, 1017.84it/s]
 94%|█████████▍| 3365/3568 [00:03<00:00, 1025.86it/s]
 97%|█████████▋| 3469/3568 [00:03<00:00, 1027.12it/s]
100%|██████████| 3568/3568 [00:03<00:00, 1087.04it/s]
2025-12-29 21:21:43.084626: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:21:50.494824: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:21:50.495994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:21:50.523825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:21:50.523883: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:21:50.529519: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:21:50.529611: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:21:50.532203: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:21:50.533904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:21:50.537519: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:21:50.539374: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:21:50.540842: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:21:50.541242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:21:50.541589: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:21:50.543233: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:21:50.543515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:21:50.543553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:21:50.543582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:21:50.543596: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:21:50.543609: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:21:50.543622: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:21:50.543634: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:21:50.543647: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:21:50.543660: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:21:50.543962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:21:50.543992: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:21:50.963730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:21:50.963832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:21:50.963842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:21:50.964588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 21:21:51.004660: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 21:21:51.018233: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:21:54.159965: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:21:54.639235: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:21:54.645269: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:21:56.298102: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:21:56.424537: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:22:19.211139: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)