Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 22:01:35.666435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:01:47.587206: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:01:47.588596: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:01:47.615031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 22:01:47.615081: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:01:47.620020: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:01:47.620088: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:01:47.622495: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:01:47.624108: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:01:47.627744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:01:47.629579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:01:47.631076: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:01:47.631456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:01:47.631858: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:01:47.633525: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:01:47.633782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 22:01:47.633803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:01:47.633817: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:01:47.633827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:01:47.633836: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:01:47.633845: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:01:47.633854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:01:47.633863: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:01:47.633882: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:01:47.634163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:01:47.634182: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:01:48.065235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:01:48.065331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:01:48.065343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:01:48.066046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 22:01:49.192055: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:01:49.192751: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 22:01:50.550790: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:01:51.061594: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:01:51.066999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:01:52.657810: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:01:52.791290: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:02:19.685738: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 22:06:59.849979: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 22:07:03.969553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:10.658100: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:07:10.658994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:07:10.686394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 22:07:10.686442: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:10.691313: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:07:10.691397: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:07:10.693852: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:07:10.695326: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:07:10.698971: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:07:10.700805: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:07:10.702258: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:07:10.702647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:07:10.702885: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:07:10.704556: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:07:10.704814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 22:07:10.704837: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:10.704851: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:07:10.704861: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:07:10.704870: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:07:10.704881: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:07:10.704890: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:07:10.704900: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:07:10.704910: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:07:10.705195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:07:10.705217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:11.132906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:07:11.133009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:07:11.133019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:07:11.133719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-12-29 22:07:12.727141: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:07:12.727626: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 22:07:12.923218: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:07:13.423123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:07:13.425067: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:07:14.927571: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:07:15.019147: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:27<23:39, 27.83s/it]
batch:   4%|▍         | 2/52 [00:28<09:40, 11.60s/it]
batch:   6%|▌         | 3/52 [00:28<05:14,  6.41s/it]
batch:   8%|▊         | 4/52 [00:28<03:10,  3.97s/it]
batch:  10%|▉         | 5/52 [00:28<02:03,  2.62s/it]
batch:  12%|█▏        | 6/52 [00:29<01:23,  1.81s/it]
batch:  13%|█▎        | 7/52 [00:29<00:58,  1.30s/it]
batch:  15%|█▌        | 8/52 [00:29<00:42,  1.04it/s]
batch:  17%|█▋        | 9/52 [00:29<00:31,  1.36it/s]
batch:  19%|█▉        | 10/52 [00:29<00:24,  1.73it/s]
batch:  21%|██        | 11/52 [00:30<00:19,  2.12it/s]
batch:  23%|██▎       | 12/52 [00:30<00:15,  2.50it/s]
batch:  25%|██▌       | 13/52 [00:30<00:13,  2.88it/s]
batch:  27%|██▋       | 14/52 [00:30<00:11,  3.21it/s]
batch:  29%|██▉       | 15/52 [00:31<00:10,  3.48it/s]
batch:  31%|███       | 16/52 [00:31<00:09,  3.70it/s]
batch:  33%|███▎      | 17/52 [00:31<00:09,  3.89it/s]
batch:  35%|███▍      | 18/52 [00:31<00:08,  4.01it/s]
batch:  37%|███▋      | 19/52 [00:32<00:08,  4.12it/s]
batch:  38%|███▊      | 20/52 [00:32<00:07,  4.16it/s]
batch:  40%|████      | 21/52 [00:32<00:07,  4.20it/s]
batch:  42%|████▏     | 22/52 [00:32<00:07,  4.22it/s]
batch:  44%|████▍     | 23/52 [00:32<00:06,  4.25it/s]
batch:  46%|████▌     | 24/52 [00:33<00:06,  4.27it/s]
batch:  48%|████▊     | 25/52 [00:33<00:06,  4.28it/s]
batch:  50%|█████     | 26/52 [00:33<00:06,  4.32it/s]
batch:  52%|█████▏    | 27/52 [00:33<00:05,  4.31it/s]
batch:  54%|█████▍    | 28/52 [00:34<00:05,  4.34it/s]
batch:  56%|█████▌    | 29/52 [00:34<00:05,  4.33it/s]
batch:  58%|█████▊    | 30/52 [00:34<00:05,  4.34it/s]
batch:  60%|█████▉    | 31/52 [00:34<00:04,  4.37it/s]
batch:  62%|██████▏   | 32/52 [00:35<00:04,  4.34it/s]
batch:  63%|██████▎   | 33/52 [00:35<00:04,  4.35it/s]
batch:  65%|██████▌   | 34/52 [00:35<00:04,  4.33it/s]
batch:  67%|██████▋   | 35/52 [00:35<00:03,  4.35it/s]
batch:  69%|██████▉   | 36/52 [00:35<00:03,  4.33it/s]
batch:  71%|███████   | 37/52 [00:36<00:03,  4.30it/s]
batch:  73%|███████▎  | 38/52 [00:36<00:03,  4.32it/s]
batch:  75%|███████▌  | 39/52 [00:36<00:03,  4.31it/s]
batch:  77%|███████▋  | 40/52 [00:36<00:02,  4.32it/s]
batch:  79%|███████▉  | 41/52 [00:37<00:02,  4.33it/s]
batch:  81%|████████  | 42/52 [00:37<00:02,  4.33it/s]
batch:  83%|████████▎ | 43/52 [00:37<00:02,  4.35it/s]
batch:  85%|████████▍ | 44/52 [00:37<00:01,  4.35it/s]
batch:  87%|████████▋ | 45/52 [00:38<00:01,  4.32it/s]
batch:  88%|████████▊ | 46/52 [00:38<00:01,  4.33it/s]
batch:  90%|█████████ | 47/52 [00:38<00:01,  4.33it/s]
batch:  92%|█████████▏| 48/52 [00:38<00:00,  4.35it/s]
batch:  94%|█████████▍| 49/52 [00:38<00:00,  4.37it/s]
batch:  96%|█████████▌| 50/52 [00:39<00:00,  4.37it/s]
batch:  98%|█████████▊| 51/52 [00:39<00:00,  4.38it/s]
batch: 100%|██████████| 52/52 [00:39<00:00,  1.32it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  4%|▎         | 121/3269 [00:00<00:02, 1199.49it/s]
  7%|▋         | 243/3269 [00:00<00:02, 1211.47it/s]
 11%|█         | 365/3269 [00:00<00:02, 1204.46it/s]
 15%|█▍        | 486/3269 [00:00<00:02, 1200.81it/s]
 19%|█▊        | 607/3269 [00:00<00:02, 1191.31it/s]
 22%|██▏       | 727/3269 [00:00<00:02, 1186.76it/s]
 26%|██▌       | 846/3269 [00:00<00:02, 1181.93it/s]
 30%|██▉       | 965/3269 [00:00<00:01, 1172.55it/s]
 33%|███▎      | 1083/3269 [00:00<00:01, 1168.40it/s]
 37%|███▋      | 1200/3269 [00:01<00:01, 1162.00it/s]
 40%|████      | 1317/3269 [00:01<00:01, 1151.63it/s]
 44%|████▍     | 1433/3269 [00:01<00:01, 1147.58it/s]
 47%|████▋     | 1548/3269 [00:01<00:01, 1136.56it/s]
 51%|█████     | 1662/3269 [00:01<00:01, 1126.70it/s]
 54%|█████▍    | 1775/3269 [00:01<00:01, 1123.43it/s]
 58%|█████▊    | 1888/3269 [00:01<00:01, 1116.67it/s]
 61%|██████    | 2000/3269 [00:01<00:01, 1107.76it/s]
 65%|██████▍   | 2111/3269 [00:01<00:01, 1105.39it/s]
 68%|██████▊   | 2222/3269 [00:01<00:00, 1095.77it/s]
 71%|███████▏  | 2332/3269 [00:02<00:00, 1093.36it/s]
 75%|███████▍  | 2442/3269 [00:02<00:00, 1090.94it/s]
 78%|███████▊  | 2552/3269 [00:02<00:00, 1081.64it/s]
 81%|████████▏ | 2661/3269 [00:02<00:00, 1076.92it/s]
 85%|████████▍ | 2769/3269 [00:02<00:00, 1071.16it/s]
 88%|████████▊ | 2877/3269 [00:02<00:00, 1070.99it/s]
 91%|█████████▏| 2985/3269 [00:02<00:00, 1062.17it/s]
 95%|█████████▍| 3092/3269 [00:02<00:00, 1061.21it/s]
 98%|█████████▊| 3199/3269 [00:02<00:00, 1054.57it/s]
100%|██████████| 3269/3269 [00:02<00:00, 1116.69it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  4%|▎         | 121/3269 [00:00<00:02, 1195.92it/s]
  7%|▋         | 242/3269 [00:00<00:02, 1203.26it/s]
 11%|█         | 363/3269 [00:00<00:02, 1204.94it/s]
 15%|█▍        | 484/3269 [00:00<00:02, 1192.48it/s]
 18%|█▊        | 604/3269 [00:00<00:02, 1190.11it/s]
 22%|██▏       | 724/3269 [00:00<00:02, 1181.59it/s]
 26%|██▌       | 843/3269 [00:00<00:02, 1178.26it/s]
 29%|██▉       | 961/3269 [00:00<00:01, 1167.17it/s]
 33%|███▎      | 1079/3269 [00:00<00:01, 1164.76it/s]
 37%|███▋      | 1196/3269 [00:01<00:01, 1159.87it/s]
 40%|████      | 1312/3269 [00:01<00:01, 1153.47it/s]
 44%|████▎     | 1428/3269 [00:01<00:01, 1144.22it/s]
 47%|████▋     | 1543/3269 [00:01<00:01, 1140.37it/s]
 51%|█████     | 1658/3269 [00:01<00:01, 1131.35it/s]
 54%|█████▍    | 1772/3269 [00:01<00:01, 1123.55it/s]
 58%|█████▊    | 1885/3269 [00:01<00:01, 1117.49it/s]
 61%|██████    | 1997/3269 [00:01<00:01, 1108.20it/s]
 64%|██████▍   | 2108/3269 [00:01<00:01, 1106.02it/s]
 68%|██████▊   | 2219/3269 [00:01<00:00, 1096.83it/s]
 71%|███████   | 2329/3269 [00:02<00:00, 1093.98it/s]
 75%|███████▍  | 2439/3269 [00:02<00:00, 1091.92it/s]
 78%|███████▊  | 2549/3269 [00:02<00:00, 1082.42it/s]
 81%|████████▏ | 2658/3269 [00:02<00:00, 1077.40it/s]
 85%|████████▍ | 2766/3269 [00:02<00:00, 1077.64it/s]
 88%|████████▊ | 2874/3269 [00:02<00:00, 1070.37it/s]
 91%|█████████ | 2982/3269 [00:02<00:00, 1066.93it/s]
 94%|█████████▍| 3089/3269 [00:02<00:00, 1059.98it/s]
 98%|█████████▊| 3196/3269 [00:02<00:00, 1054.51it/s]
100%|██████████| 3269/3269 [00:02<00:00, 1116.40it/s]
2025-12-29 22:08:08.763593: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:16.060543: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:08:16.092014: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:08:16.120499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 22:08:16.120588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:16.125748: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:08:16.125845: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:08:16.128329: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:08:16.129879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:08:16.133565: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:08:16.135465: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:08:16.136943: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:08:16.137361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:08:16.138031: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:08:16.139721: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:08:16.140019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 22:08:16.140053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:16.140070: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:08:16.140083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:08:16.140096: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:08:16.140109: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:08:16.140121: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:08:16.140145: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:08:16.140159: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:08:16.140460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:08:16.140487: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:16.570643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:08:16.570746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:08:16.570757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:08:16.571556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 22:08:16.615145: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 22:08:16.629447: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 22:08:19.595623: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:08:20.105489: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:08:20.111619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:08:21.746974: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:08:21.881152: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:08:45.869148: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)