Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 21:49:15.998856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:49:27.587834: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:49:27.589198: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:49:27.615864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:49:27.615913: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:49:27.620744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:49:27.620811: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:49:27.623062: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:49:27.624628: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:49:27.628100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:49:27.630009: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:49:27.631803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:49:27.632185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:49:27.632567: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:49:27.634091: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:49:27.634350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:49:27.634373: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:49:27.634389: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:49:27.634398: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:49:27.634408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:49:27.634417: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:49:27.634426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:49:27.634435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:49:27.634456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:49:27.634748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:49:27.634768: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:49:28.076411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:49:28.076514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:49:28.076526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:49:28.077238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 21:49:29.721669: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:49:29.722221: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:49:30.893179: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:49:31.412076: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:49:31.417425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:49:33.028603: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:49:33.150686: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:49:58.693970: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 21:55:23.455104: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 21:55:27.618514: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:34.530990: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:55:34.531879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:55:34.559183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:55:34.559232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:34.564071: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:55:34.564155: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:55:34.566523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:55:34.568134: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:55:34.571726: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:55:34.573495: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:55:34.574882: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:55:34.575261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:55:34.575499: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:55:34.576967: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:55:34.577230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:55:34.577255: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:34.577270: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:55:34.577280: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:55:34.577290: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:55:34.577300: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:55:34.577310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:55:34.577320: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:55:34.577330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:55:34.577617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:55:34.577639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:34.996286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:55:34.996385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:55:34.996395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:55:34.997080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/54 [00:00<?, ?it/s]2025-12-29 21:55:36.582074: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:55:36.582583: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:55:36.781047: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:55:37.270377: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:55:37.272263: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:55:38.852425: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:55:38.968341: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/54 [00:28<25:27, 28.82s/it]
batch:   4%|▎         | 2/54 [00:29<10:24, 12.01s/it]
batch:   6%|▌         | 3/54 [00:29<05:38,  6.63s/it]
batch:   7%|▋         | 4/54 [00:29<03:25,  4.11s/it]
batch:   9%|▉         | 5/54 [00:29<02:12,  2.71s/it]
batch:  11%|█         | 6/54 [00:30<01:29,  1.87s/it]
batch:  13%|█▎        | 7/54 [00:30<01:02,  1.34s/it]
batch:  15%|█▍        | 8/54 [00:30<00:45,  1.02it/s]
batch:  17%|█▋        | 9/54 [00:30<00:33,  1.33it/s]
batch:  19%|█▊        | 10/54 [00:30<00:26,  1.69it/s]
batch:  20%|██        | 11/54 [00:31<00:20,  2.07it/s]
batch:  22%|██▏       | 12/54 [00:31<00:17,  2.46it/s]
batch:  24%|██▍       | 13/54 [00:31<00:14,  2.82it/s]
batch:  26%|██▌       | 14/54 [00:31<00:12,  3.14it/s]
batch:  28%|██▊       | 15/54 [00:32<00:11,  3.41it/s]
batch:  30%|██▉       | 16/54 [00:32<00:10,  3.62it/s]
batch:  31%|███▏      | 17/54 [00:32<00:09,  3.79it/s]
batch:  33%|███▎      | 18/54 [00:32<00:09,  3.92it/s]
batch:  35%|███▌      | 19/54 [00:33<00:08,  4.02it/s]
batch:  37%|███▋      | 20/54 [00:33<00:08,  4.07it/s]
batch:  39%|███▉      | 21/54 [00:33<00:07,  4.15it/s]
batch:  41%|████      | 22/54 [00:33<00:07,  4.20it/s]
batch:  43%|████▎     | 23/54 [00:33<00:07,  4.21it/s]
batch:  44%|████▍     | 24/54 [00:34<00:07,  4.25it/s]
batch:  46%|████▋     | 25/54 [00:34<00:06,  4.24it/s]
batch:  48%|████▊     | 26/54 [00:34<00:06,  4.26it/s]
batch:  50%|█████     | 27/54 [00:34<00:06,  4.26it/s]
batch:  52%|█████▏    | 28/54 [00:35<00:06,  4.26it/s]
batch:  54%|█████▎    | 29/54 [00:35<00:05,  4.28it/s]
batch:  56%|█████▌    | 30/54 [00:35<00:05,  4.27it/s]
batch:  57%|█████▋    | 31/54 [00:35<00:05,  4.30it/s]
batch:  59%|█████▉    | 32/54 [00:36<00:05,  4.25it/s]
batch:  61%|██████    | 33/54 [00:36<00:04,  4.26it/s]
batch:  63%|██████▎   | 34/54 [00:36<00:04,  4.28it/s]
batch:  65%|██████▍   | 35/54 [00:36<00:04,  4.29it/s]
batch:  67%|██████▋   | 36/54 [00:37<00:04,  4.29it/s]
batch:  69%|██████▊   | 37/54 [00:37<00:03,  4.28it/s]
batch:  70%|███████   | 38/54 [00:37<00:03,  4.28it/s]
batch:  72%|███████▏  | 39/54 [00:37<00:03,  4.26it/s]
batch:  74%|███████▍  | 40/54 [00:37<00:03,  4.25it/s]
batch:  76%|███████▌  | 41/54 [00:38<00:03,  4.26it/s]
batch:  78%|███████▊  | 42/54 [00:38<00:02,  4.23it/s]
batch:  80%|███████▉  | 43/54 [00:38<00:02,  4.23it/s]
batch:  81%|████████▏ | 44/54 [00:38<00:02,  4.21it/s]
batch:  83%|████████▎ | 45/54 [00:39<00:02,  4.22it/s]
batch:  85%|████████▌ | 46/54 [00:39<00:01,  4.23it/s]
batch:  87%|████████▋ | 47/54 [00:39<00:01,  4.24it/s]
batch:  89%|████████▉ | 48/54 [00:39<00:01,  4.24it/s]
batch:  91%|█████████ | 49/54 [00:40<00:01,  4.24it/s]
batch:  93%|█████████▎| 50/54 [00:40<00:00,  4.24it/s]
batch:  94%|█████████▍| 51/54 [00:40<00:00,  4.24it/s]
batch:  96%|█████████▋| 52/54 [00:40<00:00,  5.06it/s]
batch:  98%|█████████▊| 53/54 [00:40<00:00,  4.78it/s]
batch: 100%|██████████| 54/54 [00:40<00:00,  1.32it/s]

  0%|          | 0/3350 [00:00<?, ?it/s]
  4%|▎         | 119/3350 [00:00<00:02, 1189.84it/s]
  7%|▋         | 238/3350 [00:00<00:02, 1179.76it/s]
 11%|█         | 356/3350 [00:00<00:02, 1172.77it/s]
 14%|█▍        | 476/3350 [00:00<00:02, 1183.01it/s]
 18%|█▊        | 595/3350 [00:00<00:02, 1179.33it/s]
 21%|██▏       | 713/3350 [00:00<00:02, 1171.43it/s]
 25%|██▍       | 831/3350 [00:00<00:02, 1173.12it/s]
 28%|██▊       | 949/3350 [00:00<00:02, 1154.81it/s]
 32%|███▏      | 1065/3350 [00:00<00:01, 1148.63it/s]
 35%|███▌      | 1181/3350 [00:01<00:01, 1148.21it/s]
 39%|███▊      | 1296/3350 [00:01<00:01, 1144.95it/s]
 42%|████▏     | 1411/3350 [00:01<00:01, 1139.68it/s]
 46%|████▌     | 1525/3350 [00:01<00:01, 1134.53it/s]
 49%|████▉     | 1639/3350 [00:01<00:01, 1124.84it/s]
 52%|█████▏    | 1752/3350 [00:01<00:01, 1122.09it/s]
 56%|█████▌    | 1865/3350 [00:01<00:01, 1115.01it/s]
 59%|█████▉    | 1977/3350 [00:01<00:01, 1102.72it/s]
 62%|██████▏   | 2088/3350 [00:01<00:01, 1097.42it/s]
 66%|██████▌   | 2198/3350 [00:01<00:01, 1093.25it/s]
 69%|██████▉   | 2308/3350 [00:02<00:00, 1084.86it/s]
 72%|███████▏  | 2417/3350 [00:02<00:00, 1082.13it/s]
 75%|███████▌  | 2526/3350 [00:02<00:00, 1072.61it/s]
 79%|███████▊  | 2634/3350 [00:02<00:00, 1072.21it/s]
 82%|████████▏ | 2742/3350 [00:02<00:00, 1065.27it/s]
 85%|████████▌ | 2849/3350 [00:02<00:00, 1057.75it/s]
 88%|████████▊ | 2955/3350 [00:02<00:00, 1056.36it/s]
 91%|█████████▏| 3061/3350 [00:02<00:00, 1049.79it/s]
 95%|█████████▍| 3166/3350 [00:02<00:00, 1043.43it/s]
 98%|█████████▊| 3271/3350 [00:02<00:00, 1043.58it/s]
100%|██████████| 3350/3350 [00:03<00:00, 1104.22it/s]

  0%|          | 0/3350 [00:00<?, ?it/s]
  4%|▎         | 118/3350 [00:00<00:02, 1171.60it/s]
  7%|▋         | 236/3350 [00:00<00:02, 1167.65it/s]
 11%|█         | 356/3350 [00:00<00:02, 1172.53it/s]
 14%|█▍        | 476/3350 [00:00<00:02, 1182.31it/s]
 18%|█▊        | 595/3350 [00:00<00:02, 1179.31it/s]
 21%|██▏       | 713/3350 [00:00<00:02, 1171.77it/s]
 25%|██▍       | 831/3350 [00:00<00:02, 1173.36it/s]
 28%|██▊       | 949/3350 [00:00<00:02, 1155.16it/s]
 32%|███▏      | 1065/3350 [00:00<00:01, 1148.64it/s]
 35%|███▌      | 1181/3350 [00:01<00:01, 1147.67it/s]
 39%|███▊      | 1296/3350 [00:01<00:01, 1143.88it/s]
 42%|████▏     | 1411/3350 [00:01<00:01, 1139.04it/s]
 46%|████▌     | 1525/3350 [00:01<00:01, 1134.78it/s]
 49%|████▉     | 1639/3350 [00:01<00:01, 1125.18it/s]
 52%|█████▏    | 1752/3350 [00:01<00:01, 1121.94it/s]
 56%|█████▌    | 1865/3350 [00:01<00:01, 1119.82it/s]
 59%|█████▉    | 1977/3350 [00:01<00:01, 1106.90it/s]
 62%|██████▏   | 2088/3350 [00:01<00:01, 1101.37it/s]
 66%|██████▌   | 2199/3350 [00:01<00:01, 1092.86it/s]
 69%|██████▉   | 2309/3350 [00:02<00:00, 1089.67it/s]
 72%|███████▏  | 2418/3350 [00:02<00:00, 1085.49it/s]
 75%|███████▌  | 2527/3350 [00:02<00:00, 1076.43it/s]
 79%|███████▊  | 2635/3350 [00:02<00:00, 1071.57it/s]
 82%|████████▏ | 2743/3350 [00:02<00:00, 1070.12it/s]
 85%|████████▌ | 2851/3350 [00:02<00:00, 1063.41it/s]
 88%|████████▊ | 2958/3350 [00:02<00:00, 1046.72it/s]
 91%|█████████▏| 3063/3350 [00:02<00:00, 1045.69it/s]
 95%|█████████▍| 3168/3350 [00:02<00:00, 1045.71it/s]
 98%|█████████▊| 3273/3350 [00:02<00:00, 1042.66it/s]
100%|██████████| 3350/3350 [00:03<00:00, 1104.40it/s]
2025-12-29 21:56:34.534862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:41.349215: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:56:41.350476: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:56:41.378436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:56:41.378493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:41.383259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:56:41.383348: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:56:41.385675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:56:41.387124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:56:41.390639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:56:41.392658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:56:41.394058: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:56:41.394462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:56:41.394809: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:56:41.396336: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:56:41.396622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:56:41.396654: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:41.396670: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:56:41.396683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:56:41.396696: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:56:41.396723: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:56:41.396736: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:56:41.396748: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:56:41.396761: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:56:41.397064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:56:41.397091: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:41.816577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:56:41.816680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:56:41.816690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:56:41.817426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 21:56:41.857743: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 21:56:41.871375: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:56:44.494222: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:56:45.008835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:56:45.013008: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:56:46.582763: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:56:46.711296: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:57:10.000825: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)