Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 22:03:44.358795: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:03:58.232932: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:03:58.234693: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:03:58.423965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:03:58.424053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:03:58.429872: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:03:58.429927: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:03:58.432779: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:03:58.434632: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:03:58.439174: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:03:58.441618: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:03:58.443350: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:03:58.449512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:03:58.450118: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:03:58.453040: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:03:58.455522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:03:58.455569: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:03:58.455598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:03:58.455620: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:03:58.455641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:03:58.455661: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:03:58.455681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:03:58.455702: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:03:58.455744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:03:58.460485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:03:58.460523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:03:58.976936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:03:58.977076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:03:58.977095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:03:58.981013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0)
2025-12-29 22:04:00.437693: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:04:00.438490: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 22:04:02.067801: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:04:02.582793: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:04:02.589131: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:04:04.186093: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:04:04.364239: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:04:42.867703: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 22:07:02.798805: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 22:07:07.908381: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:17.292189: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:07:17.293150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:07:17.498345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:07:17.498443: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:17.504358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:07:17.504433: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:07:17.507559: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:07:17.509428: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:07:17.513902: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:07:17.516420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:07:17.518256: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:07:17.526004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:07:17.526396: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:07:17.529015: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:07:17.531518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:07:17.531569: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:17.531601: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:07:17.531627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:07:17.531650: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:07:17.531672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:07:17.531694: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:07:17.531716: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:07:17.531738: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:07:17.537868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:07:17.537909: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:07:18.169386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:07:18.169563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:07:18.169579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:07:18.174814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-12-29 22:07:20.268017: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:07:20.268498: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 22:07:20.506281: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:07:20.994175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:07:20.996315: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:07:22.627097: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:07:22.851706: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:48<41:14, 48.51s/it]
batch:   4%|▍         | 2/52 [00:48<16:48, 20.16s/it]
batch:   6%|▌         | 3/52 [00:49<09:03, 11.09s/it]
batch:   8%|▊         | 4/52 [00:49<05:28,  6.83s/it]
batch:  10%|▉         | 5/52 [00:49<03:30,  4.48s/it]
batch:  12%|█▏        | 6/52 [00:50<02:20,  3.06s/it]
batch:  13%|█▎        | 7/52 [00:50<01:37,  2.16s/it]
batch:  15%|█▌        | 8/52 [00:50<01:09,  1.57s/it]
batch:  17%|█▋        | 9/52 [00:50<00:50,  1.17s/it]
batch:  19%|█▉        | 10/52 [00:51<00:37,  1.11it/s]
batch:  21%|██        | 11/52 [00:51<00:29,  1.39it/s]
batch:  23%|██▎       | 12/52 [00:51<00:23,  1.69it/s]
batch:  25%|██▌       | 13/52 [00:52<00:19,  1.97it/s]
batch:  27%|██▋       | 14/52 [00:52<00:16,  2.24it/s]
batch:  29%|██▉       | 15/52 [00:52<00:14,  2.47it/s]
batch:  31%|███       | 16/52 [00:53<00:13,  2.67it/s]
batch:  33%|███▎      | 17/52 [00:53<00:12,  2.82it/s]
batch:  35%|███▍      | 18/52 [00:53<00:11,  2.95it/s]
batch:  37%|███▋      | 19/52 [00:54<00:10,  3.03it/s]
batch:  38%|███▊      | 20/52 [00:54<00:10,  3.09it/s]
batch:  40%|████      | 21/52 [00:54<00:09,  3.13it/s]
batch:  42%|████▏     | 22/52 [00:54<00:09,  3.19it/s]
batch:  44%|████▍     | 23/52 [00:55<00:09,  3.21it/s]
batch:  46%|████▌     | 24/52 [00:55<00:08,  3.24it/s]
batch:  48%|████▊     | 25/52 [00:55<00:08,  3.24it/s]
batch:  50%|█████     | 26/52 [00:56<00:07,  3.27it/s]
batch:  52%|█████▏    | 27/52 [00:56<00:07,  3.25it/s]
batch:  54%|█████▍    | 28/52 [00:56<00:07,  3.27it/s]
batch:  56%|█████▌    | 29/52 [00:57<00:07,  3.26it/s]
batch:  58%|█████▊    | 30/52 [00:57<00:06,  3.27it/s]
batch:  60%|█████▉    | 31/52 [00:57<00:06,  3.26it/s]
batch:  62%|██████▏   | 32/52 [00:57<00:06,  3.27it/s]
batch:  63%|██████▎   | 33/52 [00:58<00:05,  3.26it/s]
batch:  65%|██████▌   | 34/52 [00:58<00:05,  3.26it/s]
batch:  67%|██████▋   | 35/52 [00:58<00:05,  3.24it/s]
batch:  69%|██████▉   | 36/52 [00:59<00:04,  3.28it/s]
batch:  71%|███████   | 37/52 [00:59<00:04,  3.26it/s]
batch:  73%|███████▎  | 38/52 [00:59<00:04,  3.29it/s]
batch:  75%|███████▌  | 39/52 [01:00<00:03,  3.27it/s]
batch:  77%|███████▋  | 40/52 [01:00<00:03,  3.25it/s]
batch:  79%|███████▉  | 41/52 [01:00<00:03,  3.24it/s]
batch:  81%|████████  | 42/52 [01:01<00:03,  3.25it/s]
batch:  83%|████████▎ | 43/52 [01:01<00:02,  3.25it/s]
batch:  85%|████████▍ | 44/52 [01:01<00:02,  3.27it/s]
batch:  87%|████████▋ | 45/52 [01:01<00:02,  3.27it/s]
batch:  88%|████████▊ | 46/52 [01:02<00:01,  3.28it/s]
batch:  90%|█████████ | 47/52 [01:02<00:01,  3.28it/s]
batch:  92%|█████████▏| 48/52 [01:02<00:01,  3.29it/s]
batch:  94%|█████████▍| 49/52 [01:03<00:00,  3.28it/s]
batch:  96%|█████████▌| 50/52 [01:03<00:00,  3.29it/s]
batch:  98%|█████████▊| 51/52 [01:03<00:00,  3.28it/s]
batch: 100%|██████████| 52/52 [01:03<00:00,  1.23s/it]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 94/3269 [00:00<00:03, 935.98it/s]
  6%|▌         | 195/3269 [00:00<00:03, 967.19it/s]
  9%|▉         | 292/3269 [00:00<00:03, 967.72it/s]
 12%|█▏        | 392/3269 [00:00<00:02, 972.56it/s]
 15%|█▍        | 490/3269 [00:00<00:02, 971.79it/s]
 18%|█▊        | 588/3269 [00:00<00:02, 972.64it/s]
 21%|██        | 686/3269 [00:00<00:02, 973.35it/s]
 24%|██▍       | 784/3269 [00:00<00:02, 970.63it/s]
 27%|██▋       | 882/3269 [00:00<00:02, 964.66it/s]
 30%|██▉       | 980/3269 [00:01<00:02, 968.84it/s]
 33%|███▎      | 1077/3269 [00:01<00:02, 962.24it/s]
 36%|███▌      | 1174/3269 [00:01<00:02, 960.23it/s]
 39%|███▉      | 1271/3269 [00:01<00:02, 955.95it/s]
 42%|████▏     | 1367/3269 [00:01<00:02, 949.81it/s]
 45%|████▍     | 1462/3269 [00:01<00:01, 944.26it/s]
 48%|████▊     | 1557/3269 [00:01<00:01, 942.53it/s]
 51%|█████     | 1652/3269 [00:01<00:01, 936.53it/s]
 53%|█████▎    | 1746/3269 [00:01<00:01, 929.02it/s]
 56%|█████▋    | 1839/3269 [00:01<00:01, 928.89it/s]
 59%|█████▉    | 1932/3269 [00:02<00:01, 923.27it/s]
 62%|██████▏   | 2025/3269 [00:02<00:01, 918.14it/s]
 65%|██████▍   | 2118/3269 [00:02<00:01, 915.67it/s]
 68%|██████▊   | 2210/3269 [00:02<00:01, 915.25it/s]
 70%|███████   | 2302/3269 [00:02<00:01, 909.39it/s]
 73%|███████▎  | 2393/3269 [00:02<00:00, 899.99it/s]
 76%|███████▌  | 2484/3269 [00:02<00:00, 901.14it/s]
 79%|███████▉  | 2575/3269 [00:02<00:00, 900.52it/s]
 82%|████████▏ | 2666/3269 [00:02<00:00, 895.16it/s]
 84%|████████▍ | 2756/3269 [00:02<00:00, 893.66it/s]
 87%|████████▋ | 2846/3269 [00:03<00:00, 889.92it/s]
 90%|████████▉ | 2936/3269 [00:03<00:00, 886.90it/s]
 93%|█████████▎| 3026/3269 [00:03<00:00, 884.82it/s]
 95%|█████████▌| 3115/3269 [00:03<00:00, 882.57it/s]
 98%|█████████▊| 3204/3269 [00:03<00:00, 880.73it/s]
100%|██████████| 3269/3269 [00:03<00:00, 925.05it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 97/3269 [00:00<00:03, 969.64it/s]
  6%|▌         | 198/3269 [00:00<00:03, 982.70it/s]
  9%|▉         | 297/3269 [00:00<00:03, 984.63it/s]
 12%|█▏        | 397/3269 [00:00<00:02, 982.33it/s]
 15%|█▌        | 496/3269 [00:00<00:02, 979.94it/s]
 18%|█▊        | 594/3269 [00:00<00:02, 978.46it/s]
 21%|██        | 692/3269 [00:00<00:02, 975.44it/s]
 24%|██▍       | 790/3269 [00:00<00:02, 970.41it/s]
 27%|██▋       | 888/3269 [00:00<00:02, 966.23it/s]
 30%|███       | 985/3269 [00:01<00:02, 966.01it/s]
 33%|███▎      | 1082/3269 [00:01<00:02, 966.30it/s]
 36%|███▌      | 1179/3269 [00:01<00:02, 956.37it/s]
 39%|███▉      | 1275/3269 [00:01<00:02, 951.10it/s]
 42%|████▏     | 1371/3269 [00:01<00:02, 946.35it/s]
 45%|████▍     | 1466/3269 [00:01<00:01, 946.74it/s]
 48%|████▊     | 1561/3269 [00:01<00:01, 937.05it/s]
 51%|█████     | 1655/3269 [00:01<00:01, 935.40it/s]
 54%|█████▎    | 1749/3269 [00:01<00:01, 928.12it/s]
 56%|█████▋    | 1843/3269 [00:01<00:01, 925.36it/s]
 59%|█████▉    | 1936/3269 [00:02<00:01, 921.46it/s]
 62%|██████▏   | 2029/3269 [00:02<00:01, 917.69it/s]
 65%|██████▍   | 2123/3269 [00:02<00:01, 918.05it/s]
 68%|██████▊   | 2215/3269 [00:02<00:01, 917.11it/s]
 71%|███████   | 2307/3269 [00:02<00:01, 911.05it/s]
 73%|███████▎  | 2399/3269 [00:02<00:00, 908.32it/s]
 76%|███████▌  | 2490/3269 [00:02<00:00, 907.23it/s]
 79%|███████▉  | 2581/3269 [00:02<00:00, 894.54it/s]
 82%|████████▏ | 2671/3269 [00:02<00:00, 894.38it/s]
 84%|████████▍ | 2761/3269 [00:02<00:00, 894.34it/s]
 87%|████████▋ | 2851/3269 [00:03<00:00, 893.08it/s]
 90%|████████▉ | 2942/3269 [00:03<00:00, 893.36it/s]
 93%|█████████▎| 3032/3269 [00:03<00:00, 890.80it/s]
 96%|█████████▌| 3122/3269 [00:03<00:00, 889.50it/s]
 98%|█████████▊| 3211/3269 [00:03<00:00, 886.42it/s]
100%|██████████| 3269/3269 [00:03<00:00, 927.42it/s]
2025-12-29 22:08:44.866741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:58.404902: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:08:58.406369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:08:58.555195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:08:58.555335: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:58.564606: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:08:58.564674: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:08:58.569488: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:08:58.572805: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:08:58.579185: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:08:58.583185: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:08:58.586576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:08:58.591779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:08:58.592363: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:08:58.594838: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:08:58.597343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:08:58.597396: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:58.597443: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:08:58.597473: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:08:58.597498: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:08:58.597523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:08:58.597547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:08:58.597570: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:08:58.597594: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:08:58.602359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:08:58.602427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:08:59.112995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:08:59.113098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:08:59.113115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:08:59.118256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0)
2025-12-29 22:08:59.166454: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 22:08:59.179058: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 22:09:02.872446: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:09:03.394602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:09:03.401936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:09:05.056867: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:09:05.261104: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:09:41.750258: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)