Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 22:05:52.148495: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:06:06.217069: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:06:06.218920: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:06:06.414203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:06:06.414311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:06:06.419617: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:06:06.419674: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:06:06.422164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:06:06.423737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:06:06.427757: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:06:06.429776: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:06:06.431315: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:06:06.436725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:06:06.437423: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:06:06.440739: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:06:06.444071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:06:06.444118: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:06:06.444149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:06:06.444174: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:06:06.444196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:06:06.444217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:06:06.444237: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:06:06.444258: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:06:06.444304: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:06:06.448483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:06:06.448522: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:06:06.934503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:06:06.934616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:06:06.934635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:06:06.938576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2025-12-29 22:06:08.191421: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:06:08.192785: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 22:06:09.942118: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:06:10.412695: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:06:10.428445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:06:12.027208: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:06:12.204302: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:06:45.272469: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 22:08:54.757812: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 22:09:01.254664: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:09:15.367987: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:09:15.369276: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:09:15.539060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:09:15.539170: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:09:15.543824: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:09:15.543899: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:09:15.546320: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:09:15.547827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:09:15.551926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:09:15.553787: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:09:15.555205: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:09:15.561421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:09:15.561985: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:09:15.565213: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:09:15.569946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:09:15.569982: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:09:15.570004: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:09:15.570020: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:09:15.570036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:09:15.570051: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:09:15.570066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:09:15.570080: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:09:15.570095: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:09:15.574817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:09:15.574860: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:09:16.052753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:09:16.052958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:09:16.052977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:09:16.058121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-12-29 22:09:18.127277: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:09:18.128021: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 22:09:18.381033: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:09:18.874308: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:09:18.876667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:09:20.481472: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:09:20.658924: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:44<38:05, 44.81s/it]
batch:   4%|▍         | 2/52 [00:45<15:31, 18.64s/it]
batch:   6%|▌         | 3/52 [00:45<08:23, 10.27s/it]
batch:   8%|▊         | 4/52 [00:45<05:04,  6.34s/it]
batch:  10%|▉         | 5/52 [00:46<03:15,  4.17s/it]
batch:  12%|█▏        | 6/52 [00:46<02:11,  2.85s/it]
batch:  13%|█▎        | 7/52 [00:46<01:30,  2.02s/it]
batch:  15%|█▌        | 8/52 [00:46<01:04,  1.48s/it]
batch:  17%|█▋        | 9/52 [00:47<00:47,  1.11s/it]
batch:  19%|█▉        | 10/52 [00:47<00:36,  1.16it/s]
batch:  21%|██        | 11/52 [00:47<00:28,  1.44it/s]
batch:  23%|██▎       | 12/52 [00:48<00:23,  1.74it/s]
batch:  25%|██▌       | 13/52 [00:48<00:19,  2.03it/s]
batch:  27%|██▋       | 14/52 [00:48<00:16,  2.30it/s]
batch:  29%|██▉       | 15/52 [00:49<00:14,  2.53it/s]
batch:  31%|███       | 16/52 [00:49<00:13,  2.70it/s]
batch:  33%|███▎      | 17/52 [00:49<00:12,  2.80it/s]
batch:  35%|███▍      | 18/52 [00:50<00:11,  2.93it/s]
batch:  37%|███▋      | 19/52 [00:50<00:10,  3.02it/s]
batch:  38%|███▊      | 20/52 [00:50<00:10,  3.09it/s]
batch:  40%|████      | 21/52 [00:50<00:09,  3.14it/s]
batch:  42%|████▏     | 22/52 [00:51<00:09,  3.19it/s]
batch:  44%|████▍     | 23/52 [00:51<00:09,  3.21it/s]
batch:  46%|████▌     | 24/52 [00:51<00:08,  3.23it/s]
batch:  48%|████▊     | 25/52 [00:52<00:08,  3.26it/s]
batch:  50%|█████     | 26/52 [00:52<00:07,  3.28it/s]
batch:  52%|█████▏    | 27/52 [00:52<00:07,  3.27it/s]
batch:  54%|█████▍    | 28/52 [00:53<00:07,  3.29it/s]
batch:  56%|█████▌    | 29/52 [00:53<00:06,  3.30it/s]
batch:  58%|█████▊    | 30/52 [00:53<00:06,  3.27it/s]
batch:  60%|█████▉    | 31/52 [00:54<00:06,  3.26it/s]
batch:  62%|██████▏   | 32/52 [00:54<00:06,  3.27it/s]
batch:  63%|██████▎   | 33/52 [00:54<00:05,  3.26it/s]
batch:  65%|██████▌   | 34/52 [00:54<00:05,  3.27it/s]
batch:  67%|██████▋   | 35/52 [00:55<00:05,  3.28it/s]
batch:  69%|██████▉   | 36/52 [00:55<00:04,  3.28it/s]
batch:  71%|███████   | 37/52 [00:55<00:04,  3.26it/s]
batch:  73%|███████▎  | 38/52 [00:56<00:04,  3.26it/s]
batch:  75%|███████▌  | 39/52 [00:56<00:03,  3.25it/s]
batch:  77%|███████▋  | 40/52 [00:56<00:03,  3.24it/s]
batch:  79%|███████▉  | 41/52 [00:57<00:03,  3.24it/s]
batch:  81%|████████  | 42/52 [00:57<00:03,  3.25it/s]
batch:  83%|████████▎ | 43/52 [00:57<00:02,  3.24it/s]
batch:  85%|████████▍ | 44/52 [00:58<00:02,  3.23it/s]
batch:  87%|████████▋ | 45/52 [00:58<00:02,  3.21it/s]
batch:  88%|████████▊ | 46/52 [00:58<00:01,  3.23it/s]
batch:  90%|█████████ | 47/52 [00:58<00:01,  3.24it/s]
batch:  92%|█████████▏| 48/52 [00:59<00:01,  3.24it/s]
batch:  94%|█████████▍| 49/52 [00:59<00:00,  3.26it/s]
batch:  96%|█████████▌| 50/52 [00:59<00:00,  3.28it/s]
batch:  98%|█████████▊| 51/52 [01:00<00:00,  3.25it/s]
batch: 100%|██████████| 52/52 [01:00<00:00,  1.16s/it]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 94/3269 [00:00<00:03, 938.70it/s]
  6%|▌         | 195/3269 [00:00<00:03, 970.29it/s]
  9%|▉         | 294/3269 [00:00<00:03, 977.16it/s]
 12%|█▏        | 394/3269 [00:00<00:02, 977.77it/s]
 15%|█▌        | 493/3269 [00:00<00:02, 977.35it/s]
 18%|█▊        | 591/3269 [00:00<00:02, 976.27it/s]
 21%|██        | 689/3269 [00:00<00:02, 973.96it/s]
 24%|██▍       | 787/3269 [00:00<00:02, 969.22it/s]
 27%|██▋       | 884/3269 [00:00<00:02, 967.76it/s]
 30%|███       | 981/3269 [00:01<00:02, 957.67it/s]
 33%|███▎      | 1077/3269 [00:01<00:02, 957.41it/s]
 36%|███▌      | 1173/3269 [00:01<00:02, 954.30it/s]
 39%|███▉      | 1269/3269 [00:01<00:02, 949.05it/s]
 42%|████▏     | 1364/3269 [00:01<00:02, 948.57it/s]
 45%|████▍     | 1459/3269 [00:01<00:01, 943.85it/s]
 48%|████▊     | 1554/3269 [00:01<00:01, 940.74it/s]
 50%|█████     | 1649/3269 [00:01<00:01, 934.14it/s]
 53%|█████▎    | 1743/3269 [00:01<00:01, 932.30it/s]
 56%|█████▌    | 1837/3269 [00:01<00:01, 924.33it/s]
 59%|█████▉    | 1930/3269 [00:02<00:01, 922.67it/s]
 62%|██████▏   | 2023/3269 [00:02<00:01, 917.45it/s]
 65%|██████▍   | 2115/3269 [00:02<00:01, 913.06it/s]
 68%|██████▊   | 2207/3269 [00:02<00:01, 909.27it/s]
 70%|███████   | 2298/3269 [00:02<00:01, 908.60it/s]
 73%|███████▎  | 2389/3269 [00:02<00:00, 904.05it/s]
 76%|███████▌  | 2480/3269 [00:02<00:00, 904.26it/s]
 79%|███████▊  | 2571/3269 [00:02<00:00, 898.64it/s]
 81%|████████▏ | 2661/3269 [00:02<00:00, 897.25it/s]
 84%|████████▍ | 2751/3269 [00:02<00:00, 896.02it/s]
 87%|████████▋ | 2841/3269 [00:03<00:00, 893.66it/s]
 90%|████████▉ | 2931/3269 [00:03<00:00, 890.79it/s]
 92%|█████████▏| 3021/3269 [00:03<00:00, 888.58it/s]
 95%|█████████▌| 3110/3269 [00:03<00:00, 885.90it/s]
 98%|█████████▊| 3199/3269 [00:03<00:00, 884.62it/s]
100%|██████████| 3269/3269 [00:03<00:00, 925.42it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 91/3269 [00:00<00:03, 901.96it/s]
  6%|▌         | 193/3269 [00:00<00:03, 958.10it/s]
  9%|▉         | 291/3269 [00:00<00:03, 966.43it/s]
 12%|█▏        | 391/3269 [00:00<00:02, 979.35it/s]
 15%|█▍        | 489/3269 [00:00<00:02, 976.53it/s]
 18%|█▊        | 587/3269 [00:00<00:02, 969.07it/s]
 21%|██        | 684/3269 [00:00<00:02, 967.28it/s]
 24%|██▍       | 782/3269 [00:00<00:02, 970.79it/s]
 27%|██▋       | 880/3269 [00:00<00:02, 965.31it/s]
 30%|██▉       | 977/3269 [00:01<00:02, 966.24it/s]
 33%|███▎      | 1074/3269 [00:01<00:02, 960.00it/s]
 36%|███▌      | 1171/3269 [00:01<00:02, 958.14it/s]
 39%|███▉      | 1267/3269 [00:01<00:02, 952.15it/s]
 42%|████▏     | 1363/3269 [00:01<00:02, 940.75it/s]
 45%|████▍     | 1458/3269 [00:01<00:01, 937.90it/s]
 48%|████▊     | 1553/3269 [00:01<00:01, 937.00it/s]
 50%|█████     | 1647/3269 [00:01<00:01, 935.20it/s]
 53%|█████▎    | 1741/3269 [00:01<00:01, 927.88it/s]
 56%|█████▌    | 1834/3269 [00:01<00:01, 925.04it/s]
 59%|█████▉    | 1927/3269 [00:02<00:01, 923.98it/s]
 62%|██████▏   | 2020/3269 [00:02<00:01, 918.29it/s]
 65%|██████▍   | 2112/3269 [00:02<00:01, 913.72it/s]
 67%|██████▋   | 2204/3269 [00:02<00:01, 914.41it/s]
 70%|███████   | 2296/3269 [00:02<00:01, 908.94it/s]
 73%|███████▎  | 2388/3269 [00:02<00:00, 906.84it/s]
 76%|███████▌  | 2479/3269 [00:02<00:00, 905.73it/s]
 79%|███████▊  | 2570/3269 [00:02<00:00, 894.26it/s]
 81%|████████▏ | 2660/3269 [00:02<00:00, 893.98it/s]
 84%|████████▍ | 2750/3269 [00:02<00:00, 893.94it/s]
 87%|████████▋ | 2840/3269 [00:03<00:00, 892.04it/s]
 90%|████████▉ | 2930/3269 [00:03<00:00, 886.49it/s]
 92%|█████████▏| 3019/3269 [00:03<00:00, 883.17it/s]
 95%|█████████▌| 3108/3269 [00:03<00:00, 882.41it/s]
 98%|█████████▊| 3197/3269 [00:03<00:00, 882.31it/s]
100%|██████████| 3269/3269 [00:03<00:00, 923.52it/s]
2025-12-29 22:10:39.421747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:10:48.342682: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:10:48.345264: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:10:48.526340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:10:48.526453: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:10:48.533173: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:10:48.533251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:10:48.536409: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:10:48.538930: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:10:48.543956: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:10:48.546662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:10:48.548859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:10:48.555825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:10:48.556373: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:10:48.558978: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:10:48.561514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:10:48.561577: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:10:48.561623: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:10:48.561652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:10:48.561677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:10:48.561700: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:10:48.561724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:10:48.561747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:10:48.561770: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:10:48.566553: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:10:48.566604: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:10:49.045813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:10:49.045891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:10:49.045907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:10:49.050811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2025-12-29 22:10:49.098454: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 22:10:49.111486: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 22:10:52.777754: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:10:53.245899: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:10:53.252493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:10:54.806671: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:10:54.976866: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:11:28.482085: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)