Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 22:14:42.347901: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:05.902251: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:15:05.906131: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:15:06.145521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:8a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:15:06.145662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:06.689138: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:15:06.689297: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:15:07.099324: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:15:07.624417: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:15:08.143483: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:15:08.336436: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:15:08.542819: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:15:08.548133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:15:08.548761: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:15:08.550826: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:15:08.553493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:8a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:15:08.553532: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:08.553555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:15:08.553572: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:15:08.553588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:15:08.553603: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:15:08.553618: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:15:08.553633: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:15:08.553679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:15:08.560304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:15:08.560347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:11.064103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:15:11.064296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:15:11.064313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:15:11.069161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:8a:00.0, compute capability: 8.0)
2025-12-29 22:15:22.119886: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:15:22.120403: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450010000 Hz
2025-12-29 22:15:23.592524: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:15:25.408691: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:15:25.413702: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:15:39.141513: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:15:39.334574: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:16:19.203407: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 22:19:12.276907: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 22:19:17.073658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:24.366654: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:19:24.368278: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:19:24.605614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:8a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:19:24.605746: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:24.610326: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:19:24.610415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:19:24.612549: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:19:24.613851: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:19:24.618000: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:19:24.619767: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:19:24.621006: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:19:24.626323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:19:24.626721: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:19:24.629196: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:19:24.632488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:8a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:19:24.632523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:24.632546: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:19:24.632564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:19:24.632581: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:19:24.632597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:19:24.632613: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:19:24.632630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:19:24.632646: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:19:24.636692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:19:24.636732: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:25.188923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:19:25.189098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:19:25.189113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:19:25.193290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:8a:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-12-29 22:19:27.433176: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:19:27.433737: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450010000 Hz
2025-12-29 22:19:27.736178: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:19:28.248258: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:19:28.251183: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:19:30.152042: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:19:30.402672: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:56<47:52, 56.33s/it]
batch:   4%|▍         | 2/52 [00:56<19:29, 23.38s/it]
batch:   6%|▌         | 3/52 [00:56<10:29, 12.85s/it]
batch:   8%|▊         | 4/52 [00:57<06:19,  7.90s/it]
batch:  10%|▉         | 5/52 [00:57<04:02,  5.16s/it]
batch:  12%|█▏        | 6/52 [00:57<02:41,  3.51s/it]
batch:  13%|█▎        | 7/52 [00:58<01:51,  2.47s/it]
batch:  15%|█▌        | 8/52 [00:58<01:18,  1.78s/it]
batch:  17%|█▋        | 9/52 [00:58<00:57,  1.33s/it]
batch:  19%|█▉        | 10/52 [00:59<00:42,  1.01s/it]
batch:  21%|██        | 11/52 [00:59<00:32,  1.25it/s]
batch:  23%|██▎       | 12/52 [00:59<00:26,  1.52it/s]
batch:  25%|██▌       | 13/52 [01:00<00:21,  1.79it/s]
batch:  27%|██▋       | 14/52 [01:00<00:18,  2.05it/s]
batch:  29%|██▉       | 15/52 [01:00<00:16,  2.27it/s]
batch:  31%|███       | 16/52 [01:01<00:14,  2.48it/s]
batch:  33%|███▎      | 17/52 [01:01<00:13,  2.64it/s]
batch:  35%|███▍      | 18/52 [01:01<00:12,  2.77it/s]
batch:  37%|███▋      | 19/52 [01:02<00:11,  2.89it/s]
batch:  38%|███▊      | 20/52 [01:02<00:10,  2.97it/s]
batch:  40%|████      | 21/52 [01:02<00:10,  2.99it/s]
batch:  42%|████▏     | 22/52 [01:03<00:09,  3.06it/s]
batch:  44%|████▍     | 23/52 [01:03<00:09,  3.11it/s]
batch:  46%|████▌     | 24/52 [01:03<00:08,  3.15it/s]
batch:  48%|████▊     | 25/52 [01:03<00:08,  3.17it/s]
batch:  50%|█████     | 26/52 [01:04<00:08,  3.18it/s]
batch:  52%|█████▏    | 27/52 [01:04<00:07,  3.15it/s]
batch:  54%|█████▍    | 28/52 [01:04<00:07,  3.15it/s]
batch:  56%|█████▌    | 29/52 [01:05<00:07,  3.16it/s]
batch:  58%|█████▊    | 30/52 [01:05<00:06,  3.17it/s]
batch:  60%|█████▉    | 31/52 [01:05<00:06,  3.18it/s]
batch:  62%|██████▏   | 32/52 [01:06<00:06,  3.18it/s]
batch:  63%|██████▎   | 33/52 [01:06<00:05,  3.18it/s]
batch:  65%|██████▌   | 34/52 [01:06<00:05,  3.16it/s]
batch:  67%|██████▋   | 35/52 [01:07<00:05,  3.17it/s]
batch:  69%|██████▉   | 36/52 [01:07<00:05,  3.20it/s]
batch:  71%|███████   | 37/52 [01:07<00:04,  3.10it/s]
batch:  73%|███████▎  | 38/52 [01:08<00:04,  3.10it/s]
batch:  75%|███████▌  | 39/52 [01:08<00:04,  3.08it/s]
batch:  77%|███████▋  | 40/52 [01:08<00:03,  3.09it/s]
batch:  79%|███████▉  | 41/52 [01:09<00:03,  3.09it/s]
batch:  81%|████████  | 42/52 [01:09<00:03,  3.09it/s]
batch:  83%|████████▎ | 43/52 [01:09<00:02,  3.12it/s]
batch:  85%|████████▍ | 44/52 [01:10<00:02,  3.13it/s]
batch:  87%|████████▋ | 45/52 [01:10<00:02,  3.15it/s]
batch:  88%|████████▊ | 46/52 [01:10<00:01,  3.13it/s]
batch:  90%|█████████ | 47/52 [01:10<00:01,  3.11it/s]
batch:  92%|█████████▏| 48/52 [01:11<00:01,  3.12it/s]
batch:  94%|█████████▍| 49/52 [01:11<00:00,  3.11it/s]
batch:  96%|█████████▌| 50/52 [01:11<00:00,  3.12it/s]
batch:  98%|█████████▊| 51/52 [01:12<00:00,  3.12it/s]
batch: 100%|██████████| 52/52 [01:12<00:00,  1.39s/it]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 91/3269 [00:00<00:03, 907.79it/s]
  6%|▌         | 187/3269 [00:00<00:03, 938.34it/s]
  9%|▊         | 285/3269 [00:00<00:03, 956.27it/s]
 12%|█▏        | 381/3269 [00:00<00:03, 949.41it/s]
 15%|█▍        | 476/3269 [00:00<00:02, 943.54it/s]
 17%|█▋        | 571/3269 [00:00<00:02, 938.21it/s]
 20%|██        | 667/3269 [00:00<00:02, 945.10it/s]
 23%|██▎       | 762/3269 [00:00<00:02, 942.72it/s]
 26%|██▌       | 857/3269 [00:00<00:02, 940.88it/s]
 29%|██▉       | 952/3269 [00:01<00:02, 940.16it/s]
 32%|███▏      | 1047/3269 [00:01<00:02, 934.83it/s]
 35%|███▍      | 1141/3269 [00:01<00:02, 935.93it/s]
 38%|███▊      | 1235/3269 [00:01<00:02, 920.12it/s]
 41%|████      | 1328/3269 [00:01<00:02, 919.73it/s]
 43%|████▎     | 1421/3269 [00:01<00:02, 911.25it/s]
 46%|████▋     | 1514/3269 [00:01<00:01, 911.10it/s]
 49%|████▉     | 1607/3269 [00:01<00:01, 910.95it/s]
 52%|█████▏    | 1699/3269 [00:01<00:01, 902.97it/s]
 55%|█████▍    | 1790/3269 [00:01<00:01, 901.29it/s]
 58%|█████▊    | 1881/3269 [00:02<00:01, 888.20it/s]
 60%|██████    | 1972/3269 [00:02<00:01, 890.50it/s]
 63%|██████▎   | 2062/3269 [00:02<00:01, 889.99it/s]
 66%|██████▌   | 2152/3269 [00:02<00:01, 871.82it/s]
 69%|██████▊   | 2240/3269 [00:02<00:01, 873.09it/s]
 71%|███████   | 2329/3269 [00:02<00:01, 876.57it/s]
 74%|███████▍  | 2417/3269 [00:02<00:00, 866.23it/s]
 77%|███████▋  | 2505/3269 [00:02<00:00, 868.36it/s]
 79%|███████▉  | 2594/3269 [00:02<00:00, 874.16it/s]
 82%|████████▏ | 2682/3269 [00:02<00:00, 842.93it/s]
 85%|████████▍ | 2772/3269 [00:03<00:00, 854.36it/s]
 87%|████████▋ | 2860/3269 [00:03<00:00, 861.09it/s]
 90%|█████████ | 2947/3269 [00:03<00:00, 862.47it/s]
 93%|█████████▎| 3034/3269 [00:03<00:00, 863.61it/s]
 95%|█████████▌| 3121/3269 [00:03<00:00, 859.45it/s]
 98%|█████████▊| 3209/3269 [00:03<00:00, 865.46it/s]
100%|██████████| 3269/3269 [00:03<00:00, 895.85it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 92/3269 [00:00<00:03, 919.03it/s]
  6%|▌         | 190/3269 [00:00<00:03, 950.10it/s]
  9%|▊         | 286/3269 [00:00<00:03, 929.10it/s]
 12%|█▏        | 383/3269 [00:00<00:03, 940.98it/s]
 15%|█▍        | 479/3269 [00:00<00:02, 947.08it/s]
 18%|█▊        | 576/3269 [00:00<00:02, 948.66it/s]
 21%|██        | 671/3269 [00:00<00:02, 866.65it/s]
 23%|██▎       | 767/3269 [00:00<00:02, 890.97it/s]
 26%|██▋       | 863/3269 [00:00<00:02, 905.61it/s]
 29%|██▉       | 958/3269 [00:01<00:02, 915.29it/s]
 32%|███▏      | 1054/3269 [00:01<00:02, 928.43it/s]
 35%|███▌      | 1148/3269 [00:01<00:02, 931.07it/s]
 38%|███▊      | 1242/3269 [00:01<00:02, 923.88it/s]
 41%|████      | 1335/3269 [00:01<00:02, 922.92it/s]
 44%|████▎     | 1428/3269 [00:01<00:01, 922.80it/s]
 47%|████▋     | 1521/3269 [00:01<00:01, 922.19it/s]
 49%|████▉     | 1614/3269 [00:01<00:01, 894.96it/s]
 52%|█████▏    | 1705/3269 [00:01<00:01, 895.08it/s]
 55%|█████▌    | 1798/3269 [00:01<00:01, 899.79it/s]
 58%|█████▊    | 1889/3269 [00:02<00:01, 898.31it/s]
 61%|██████    | 1980/3269 [00:02<00:01, 897.03it/s]
 63%|██████▎   | 2070/3269 [00:02<00:01, 859.32it/s]
 66%|██████▌   | 2162/3269 [00:02<00:01, 872.09it/s]
 69%|██████▉   | 2252/3269 [00:02<00:01, 876.99it/s]
 72%|███████▏  | 2342/3269 [00:02<00:01, 878.21it/s]
 74%|███████▍  | 2432/3269 [00:02<00:00, 881.46it/s]
 77%|███████▋  | 2522/3269 [00:02<00:00, 881.10it/s]
 80%|███████▉  | 2611/3269 [00:02<00:00, 864.66it/s]
 83%|████████▎ | 2700/3269 [00:03<00:00, 871.92it/s]
 85%|████████▌ | 2788/3269 [00:03<00:00, 869.42it/s]
 88%|████████▊ | 2876/3269 [00:03<00:00, 868.07it/s]
 91%|█████████ | 2964/3269 [00:03<00:00, 867.88it/s]
 93%|█████████▎| 3051/3269 [00:03<00:00, 867.26it/s]
 96%|█████████▌| 3138/3269 [00:03<00:00, 860.39it/s]
 99%|█████████▊| 3226/3269 [00:03<00:00, 860.17it/s]
100%|██████████| 3269/3269 [00:03<00:00, 892.72it/s]
2025-12-29 22:21:03.710235: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:21:11.231489: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:21:11.233126: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:21:11.483057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:8a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:21:11.483156: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:21:11.487604: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:21:11.487678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:21:11.489887: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:21:11.491158: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:21:11.495450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:21:11.497236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:21:11.498457: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:21:11.504824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:21:11.505316: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:21:11.507555: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:21:11.510105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:8a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 22:21:11.510199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:21:11.510245: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:21:11.510281: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:21:11.510315: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:21:11.510358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:21:11.510393: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:21:11.510425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:21:11.510458: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:21:11.515308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:21:11.515366: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:21:12.067437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:21:12.067608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:21:12.067623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:21:12.072567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:8a:00.0, compute capability: 8.0)
2025-12-29 22:21:12.131802: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 22:21:12.147481: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450010000 Hz
2025-12-29 22:21:16.133287: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:21:16.627018: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:21:16.637900: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:21:18.703652: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:21:18.884302: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:21:50.210861: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)