Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 22:14:43.863052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:05.846149: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:15:05.849936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:15:06.170079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 22:15:06.170222: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:06.740807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:15:06.741073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:15:07.152631: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:15:07.671716: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:15:08.226370: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:15:08.428537: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:15:08.613595: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:15:08.619457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:15:08.620357: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:15:08.624537: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:15:08.627195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 22:15:08.627228: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:08.627261: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:15:08.627276: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:15:08.627291: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:15:08.627306: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:15:08.627321: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:15:08.627335: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:15:08.627369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:15:08.634772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:15:08.634810: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:15:11.153976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:15:11.154116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:15:11.154140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:15:11.158471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37373 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:4d:00.0, compute capability: 8.0)
2025-12-29 22:15:25.929870: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:15:25.931647: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000035000 Hz
2025-12-29 22:15:27.478676: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:15:29.312123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:15:29.322190: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:15:43.410296: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:15:43.531479: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:16:08.741016: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 22:19:07.583874: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 22:19:13.102522: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:22.616478: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:19:22.618362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:19:22.935039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 22:19:22.935193: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:22.940815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:19:22.940903: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:19:22.943380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:19:22.945339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:19:22.950387: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:19:22.952555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:19:22.954339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:19:22.962322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:19:22.962922: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:19:22.966227: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:19:22.968546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 22:19:22.968578: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:22.968597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:19:22.968614: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:19:22.968629: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:19:22.968644: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:19:22.968658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:19:22.968673: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:19:22.968688: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:19:22.973057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:19:22.973098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:19:23.509204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:19:23.509413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:19:23.509443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:19:23.515343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37373 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:4d:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-12-29 22:19:25.395542: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 22:19:25.396207: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000035000 Hz
2025-12-29 22:19:25.669598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:19:26.164202: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:19:26.167278: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:19:27.801593: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:19:27.919319: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:33<28:14, 33.23s/it]
batch:   4%|▍         | 2/52 [00:33<11:33, 13.87s/it]
batch:   6%|▌         | 3/52 [00:33<06:16,  7.67s/it]
batch:   8%|▊         | 4/52 [00:34<03:48,  4.77s/it]
batch:  10%|▉         | 5/52 [00:34<02:28,  3.16s/it]
batch:  12%|█▏        | 6/52 [00:34<01:40,  2.19s/it]
batch:  13%|█▎        | 7/52 [00:35<01:10,  1.58s/it]
batch:  15%|█▌        | 8/52 [00:35<00:51,  1.17s/it]
batch:  17%|█▋        | 9/52 [00:35<00:38,  1.11it/s]
batch:  19%|█▉        | 10/52 [00:36<00:30,  1.39it/s]
batch:  21%|██        | 11/52 [00:36<00:24,  1.69it/s]
batch:  23%|██▎       | 12/52 [00:36<00:20,  1.99it/s]
batch:  25%|██▌       | 13/52 [00:36<00:17,  2.27it/s]
batch:  27%|██▋       | 14/52 [00:37<00:15,  2.50it/s]
batch:  29%|██▉       | 15/52 [00:37<00:13,  2.69it/s]
batch:  31%|███       | 16/52 [00:37<00:12,  2.85it/s]
batch:  33%|███▎      | 17/52 [00:38<00:11,  2.95it/s]
batch:  35%|███▍      | 18/52 [00:38<00:11,  3.04it/s]
batch:  37%|███▋      | 19/52 [00:38<00:10,  3.11it/s]
batch:  38%|███▊      | 20/52 [00:39<00:10,  3.14it/s]
batch:  40%|████      | 21/52 [00:39<00:09,  3.18it/s]
batch:  42%|████▏     | 22/52 [00:39<00:09,  3.20it/s]
batch:  44%|████▍     | 23/52 [00:39<00:09,  3.21it/s]
batch:  46%|████▌     | 24/52 [00:40<00:08,  3.24it/s]
batch:  48%|████▊     | 25/52 [00:40<00:08,  3.26it/s]
batch:  50%|█████     | 26/52 [00:40<00:07,  3.27it/s]
batch:  52%|█████▏    | 27/52 [00:41<00:07,  3.24it/s]
batch:  54%|█████▍    | 28/52 [00:41<00:07,  3.25it/s]
batch:  56%|█████▌    | 29/52 [00:41<00:07,  3.24it/s]
batch:  58%|█████▊    | 30/52 [00:42<00:06,  3.24it/s]
batch:  60%|█████▉    | 31/52 [00:42<00:06,  3.21it/s]
batch:  62%|██████▏   | 32/52 [00:42<00:06,  3.23it/s]
batch:  63%|██████▎   | 33/52 [00:43<00:05,  3.23it/s]
batch:  65%|██████▌   | 34/52 [00:43<00:05,  3.20it/s]
batch:  67%|██████▋   | 35/52 [00:43<00:05,  3.21it/s]
batch:  69%|██████▉   | 36/52 [00:43<00:04,  3.24it/s]
batch:  71%|███████   | 37/52 [00:44<00:04,  3.23it/s]
batch:  73%|███████▎  | 38/52 [00:44<00:04,  3.25it/s]
batch:  75%|███████▌  | 39/52 [00:44<00:03,  3.27it/s]
batch:  77%|███████▋  | 40/52 [00:45<00:03,  3.28it/s]
batch:  79%|███████▉  | 41/52 [00:45<00:03,  3.27it/s]
batch:  81%|████████  | 42/52 [00:45<00:03,  3.28it/s]
batch:  83%|████████▎ | 43/52 [00:46<00:02,  3.27it/s]
batch:  85%|████████▍ | 44/52 [00:46<00:02,  3.29it/s]
batch:  87%|████████▋ | 45/52 [00:46<00:02,  3.29it/s]
batch:  88%|████████▊ | 46/52 [00:47<00:01,  3.30it/s]
batch:  90%|█████████ | 47/52 [00:47<00:01,  3.29it/s]
batch:  92%|█████████▏| 48/52 [00:47<00:01,  3.30it/s]
batch:  94%|█████████▍| 49/52 [00:47<00:00,  3.29it/s]
batch:  96%|█████████▌| 50/52 [00:48<00:00,  3.31it/s]
batch:  98%|█████████▊| 51/52 [00:48<00:00,  3.30it/s]
batch: 100%|██████████| 52/52 [00:48<00:00,  1.07it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 97/3269 [00:00<00:03, 949.81it/s]
  6%|▌         | 198/3269 [00:00<00:03, 978.87it/s]
  9%|▉         | 298/3269 [00:00<00:03, 988.33it/s]
 12%|█▏        | 397/3269 [00:00<00:02, 984.42it/s]
 15%|█▌        | 496/3269 [00:00<00:02, 983.44it/s]
 18%|█▊        | 595/3269 [00:00<00:02, 977.47it/s]
 21%|██        | 693/3269 [00:00<00:02, 976.62it/s]
 24%|██▍       | 792/3269 [00:00<00:02, 974.06it/s]
 27%|██▋       | 890/3269 [00:00<00:02, 968.17it/s]
 30%|███       | 987/3269 [00:01<00:02, 961.77it/s]
 33%|███▎      | 1084/3269 [00:01<00:02, 961.87it/s]
 36%|███▌      | 1181/3269 [00:01<00:02, 951.60it/s]
 39%|███▉      | 1277/3269 [00:01<00:02, 952.90it/s]
 42%|████▏     | 1373/3269 [00:01<00:02, 942.63it/s]
 45%|████▍     | 1469/3269 [00:01<00:01, 941.63it/s]
 48%|████▊     | 1564/3269 [00:01<00:01, 939.55it/s]
 51%|█████     | 1658/3269 [00:01<00:01, 936.36it/s]
 54%|█████▎    | 1752/3269 [00:01<00:01, 929.51it/s]
 56%|█████▋    | 1845/3269 [00:01<00:01, 929.61it/s]
 59%|█████▉    | 1938/3269 [00:02<00:01, 923.92it/s]
 62%|██████▏   | 2031/3269 [00:02<00:01, 918.70it/s]
 65%|██████▍   | 2123/3269 [00:02<00:01, 915.22it/s]
 68%|██████▊   | 2215/3269 [00:02<00:01, 914.59it/s]
 71%|███████   | 2307/3269 [00:02<00:01, 909.47it/s]
 73%|███████▎  | 2399/3269 [00:02<00:00, 906.92it/s]
 76%|███████▌  | 2490/3269 [00:02<00:00, 905.54it/s]
 79%|███████▉  | 2581/3269 [00:02<00:00, 894.52it/s]
 82%|████████▏ | 2671/3269 [00:02<00:00, 887.39it/s]
 84%|████████▍ | 2760/3269 [00:02<00:00, 884.63it/s]
 87%|████████▋ | 2849/3269 [00:03<00:00, 876.04it/s]
 90%|████████▉ | 2937/3269 [00:03<00:00, 871.73it/s]
 93%|█████████▎| 3025/3269 [00:03<00:00, 868.58it/s]
 95%|█████████▌| 3112/3269 [00:03<00:00, 865.68it/s]
 98%|█████████▊| 3199/3269 [00:03<00:00, 859.72it/s]
100%|██████████| 3269/3269 [00:03<00:00, 920.64it/s]

  0%|          | 0/3269 [00:00<?, ?it/s]
  3%|▎         | 94/3269 [00:00<00:03, 921.15it/s]
  6%|▌         | 195/3269 [00:00<00:03, 965.71it/s]
  9%|▉         | 294/3269 [00:00<00:03, 976.56it/s]
 12%|█▏        | 394/3269 [00:00<00:02, 979.40it/s]
 15%|█▌        | 493/3269 [00:00<00:02, 979.83it/s]
 18%|█▊        | 591/3269 [00:00<00:02, 978.55it/s]
 21%|██        | 689/3269 [00:00<00:02, 976.63it/s]
 24%|██▍       | 787/3269 [00:00<00:02, 972.19it/s]
 27%|██▋       | 885/3269 [00:00<00:02, 967.40it/s]
 30%|███       | 982/3269 [00:01<00:02, 966.02it/s]
 33%|███▎      | 1079/3269 [00:01<00:02, 959.37it/s]
 36%|███▌      | 1176/3269 [00:01<00:02, 957.65it/s]
 39%|███▉      | 1272/3269 [00:01<00:02, 951.53it/s]
 42%|████▏     | 1368/3269 [00:01<00:02, 941.69it/s]
 45%|████▍     | 1463/3269 [00:01<00:01, 943.86it/s]
 48%|████▊     | 1558/3269 [00:01<00:01, 934.70it/s]
 51%|█████     | 1652/3269 [00:01<00:01, 932.93it/s]
 53%|█████▎    | 1746/3269 [00:01<00:01, 925.82it/s]
 56%|█████▋    | 1839/3269 [00:01<00:01, 926.17it/s]
 59%|█████▉    | 1932/3269 [00:02<00:01, 920.34it/s]
 62%|██████▏   | 2025/3269 [00:02<00:01, 915.96it/s]
 65%|██████▍   | 2118/3269 [00:02<00:01, 914.34it/s]
 68%|██████▊   | 2210/3269 [00:02<00:01, 914.00it/s]
 70%|███████   | 2302/3269 [00:02<00:01, 908.46it/s]
 73%|███████▎  | 2393/3269 [00:02<00:00, 903.59it/s]
 76%|███████▌  | 2484/3269 [00:02<00:00, 902.76it/s]
 79%|███████▉  | 2575/3269 [00:02<00:00, 898.28it/s]
 82%|████████▏ | 2665/3269 [00:02<00:00, 890.50it/s]
 84%|████████▍ | 2755/3269 [00:02<00:00, 884.05it/s]
 87%|████████▋ | 2844/3269 [00:03<00:00, 876.28it/s]
 90%|████████▉ | 2932/3269 [00:03<00:00, 872.80it/s]
 92%|█████████▏| 3020/3269 [00:03<00:00, 869.70it/s]
 95%|█████████▌| 3107/3269 [00:03<00:00, 862.52it/s]
 98%|█████████▊| 3194/3269 [00:03<00:00, 861.98it/s]
100%|██████████| 3269/3269 [00:03<00:00, 919.16it/s]
2025-12-29 22:20:42.510814: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:20:55.322259: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:20:55.323592: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 22:20:55.652418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 22:20:55.652650: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:20:55.658301: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:20:55.658369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:20:55.661053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:20:55.662830: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:20:55.668096: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:20:55.670674: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:20:55.672434: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:20:55.679756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:20:55.680398: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 22:20:55.684514: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 22:20:55.689578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 22:20:55.689621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:20:55.689669: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:20:55.689692: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:20:55.689713: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 22:20:55.689733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 22:20:55.689753: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 22:20:55.689773: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 22:20:55.689793: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:20:55.694957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 22:20:55.695007: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 22:20:56.284095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 22:20:56.284248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 22:20:56.284262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 22:20:56.289875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37373 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:4d:00.0, compute capability: 8.0)
2025-12-29 22:20:56.343229: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 22:20:56.360676: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000035000 Hz
2025-12-29 22:21:00.104976: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 22:21:00.627983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 22:21:00.636733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 22:21:02.277326: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 22:21:02.460200: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 22:21:36.942440: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)