Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-11-17 16:26:58.510766: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:28:44.789557: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:28:44.819344: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:28:44.979960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:28:44.980100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:28:45.709517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:28:45.709675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:28:45.921245: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:28:46.505096: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:28:47.092436: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:28:47.298949: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:28:47.435410: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:28:47.440568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:28:47.441274: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:28:47.445356: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:28:47.449162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:28:47.449188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:28:47.449204: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:28:47.449217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:28:47.449229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:28:47.449240: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:28:47.449251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:28:47.449261: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:28:47.449288: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:28:47.453915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:28:47.453946: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:28:50.033118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:28:50.033250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:28:50.033265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:28:50.037085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75641 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:01:00.0, compute capability: 8.0)
2025-11-17 16:29:13.257043: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 16:29:13.257897: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799920000 Hz
2025-11-17 16:29:14.433270: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:29:16.111087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:29:16.119247: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:29:30.074474: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:29:30.230684: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 16:30:06.413410: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-11-17 16:35:53.681744: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-11-17 16:36:03.919695: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:36:17.201258: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:36:17.202724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:36:17.354903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:36:17.355046: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:36:17.362553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:36:17.362627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:36:17.366194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:36:17.368705: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:36:17.374193: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:36:17.376951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:36:17.379175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:36:17.385243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:36:17.385763: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:36:17.389661: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:36:17.393466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:36:17.393517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:36:17.393568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:36:17.393594: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:36:17.393617: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:36:17.393638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:36:17.393659: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:36:17.393680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:36:17.393701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:36:17.398349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:36:17.398387: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:36:17.849774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:36:17.849994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:36:17.850010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:36:17.855586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75641 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:01:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-11-17 16:36:19.554647: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 16:36:19.555300: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799920000 Hz
2025-11-17 16:36:19.766993: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:36:20.166642: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:36:20.169713: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:36:21.376515: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:36:21.498309: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:39<33:24, 39.31s/it]
batch:   4%|▍         | 2/52 [00:39<13:37, 16.35s/it]
batch:   6%|▌         | 3/52 [00:39<07:21,  9.01s/it]
batch:   8%|▊         | 4/52 [00:40<04:26,  5.56s/it]
batch:  10%|▉         | 5/52 [00:40<02:51,  3.65s/it]
batch:  12%|█▏        | 6/52 [00:40<01:54,  2.50s/it]
batch:  13%|█▎        | 7/52 [00:40<01:19,  1.77s/it]
batch:  15%|█▌        | 8/52 [00:41<00:56,  1.29s/it]
batch:  17%|█▋        | 9/52 [00:41<00:41,  1.03it/s]
batch:  19%|█▉        | 10/52 [00:41<00:31,  1.32it/s]
batch:  21%|██        | 11/52 [00:42<00:24,  1.65it/s]
batch:  23%|██▎       | 12/52 [00:42<00:20,  1.98it/s]
batch:  25%|██▌       | 13/52 [00:42<00:16,  2.32it/s]
batch:  27%|██▋       | 14/52 [00:42<00:14,  2.63it/s]
batch:  29%|██▉       | 15/52 [00:43<00:12,  2.88it/s]
batch:  31%|███       | 16/52 [00:43<00:11,  3.10it/s]
batch:  33%|███▎      | 17/52 [00:43<00:10,  3.28it/s]
batch:  35%|███▍      | 18/52 [00:43<00:10,  3.38it/s]
batch:  37%|███▋      | 19/52 [00:44<00:09,  3.47it/s]
batch:  38%|███▊      | 20/52 [00:44<00:09,  3.53it/s]
batch:  40%|████      | 21/52 [00:44<00:08,  3.57it/s]
batch:  42%|████▏     | 22/52 [00:44<00:08,  3.66it/s]
batch:  44%|████▍     | 23/52 [00:45<00:07,  3.66it/s]
batch:  46%|████▌     | 24/52 [00:45<00:07,  3.66it/s]
batch:  48%|████▊     | 25/52 [00:45<00:07,  3.56it/s]
batch:  50%|█████     | 26/52 [00:46<00:07,  3.59it/s]
batch:  52%|█████▏    | 27/52 [00:46<00:06,  3.59it/s]
batch:  54%|█████▍    | 28/52 [00:46<00:06,  3.64it/s]
batch:  56%|█████▌    | 29/52 [00:46<00:06,  3.64it/s]
batch:  58%|█████▊    | 30/52 [00:47<00:05,  3.67it/s]
batch:  60%|█████▉    | 31/52 [00:47<00:05,  3.64it/s]
batch:  62%|██████▏   | 32/52 [00:47<00:05,  3.67it/s]
batch:  63%|██████▎   | 33/52 [00:47<00:05,  3.70it/s]
batch:  65%|██████▌   | 34/52 [00:48<00:04,  3.74it/s]
batch:  67%|██████▋   | 35/52 [00:48<00:04,  3.72it/s]
batch:  69%|██████▉   | 36/52 [00:48<00:04,  3.75it/s]
batch:  71%|███████   | 37/52 [00:49<00:04,  3.71it/s]
batch:  73%|███████▎  | 38/52 [00:49<00:03,  3.75it/s]
batch:  75%|███████▌  | 39/52 [00:49<00:03,  3.73it/s]
batch:  77%|███████▋  | 40/52 [00:49<00:03,  3.72it/s]
batch:  79%|███████▉  | 41/52 [00:50<00:02,  3.71it/s]
batch:  81%|████████  | 42/52 [00:50<00:02,  3.75it/s]
batch:  83%|████████▎ | 43/52 [00:50<00:02,  3.75it/s]
batch:  85%|████████▍ | 44/52 [00:50<00:02,  3.73it/s]
batch:  87%|████████▋ | 45/52 [00:51<00:01,  3.78it/s]
batch:  88%|████████▊ | 46/52 [00:51<00:01,  3.77it/s]
batch:  90%|█████████ | 47/52 [00:51<00:01,  3.79it/s]
batch:  92%|█████████▏| 48/52 [00:51<00:01,  3.78it/s]
batch:  94%|█████████▍| 49/52 [00:52<00:00,  3.80it/s]
batch:  96%|█████████▌| 50/52 [00:52<00:00,  3.84it/s]
batch:  98%|█████████▊| 51/52 [00:52<00:00,  4.28it/s]
batch: 100%|██████████| 52/52 [00:52<00:00,  1.01s/it]

  0%|          | 0/3229 [00:00<?, ?it/s]
  4%|▎         | 120/3229 [00:00<00:02, 1197.75it/s]
  7%|▋         | 240/3229 [00:00<00:02, 1187.31it/s]
 11%|█         | 359/3229 [00:00<00:02, 1183.71it/s]
 15%|█▍        | 478/3229 [00:00<00:02, 1181.71it/s]
 18%|█▊        | 597/3229 [00:00<00:02, 1172.39it/s]
 22%|██▏       | 715/3229 [00:00<00:02, 1173.60it/s]
 26%|██▌       | 833/3229 [00:00<00:02, 1170.67it/s]
 29%|██▉       | 951/3229 [00:00<00:01, 1159.35it/s]
 33%|███▎      | 1067/3229 [00:00<00:01, 1153.69it/s]
 37%|███▋      | 1183/3229 [00:01<00:01, 1151.29it/s]
 40%|████      | 1299/3229 [00:01<00:01, 1146.12it/s]
 44%|████▍     | 1414/3229 [00:01<00:01, 1138.92it/s]
 47%|████▋     | 1528/3229 [00:01<00:01, 1136.54it/s]
 51%|█████     | 1642/3229 [00:01<00:01, 1121.48it/s]
 54%|█████▍    | 1755/3229 [00:01<00:01, 1114.78it/s]
 58%|█████▊    | 1867/3229 [00:01<00:01, 1115.59it/s]
 61%|██████▏   | 1979/3229 [00:01<00:01, 1113.40it/s]
 65%|██████▍   | 2091/3229 [00:01<00:01, 1108.21it/s]
 68%|██████▊   | 2202/3229 [00:01<00:00, 1102.40it/s]
 72%|███████▏  | 2313/3229 [00:02<00:00, 1098.88it/s]
 75%|███████▌  | 2423/3229 [00:02<00:00, 1098.99it/s]
 78%|███████▊  | 2533/3229 [00:02<00:00, 1094.07it/s]
 82%|████████▏ | 2643/3229 [00:02<00:00, 1090.63it/s]
 85%|████████▌ | 2753/3229 [00:02<00:00, 1083.61it/s]
 89%|████████▊ | 2862/3229 [00:02<00:00, 1077.52it/s]
 92%|█████████▏| 2970/3229 [00:02<00:00, 1077.91it/s]
 95%|█████████▌| 3078/3229 [00:02<00:00, 1071.69it/s]
 99%|█████████▊| 3186/3229 [00:02<00:00, 1070.18it/s]
100%|██████████| 3229/3229 [00:02<00:00, 1117.57it/s]

  0%|          | 0/3229 [00:00<?, ?it/s]
  4%|▎         | 115/3229 [00:00<00:02, 1143.37it/s]
  7%|▋         | 235/3229 [00:00<00:02, 1166.47it/s]
 11%|█         | 355/3229 [00:00<00:02, 1175.29it/s]
 15%|█▍        | 474/3229 [00:00<00:02, 1178.71it/s]
 18%|█▊        | 592/3229 [00:00<00:02, 1176.10it/s]
 22%|██▏       | 710/3229 [00:00<00:02, 1176.48it/s]
 26%|██▌       | 828/3229 [00:00<00:02, 1167.05it/s]
 29%|██▉       | 945/3229 [00:00<00:01, 1161.41it/s]
 33%|███▎      | 1062/3229 [00:00<00:01, 1156.62it/s]
 36%|███▋      | 1178/3229 [00:01<00:01, 1152.80it/s]
 40%|████      | 1294/3229 [00:01<00:01, 1147.74it/s]
 44%|████▎     | 1409/3229 [00:02<00:06, 295.07it/s] 
 47%|████▋     | 1521/3229 [00:02<00:04, 376.99it/s]
 51%|█████     | 1634/3229 [00:02<00:03, 469.57it/s]
 54%|█████▍    | 1746/3229 [00:02<00:02, 567.26it/s]
 58%|█████▊    | 1859/3229 [00:02<00:02, 666.56it/s]
 61%|██████    | 1970/3229 [00:02<00:01, 755.27it/s]
 64%|██████▍   | 2082/3229 [00:02<00:01, 833.34it/s]
 68%|██████▊   | 2190/3229 [00:02<00:01, 890.55it/s]
 71%|███████   | 2300/3229 [00:02<00:00, 941.16it/s]
 75%|███████▍  | 2411/3229 [00:03<00:00, 981.62it/s]
 78%|███████▊  | 2521/3229 [00:03<00:00, 1010.81it/s]
 81%|████████▏ | 2630/3229 [00:03<00:00, 1029.74it/s]
 85%|████████▍ | 2739/3229 [00:03<00:00, 1044.94it/s]
 88%|████████▊ | 2848/3229 [00:03<00:00, 1051.17it/s]
 92%|█████████▏| 2958/3229 [00:03<00:00, 1059.34it/s]
 95%|█████████▍| 3066/3229 [00:03<00:00, 1064.21it/s]
 98%|█████████▊| 3174/3229 [00:03<00:00, 1061.74it/s]
100%|██████████| 3229/3229 [00:03<00:00, 839.14it/s] 
2025-11-17 16:37:39.311346: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:37:51.064700: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:37:51.066031: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:37:51.225474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:37:51.225654: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:37:51.231890: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:37:51.231941: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:37:51.234716: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:37:51.236935: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:37:51.241658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:37:51.244506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:37:51.246717: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:37:51.255557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:37:51.256001: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:37:51.259799: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:37:51.262863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:37:51.262898: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:37:51.262919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:37:51.262936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:37:51.262952: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:37:51.262968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:37:51.262984: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:37:51.263017: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:37:51.263035: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:37:51.267661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:37:51.267701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:37:51.737493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:37:51.737614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:37:51.737628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:37:51.742642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75641 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:01:00.0, compute capability: 8.0)
2025-11-17 16:37:51.782334: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-11-17 16:37:51.793154: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799920000 Hz
2025-11-17 16:37:54.892596: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:37:55.383443: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:37:55.392501: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:37:56.879061: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:37:57.087818: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 16:38:34.422485: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)