Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-11-17 16:45:53.775224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:46:34.738209: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:46:34.740602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:46:34.903172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:46:34.903308: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:46:34.909234: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:46:34.909282: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:46:34.911967: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:46:34.914078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:46:34.918091: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:46:34.920434: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:46:34.922391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:46:34.928992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:46:34.930027: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:46:34.934018: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:46:34.936191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:46:34.936285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:46:34.936320: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:46:34.936333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:46:34.936346: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:46:34.936359: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:46:34.936370: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:46:34.936382: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:46:34.936412: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:46:34.940238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:46:34.940271: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:46:35.406831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:46:35.406955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:46:35.406972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:46:35.411901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75641 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:01:00.0, compute capability: 8.0)
2025-11-17 16:46:47.609651: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 16:46:47.610707: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799920000 Hz
2025-11-17 16:46:48.828657: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:46:49.245421: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:46:49.255091: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:46:50.784644: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:46:50.950326: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 16:47:25.900364: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-11-17 16:52:48.150394: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-11-17 16:52:53.806060: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:53:04.403921: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:53:04.405190: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:53:04.555954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:53:04.556124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:53:04.562414: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:53:04.562490: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:53:04.565865: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:53:04.567993: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:53:04.573449: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:53:04.576092: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:53:04.578114: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:53:04.583019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:53:04.583567: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:53:04.587169: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:53:04.590953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:53:04.591006: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:53:04.591037: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:53:04.591062: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:53:04.591085: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:53:04.591107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:53:04.591128: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:53:04.591150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:53:04.591171: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:53:04.595867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:53:04.595906: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:53:05.039762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:53:05.040010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:53:05.040026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:53:05.047428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75641 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:01:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/58 [00:00<?, ?it/s]2025-11-17 16:53:06.752244: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 16:53:06.752981: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799920000 Hz
2025-11-17 16:53:06.997432: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:53:07.440766: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:53:07.443744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:53:08.969034: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:53:09.149199: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/58 [00:43<41:47, 43.98s/it]
batch:   3%|▎         | 2/58 [00:44<17:03, 18.28s/it]
batch:   5%|▌         | 3/58 [00:44<09:13, 10.06s/it]
batch:   7%|▋         | 4/58 [00:44<05:34,  6.19s/it]
batch:   9%|▊         | 5/58 [00:45<03:35,  4.06s/it]
batch:  10%|█         | 6/58 [00:45<02:24,  2.77s/it]
batch:  12%|█▏        | 7/58 [00:45<01:39,  1.95s/it]
batch:  14%|█▍        | 8/58 [00:45<01:10,  1.42s/it]
batch:  16%|█▌        | 9/58 [00:46<00:51,  1.06s/it]
batch:  17%|█▋        | 10/58 [00:46<00:39,  1.22it/s]
batch:  19%|█▉        | 11/58 [00:46<00:30,  1.53it/s]
batch:  21%|██        | 12/58 [00:47<00:24,  1.85it/s]
batch:  22%|██▏       | 13/58 [00:47<00:20,  2.18it/s]
batch:  24%|██▍       | 14/58 [00:47<00:17,  2.49it/s]
batch:  26%|██▌       | 15/58 [00:47<00:15,  2.74it/s]
batch:  28%|██▊       | 16/58 [00:48<00:14,  2.96it/s]
batch:  29%|██▉       | 17/58 [00:48<00:12,  3.16it/s]
batch:  31%|███       | 18/58 [00:48<00:12,  3.30it/s]
batch:  33%|███▎      | 19/58 [00:48<00:11,  3.41it/s]
batch:  34%|███▍      | 20/58 [00:49<00:10,  3.46it/s]
batch:  36%|███▌      | 21/58 [00:49<00:10,  3.53it/s]
batch:  38%|███▊      | 22/58 [00:49<00:10,  3.58it/s]
batch:  40%|███▉      | 23/58 [00:50<00:09,  3.60it/s]
batch:  41%|████▏     | 24/58 [00:50<00:09,  3.62it/s]
batch:  43%|████▎     | 25/58 [00:50<00:09,  3.63it/s]
batch:  45%|████▍     | 26/58 [00:50<00:08,  3.65it/s]
batch:  47%|████▋     | 27/58 [00:51<00:08,  3.65it/s]
batch:  48%|████▊     | 28/58 [00:51<00:08,  3.68it/s]
batch:  50%|█████     | 29/58 [00:51<00:07,  3.69it/s]
batch:  52%|█████▏    | 30/58 [00:51<00:07,  3.70it/s]
batch:  53%|█████▎    | 31/58 [00:52<00:07,  3.70it/s]
batch:  55%|█████▌    | 32/58 [00:52<00:06,  3.72it/s]
batch:  57%|█████▋    | 33/58 [00:52<00:06,  3.71it/s]
batch:  59%|█████▊    | 34/58 [00:52<00:06,  3.69it/s]
batch:  60%|██████    | 35/58 [00:53<00:06,  3.68it/s]
batch:  62%|██████▏   | 36/58 [00:53<00:05,  3.71it/s]
batch:  64%|██████▍   | 37/58 [00:53<00:05,  3.70it/s]
batch:  66%|██████▌   | 38/58 [00:54<00:05,  3.71it/s]
batch:  67%|██████▋   | 39/58 [00:54<00:05,  3.70it/s]
batch:  69%|██████▉   | 40/58 [00:54<00:04,  3.71it/s]
batch:  71%|███████   | 41/58 [00:54<00:04,  3.70it/s]
batch:  72%|███████▏  | 42/58 [00:55<00:04,  3.67it/s]
batch:  74%|███████▍  | 43/58 [00:55<00:04,  3.66it/s]
batch:  76%|███████▌  | 44/58 [00:55<00:03,  3.67it/s]
batch:  78%|███████▊  | 45/58 [00:55<00:03,  3.70it/s]
batch:  79%|███████▉  | 46/58 [00:56<00:03,  3.71it/s]
batch:  81%|████████  | 47/58 [00:56<00:02,  3.72it/s]
batch:  83%|████████▎ | 48/58 [00:56<00:02,  3.71it/s]
batch:  84%|████████▍ | 49/58 [00:57<00:02,  3.70it/s]
batch:  86%|████████▌ | 50/58 [00:57<00:02,  3.73it/s]
batch:  88%|████████▊ | 51/58 [00:57<00:01,  3.71it/s]
batch:  90%|████████▉ | 52/58 [00:57<00:01,  3.68it/s]
batch:  91%|█████████▏| 53/58 [00:58<00:01,  3.67it/s]
batch:  93%|█████████▎| 54/58 [00:58<00:01,  3.69it/s]
batch:  95%|█████████▍| 55/58 [00:58<00:00,  3.71it/s]
batch:  97%|█████████▋| 56/58 [00:58<00:00,  3.69it/s]
batch:  98%|█████████▊| 57/58 [00:59<00:00,  3.69it/s]
batch: 100%|██████████| 58/58 [00:59<00:00,  1.02s/it]

  0%|          | 0/3656 [00:00<?, ?it/s]
  3%|▎         | 116/3656 [00:00<00:03, 1145.83it/s]
  6%|▋         | 235/3656 [00:00<00:02, 1171.11it/s]
 10%|▉         | 355/3656 [00:00<00:02, 1178.79it/s]
 13%|█▎        | 473/3656 [00:00<00:02, 1174.79it/s]
 16%|█▌        | 591/3656 [00:00<00:02, 1171.57it/s]
 19%|█▉        | 709/3656 [00:00<00:02, 1166.31it/s]
 23%|██▎       | 826/3656 [00:00<00:02, 1159.15it/s]
 26%|██▌       | 942/3656 [00:00<00:02, 1155.29it/s]
 29%|██▉       | 1058/3656 [00:00<00:02, 1153.17it/s]
 32%|███▏      | 1174/3656 [00:01<00:02, 1147.52it/s]
 35%|███▌      | 1289/3656 [00:01<00:02, 1138.61it/s]
 38%|███▊      | 1403/3656 [00:01<00:01, 1131.37it/s]
 41%|████▏     | 1517/3656 [00:01<00:01, 1127.74it/s]
 45%|████▍     | 1630/3656 [00:01<00:01, 1116.97it/s]
 48%|████▊     | 1742/3656 [00:01<00:01, 1110.76it/s]
 51%|█████     | 1854/3656 [00:01<00:01, 1111.71it/s]
 54%|█████▍    | 1966/3656 [00:01<00:01, 1108.00it/s]
 57%|█████▋    | 2077/3656 [00:01<00:01, 1100.98it/s]
 60%|█████▉    | 2188/3656 [00:01<00:01, 1101.30it/s]
 63%|██████▎   | 2299/3656 [00:02<00:01, 1096.68it/s]
 66%|██████▌   | 2409/3656 [00:02<00:01, 1087.04it/s]
 69%|██████▉   | 2518/3656 [00:02<00:01, 1081.50it/s]
 72%|███████▏  | 2627/3656 [00:02<00:00, 1076.00it/s]
 75%|███████▍  | 2735/3656 [00:02<00:00, 1070.88it/s]
 78%|███████▊  | 2843/3656 [00:02<00:00, 1063.55it/s]
 81%|████████  | 2950/3656 [00:02<00:00, 1057.80it/s]
 84%|████████▎ | 3056/3656 [00:02<00:00, 1051.62it/s]
 86%|████████▋ | 3162/3656 [00:02<00:00, 1050.64it/s]
 89%|████████▉ | 3268/3656 [00:02<00:00, 1042.52it/s]
 92%|█████████▏| 3373/3656 [00:03<00:00, 1038.85it/s]
 95%|█████████▌| 3477/3656 [00:03<00:00, 1035.12it/s]
 98%|█████████▊| 3581/3656 [00:03<00:00, 1031.39it/s]
100%|██████████| 3656/3656 [00:03<00:00, 1096.08it/s]

  0%|          | 0/3656 [00:00<?, ?it/s]
  3%|▎         | 119/3656 [00:00<00:02, 1183.85it/s]
  7%|▋         | 238/3656 [00:00<00:02, 1186.51it/s]
 10%|▉         | 357/3656 [00:00<00:02, 1184.61it/s]
 13%|█▎        | 476/3656 [00:00<00:02, 1181.97it/s]
 16%|█▋        | 595/3656 [00:00<00:02, 1178.59it/s]
 20%|█▉        | 713/3656 [00:00<00:03, 823.43it/s] 
 23%|██▎       | 830/3656 [00:00<00:03, 907.27it/s]
 26%|██▌       | 945/3656 [00:00<00:02, 971.15it/s]
 29%|██▉       | 1061/3656 [00:01<00:02, 1020.31it/s]
 32%|███▏      | 1176/3656 [00:01<00:02, 1055.97it/s]
 35%|███▌      | 1287/3656 [00:01<00:02, 1071.28it/s]
 38%|███▊      | 1399/3656 [00:01<00:02, 1084.54it/s]
 41%|████▏     | 1511/3656 [00:01<00:01, 1089.22it/s]
 44%|████▍     | 1622/3656 [00:01<00:01, 1089.77it/s]
 47%|████▋     | 1733/3656 [00:01<00:01, 1092.39it/s]
 50%|█████     | 1844/3656 [00:01<00:01, 1093.97it/s]
 53%|█████▎    | 1955/3656 [00:01<00:01, 1089.34it/s]
 56%|█████▋    | 2065/3656 [00:01<00:01, 1091.22it/s]
 60%|█████▉    | 2177/3656 [00:02<00:01, 1096.41it/s]
 63%|██████▎   | 2287/3656 [00:02<00:01, 1093.57it/s]
 66%|██████▌   | 2397/3656 [00:02<00:01, 1091.32it/s]
 69%|██████▊   | 2507/3656 [00:02<00:01, 1082.01it/s]
 72%|███████▏  | 2616/3656 [00:02<00:00, 1075.62it/s]
 75%|███████▍  | 2724/3656 [00:02<00:00, 1070.70it/s]
 77%|███████▋  | 2832/3656 [00:02<00:00, 1063.41it/s]
 80%|████████  | 2939/3656 [00:02<00:00, 1057.96it/s]
 83%|████████▎ | 3045/3656 [00:02<00:00, 1052.49it/s]
 86%|████████▌ | 3151/3656 [00:02<00:00, 1051.66it/s]
 89%|████████▉ | 3257/3656 [00:03<00:00, 1043.76it/s]
 92%|█████████▏| 3362/3656 [00:03<00:00, 1039.70it/s]
 95%|█████████▍| 3466/3656 [00:03<00:00, 1037.06it/s]
 98%|█████████▊| 3570/3656 [00:03<00:00, 1032.10it/s]
100%|██████████| 3656/3656 [00:03<00:00, 1057.27it/s]
2025-11-17 16:54:28.669108: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:11.921705: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:55:11.923048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:55:12.071620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:55:12.071837: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:12.080975: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:55:12.081073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:55:12.086330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:55:12.089467: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:55:12.097434: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:55:12.101273: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:55:12.103824: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:55:12.112182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:55:12.112836: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:55:12.116957: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:55:12.119443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-11-17 16:55:12.119509: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:12.119542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:55:12.119564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:55:12.119582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:55:12.119599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:55:12.119616: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:55:12.119633: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:55:12.119649: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:55:12.124252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:55:12.124299: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:12.566757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:55:12.566950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:55:12.566966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:55:12.570940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75641 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:01:00.0, compute capability: 8.0)
2025-11-17 16:55:12.611077: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-11-17 16:55:12.621464: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2799920000 Hz
2025-11-17 16:55:15.984543: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:55:16.459444: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:55:16.468232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:55:18.259785: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:55:18.459152: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 16:55:52.030095: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)