Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 21:51:11.180130: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:51:28.816789: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:51:28.818614: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:51:29.050059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:51:29.050156: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:51:29.055013: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:51:29.055083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:51:29.057402: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:51:29.058965: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:51:29.062685: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:51:29.064552: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:51:29.065797: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:51:29.071744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:51:29.072390: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:51:29.075123: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:51:29.077486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:51:29.077517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:51:29.077536: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:51:29.077551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:51:29.077565: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:51:29.077578: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:51:29.077592: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:51:29.077607: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:51:29.077638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:51:29.083416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:51:29.083455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:51:29.597083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:51:29.597199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:51:29.597218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:51:29.602014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2025-12-29 21:51:31.199968: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:51:31.200661: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 21:51:33.004815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:51:33.516635: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:51:33.524273: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:51:35.279669: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:51:35.561449: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:52:25.015737: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 21:55:36.699579: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 21:55:41.744181: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:51.084164: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:55:51.085242: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:55:51.318485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:55:51.318588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:51.323348: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:55:51.323425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:55:51.326230: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:55:51.327901: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:55:51.332173: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:55:51.334102: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:55:51.335597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:55:51.342165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:55:51.342651: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:55:51.345217: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:55:51.347534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:55:51.347568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:51.347588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:55:51.347604: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:55:51.347619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:55:51.347634: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:55:51.347649: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:55:51.347664: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:55:51.347678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:55:51.354040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:55:51.354081: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:55:51.902080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:55:51.902256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:55:51.902273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:55:51.906990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/54 [00:00<?, ?it/s]2025-12-29 21:55:54.092140: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:55:54.092809: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 21:55:54.342149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:55:54.850082: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:55:54.852880: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:55:56.498838: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:55:56.689110: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/54 [00:52<46:01, 52.10s/it]
batch:   4%|▎         | 2/54 [00:52<18:45, 21.64s/it]
batch:   6%|▌         | 3/54 [00:52<10:07, 11.90s/it]
batch:   7%|▋         | 4/54 [00:53<06:06,  7.33s/it]
batch:   9%|▉         | 5/54 [00:53<03:55,  4.80s/it]
batch:  11%|█         | 6/54 [00:53<02:37,  3.28s/it]
batch:  13%|█▎        | 7/54 [00:53<01:48,  2.31s/it]
batch:  15%|█▍        | 8/54 [00:54<01:16,  1.67s/it]
batch:  17%|█▋        | 9/54 [00:54<00:56,  1.25s/it]
batch:  19%|█▊        | 10/54 [00:54<00:42,  1.04it/s]
batch:  20%|██        | 11/54 [00:55<00:32,  1.31it/s]
batch:  22%|██▏       | 12/54 [00:55<00:26,  1.60it/s]
batch:  24%|██▍       | 13/54 [00:55<00:21,  1.89it/s]
batch:  26%|██▌       | 14/54 [00:56<00:18,  2.15it/s]
batch:  28%|██▊       | 15/54 [00:56<00:16,  2.36it/s]
batch:  30%|██▉       | 16/54 [00:56<00:14,  2.56it/s]
batch:  31%|███▏      | 17/54 [00:57<00:13,  2.71it/s]
batch:  33%|███▎      | 18/54 [00:57<00:12,  2.84it/s]
batch:  35%|███▌      | 19/54 [00:57<00:11,  2.93it/s]
batch:  37%|███▋      | 20/54 [00:58<00:11,  2.97it/s]
batch:  39%|███▉      | 21/54 [00:58<00:10,  3.04it/s]
batch:  41%|████      | 22/54 [00:58<00:10,  3.08it/s]
batch:  43%|████▎     | 23/54 [00:59<00:09,  3.11it/s]
batch:  44%|████▍     | 24/54 [00:59<00:09,  3.12it/s]
batch:  46%|████▋     | 25/54 [00:59<00:09,  3.12it/s]
batch:  48%|████▊     | 26/54 [00:59<00:08,  3.14it/s]
batch:  50%|█████     | 27/54 [01:00<00:08,  3.15it/s]
batch:  52%|█████▏    | 28/54 [01:00<00:08,  3.18it/s]
batch:  54%|█████▎    | 29/54 [01:00<00:07,  3.20it/s]
batch:  56%|█████▌    | 30/54 [01:01<00:07,  3.20it/s]
batch:  57%|█████▋    | 31/54 [01:01<00:07,  3.23it/s]
batch:  59%|█████▉    | 32/54 [01:01<00:06,  3.22it/s]
batch:  61%|██████    | 33/54 [01:02<00:06,  3.24it/s]
batch:  63%|██████▎   | 34/54 [01:02<00:06,  3.21it/s]
batch:  65%|██████▍   | 35/54 [01:02<00:05,  3.23it/s]
batch:  67%|██████▋   | 36/54 [01:03<00:05,  3.22it/s]
batch:  69%|██████▊   | 37/54 [01:03<00:05,  3.21it/s]
batch:  70%|███████   | 38/54 [01:03<00:05,  3.14it/s]
batch:  72%|███████▏  | 39/54 [01:04<00:04,  3.17it/s]
batch:  74%|███████▍  | 40/54 [01:04<00:04,  3.19it/s]
batch:  76%|███████▌  | 41/54 [01:04<00:04,  3.20it/s]
batch:  78%|███████▊  | 42/54 [01:04<00:03,  3.21it/s]
batch:  80%|███████▉  | 43/54 [01:05<00:03,  3.17it/s]
batch:  81%|████████▏ | 44/54 [01:05<00:03,  3.13it/s]
batch:  83%|████████▎ | 45/54 [01:05<00:02,  3.16it/s]
batch:  85%|████████▌ | 46/54 [01:06<00:02,  3.20it/s]
batch:  87%|████████▋ | 47/54 [01:06<00:02,  3.20it/s]
batch:  89%|████████▉ | 48/54 [01:06<00:01,  3.20it/s]
batch:  91%|█████████ | 49/54 [01:07<00:01,  3.22it/s]
batch:  93%|█████████▎| 50/54 [01:07<00:01,  3.25it/s]
batch:  94%|█████████▍| 51/54 [01:07<00:00,  3.22it/s]
batch:  96%|█████████▋| 52/54 [01:08<00:00,  3.22it/s]
batch:  98%|█████████▊| 53/54 [01:08<00:00,  3.81it/s]
batch: 100%|██████████| 54/54 [01:08<00:00,  1.27s/it]

  0%|          | 0/3350 [00:00<?, ?it/s]
  3%|▎         | 94/3350 [00:00<00:03, 938.02it/s]
  6%|▌         | 192/3350 [00:00<00:03, 951.24it/s]
  9%|▊         | 288/3350 [00:00<00:03, 951.10it/s]
 12%|█▏        | 386/3350 [00:00<00:03, 959.98it/s]
 14%|█▍        | 484/3350 [00:00<00:02, 966.49it/s]
 17%|█▋        | 581/3350 [00:00<00:02, 967.00it/s]
 20%|██        | 678/3350 [00:00<00:02, 960.88it/s]
 23%|██▎       | 775/3350 [00:00<00:02, 952.59it/s]
 26%|██▌       | 871/3350 [00:00<00:02, 949.67it/s]
 29%|██▉       | 966/3350 [00:01<00:02, 941.66it/s]
 32%|███▏      | 1061/3350 [00:01<00:02, 938.03it/s]
 35%|███▍      | 1156/3350 [00:01<00:02, 938.82it/s]
 37%|███▋      | 1250/3350 [00:01<00:02, 934.27it/s]
 40%|████      | 1345/3350 [00:01<00:02, 936.12it/s]
 43%|████▎     | 1439/3350 [00:01<00:02, 932.22it/s]
 46%|████▌     | 1533/3350 [00:01<00:01, 930.33it/s]
 49%|████▊     | 1627/3350 [00:01<00:01, 925.44it/s]
 51%|█████▏    | 1720/3350 [00:01<00:01, 925.42it/s]
 54%|█████▍    | 1813/3350 [00:01<00:01, 920.67it/s]
 57%|█████▋    | 1906/3350 [00:02<00:01, 916.20it/s]
 60%|█████▉    | 1998/3350 [00:02<00:01, 884.66it/s]
 62%|██████▏   | 2089/3350 [00:02<00:01, 887.02it/s]
 65%|██████▌   | 2180/3350 [00:02<00:01, 892.76it/s]
 68%|██████▊   | 2270/3350 [00:02<00:01, 893.55it/s]
 71%|███████   | 2362/3350 [00:02<00:01, 896.76it/s]
 73%|███████▎  | 2452/3350 [00:02<00:01, 888.27it/s]
 76%|███████▌  | 2541/3350 [00:02<00:00, 888.62it/s]
 79%|███████▊  | 2630/3350 [00:02<00:00, 851.77it/s]
 81%|████████  | 2720/3350 [00:02<00:00, 862.49it/s]
 84%|████████▍ | 2809/3350 [00:03<00:00, 869.84it/s]
 86%|████████▋ | 2897/3350 [00:03<00:00, 871.92it/s]
 89%|████████▉ | 2985/3350 [00:03<00:00, 871.04it/s]
 92%|█████████▏| 3074/3350 [00:03<00:00, 872.48it/s]
 94%|█████████▍| 3162/3350 [00:03<00:00, 860.45it/s]
 97%|█████████▋| 3249/3350 [00:03<00:00, 862.07it/s]
100%|█████████▉| 3337/3350 [00:03<00:00, 862.17it/s]
100%|██████████| 3350/3350 [00:03<00:00, 906.51it/s]

  0%|          | 0/3350 [00:00<?, ?it/s]
  3%|▎         | 88/3350 [00:00<00:03, 863.01it/s]
  6%|▌         | 186/3350 [00:00<00:03, 921.00it/s]
  8%|▊         | 283/3350 [00:00<00:03, 939.30it/s]
 11%|█▏        | 383/3350 [00:00<00:03, 954.86it/s]
 14%|█▍        | 483/3350 [00:00<00:02, 965.54it/s]
 17%|█▋        | 580/3350 [00:00<00:02, 967.02it/s]
 20%|██        | 677/3350 [00:00<00:02, 950.11it/s]
 23%|██▎       | 773/3350 [00:00<00:02, 953.12it/s]
 26%|██▌       | 870/3350 [00:00<00:02, 955.06it/s]
 29%|██▉       | 966/3350 [00:01<00:02, 908.11it/s]
 32%|███▏      | 1060/3350 [00:01<00:02, 916.59it/s]
 35%|███▍      | 1156/3350 [00:01<00:02, 925.55it/s]
 37%|███▋      | 1251/3350 [00:01<00:02, 932.05it/s]
 40%|████      | 1345/3350 [00:01<00:02, 931.77it/s]
 43%|████▎     | 1440/3350 [00:01<00:02, 933.07it/s]
 46%|████▌     | 1535/3350 [00:01<00:01, 932.52it/s]
 49%|████▊     | 1629/3350 [00:01<00:01, 916.88it/s]
 51%|█████▏    | 1721/3350 [00:01<00:01, 915.87it/s]
 54%|█████▍    | 1814/3350 [00:01<00:01, 918.63it/s]
 57%|█████▋    | 1906/3350 [00:02<00:01, 917.70it/s]
 60%|█████▉    | 1998/3350 [00:02<00:01, 915.34it/s]
 62%|██████▏   | 2090/3350 [00:02<00:01, 902.65it/s]
 65%|██████▌   | 2181/3350 [00:02<00:01, 903.34it/s]
 68%|██████▊   | 2272/3350 [00:02<00:01, 898.21it/s]
 71%|███████   | 2364/3350 [00:02<00:01, 899.14it/s]
 73%|███████▎  | 2454/3350 [00:02<00:01, 889.57it/s]
 76%|███████▌  | 2545/3350 [00:02<00:00, 889.56it/s]
 79%|███████▊  | 2635/3350 [00:02<00:00, 891.60it/s]
 81%|████████▏ | 2725/3350 [00:02<00:00, 889.76it/s]
 84%|████████▍ | 2814/3350 [00:03<00:00, 888.44it/s]
 87%|████████▋ | 2903/3350 [00:03<00:00, 882.33it/s]
 89%|████████▉ | 2992/3350 [00:03<00:00, 876.28it/s]
 92%|█████████▏| 3080/3350 [00:03<00:00, 877.30it/s]
 95%|█████████▍| 3168/3350 [00:03<00:00, 877.23it/s]
 97%|█████████▋| 3256/3350 [00:03<00:00, 859.21it/s]
100%|█████████▉| 3344/3350 [00:03<00:00, 864.96it/s]
100%|██████████| 3350/3350 [00:03<00:00, 908.02it/s]
2025-12-29 21:57:25.316862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:57:45.538463: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:57:45.544550: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:57:45.774375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:57:45.774496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:57:45.778763: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:57:45.778827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:57:45.781141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:57:45.782450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:57:45.786051: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:57:45.788169: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:57:45.789641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:57:45.795876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:57:45.796493: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:57:45.798913: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:57:45.802814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:57:45.802879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:57:45.802917: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:57:45.802958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:57:45.802990: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:57:45.803020: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:57:45.803049: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:57:45.803078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:57:45.803107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:57:45.807890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:57:45.807939: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:57:46.284388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:57:46.284508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:57:46.284524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:57:46.289423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2025-12-29 21:57:46.335470: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 21:57:46.347603: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 21:57:49.738504: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:57:50.274567: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:57:50.281339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:57:51.793737: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:57:52.010755: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:58:28.150818: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)