Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-28 11:52:44.897052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:53:04.312717: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:53:04.314463: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:53:04.623818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 11:53:04.623930: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:53:04.628456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:53:04.628511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:53:04.630606: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:53:04.632040: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:53:04.635859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:53:04.637573: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:53:04.638819: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:53:04.645993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:53:04.646488: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:53:04.646766: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:53:04.649516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 11:53:04.649576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:53:04.649613: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:53:04.649641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:53:04.649668: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:53:04.649694: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:53:04.649719: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:53:04.649745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:53:04.649799: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:53:04.656103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:53:04.656139: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:53:05.161065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:53:05.161198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:53:05.161215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:53:05.165210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:84:00.0, compute capability: 8.0)
2026-05-28 11:53:07.049279: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:53:07.049891: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2449805000 Hz
2026-05-28 11:53:11.227124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:53:11.682266: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:53:11.687394: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:53:13.121333: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:53:13.259641: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 11:53:39.322881: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-28 12:00:56.567023: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-28 12:01:00.458693: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:01:06.146320: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 12:01:06.147764: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 12:01:06.462866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 12:01:06.462990: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:01:06.466870: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:01:06.466959: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:01:06.468804: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 12:01:06.469964: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 12:01:06.473856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 12:01:06.475968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 12:01:06.477156: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:01:06.483457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 12:01:06.483852: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 12:01:06.484095: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 12:01:06.486762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 12:01:06.486812: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:01:06.486843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:01:06.486869: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:01:06.486893: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 12:01:06.486923: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 12:01:06.486948: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 12:01:06.486971: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 12:01:06.486993: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:01:06.491867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 12:01:06.491906: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:01:06.995604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 12:01:06.995717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 12:01:06.995732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 12:01:06.999837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:84:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/54 [00:00<?, ?it/s]2026-05-28 12:01:09.015052: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 12:01:09.015565: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2449805000 Hz
2026-05-28 12:01:09.266345: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:01:09.773627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:01:09.775631: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:01:11.399975: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 12:01:11.540027: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/54 [00:31<27:28, 31.11s/it]
batch:   4%|▎         | 2/54 [00:31<11:15, 12.99s/it]
batch:   6%|▌         | 3/54 [00:31<06:07,  7.20s/it]
batch:   7%|▋         | 4/54 [00:32<03:43,  4.47s/it]
batch:   9%|▉         | 5/54 [00:32<02:25,  2.97s/it]
batch:  11%|█         | 6/54 [00:32<01:38,  2.06s/it]
batch:  13%|█▎        | 7/54 [00:32<01:09,  1.48s/it]
batch:  15%|█▍        | 8/54 [00:33<00:50,  1.11s/it]
batch:  17%|█▋        | 9/54 [00:33<00:38,  1.17it/s]
batch:  19%|█▊        | 10/54 [00:33<00:30,  1.46it/s]
batch:  20%|██        | 11/54 [00:34<00:24,  1.77it/s]
batch:  22%|██▏       | 12/54 [00:34<00:20,  2.06it/s]
batch:  24%|██▍       | 13/54 [00:34<00:17,  2.33it/s]
batch:  26%|██▌       | 14/54 [00:35<00:15,  2.56it/s]
batch:  28%|██▊       | 15/54 [00:35<00:14,  2.76it/s]
batch:  30%|██▉       | 16/54 [00:35<00:13,  2.91it/s]
batch:  31%|███▏      | 17/54 [00:35<00:12,  3.03it/s]
batch:  33%|███▎      | 18/54 [00:36<00:11,  3.12it/s]
batch:  35%|███▌      | 19/54 [00:36<00:10,  3.18it/s]
batch:  37%|███▋      | 20/54 [00:36<00:10,  3.22it/s]
batch:  39%|███▉      | 21/54 [00:37<00:10,  3.25it/s]
batch:  41%|████      | 22/54 [00:37<00:09,  3.28it/s]
batch:  43%|████▎     | 23/54 [00:37<00:09,  3.30it/s]
batch:  44%|████▍     | 24/54 [00:38<00:09,  3.31it/s]
batch:  46%|████▋     | 25/54 [00:38<00:08,  3.32it/s]
batch:  48%|████▊     | 26/54 [00:38<00:08,  3.33it/s]
batch:  50%|█████     | 27/54 [00:38<00:08,  3.34it/s]
batch:  52%|█████▏    | 28/54 [00:39<00:07,  3.35it/s]
batch:  54%|█████▎    | 29/54 [00:39<00:07,  3.35it/s]
batch:  56%|█████▌    | 30/54 [00:39<00:07,  3.35it/s]
batch:  57%|█████▋    | 31/54 [00:40<00:06,  3.35it/s]
batch:  59%|█████▉    | 32/54 [00:40<00:06,  3.35it/s]
batch:  61%|██████    | 33/54 [00:40<00:06,  3.35it/s]
batch:  63%|██████▎   | 34/54 [00:40<00:05,  3.35it/s]
batch:  65%|██████▍   | 35/54 [00:41<00:05,  3.36it/s]
batch:  67%|██████▋   | 36/54 [00:41<00:05,  3.35it/s]
batch:  69%|██████▊   | 37/54 [00:41<00:05,  3.34it/s]
batch:  70%|███████   | 38/54 [00:42<00:04,  3.34it/s]
batch:  72%|███████▏  | 39/54 [00:42<00:04,  3.33it/s]
batch:  74%|███████▍  | 40/54 [00:42<00:04,  3.33it/s]
batch:  76%|███████▌  | 41/54 [00:43<00:03,  3.33it/s]
batch:  78%|███████▊  | 42/54 [00:43<00:03,  3.34it/s]
batch:  80%|███████▉  | 43/54 [00:43<00:03,  3.35it/s]
batch:  81%|████████▏ | 44/54 [00:43<00:03,  3.32it/s]
batch:  83%|████████▎ | 45/54 [00:44<00:02,  3.32it/s]
batch:  85%|████████▌ | 46/54 [00:44<00:02,  3.33it/s]
batch:  87%|████████▋ | 47/54 [00:44<00:02,  3.34it/s]
batch:  89%|████████▉ | 48/54 [00:45<00:01,  3.34it/s]
batch:  91%|█████████ | 49/54 [00:45<00:01,  3.34it/s]
batch:  93%|█████████▎| 50/54 [00:45<00:01,  3.33it/s]
batch:  94%|█████████▍| 51/54 [00:46<00:00,  3.33it/s]
batch:  96%|█████████▋| 52/54 [00:46<00:00,  3.78it/s]
batch:  98%|█████████▊| 53/54 [00:46<00:00,  3.64it/s]
batch: 100%|██████████| 54/54 [00:46<00:00,  1.16it/s]

  0%|          | 0/3361 [00:00<?, ?it/s]
  3%|▎         | 92/3361 [00:00<00:03, 917.28it/s]
  6%|▌         | 190/3361 [00:00<00:03, 953.21it/s]
  9%|▊         | 288/3361 [00:00<00:03, 964.55it/s]
 11%|█▏        | 385/3361 [00:00<00:03, 964.72it/s]
 14%|█▍        | 484/3361 [00:00<00:02, 969.89it/s]
 17%|█▋        | 583/3361 [00:00<00:02, 970.77it/s]
 20%|██        | 681/3361 [00:00<00:02, 968.18it/s]
 23%|██▎       | 779/3361 [00:00<00:02, 969.11it/s]
 26%|██▌       | 876/3361 [00:00<00:02, 969.08it/s]
 29%|██▉       | 973/3361 [00:01<00:02, 966.74it/s]
 32%|███▏      | 1070/3361 [00:01<00:02, 960.83it/s]
 35%|███▍      | 1167/3361 [00:01<00:02, 958.68it/s]
 38%|███▊      | 1263/3361 [00:01<00:02, 957.02it/s]
 40%|████      | 1359/3361 [00:01<00:02, 952.28it/s]
 43%|████▎     | 1455/3361 [00:01<00:02, 945.95it/s]
 46%|████▌     | 1550/3361 [00:01<00:01, 944.12it/s]
 49%|████▉     | 1645/3361 [00:01<00:01, 938.54it/s]
 52%|█████▏    | 1740/3361 [00:01<00:01, 941.90it/s]
 55%|█████▍    | 1835/3361 [00:01<00:01, 941.96it/s]
 57%|█████▋    | 1930/3361 [00:02<00:01, 936.34it/s]
 60%|██████    | 2024/3361 [00:02<00:01, 935.62it/s]
 63%|██████▎   | 2118/3361 [00:02<00:01, 929.48it/s]
 66%|██████▌   | 2211/3361 [00:02<00:01, 895.14it/s]
 69%|██████▊   | 2303/3361 [00:02<00:01, 896.84it/s]
 71%|███████▏  | 2397/3361 [00:02<00:01, 903.77it/s]
 74%|███████▍  | 2488/3361 [00:02<00:00, 902.89it/s]
 77%|███████▋  | 2580/3361 [00:02<00:00, 907.10it/s]
 79%|███████▉  | 2671/3361 [00:02<00:00, 903.52it/s]
 82%|████████▏ | 2762/3361 [00:02<00:00, 902.34it/s]
 85%|████████▍ | 2853/3361 [00:03<00:00, 902.61it/s]
 88%|████████▊ | 2944/3361 [00:03<00:00, 899.53it/s]
 90%|█████████ | 3034/3361 [00:03<00:00, 898.01it/s]
 93%|█████████▎| 3124/3361 [00:03<00:00, 893.29it/s]
 96%|█████████▌| 3214/3361 [00:03<00:00, 888.61it/s]
 98%|█████████▊| 3303/3361 [00:03<00:00, 886.69it/s]
100%|██████████| 3361/3361 [00:03<00:00, 927.31it/s]

  0%|          | 0/3361 [00:00<?, ?it/s]
  3%|▎         | 99/3361 [00:00<00:03, 977.53it/s]
  6%|▌         | 197/3361 [00:00<00:03, 976.73it/s]
  9%|▉         | 295/3361 [00:00<00:03, 975.57it/s]
 12%|█▏        | 393/3361 [00:00<00:03, 972.58it/s]
 15%|█▍        | 492/3361 [00:00<00:02, 974.30it/s]
 18%|█▊        | 591/3361 [00:00<00:02, 973.97it/s]
 20%|██        | 689/3361 [00:00<00:02, 971.42it/s]
 23%|██▎       | 787/3361 [00:00<00:02, 968.02it/s]
 26%|██▋       | 884/3361 [00:00<00:02, 968.05it/s]
 29%|██▉       | 981/3361 [00:01<00:02, 965.18it/s]
 32%|███▏      | 1078/3361 [00:01<00:02, 958.62it/s]
 35%|███▍      | 1174/3361 [00:01<00:02, 954.91it/s]
 38%|███▊      | 1270/3361 [00:01<00:02, 952.14it/s]
 41%|████      | 1366/3361 [00:01<00:02, 947.67it/s]
 43%|████▎     | 1461/3361 [00:01<00:02, 945.30it/s]
 46%|████▋     | 1556/3361 [00:01<00:01, 943.13it/s]
 49%|████▉     | 1651/3361 [00:01<00:01, 936.19it/s]
 52%|█████▏    | 1746/3361 [00:01<00:01, 939.60it/s]
 55%|█████▍    | 1840/3361 [00:01<00:01, 936.83it/s]
 58%|█████▊    | 1934/3361 [00:02<00:01, 930.22it/s]
 60%|██████    | 2028/3361 [00:02<00:01, 930.04it/s]
 63%|██████▎   | 2122/3361 [00:02<00:01, 927.78it/s]
 66%|██████▌   | 2215/3361 [00:02<00:01, 921.89it/s]
 69%|██████▊   | 2308/3361 [00:02<00:01, 917.56it/s]
 71%|███████▏  | 2400/3361 [00:02<00:01, 913.18it/s]
 74%|███████▍  | 2492/3361 [00:02<00:00, 910.23it/s]
 77%|███████▋  | 2584/3361 [00:02<00:00, 906.55it/s]
 80%|███████▉  | 2676/3361 [00:02<00:00, 904.54it/s]
 82%|████████▏ | 2768/3361 [00:02<00:00, 903.94it/s]
 85%|████████▌ | 2859/3361 [00:03<00:00, 902.18it/s]
 88%|████████▊ | 2950/3361 [00:03<00:00, 897.91it/s]
 90%|█████████ | 3040/3361 [00:03<00:00, 895.63it/s]
 93%|█████████▎| 3130/3361 [00:03<00:00, 891.04it/s]
 96%|█████████▌| 3220/3361 [00:03<00:00, 885.89it/s]
 98%|█████████▊| 3309/3361 [00:03<00:00, 884.10it/s]
100%|██████████| 3361/3361 [00:03<00:00, 928.90it/s]
2026-05-28 12:02:13.981637: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:02:20.473257: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 12:02:20.474820: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 12:02:20.789121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 12:02:20.789225: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:02:20.793737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:02:20.793800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:02:20.795951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 12:02:20.797167: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 12:02:20.800353: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 12:02:20.802024: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 12:02:20.803249: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:02:20.808873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 12:02:20.809337: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 12:02:20.809618: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 12:02:20.812211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 12:02:20.812285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:02:20.812318: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:02:20.812344: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:02:20.812368: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 12:02:20.812392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 12:02:20.812416: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 12:02:20.812439: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 12:02:20.812464: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:02:20.817997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 12:02:20.818078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:02:21.317171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 12:02:21.317306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 12:02:21.317322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 12:02:21.322522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:84:00.0, compute capability: 8.0)
2026-05-28 12:02:21.368386: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-28 12:02:21.380651: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2449805000 Hz
2026-05-28 12:02:25.205079: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:02:25.651646: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:02:25.657704: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:02:27.069185: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 12:02:27.203571: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 12:02:49.986884: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)