Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-28 11:52:19.050181: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:52:50.057433: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:52:50.059297: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:52:50.345816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 11:52:50.345982: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:52:50.353711: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:52:50.353814: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:52:50.357280: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:52:50.359977: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:52:50.365875: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:52:50.369994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:52:50.372030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:52:50.379635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:52:50.380337: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:52:50.380588: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:52:50.383161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 11:52:50.383217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:52:50.383244: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:52:50.383262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:52:50.383278: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:52:50.383294: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:52:50.383310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:52:50.383326: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:52:50.383371: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:52:50.388142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:52:50.388190: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:52:50.874060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:52:50.874188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:52:50.874205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:52:50.879024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2026-05-28 11:52:52.884012: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:52:52.884574: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2449805000 Hz
2026-05-28 11:52:56.520944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:52:56.979016: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:52:56.984586: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:52:58.625613: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:52:58.716627: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 11:53:16.197693: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-28 11:59:28.085076: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-28 11:59:32.312506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:59:39.277081: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:59:39.278306: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:59:39.606685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 11:59:39.606804: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:59:39.613789: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:59:39.613907: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:59:39.618129: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:59:39.619827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:59:39.625679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:59:39.629302: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:59:39.631372: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:59:39.637183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:59:39.638089: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:59:39.638392: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:59:39.640534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 11:59:39.640567: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:59:39.640587: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:59:39.640603: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:59:39.640618: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:59:39.640633: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:59:39.640648: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:59:39.640662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:59:39.640678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:59:39.648170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:59:39.648210: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:59:40.174192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:59:40.174400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:59:40.174418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:59:40.180876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/54 [00:00<?, ?it/s]2026-05-28 11:59:42.328043: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:59:42.328587: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2449805000 Hz
2026-05-28 11:59:42.587622: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:59:43.141291: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:59:43.143610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:59:44.911439: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:59:45.081790: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/54 [00:33<29:34, 33.48s/it]
batch:   4%|▎         | 2/54 [00:33<12:06, 13.97s/it]
batch:   6%|▌         | 3/54 [00:34<06:34,  7.73s/it]
batch:   7%|▋         | 4/54 [00:34<03:59,  4.80s/it]
batch:   9%|▉         | 5/54 [00:34<02:35,  3.18s/it]
batch:  11%|█         | 6/54 [00:34<01:45,  2.20s/it]
batch:  13%|█▎        | 7/54 [00:35<01:14,  1.58s/it]
batch:  15%|█▍        | 8/54 [00:35<00:53,  1.17s/it]
batch:  17%|█▋        | 9/54 [00:35<00:40,  1.11it/s]
batch:  19%|█▊        | 10/54 [00:36<00:31,  1.40it/s]
batch:  20%|██        | 11/54 [00:36<00:25,  1.70it/s]
batch:  22%|██▏       | 12/54 [00:36<00:21,  2.00it/s]
batch:  24%|██▍       | 13/54 [00:37<00:18,  2.27it/s]
batch:  26%|██▌       | 14/54 [00:37<00:15,  2.51it/s]
batch:  28%|██▊       | 15/54 [00:37<00:14,  2.70it/s]
batch:  30%|██▉       | 16/54 [00:38<00:13,  2.86it/s]
batch:  31%|███▏      | 17/54 [00:38<00:12,  3.00it/s]
batch:  33%|███▎      | 18/54 [00:38<00:11,  3.10it/s]
batch:  35%|███▌      | 19/54 [00:38<00:10,  3.20it/s]
batch:  37%|███▋      | 20/54 [00:39<00:10,  3.26it/s]
batch:  39%|███▉      | 21/54 [00:39<00:10,  3.29it/s]
batch:  41%|████      | 22/54 [00:39<00:09,  3.31it/s]
batch:  43%|████▎     | 23/54 [00:40<00:09,  3.32it/s]
batch:  44%|████▍     | 24/54 [00:40<00:09,  3.32it/s]
batch:  46%|████▋     | 25/54 [00:40<00:08,  3.33it/s]
batch:  48%|████▊     | 26/54 [00:40<00:08,  3.33it/s]
batch:  50%|█████     | 27/54 [00:41<00:08,  3.34it/s]
batch:  52%|█████▏    | 28/54 [00:41<00:07,  3.35it/s]
batch:  54%|█████▎    | 29/54 [00:41<00:07,  3.35it/s]
batch:  56%|█████▌    | 30/54 [00:42<00:07,  3.35it/s]
batch:  57%|█████▋    | 31/54 [00:42<00:06,  3.36it/s]
batch:  59%|█████▉    | 32/54 [00:42<00:06,  3.34it/s]
batch:  61%|██████    | 33/54 [00:43<00:06,  3.34it/s]
batch:  63%|██████▎   | 34/54 [00:43<00:05,  3.34it/s]
batch:  65%|██████▍   | 35/54 [00:43<00:05,  3.34it/s]
batch:  67%|██████▋   | 36/54 [00:43<00:05,  3.34it/s]
batch:  69%|██████▊   | 37/54 [00:44<00:05,  3.32it/s]
batch:  70%|███████   | 38/54 [00:44<00:04,  3.33it/s]
batch:  72%|███████▏  | 39/54 [00:44<00:04,  3.33it/s]
batch:  74%|███████▍  | 40/54 [00:45<00:04,  3.33it/s]
batch:  76%|███████▌  | 41/54 [00:45<00:03,  3.34it/s]
batch:  78%|███████▊  | 42/54 [00:45<00:03,  3.32it/s]
batch:  80%|███████▉  | 43/54 [00:46<00:03,  3.32it/s]
batch:  81%|████████▏ | 44/54 [00:46<00:03,  3.32it/s]
batch:  83%|████████▎ | 45/54 [00:46<00:02,  3.33it/s]
batch:  85%|████████▌ | 46/54 [00:46<00:02,  3.34it/s]
batch:  87%|████████▋ | 47/54 [00:47<00:02,  3.31it/s]
batch:  89%|████████▉ | 48/54 [00:47<00:01,  3.34it/s]
batch:  91%|█████████ | 49/54 [00:47<00:01,  3.34it/s]
batch:  93%|█████████▎| 50/54 [00:48<00:01,  3.34it/s]
batch:  94%|█████████▍| 51/54 [00:48<00:00,  3.38it/s]
batch:  96%|█████████▋| 52/54 [00:48<00:00,  3.40it/s]
batch:  98%|█████████▊| 53/54 [00:48<00:00,  3.84it/s]
batch: 100%|██████████| 54/54 [00:48<00:00,  1.10it/s]

  0%|          | 0/3361 [00:00<?, ?it/s]
  3%|▎         | 96/3361 [00:00<00:03, 950.76it/s]
  6%|▌         | 195/3361 [00:00<00:03, 968.17it/s]
  9%|▉         | 295/3361 [00:00<00:03, 974.16it/s]
 12%|█▏        | 394/3361 [00:00<00:03, 979.57it/s]
 15%|█▍        | 492/3361 [00:00<00:02, 978.81it/s]
 18%|█▊        | 591/3361 [00:00<00:02, 976.30it/s]
 20%|██        | 689/3361 [00:00<00:02, 976.34it/s]
 23%|██▎       | 788/3361 [00:00<00:02, 979.96it/s]
 27%|██▋       | 892/3361 [00:00<00:02, 994.83it/s]
 30%|██▉       | 992/3361 [00:01<00:02, 990.63it/s]
 32%|███▏      | 1092/3361 [00:01<00:02, 989.39it/s]
 35%|███▌      | 1191/3361 [00:01<00:02, 977.80it/s]
 38%|███▊      | 1289/3361 [00:01<00:02, 976.66it/s]
 41%|████▏     | 1387/3361 [00:01<00:02, 974.76it/s]
 44%|████▍     | 1485/3361 [00:01<00:01, 960.51it/s]
 47%|████▋     | 1582/3361 [00:01<00:01, 952.62it/s]
 50%|████▉     | 1678/3361 [00:01<00:01, 949.84it/s]
 53%|█████▎    | 1774/3361 [00:01<00:01, 945.12it/s]
 56%|█████▌    | 1869/3361 [00:01<00:01, 938.41it/s]
 58%|█████▊    | 1963/3361 [00:02<00:01, 938.06it/s]
 61%|██████    | 2057/3361 [00:02<00:01, 928.02it/s]
 64%|██████▍   | 2151/3361 [00:02<00:01, 931.07it/s]
 67%|██████▋   | 2245/3361 [00:02<00:01, 923.82it/s]
 70%|██████▉   | 2339/3361 [00:02<00:01, 928.44it/s]
 73%|███████▎  | 2438/3361 [00:02<00:00, 941.21it/s]
 75%|███████▌  | 2533/3361 [00:02<00:00, 931.04it/s]
 78%|███████▊  | 2627/3361 [00:02<00:00, 922.05it/s]
 81%|████████  | 2720/3361 [00:02<00:00, 920.05it/s]
 84%|████████▍ | 2817/3361 [00:02<00:00, 931.51it/s]
 87%|████████▋ | 2911/3361 [00:03<00:00, 933.66it/s]
 89%|████████▉ | 3005/3361 [00:03<00:00, 918.51it/s]
 92%|█████████▏| 3097/3361 [00:03<00:00, 913.26it/s]
 95%|█████████▍| 3189/3361 [00:03<00:00, 900.69it/s]
 98%|█████████▊| 3281/3361 [00:03<00:00, 906.07it/s]
100%|██████████| 3361/3361 [00:03<00:00, 944.77it/s]

  0%|          | 0/3361 [00:00<?, ?it/s]
  3%|▎         | 99/3361 [00:00<00:03, 979.82it/s]
  6%|▌         | 198/3361 [00:00<00:03, 984.09it/s]
  9%|▉         | 297/3361 [00:00<00:03, 984.65it/s]
 12%|█▏        | 396/3361 [00:00<00:03, 974.95it/s]
 15%|█▍        | 495/3361 [00:00<00:02, 976.84it/s]
 18%|█▊        | 593/3361 [00:00<00:02, 975.36it/s]
 21%|██        | 691/3361 [00:00<00:02, 967.12it/s]
 24%|██▎       | 790/3361 [00:00<00:02, 967.63it/s]
 26%|██▋       | 887/3361 [00:00<00:02, 967.34it/s]
 29%|██▉       | 984/3361 [00:01<00:02, 966.02it/s]
 32%|███▏      | 1081/3361 [00:01<00:02, 959.84it/s]
 35%|███▌      | 1177/3361 [00:01<00:02, 956.59it/s]
 38%|███▊      | 1273/3361 [00:01<00:02, 954.76it/s]
 41%|████      | 1369/3361 [00:01<00:02, 950.49it/s]
 44%|████▎     | 1465/3361 [00:01<00:02, 944.79it/s]
 46%|████▋     | 1561/3361 [00:01<00:01, 946.56it/s]
 49%|████▉     | 1656/3361 [00:01<00:01, 938.75it/s]
 52%|█████▏    | 1752/3361 [00:01<00:01, 939.35it/s]
 55%|█████▍    | 1846/3361 [00:01<00:01, 937.81it/s]
 58%|█████▊    | 1940/3361 [00:02<00:01, 936.69it/s]
 61%|██████    | 2034/3361 [00:02<00:01, 934.84it/s]
 63%|██████▎   | 2128/3361 [00:02<00:01, 930.88it/s]
 66%|██████▌   | 2222/3361 [00:02<00:01, 926.81it/s]
 69%|██████▉   | 2315/3361 [00:02<00:01, 921.46it/s]
 72%|███████▏  | 2408/3361 [00:02<00:01, 918.59it/s]
 74%|███████▍  | 2500/3361 [00:02<00:00, 915.17it/s]
 77%|███████▋  | 2592/3361 [00:02<00:00, 914.10it/s]
 80%|███████▉  | 2684/3361 [00:02<00:00, 905.08it/s]
 83%|████████▎ | 2776/3361 [00:02<00:00, 904.96it/s]
 85%|████████▌ | 2868/3361 [00:03<00:00, 908.57it/s]
 88%|████████▊ | 2962/3361 [00:03<00:00, 913.65it/s]
 91%|█████████ | 3057/3361 [00:03<00:00, 921.24it/s]
 94%|█████████▎| 3150/3361 [00:03<00:00, 919.03it/s]
 96%|█████████▋| 3242/3361 [00:03<00:00, 908.37it/s]
 99%|█████████▉| 3333/3361 [00:03<00:00, 908.14it/s]
100%|██████████| 3361/3361 [00:03<00:00, 936.56it/s]
2026-05-28 12:00:49.673715: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:00:56.244564: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 12:00:56.245786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 12:00:56.554297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 12:00:56.554415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:00:56.559860: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:00:56.559956: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:00:56.562884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 12:00:56.564534: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 12:00:56.569620: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 12:00:56.571490: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 12:00:56.572761: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:00:56.578131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 12:00:56.578771: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 12:00:56.579160: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 12:00:56.581357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-28 12:00:56.581425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:00:56.581453: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:00:56.581475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:00:56.581496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 12:00:56.581515: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 12:00:56.581535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 12:00:56.581554: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 12:00:56.581575: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:00:56.586785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 12:00:56.586837: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 12:00:57.103871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 12:00:57.104057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 12:00:57.104076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 12:00:57.108127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2026-05-28 12:00:57.152786: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-28 12:00:57.165365: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2449805000 Hz
2026-05-28 12:01:01.210934: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 12:01:01.697233: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 12:01:01.702557: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 12:01:03.419420: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 12:01:03.567275: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 12:01:27.688828: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)