Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:36:38.106732: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:25.703085: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:25.939476: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:51:26.488302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:26.488387: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:38.888018: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:38.888280: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:39.171093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:39.813831: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:41.748587: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.243477: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.501908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.521648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.522111: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:51:42.522285: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:42.551428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:42.551595: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:42.551651: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:42.551675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:42.551710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:42.551734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:42.551754: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.551776: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.551834: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.576937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.577021: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:45.451825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:51:45.452025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:51:45.452044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:51:45.457334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0)
2026-05-11 03:51:49.232250: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:51:49.232899: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 03:51:57.009580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:59.305301: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:59.318741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:10.726946: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:53:10.850766: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:54:42.245499: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 04:09:56.424506: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 04:12:41.460679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:54.629739: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:16:54.630894: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:16:54.896518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:16:54.896629: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:54.980046: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:54.980209: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:55.022687: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:16:55.059639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:16:55.099026: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:16:55.136105: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:16:55.174093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:55.178958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:16:55.179468: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:16:55.179761: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:16:55.182119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:16:55.182182: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:55.182212: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:55.182231: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:55.182250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:16:55.182268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:16:55.182285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:16:55.182303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:16:55.182321: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:55.187779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:16:55.187902: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:55.674131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:16:55.674265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:16:55.674279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:16:55.679928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/126 [00:00<?, ?it/s]2026-05-11 04:16:58.226011: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 04:16:58.226861: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:16:58.738506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:59.284125: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:59.291487: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:17:01.526556: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:17:01.625245: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/126 [01:44<3:37:00, 104.17s/it]
batch:   2%|▏         | 2/126 [01:44<1:29:01, 43.08s/it] 
batch:   2%|▏         | 3/126 [01:44<48:16, 23.55s/it]  
batch:   3%|▎         | 4/126 [01:45<29:13, 14.37s/it]
batch:   4%|▍         | 5/126 [01:45<18:45,  9.30s/it]
batch:   5%|▍         | 6/126 [01:45<12:28,  6.24s/it]
batch:   6%|▌         | 7/126 [01:46<08:31,  4.30s/it]
batch:   6%|▋         | 8/126 [01:46<05:57,  3.03s/it]
batch:   7%|▋         | 9/126 [01:46<04:14,  2.18s/it]
batch:   8%|▊         | 10/126 [01:46<03:05,  1.60s/it]
batch:   9%|▊         | 11/126 [01:47<02:17,  1.20s/it]
batch:  10%|▉         | 12/126 [01:47<01:45,  1.08it/s]
batch:  10%|█         | 13/126 [01:47<01:23,  1.36it/s]
batch:  11%|█         | 14/126 [01:48<01:07,  1.66it/s]
batch:  12%|█▏        | 15/126 [01:48<00:56,  1.96it/s]
batch:  13%|█▎        | 16/126 [01:48<00:48,  2.25it/s]
batch:  13%|█▎        | 17/126 [01:48<00:43,  2.49it/s]
batch:  14%|█▍        | 18/126 [01:49<00:40,  2.69it/s]
batch:  15%|█▌        | 19/126 [01:49<00:37,  2.87it/s]
batch:  16%|█▌        | 20/126 [01:49<00:35,  3.00it/s]
batch:  17%|█▋        | 21/126 [01:50<00:33,  3.11it/s]
batch:  17%|█▋        | 22/126 [01:50<00:32,  3.17it/s]
batch:  18%|█▊        | 23/126 [01:50<00:32,  3.22it/s]
batch:  19%|█▉        | 24/126 [01:51<00:31,  3.25it/s]
batch:  20%|█▉        | 25/126 [01:51<00:31,  3.26it/s]
batch:  21%|██        | 26/126 [01:51<00:30,  3.25it/s]
batch:  21%|██▏       | 27/126 [01:51<00:29,  3.30it/s]
batch:  22%|██▏       | 28/126 [01:52<00:29,  3.32it/s]
batch:  23%|██▎       | 29/126 [01:52<00:29,  3.33it/s]
batch:  24%|██▍       | 30/126 [01:52<00:28,  3.32it/s]
batch:  25%|██▍       | 31/126 [01:53<00:28,  3.31it/s]
batch:  25%|██▌       | 32/126 [01:53<00:28,  3.33it/s]
batch:  26%|██▌       | 33/126 [01:53<00:27,  3.33it/s]
batch:  27%|██▋       | 34/126 [01:54<00:27,  3.36it/s]
batch:  28%|██▊       | 35/126 [01:54<00:26,  3.39it/s]
batch:  29%|██▊       | 36/126 [01:54<00:26,  3.42it/s]
batch:  29%|██▉       | 37/126 [01:54<00:26,  3.40it/s]
batch:  30%|███       | 38/126 [01:55<00:25,  3.43it/s]
batch:  31%|███       | 39/126 [01:55<00:25,  3.43it/s]
batch:  32%|███▏      | 40/126 [01:55<00:25,  3.42it/s]
batch:  33%|███▎      | 41/126 [01:56<00:24,  3.43it/s]
batch:  33%|███▎      | 42/126 [01:56<00:24,  3.41it/s]
batch:  34%|███▍      | 43/126 [01:56<00:24,  3.40it/s]
batch:  35%|███▍      | 44/126 [01:56<00:24,  3.41it/s]
batch:  36%|███▌      | 45/126 [01:57<00:23,  3.42it/s]
batch:  37%|███▋      | 46/126 [01:57<00:23,  3.40it/s]
batch:  37%|███▋      | 47/126 [01:57<00:23,  3.40it/s]
batch:  38%|███▊      | 48/126 [01:58<00:22,  3.40it/s]
batch:  39%|███▉      | 49/126 [01:58<00:22,  3.40it/s]
batch:  40%|███▉      | 50/126 [01:58<00:22,  3.40it/s]
batch:  40%|████      | 51/126 [01:59<00:21,  3.46it/s]
batch:  41%|████▏     | 52/126 [01:59<00:21,  3.48it/s]
batch:  42%|████▏     | 53/126 [01:59<00:21,  3.46it/s]
batch:  43%|████▎     | 54/126 [01:59<00:21,  3.42it/s]
batch:  44%|████▎     | 55/126 [02:00<00:20,  3.41it/s]
batch:  44%|████▍     | 56/126 [02:00<00:20,  3.39it/s]
batch:  45%|████▌     | 57/126 [02:00<00:20,  3.38it/s]
batch:  46%|████▌     | 58/126 [02:01<00:20,  3.35it/s]
batch:  47%|████▋     | 59/126 [02:01<00:20,  3.34it/s]
batch:  48%|████▊     | 60/126 [02:01<00:19,  3.35it/s]
batch:  48%|████▊     | 61/126 [02:02<00:19,  3.38it/s]
batch:  49%|████▉     | 62/126 [02:02<00:18,  3.41it/s]
batch:  50%|█████     | 63/126 [02:02<00:18,  3.45it/s]
batch:  51%|█████     | 64/126 [02:02<00:17,  3.48it/s]
batch:  52%|█████▏    | 65/126 [02:03<00:17,  3.48it/s]
batch:  52%|█████▏    | 66/126 [02:03<00:19,  3.07it/s]
batch:  53%|█████▎    | 67/126 [02:03<00:18,  3.14it/s]
batch:  54%|█████▍    | 68/126 [02:04<00:18,  3.21it/s]
batch:  55%|█████▍    | 69/126 [02:04<00:17,  3.29it/s]
batch:  56%|█████▌    | 70/126 [02:04<00:16,  3.35it/s]
batch:  56%|█████▋    | 71/126 [02:05<00:16,  3.39it/s]
batch:  57%|█████▋    | 72/126 [02:05<00:15,  3.41it/s]
batch:  58%|█████▊    | 73/126 [02:05<00:15,  3.43it/s]
batch:  59%|█████▊    | 74/126 [02:05<00:15,  3.42it/s]
batch:  60%|█████▉    | 75/126 [02:06<00:15,  3.40it/s]
batch:  60%|██████    | 76/126 [02:06<00:14,  3.39it/s]
batch:  61%|██████    | 77/126 [02:06<00:14,  3.36it/s]
batch:  62%|██████▏   | 78/126 [02:07<00:14,  3.34it/s]
batch:  63%|██████▎   | 79/126 [02:07<00:14,  3.34it/s]
batch:  63%|██████▎   | 80/126 [02:07<00:13,  3.35it/s]
batch:  64%|██████▍   | 81/126 [02:07<00:13,  3.35it/s]
batch:  65%|██████▌   | 82/126 [02:08<00:13,  3.34it/s]
batch:  66%|██████▌   | 83/126 [02:08<00:12,  3.32it/s]
batch:  67%|██████▋   | 84/126 [02:08<00:12,  3.32it/s]
batch:  67%|██████▋   | 85/126 [02:09<00:12,  3.35it/s]
batch:  68%|██████▊   | 86/126 [02:09<00:11,  3.36it/s]
batch:  69%|██████▉   | 87/126 [02:09<00:11,  3.36it/s]
batch:  70%|██████▉   | 88/126 [02:10<00:11,  3.37it/s]
batch:  71%|███████   | 89/126 [02:10<00:10,  3.41it/s]
batch:  71%|███████▏  | 90/126 [02:10<00:10,  3.43it/s]
batch:  72%|███████▏  | 91/126 [02:10<00:10,  3.42it/s]
batch:  73%|███████▎  | 92/126 [02:11<00:09,  3.44it/s]
batch:  74%|███████▍  | 93/126 [02:11<00:09,  3.43it/s]
batch:  75%|███████▍  | 94/126 [02:11<00:09,  3.41it/s]
batch:  75%|███████▌  | 95/126 [02:12<00:08,  3.45it/s]
batch:  76%|███████▌  | 96/126 [02:12<00:08,  3.47it/s]
batch:  77%|███████▋  | 97/126 [02:12<00:08,  3.44it/s]
batch:  78%|███████▊  | 98/126 [02:12<00:08,  3.42it/s]
batch:  79%|███████▊  | 99/126 [02:13<00:07,  3.44it/s]
batch:  79%|███████▉  | 100/126 [02:13<00:07,  3.44it/s]
batch:  80%|████████  | 101/126 [02:13<00:07,  3.41it/s]
batch:  81%|████████  | 102/126 [02:14<00:07,  3.40it/s]
batch:  82%|████████▏ | 103/126 [02:14<00:06,  3.44it/s]
batch:  83%|████████▎ | 104/126 [02:14<00:06,  3.43it/s]
batch:  83%|████████▎ | 105/126 [02:15<00:06,  3.42it/s]
batch:  84%|████████▍ | 106/126 [02:15<00:05,  3.41it/s]
batch:  85%|████████▍ | 107/126 [02:15<00:05,  3.39it/s]
batch:  86%|████████▌ | 108/126 [02:15<00:05,  3.39it/s]
batch:  87%|████████▋ | 109/126 [02:16<00:04,  3.41it/s]
batch:  87%|████████▋ | 110/126 [02:16<00:04,  3.37it/s]
batch:  88%|████████▊ | 111/126 [02:16<00:04,  3.36it/s]
batch:  89%|████████▉ | 112/126 [02:17<00:04,  3.36it/s]
batch:  90%|████████▉ | 113/126 [02:17<00:03,  3.38it/s]
batch:  90%|█████████ | 114/126 [02:17<00:03,  3.39it/s]
batch:  91%|█████████▏| 115/126 [02:17<00:03,  3.39it/s]
batch:  92%|█████████▏| 116/126 [02:18<00:02,  3.42it/s]
batch:  93%|█████████▎| 117/126 [02:18<00:02,  3.41it/s]
batch:  94%|█████████▎| 118/126 [02:18<00:02,  3.41it/s]
batch:  94%|█████████▍| 119/126 [02:19<00:02,  3.41it/s]
batch:  95%|█████████▌| 120/126 [02:19<00:01,  3.43it/s]
batch:  96%|█████████▌| 121/126 [02:19<00:01,  3.44it/s]
batch:  97%|█████████▋| 122/126 [02:20<00:01,  3.41it/s]
batch:  98%|█████████▊| 123/126 [02:20<00:00,  3.41it/s]
batch:  98%|█████████▊| 124/126 [02:20<00:00,  3.85it/s]
batch:  99%|█████████▉| 125/126 [02:20<00:00,  3.67it/s]
batch: 100%|██████████| 126/126 [02:20<00:00,  1.12s/it]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 106/7971 [00:00<00:07, 1050.41it/s]
  3%|▎         | 212/7971 [00:00<00:07, 1052.98it/s]
  4%|▍         | 326/7971 [00:00<00:07, 1085.09it/s]
  5%|▌         | 435/7971 [00:00<00:07, 1049.20it/s]
  7%|▋         | 544/7971 [00:00<00:07, 1057.75it/s]
  8%|▊         | 651/7971 [00:00<00:06, 1060.68it/s]
 10%|▉         | 758/7971 [00:00<00:06, 1056.79it/s]
 11%|█         | 865/7971 [00:00<00:06, 1054.93it/s]
 12%|█▏        | 978/7971 [00:00<00:06, 1075.62it/s]
 14%|█▎        | 1086/7971 [00:01<00:06, 1071.67it/s]
 15%|█▍        | 1194/7971 [00:01<00:06, 1059.70it/s]
 16%|█▋        | 1301/7971 [00:01<00:06, 1053.06it/s]
 18%|█▊        | 1407/7971 [00:01<00:06, 1047.85it/s]
 19%|█▉        | 1525/7971 [00:01<00:05, 1086.25it/s]
 20%|██        | 1634/7971 [00:01<00:05, 1081.99it/s]
 22%|██▏       | 1743/7971 [00:01<00:05, 1079.80it/s]
 23%|██▎       | 1854/7971 [00:01<00:05, 1084.09it/s]
 25%|██▍       | 1963/7971 [00:01<00:05, 1077.51it/s]
 26%|██▌       | 2071/7971 [00:01<00:05, 1071.57it/s]
 27%|██▋       | 2182/7971 [00:02<00:05, 1082.83it/s]
 29%|██▊       | 2291/7971 [00:02<00:05, 1080.03it/s]
 30%|███       | 2400/7971 [00:02<00:05, 1077.25it/s]
 32%|███▏      | 2513/7971 [00:02<00:05, 1086.74it/s]
 33%|███▎      | 2622/7971 [00:02<00:05, 1069.05it/s]
 34%|███▍      | 2733/7971 [00:02<00:04, 1075.03it/s]
 36%|███▌      | 2841/7971 [00:02<00:04, 1064.65it/s]
 37%|███▋      | 2948/7971 [00:02<00:04, 1057.30it/s]
 38%|███▊      | 3054/7971 [00:02<00:04, 1054.53it/s]
 40%|███▉      | 3160/7971 [00:02<00:04, 1045.49it/s]
 41%|████      | 3265/7971 [00:03<00:04, 1022.26it/s]
 42%|████▏     | 3375/7971 [00:03<00:04, 1040.06it/s]
 44%|████▎     | 3480/7971 [00:03<00:04, 1037.38it/s]
 45%|████▌     | 3601/7971 [00:03<00:04, 1083.38it/s]
 47%|████▋     | 3710/7971 [00:03<00:04, 1061.21it/s]
 48%|████▊     | 3817/7971 [00:03<00:03, 1052.73it/s]
 49%|████▉     | 3925/7971 [00:03<00:03, 1059.16it/s]
 51%|█████     | 4037/7971 [00:03<00:03, 1074.66it/s]
 52%|█████▏    | 4145/7971 [00:03<00:03, 1061.97it/s]
 53%|█████▎    | 4252/7971 [00:03<00:03, 1061.54it/s]
 55%|█████▍    | 4367/7971 [00:04<00:03, 1087.59it/s]
 56%|█████▌    | 4476/7971 [00:04<00:03, 1063.28it/s]
 57%|█████▋    | 4583/7971 [00:04<00:03, 1052.92it/s]
 59%|█████▉    | 4689/7971 [00:04<00:03, 1042.51it/s]
 60%|██████    | 4797/7971 [00:04<00:03, 1048.61it/s]
 61%|██████▏   | 4902/7971 [00:04<00:02, 1047.50it/s]
 63%|██████▎   | 5007/7971 [00:04<00:02, 1045.97it/s]
 64%|██████▍   | 5114/7971 [00:04<00:02, 1052.05it/s]
 65%|██████▌   | 5220/7971 [00:04<00:02, 1037.01it/s]
 67%|██████▋   | 5326/7971 [00:05<00:02, 1036.95it/s]
 68%|██████▊   | 5437/7971 [00:05<00:02, 1055.70it/s]
 70%|██████▉   | 5543/7971 [00:05<00:02, 1047.21it/s]
 71%|███████   | 5649/7971 [00:05<00:02, 1048.11it/s]
 72%|███████▏  | 5754/7971 [00:05<00:02, 1045.70it/s]
 74%|███████▎  | 5866/7971 [00:05<00:01, 1063.31it/s]
 75%|███████▍  | 5973/7971 [00:05<00:01, 1046.92it/s]
 76%|███████▋  | 6079/7971 [00:05<00:01, 1048.91it/s]
 78%|███████▊  | 6184/7971 [00:05<00:01, 1048.31it/s]
 79%|███████▉  | 6289/7971 [00:05<00:01, 1022.99it/s]
 80%|████████  | 6392/7971 [00:06<00:01, 1017.93it/s]
 81%|████████▏ | 6494/7971 [00:06<00:01, 1010.39it/s]
 83%|████████▎ | 6596/7971 [00:06<00:01, 1012.65it/s]
 84%|████████▍ | 6707/7971 [00:06<00:01, 1036.00it/s]
 85%|████████▌ | 6811/7971 [00:06<00:01, 1028.75it/s]
 87%|████████▋ | 6917/7971 [00:06<00:01, 1031.81it/s]
 88%|████████▊ | 7022/7971 [00:06<00:00, 1036.51it/s]
 89%|████████▉ | 7126/7971 [00:06<00:00, 1025.48it/s]
 91%|█████████ | 7235/7971 [00:06<00:00, 1042.77it/s]
 92%|█████████▏| 7340/7971 [00:06<00:00, 1012.73it/s]
 93%|█████████▎| 7442/7971 [00:07<00:00, 997.20it/s] 
 95%|█████████▍| 7544/7971 [00:07<00:00, 1003.28it/s]
 96%|█████████▌| 7652/7971 [00:07<00:00, 1023.19it/s]
 97%|█████████▋| 7755/7971 [00:07<00:00, 1005.92it/s]
 99%|█████████▊| 7865/7971 [00:07<00:00, 1033.32it/s]
100%|█████████▉| 7969/7971 [00:07<00:00, 994.46it/s] 
100%|██████████| 7971/7971 [00:07<00:00, 1048.52it/s]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 106/7971 [00:00<00:07, 1050.36it/s]
  3%|▎         | 218/7971 [00:00<00:07, 1085.76it/s]
  4%|▍         | 337/7971 [00:00<00:06, 1125.42it/s]
  6%|▌         | 450/7971 [00:00<00:06, 1095.79it/s]
  7%|▋         | 563/7971 [00:00<00:06, 1103.66it/s]
  8%|▊         | 674/7971 [00:00<00:06, 1094.92it/s]
 10%|▉         | 787/7971 [00:00<00:06, 1100.30it/s]
 11%|█▏        | 904/7971 [00:00<00:06, 1115.25it/s]
 13%|█▎        | 1024/7971 [00:00<00:06, 1133.91it/s]
 14%|█▍        | 1138/7971 [00:01<00:06, 1109.37it/s]
 16%|█▌        | 1250/7971 [00:01<00:06, 1099.74it/s]
 17%|█▋        | 1361/7971 [00:01<00:06, 1077.77it/s]
 19%|█▊        | 1478/7971 [00:01<00:05, 1103.17it/s]
 20%|█▉        | 1592/7971 [00:01<00:05, 1113.56it/s]
 21%|██▏       | 1712/7971 [00:01<00:05, 1131.89it/s]
 23%|██▎       | 1826/7971 [00:01<00:05, 1110.77it/s]
 24%|██▍       | 1939/7971 [00:01<00:05, 1115.38it/s]
 26%|██▌       | 2051/7971 [00:01<00:05, 1110.33it/s]
 27%|██▋       | 2163/7971 [00:01<00:05, 1110.39it/s]
 29%|██▊       | 2275/7971 [00:02<00:05, 1084.93it/s]
 30%|██▉       | 2384/7971 [00:02<00:05, 1071.13it/s]
 31%|███▏      | 2492/7971 [00:02<00:05, 1064.25it/s]
 33%|███▎      | 2600/7971 [00:02<00:05, 1063.19it/s]
 34%|███▍      | 2710/7971 [00:02<00:04, 1072.31it/s]
 35%|███▌      | 2818/7971 [00:02<00:04, 1061.81it/s]
 37%|███▋      | 2925/7971 [00:02<00:04, 1053.49it/s]
 38%|███▊      | 3031/7971 [00:02<00:04, 1045.87it/s]
 39%|███▉      | 3136/7971 [00:02<00:04, 1028.14it/s]
 41%|████      | 3239/7971 [00:02<00:04, 1001.53it/s]
 42%|████▏     | 3344/7971 [00:03<00:04, 1011.31it/s]
 43%|████▎     | 3446/7971 [00:03<00:04, 1001.66it/s]
 45%|████▍     | 3556/7971 [00:03<00:04, 1030.21it/s]
 46%|████▌     | 3660/7971 [00:03<00:04, 1018.11it/s]
 47%|████▋     | 3762/7971 [00:03<00:04, 1005.53it/s]
 48%|████▊     | 3864/7971 [00:03<00:04, 1004.01it/s]
 50%|████▉     | 3975/7971 [00:03<00:03, 1034.95it/s]
 51%|█████     | 4082/7971 [00:03<00:03, 1045.19it/s]
 53%|█████▎    | 4188/7971 [00:03<00:03, 1048.44it/s]
 54%|█████▍    | 4297/7971 [00:04<00:03, 1059.09it/s]
 55%|█████▌    | 4410/7971 [00:04<00:03, 1079.30it/s]
 57%|█████▋    | 4518/7971 [00:04<00:03, 1040.87it/s]
 58%|█████▊    | 4625/7971 [00:04<00:03, 1048.60it/s]
 59%|█████▉    | 4731/7971 [00:04<00:03, 1019.71it/s]
 61%|██████    | 4847/7971 [00:04<00:02, 1053.31it/s]
 62%|██████▏   | 4953/7971 [00:04<00:02, 1030.78it/s]
 64%|██████▎   | 5063/7971 [00:04<00:02, 1050.52it/s]
 65%|██████▍   | 5169/7971 [00:04<00:02, 1034.73it/s]
 66%|██████▌   | 5276/7971 [00:04<00:02, 1041.90it/s]
 68%|██████▊   | 5381/7971 [00:05<00:02, 1036.01it/s]
 69%|██████▉   | 5494/7971 [00:05<00:02, 1057.52it/s]
 70%|███████   | 5600/7971 [00:05<00:02, 1046.13it/s]
 72%|███████▏  | 5705/7971 [00:05<00:02, 1043.97it/s]
 73%|███████▎  | 5816/7971 [00:05<00:02, 1061.45it/s]
 74%|███████▍  | 5923/7971 [00:05<00:01, 1060.69it/s]
 76%|███████▌  | 6033/7971 [00:05<00:01, 1067.27it/s]
 77%|███████▋  | 6140/7971 [00:05<00:01, 1056.95it/s]
 78%|███████▊  | 6246/7971 [00:05<00:01, 1050.43it/s]
 80%|███████▉  | 6352/7971 [00:05<00:01, 1039.17it/s]
 81%|████████  | 6456/7971 [00:06<00:01, 1015.94it/s]
 82%|████████▏ | 6558/7971 [00:06<00:01, 1016.62it/s]
 84%|████████▎ | 6673/7971 [00:06<00:01, 1053.73it/s]
 85%|████████▌ | 6779/7971 [00:06<00:01, 1036.46it/s]
 86%|████████▋ | 6886/7971 [00:06<00:01, 1045.63it/s]
 88%|████████▊ | 6991/7971 [00:06<00:00, 1028.50it/s]
 89%|████████▉ | 7095/7971 [00:06<00:00, 1025.42it/s]
 90%|█████████ | 7198/7971 [00:06<00:00, 1023.55it/s]
 92%|█████████▏| 7303/7971 [00:06<00:00, 1027.89it/s]
 93%|█████████▎| 7406/7971 [00:07<00:00, 998.87it/s] 
 94%|█████████▍| 7507/7971 [00:07<00:00, 983.65it/s]
 95%|█████████▌| 7606/7971 [00:07<00:00, 978.17it/s]
 97%|█████████▋| 7704/7971 [00:07<00:00, 965.37it/s]
 98%|█████████▊| 7811/7971 [00:07<00:00, 995.68it/s]
 99%|█████████▉| 7911/7971 [00:07<00:00, 973.35it/s]
100%|██████████| 7971/7971 [00:07<00:00, 1048.21it/s]
2026-05-11 04:22:21.830564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:18.722765: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:25:18.724256: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:25:19.027588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:25:19.027775: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:19.045754: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:19.045841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:19.055983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:25:19.064337: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:25:19.074677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:25:19.083462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:25:19.091268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:19.097946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:25:19.098843: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:25:19.099076: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:25:19.102916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:07:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:25:19.102968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:19.103000: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:19.103026: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:19.103050: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:25:19.103074: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:25:19.103097: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:25:19.103120: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:25:19.103143: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:19.110686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:25:19.110746: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:19.598537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:25:19.598685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:25:19.598711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:25:19.602800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0)
2026-05-11 04:25:19.651450: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:25:19.663994: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:25:26.995984: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:27.466422: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:27.473208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:29.050346: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:25:29.160958: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:26:02.630411: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)