Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:36:38.108543: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:38.395039: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:38.398036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:51:38.720701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:38.720811: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:39.027815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:39.027944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:39.171459: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:39.813841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:41.748966: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.243241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.460506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.495196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.495726: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:51:42.495919: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:42.512997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:42.513053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:42.513081: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:42.513100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:42.513117: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:42.513133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:42.513149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.513164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.513199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.549591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.549775: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:45.433285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:51:45.433455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:51:45.433472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:51:45.438006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:84:00.0, compute capability: 8.0)
2026-05-11 03:51:49.579996: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:51:49.580489: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 03:51:54.989312: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:58.751406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:58.757482: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:10.712512: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:53:10.858789: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:54:46.215975: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 04:07:43.525398: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 04:10:15.056853: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:11.900616: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:16:11.905628: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:16:12.110769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:16:12.110905: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:12.134175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:12.134301: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:12.146299: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:16:12.156219: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:16:12.170787: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:16:12.181683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:16:12.191784: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:12.199499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:16:12.199932: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:16:12.200137: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:16:12.204081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:16:12.204145: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:12.204171: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:12.204186: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:12.204199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:16:12.204212: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:16:12.204225: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:16:12.204238: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:16:12.204251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:12.209070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:16:12.209108: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:12.667242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:16:12.667390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:16:12.667404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:16:12.672200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:84:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/126 [00:00<?, ?it/s]2026-05-11 04:16:14.586982: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 04:16:14.587605: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:16:14.912812: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:15.396690: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:15.398845: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:16.940588: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:16:17.067157: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/126 [01:59<4:08:16, 119.17s/it]
batch:   2%|▏         | 2/126 [01:59<1:41:44, 49.23s/it] 
batch:   2%|▏         | 3/126 [01:59<55:05, 26.87s/it]  
batch:   3%|▎         | 4/126 [01:59<33:16, 16.37s/it]
batch:   4%|▍         | 5/126 [02:00<21:17, 10.56s/it]
batch:   5%|▍         | 6/126 [02:00<14:07,  7.06s/it]
batch:   6%|▌         | 7/126 [02:00<09:36,  4.84s/it]
batch:   6%|▋         | 8/126 [02:01<06:39,  3.39s/it]
batch:   7%|▋         | 9/126 [02:01<04:42,  2.41s/it]
batch:   8%|▊         | 10/126 [02:01<03:23,  1.75s/it]
batch:   9%|▊         | 11/126 [02:01<02:29,  1.30s/it]
batch:  10%|▉         | 12/126 [02:02<01:52,  1.02it/s]
batch:  10%|█         | 13/126 [02:02<01:26,  1.30it/s]
batch:  11%|█         | 14/126 [02:02<01:09,  1.62it/s]
batch:  12%|█▏        | 15/126 [02:02<00:56,  1.95it/s]
batch:  13%|█▎        | 16/126 [02:03<00:48,  2.27it/s]
batch:  13%|█▎        | 17/126 [02:03<00:42,  2.57it/s]
batch:  14%|█▍        | 18/126 [02:03<00:38,  2.84it/s]
batch:  15%|█▌        | 19/126 [02:04<00:37,  2.88it/s]
batch:  16%|█▌        | 20/126 [02:04<00:34,  3.08it/s]
batch:  17%|█▋        | 21/126 [02:04<00:32,  3.24it/s]
batch:  17%|█▋        | 22/126 [02:04<00:31,  3.34it/s]
batch:  18%|█▊        | 23/126 [02:05<00:29,  3.46it/s]
batch:  19%|█▉        | 24/126 [02:05<00:28,  3.52it/s]
batch:  20%|█▉        | 25/126 [02:05<00:28,  3.56it/s]
batch:  21%|██        | 26/126 [02:05<00:27,  3.61it/s]
batch:  21%|██▏       | 27/126 [02:06<00:27,  3.64it/s]
batch:  22%|██▏       | 28/126 [02:06<00:26,  3.65it/s]
batch:  23%|██▎       | 29/126 [02:06<00:26,  3.68it/s]
batch:  24%|██▍       | 30/126 [02:07<00:26,  3.66it/s]
batch:  25%|██▍       | 31/126 [02:07<00:25,  3.67it/s]
batch:  25%|██▌       | 32/126 [02:07<00:25,  3.68it/s]
batch:  26%|██▌       | 33/126 [02:07<00:25,  3.68it/s]
batch:  27%|██▋       | 34/126 [02:08<00:24,  3.70it/s]
batch:  28%|██▊       | 35/126 [02:08<00:24,  3.69it/s]
batch:  29%|██▊       | 36/126 [02:08<00:24,  3.70it/s]
batch:  29%|██▉       | 37/126 [02:08<00:24,  3.69it/s]
batch:  30%|███       | 38/126 [02:09<00:23,  3.70it/s]
batch:  31%|███       | 39/126 [02:09<00:23,  3.67it/s]
batch:  32%|███▏      | 40/126 [02:09<00:23,  3.66it/s]
batch:  33%|███▎      | 41/126 [02:10<00:23,  3.69it/s]
batch:  33%|███▎      | 42/126 [02:10<00:22,  3.69it/s]
batch:  34%|███▍      | 43/126 [02:10<00:22,  3.69it/s]
batch:  35%|███▍      | 44/126 [02:10<00:22,  3.65it/s]
batch:  36%|███▌      | 45/126 [02:11<00:22,  3.66it/s]
batch:  37%|███▋      | 46/126 [02:11<00:21,  3.66it/s]
batch:  37%|███▋      | 47/126 [02:11<00:21,  3.69it/s]
batch:  38%|███▊      | 48/126 [02:11<00:21,  3.70it/s]
batch:  39%|███▉      | 49/126 [02:12<00:20,  3.70it/s]
batch:  40%|███▉      | 50/126 [02:12<00:20,  3.70it/s]
batch:  40%|████      | 51/126 [02:12<00:20,  3.70it/s]
batch:  41%|████▏     | 52/126 [02:13<00:19,  3.71it/s]
batch:  42%|████▏     | 53/126 [02:13<00:19,  3.71it/s]
batch:  43%|████▎     | 54/126 [02:13<00:19,  3.70it/s]
batch:  44%|████▎     | 55/126 [02:13<00:19,  3.70it/s]
batch:  44%|████▍     | 56/126 [02:14<00:18,  3.69it/s]
batch:  45%|████▌     | 57/126 [02:14<00:18,  3.69it/s]
batch:  46%|████▌     | 58/126 [02:14<00:18,  3.70it/s]
batch:  47%|████▋     | 59/126 [02:14<00:18,  3.69it/s]
batch:  48%|████▊     | 60/126 [02:15<00:17,  3.72it/s]
batch:  48%|████▊     | 61/126 [02:15<00:17,  3.70it/s]
batch:  49%|████▉     | 62/126 [02:15<00:17,  3.69it/s]
batch:  50%|█████     | 63/126 [02:16<00:17,  3.70it/s]
batch:  51%|█████     | 64/126 [02:16<00:16,  3.72it/s]
batch:  52%|█████▏    | 65/126 [02:16<00:16,  3.73it/s]
batch:  52%|█████▏    | 66/126 [02:16<00:18,  3.31it/s]
batch:  53%|█████▎    | 67/126 [02:17<00:17,  3.41it/s]
batch:  54%|█████▍    | 68/126 [02:17<00:16,  3.49it/s]
batch:  55%|█████▍    | 69/126 [02:17<00:15,  3.57it/s]
batch:  56%|█████▌    | 70/126 [02:18<00:15,  3.60it/s]
batch:  56%|█████▋    | 71/126 [02:18<00:15,  3.64it/s]
batch:  57%|█████▋    | 72/126 [02:18<00:14,  3.67it/s]
batch:  58%|█████▊    | 73/126 [02:18<00:14,  3.71it/s]
batch:  59%|█████▊    | 74/126 [02:19<00:13,  3.73it/s]
batch:  60%|█████▉    | 75/126 [02:19<00:13,  3.74it/s]
batch:  60%|██████    | 76/126 [02:19<00:13,  3.72it/s]
batch:  61%|██████    | 77/126 [02:19<00:13,  3.66it/s]
batch:  62%|██████▏   | 78/126 [02:20<00:13,  3.67it/s]
batch:  63%|██████▎   | 79/126 [02:20<00:12,  3.66it/s]
batch:  63%|██████▎   | 80/126 [02:20<00:12,  3.67it/s]
batch:  64%|██████▍   | 81/126 [02:20<00:12,  3.66it/s]
batch:  65%|██████▌   | 82/126 [02:21<00:11,  3.67it/s]
batch:  66%|██████▌   | 83/126 [02:21<00:11,  3.67it/s]
batch:  67%|██████▋   | 84/126 [02:21<00:11,  3.68it/s]
batch:  67%|██████▋   | 85/126 [02:22<00:11,  3.66it/s]
batch:  68%|██████▊   | 86/126 [02:22<00:10,  3.69it/s]
batch:  69%|██████▉   | 87/126 [02:22<00:10,  3.70it/s]
batch:  70%|██████▉   | 88/126 [02:22<00:10,  3.68it/s]
batch:  71%|███████   | 89/126 [02:23<00:09,  3.71it/s]
batch:  71%|███████▏  | 90/126 [02:23<00:09,  3.70it/s]
batch:  72%|███████▏  | 91/126 [02:23<00:09,  3.71it/s]
batch:  73%|███████▎  | 92/126 [02:23<00:09,  3.70it/s]
batch:  74%|███████▍  | 93/126 [02:24<00:08,  3.68it/s]
batch:  75%|███████▍  | 94/126 [02:24<00:08,  3.70it/s]
batch:  75%|███████▌  | 95/126 [02:24<00:08,  3.72it/s]
batch:  76%|███████▌  | 96/126 [02:25<00:08,  3.72it/s]
batch:  77%|███████▋  | 97/126 [02:25<00:07,  3.74it/s]
batch:  78%|███████▊  | 98/126 [02:25<00:07,  3.72it/s]
batch:  79%|███████▊  | 99/126 [02:25<00:07,  3.74it/s]
batch:  79%|███████▉  | 100/126 [02:26<00:06,  3.74it/s]
batch:  80%|████████  | 101/126 [02:26<00:06,  3.74it/s]
batch:  81%|████████  | 102/126 [02:26<00:06,  3.73it/s]
batch:  82%|████████▏ | 103/126 [02:26<00:06,  3.73it/s]
batch:  83%|████████▎ | 104/126 [02:27<00:05,  3.74it/s]
batch:  83%|████████▎ | 105/126 [02:27<00:05,  3.72it/s]
batch:  84%|████████▍ | 106/126 [02:27<00:05,  3.72it/s]
batch:  85%|████████▍ | 107/126 [02:27<00:05,  3.73it/s]
batch:  86%|████████▌ | 108/126 [02:28<00:04,  3.68it/s]
batch:  87%|████████▋ | 109/126 [02:28<00:04,  3.70it/s]
batch:  87%|████████▋ | 110/126 [02:28<00:04,  3.68it/s]
batch:  88%|████████▊ | 111/126 [02:29<00:04,  3.70it/s]
batch:  89%|████████▉ | 112/126 [02:29<00:03,  3.71it/s]
batch:  90%|████████▉ | 113/126 [02:29<00:03,  3.71it/s]
batch:  90%|█████████ | 114/126 [02:29<00:03,  3.74it/s]
batch:  91%|█████████▏| 115/126 [02:30<00:02,  3.71it/s]
batch:  92%|█████████▏| 116/126 [02:30<00:02,  3.71it/s]
batch:  93%|█████████▎| 117/126 [02:30<00:02,  3.69it/s]
batch:  94%|█████████▎| 118/126 [02:30<00:02,  3.71it/s]
batch:  94%|█████████▍| 119/126 [02:31<00:01,  3.69it/s]
batch:  95%|█████████▌| 120/126 [02:31<00:01,  3.70it/s]
batch:  96%|█████████▌| 121/126 [02:31<00:01,  3.73it/s]
batch:  97%|█████████▋| 122/126 [02:32<00:01,  3.72it/s]
batch:  98%|█████████▊| 123/126 [02:32<00:00,  3.73it/s]
batch:  98%|█████████▊| 124/126 [02:32<00:00,  3.71it/s]
batch:  99%|█████████▉| 125/126 [02:32<00:00,  4.17it/s]
batch: 100%|██████████| 126/126 [02:32<00:00,  1.21s/it]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 114/7971 [00:00<00:06, 1139.10it/s]
  3%|▎         | 233/7971 [00:00<00:06, 1166.58it/s]
  4%|▍         | 356/7971 [00:00<00:06, 1193.90it/s]
  6%|▌         | 476/7971 [00:00<00:06, 1172.98it/s]
  7%|▋         | 597/7971 [00:00<00:06, 1181.51it/s]
  9%|▉         | 716/7971 [00:00<00:06, 1179.25it/s]
 10%|█         | 834/7971 [00:00<00:06, 1174.74it/s]
 12%|█▏        | 961/7971 [00:00<00:05, 1198.44it/s]
 14%|█▎        | 1081/7971 [00:00<00:05, 1193.03it/s]
 15%|█▌        | 1201/7971 [00:01<00:05, 1174.05it/s]
 17%|█▋        | 1319/7971 [00:01<00:05, 1156.52it/s]
 18%|█▊        | 1438/7971 [00:01<00:05, 1166.16it/s]
 20%|█▉        | 1566/7971 [00:01<00:05, 1195.95it/s]
 21%|██        | 1689/7971 [00:01<00:05, 1201.84it/s]
 23%|██▎       | 1810/7971 [00:01<00:05, 1179.85it/s]
 24%|██▍       | 1930/7971 [00:01<00:05, 1181.44it/s]
 26%|██▌       | 2049/7971 [00:01<00:05, 1182.61it/s]
 27%|██▋       | 2168/7971 [00:01<00:04, 1181.81it/s]
 29%|██▊       | 2287/7971 [00:01<00:04, 1176.50it/s]
 30%|███       | 2405/7971 [00:02<00:04, 1158.88it/s]
 32%|███▏      | 2521/7971 [00:02<00:04, 1155.56it/s]
 33%|███▎      | 2637/7971 [00:02<00:04, 1142.37it/s]
 35%|███▍      | 2752/7971 [00:02<00:04, 1133.75it/s]
 36%|███▌      | 2866/7971 [00:02<00:04, 1126.84it/s]
 37%|███▋      | 2980/7971 [00:02<00:04, 1125.10it/s]
 39%|███▉      | 3093/7971 [00:02<00:04, 1124.38it/s]
 40%|████      | 3206/7971 [00:02<00:04, 1105.54it/s]
 42%|████▏     | 3320/7971 [00:02<00:04, 1110.41it/s]
 43%|████▎     | 3432/7971 [00:02<00:04, 1110.86it/s]
 45%|████▍     | 3553/7971 [00:03<00:03, 1136.86it/s]
 46%|████▌     | 3667/7971 [00:03<00:03, 1126.73it/s]
 47%|████▋     | 3780/7971 [00:03<00:03, 1111.99it/s]
 49%|████▉     | 3892/7971 [00:03<00:03, 1102.57it/s]
 50%|█████     | 4008/7971 [00:03<00:03, 1116.98it/s]
 52%|█████▏    | 4120/7971 [00:03<00:03, 1116.49it/s]
 53%|█████▎    | 4232/7971 [00:03<00:03, 1112.42it/s]
 55%|█████▍    | 4350/7971 [00:03<00:03, 1132.15it/s]
 56%|█████▌    | 4464/7971 [00:03<00:03, 1110.24it/s]
 57%|█████▋    | 4576/7971 [00:03<00:03, 1110.17it/s]
 59%|█████▉    | 4688/7971 [00:04<00:02, 1112.32it/s]
 60%|██████    | 4807/7971 [00:04<00:02, 1134.57it/s]
 62%|██████▏   | 4921/7971 [00:04<00:02, 1128.19it/s]
 63%|██████▎   | 5036/7971 [00:04<00:02, 1130.79it/s]
 65%|██████▍   | 5150/7971 [00:04<00:02, 1131.82it/s]
 66%|██████▌   | 5267/7971 [00:04<00:02, 1138.42it/s]
 68%|██████▊   | 5381/7971 [00:04<00:02, 1122.93it/s]
 69%|██████▉   | 5499/7971 [00:04<00:02, 1134.36it/s]
 70%|███████   | 5613/7971 [00:04<00:02, 1133.66it/s]
 72%|███████▏  | 5727/7971 [00:05<00:02, 1119.67it/s]
 73%|███████▎  | 5848/7971 [00:05<00:01, 1145.78it/s]
 75%|███████▍  | 5963/7971 [00:05<00:01, 1133.89it/s]
 76%|███████▌  | 6077/7971 [00:05<00:01, 1130.67it/s]
 78%|███████▊  | 6191/7971 [00:05<00:01, 1127.17it/s]
 79%|███████▉  | 6304/7971 [00:05<00:01, 1107.86it/s]
 80%|████████  | 6415/7971 [00:05<00:01, 1089.44it/s]
 82%|████████▏ | 6525/7971 [00:05<00:01, 1085.20it/s]
 83%|████████▎ | 6643/7971 [00:05<00:01, 1111.73it/s]
 85%|████████▍ | 6755/7971 [00:05<00:01, 1113.59it/s]
 86%|████████▌ | 6867/7971 [00:06<00:00, 1106.87it/s]
 88%|████████▊ | 6981/7971 [00:06<00:00, 1116.39it/s]
 89%|████████▉ | 7093/7971 [00:06<00:00, 1105.47it/s]
 90%|█████████ | 7204/7971 [00:06<00:00, 1105.17it/s]
 92%|█████████▏| 7315/7971 [00:06<00:00, 1091.87it/s]
 93%|█████████▎| 7425/7971 [00:06<00:00, 1071.78it/s]
 95%|█████████▍| 7533/7971 [00:06<00:00, 1070.32it/s]
 96%|█████████▌| 7642/7971 [00:06<00:00, 1072.14it/s]
 97%|█████████▋| 7750/7971 [00:06<00:00, 1067.99it/s]
 99%|█████████▊| 7867/7971 [00:06<00:00, 1095.00it/s]
100%|██████████| 7971/7971 [00:07<00:00, 1129.66it/s]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 111/7971 [00:00<00:07, 1096.70it/s]
  3%|▎         | 230/7971 [00:00<00:06, 1148.22it/s]
  4%|▍         | 354/7971 [00:00<00:06, 1186.53it/s]
  6%|▌         | 473/7971 [00:00<00:06, 1166.67it/s]
  7%|▋         | 594/7971 [00:00<00:06, 1177.76it/s]
  9%|▉         | 712/7971 [00:00<00:06, 1175.81it/s]
 10%|█         | 830/7971 [00:00<00:06, 1172.45it/s]
 12%|█▏        | 958/7971 [00:00<00:05, 1199.05it/s]
 14%|█▎        | 1078/7971 [00:00<00:05, 1193.64it/s]
 15%|█▌        | 1198/7971 [00:01<00:05, 1174.13it/s]
 17%|█▋        | 1316/7971 [00:01<00:05, 1156.36it/s]
 18%|█▊        | 1435/7971 [00:01<00:05, 1166.02it/s]
 20%|█▉        | 1562/7971 [00:01<00:05, 1193.59it/s]
 21%|██        | 1687/7971 [00:01<00:05, 1204.96it/s]
 23%|██▎       | 1808/7971 [00:01<00:05, 1181.92it/s]
 24%|██▍       | 1927/7971 [00:01<00:05, 1183.03it/s]
 26%|██▌       | 2046/7971 [00:01<00:05, 1183.73it/s]
 27%|██▋       | 2165/7971 [00:01<00:04, 1183.52it/s]
 29%|██▊       | 2284/7971 [00:01<00:04, 1177.01it/s]
 30%|███       | 2402/7971 [00:02<00:04, 1160.12it/s]
 32%|███▏      | 2519/7971 [00:02<00:04, 1153.13it/s]
 33%|███▎      | 2635/7971 [00:02<00:04, 1146.83it/s]
 35%|███▍      | 2750/7971 [00:02<00:04, 1146.33it/s]
 36%|███▌      | 2865/7971 [00:02<00:04, 1137.35it/s]
 37%|███▋      | 2980/7971 [00:02<00:04, 1134.89it/s]
 39%|███▉      | 3094/7971 [00:02<00:04, 1133.40it/s]
 40%|████      | 3208/7971 [00:02<00:04, 1110.76it/s]
 42%|████▏     | 3323/7971 [00:02<00:04, 1117.09it/s]
 43%|████▎     | 3435/7971 [00:02<00:04, 1113.82it/s]
 45%|████▍     | 3556/7971 [00:03<00:03, 1139.86it/s]
 46%|████▌     | 3671/7971 [00:03<00:03, 1130.71it/s]
 47%|████▋     | 3785/7971 [00:03<00:03, 1109.83it/s]
 49%|████▉     | 3897/7971 [00:03<00:03, 1101.82it/s]
 50%|█████     | 4013/7971 [00:03<00:03, 1117.52it/s]
 52%|█████▏    | 4127/7971 [00:03<00:03, 1117.65it/s]
 53%|█████▎    | 4242/7971 [00:03<00:03, 1123.48it/s]
 55%|█████▍    | 4366/7971 [00:03<00:03, 1151.13it/s]
 56%|█████▌    | 4482/7971 [00:03<00:03, 1134.11it/s]
 58%|█████▊    | 4596/7971 [00:03<00:02, 1131.98it/s]
 59%|█████▉    | 4710/7971 [00:04<00:02, 1118.15it/s]
 61%|██████    | 4836/7971 [00:04<00:02, 1155.94it/s]
 62%|██████▏   | 4952/7971 [00:04<00:02, 1131.81it/s]
 64%|██████▎   | 5070/7971 [00:04<00:02, 1141.18it/s]
 65%|██████▌   | 5185/7971 [00:04<00:02, 1126.33it/s]
 67%|██████▋   | 5301/7971 [00:04<00:02, 1131.45it/s]
 68%|██████▊   | 5415/7971 [00:04<00:02, 1133.71it/s]
 69%|██████▉   | 5529/7971 [00:04<00:02, 1132.84it/s]
 71%|███████   | 5644/7971 [00:04<00:02, 1136.17it/s]
 72%|███████▏  | 5758/7971 [00:05<00:01, 1129.09it/s]
 74%|███████▎  | 5877/7971 [00:05<00:01, 1147.00it/s]
 75%|███████▌  | 5992/7971 [00:05<00:01, 1135.53it/s]
 77%|███████▋  | 6106/7971 [00:05<00:01, 1132.39it/s]
 78%|███████▊  | 6220/7971 [00:05<00:01, 1123.22it/s]
 79%|███████▉  | 6333/7971 [00:05<00:01, 1103.56it/s]
 81%|████████  | 6444/7971 [00:05<00:01, 1086.04it/s]
 82%|████████▏ | 6556/7971 [00:05<00:01, 1089.50it/s]
 84%|████████▍ | 6676/7971 [00:05<00:01, 1121.72it/s]
 85%|████████▌ | 6789/7971 [00:05<00:01, 1109.01it/s]
 87%|████████▋ | 6903/7971 [00:06<00:00, 1116.61it/s]
 88%|████████▊ | 7015/7971 [00:06<00:00, 1111.56it/s]
 89%|████████▉ | 7127/7971 [00:06<00:00, 1108.41it/s]
 91%|█████████ | 7241/7971 [00:06<00:00, 1116.78it/s]
 92%|█████████▏| 7353/7971 [00:06<00:00, 1093.27it/s]
 94%|█████████▎| 7463/7971 [00:06<00:00, 1073.21it/s]
 95%|█████████▍| 7571/7971 [00:06<00:00, 1073.52it/s]
 96%|█████████▋| 7681/7971 [00:06<00:00, 1079.27it/s]
 98%|█████████▊| 7794/7971 [00:06<00:00, 1093.95it/s]
 99%|█████████▉| 7904/7971 [00:06<00:00, 1092.19it/s]
100%|██████████| 7971/7971 [00:07<00:00, 1133.02it/s]
2026-05-11 04:22:12.330342: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:13.300464: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:25:13.302055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:25:13.506418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:25:13.506605: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:13.520229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:13.520422: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:13.526187: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:25:13.530817: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:25:13.538334: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:25:13.544222: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:25:13.549539: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:13.554708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:25:13.555577: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:25:13.555870: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:25:13.558401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:84:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:25:13.558449: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:13.558477: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:13.558502: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:13.558524: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:25:13.558546: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:25:13.558568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:25:13.558588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:25:13.558609: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:13.565257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:25:13.565304: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:14.016997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:25:14.017149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:25:14.017163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:25:14.021193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:84:00.0, compute capability: 8.0)
2026-05-11 04:25:14.068166: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:25:14.080247: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:25:20.795531: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:21.220001: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:21.225014: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:22.717855: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:25:22.839303: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:25:57.205622: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)