Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:36:38.106732: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:26.487142: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:26.499883: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:51:26.875031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:26.875138: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:38.888407: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:38.888583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:39.171093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:39.813821: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:41.748452: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.243477: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.481021: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.514936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.515574: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:51:42.515913: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:42.522836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:42.522913: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:42.522946: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:42.522974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:42.522999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:42.523024: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:42.523048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.523068: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.523125: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.572359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.572472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:45.590872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:51:45.591067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:51:45.591087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:51:45.595162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2026-05-11 03:51:49.770646: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:51:49.771214: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 03:51:56.818153: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:59.333082: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:59.366249: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:10.699424: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:53:10.823087: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:54:41.614415: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 04:09:40.667701: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 04:12:16.678250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:46.657397: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:16:46.658905: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:16:46.909258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:16:46.909394: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:46.951117: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:46.951248: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:46.973772: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:16:46.994099: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:16:47.017505: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:16:47.034938: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:16:47.052164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:47.060197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:16:47.060715: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:16:47.061034: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:16:47.063655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:16:47.063747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:47.063792: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:47.063825: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:47.063855: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:16:47.063886: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:16:47.063916: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:16:47.063944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:16:47.063974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:47.068840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:16:47.068883: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:16:47.542935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:16:47.543120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:16:47.543138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:16:47.547190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/126 [00:00<?, ?it/s]2026-05-11 04:16:49.666035: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 04:16:49.666662: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:16:50.006383: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:16:50.486738: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:16:50.488758: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:16:52.266893: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:16:52.412041: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/126 [02:04<4:20:06, 124.85s/it]
batch:   2%|▏         | 2/126 [02:05<1:46:37, 51.59s/it] 
batch:   2%|▏         | 3/126 [02:05<57:44, 28.17s/it]  
batch:   3%|▎         | 4/126 [02:05<34:53, 17.16s/it]
batch:   4%|▍         | 5/126 [02:06<22:20, 11.08s/it]
batch:   5%|▍         | 6/126 [02:06<14:49,  7.41s/it]
batch:   6%|▌         | 7/126 [02:06<10:05,  5.09s/it]
batch:   6%|▋         | 8/126 [02:06<07:00,  3.56s/it]
batch:   7%|▋         | 9/126 [02:07<04:57,  2.54s/it]
batch:   8%|▊         | 10/126 [02:07<03:33,  1.84s/it]
batch:   9%|▊         | 11/126 [02:07<02:37,  1.37s/it]
batch:  10%|▉         | 12/126 [02:08<01:59,  1.04s/it]
batch:  10%|█         | 13/126 [02:08<01:32,  1.22it/s]
batch:  11%|█         | 14/126 [02:08<01:13,  1.51it/s]
batch:  12%|█▏        | 15/126 [02:08<01:01,  1.81it/s]
batch:  13%|█▎        | 16/126 [02:09<00:52,  2.10it/s]
batch:  13%|█▎        | 17/126 [02:09<00:46,  2.36it/s]
batch:  14%|█▍        | 18/126 [02:09<00:41,  2.58it/s]
batch:  15%|█▌        | 19/126 [02:10<00:38,  2.77it/s]
batch:  16%|█▌        | 20/126 [02:10<00:36,  2.94it/s]
batch:  17%|█▋        | 21/126 [02:10<00:34,  3.08it/s]
batch:  17%|█▋        | 22/126 [02:11<00:32,  3.18it/s]
batch:  18%|█▊        | 23/126 [02:11<00:31,  3.25it/s]
batch:  19%|█▉        | 24/126 [02:11<00:31,  3.28it/s]
batch:  20%|█▉        | 25/126 [02:11<00:30,  3.30it/s]
batch:  21%|██        | 26/126 [02:12<00:30,  3.30it/s]
batch:  21%|██▏       | 27/126 [02:12<00:29,  3.32it/s]
batch:  22%|██▏       | 28/126 [02:12<00:29,  3.37it/s]
batch:  23%|██▎       | 29/126 [02:13<00:28,  3.37it/s]
batch:  24%|██▍       | 30/126 [02:13<00:28,  3.41it/s]
batch:  25%|██▍       | 31/126 [02:13<00:27,  3.40it/s]
batch:  25%|██▌       | 32/126 [02:14<00:27,  3.37it/s]
batch:  26%|██▌       | 33/126 [02:14<00:27,  3.34it/s]
batch:  27%|██▋       | 34/126 [02:14<00:27,  3.34it/s]
batch:  28%|██▊       | 35/126 [02:14<00:27,  3.33it/s]
batch:  29%|██▊       | 36/126 [02:15<00:27,  3.33it/s]
batch:  29%|██▉       | 37/126 [02:15<00:26,  3.30it/s]
batch:  30%|███       | 38/126 [02:15<00:26,  3.34it/s]
batch:  31%|███       | 39/126 [02:16<00:26,  3.34it/s]
batch:  32%|███▏      | 40/126 [02:16<00:25,  3.32it/s]
batch:  33%|███▎      | 41/126 [02:16<00:25,  3.32it/s]
batch:  33%|███▎      | 42/126 [02:17<00:25,  3.33it/s]
batch:  34%|███▍      | 43/126 [02:17<00:24,  3.34it/s]
batch:  35%|███▍      | 44/126 [02:17<00:24,  3.32it/s]
batch:  36%|███▌      | 45/126 [02:17<00:24,  3.31it/s]
batch:  37%|███▋      | 46/126 [02:18<00:24,  3.31it/s]
batch:  37%|███▋      | 47/126 [02:18<00:23,  3.33it/s]
batch:  38%|███▊      | 48/126 [02:18<00:23,  3.35it/s]
batch:  39%|███▉      | 49/126 [02:19<00:22,  3.35it/s]
batch:  40%|███▉      | 50/126 [02:19<00:22,  3.38it/s]
batch:  40%|████      | 51/126 [02:19<00:22,  3.39it/s]
batch:  41%|████▏     | 52/126 [02:20<00:21,  3.37it/s]
batch:  42%|████▏     | 53/126 [02:20<00:21,  3.39it/s]
batch:  43%|████▎     | 54/126 [02:20<00:21,  3.41it/s]
batch:  44%|████▎     | 55/126 [02:20<00:21,  3.37it/s]
batch:  44%|████▍     | 56/126 [02:21<00:20,  3.36it/s]
batch:  45%|████▌     | 57/126 [02:21<00:20,  3.39it/s]
batch:  46%|████▌     | 58/126 [02:21<00:20,  3.35it/s]
batch:  47%|████▋     | 59/126 [02:22<00:20,  3.33it/s]
batch:  48%|████▊     | 60/126 [02:22<00:19,  3.34it/s]
batch:  48%|████▊     | 61/126 [02:22<00:19,  3.34it/s]
batch:  49%|████▉     | 62/126 [02:22<00:19,  3.36it/s]
batch:  50%|█████     | 63/126 [02:23<00:18,  3.32it/s]
batch:  51%|█████     | 64/126 [02:23<00:18,  3.33it/s]
batch:  52%|█████▏    | 65/126 [02:23<00:18,  3.33it/s]
batch:  52%|█████▏    | 66/126 [02:24<00:20,  2.99it/s]
batch:  53%|█████▎    | 67/126 [02:24<00:19,  3.07it/s]
batch:  54%|█████▍    | 68/126 [02:24<00:18,  3.13it/s]
batch:  55%|█████▍    | 69/126 [02:25<00:17,  3.20it/s]
batch:  56%|█████▌    | 70/126 [02:25<00:17,  3.25it/s]
batch:  56%|█████▋    | 71/126 [02:25<00:16,  3.28it/s]
batch:  57%|█████▋    | 72/126 [02:26<00:16,  3.32it/s]
batch:  58%|█████▊    | 73/126 [02:26<00:15,  3.33it/s]
batch:  59%|█████▊    | 74/126 [02:26<00:15,  3.34it/s]
batch:  60%|█████▉    | 75/126 [02:27<00:15,  3.34it/s]
batch:  60%|██████    | 76/126 [02:27<00:14,  3.35it/s]
batch:  61%|██████    | 77/126 [02:27<00:14,  3.33it/s]
batch:  62%|██████▏   | 78/126 [02:27<00:14,  3.34it/s]
batch:  63%|██████▎   | 79/126 [02:28<00:14,  3.34it/s]
batch:  63%|██████▎   | 80/126 [02:28<00:13,  3.33it/s]
batch:  64%|██████▍   | 81/126 [02:28<00:13,  3.32it/s]
batch:  65%|██████▌   | 82/126 [02:29<00:13,  3.31it/s]
batch:  66%|██████▌   | 83/126 [02:29<00:13,  3.29it/s]
batch:  67%|██████▋   | 84/126 [02:29<00:12,  3.29it/s]
batch:  67%|██████▋   | 85/126 [02:30<00:12,  3.31it/s]
batch:  68%|██████▊   | 86/126 [02:30<00:12,  3.33it/s]
batch:  69%|██████▉   | 87/126 [02:30<00:11,  3.33it/s]
batch:  70%|██████▉   | 88/126 [02:30<00:11,  3.35it/s]
batch:  71%|███████   | 89/126 [02:31<00:10,  3.38it/s]
batch:  71%|███████▏  | 90/126 [02:31<00:10,  3.37it/s]
batch:  72%|███████▏  | 91/126 [02:31<00:10,  3.36it/s]
batch:  73%|███████▎  | 92/126 [02:32<00:10,  3.36it/s]
batch:  74%|███████▍  | 93/126 [02:32<00:09,  3.33it/s]
batch:  75%|███████▍  | 94/126 [02:32<00:09,  3.30it/s]
batch:  75%|███████▌  | 95/126 [02:33<00:09,  3.32it/s]
batch:  76%|███████▌  | 96/126 [02:33<00:08,  3.34it/s]
batch:  77%|███████▋  | 97/126 [02:33<00:08,  3.34it/s]
batch:  78%|███████▊  | 98/126 [02:33<00:08,  3.34it/s]
batch:  79%|███████▊  | 99/126 [02:34<00:08,  3.33it/s]
batch:  79%|███████▉  | 100/126 [02:34<00:07,  3.33it/s]
batch:  80%|████████  | 101/126 [02:34<00:07,  3.33it/s]
batch:  81%|████████  | 102/126 [02:35<00:07,  3.34it/s]
batch:  82%|████████▏ | 103/126 [02:35<00:06,  3.34it/s]
batch:  83%|████████▎ | 104/126 [02:35<00:06,  3.35it/s]
batch:  83%|████████▎ | 105/126 [02:35<00:06,  3.40it/s]
batch:  84%|████████▍ | 106/126 [02:36<00:05,  3.41it/s]
batch:  85%|████████▍ | 107/126 [02:36<00:05,  3.41it/s]
batch:  86%|████████▌ | 108/126 [02:36<00:05,  3.43it/s]
batch:  87%|████████▋ | 109/126 [02:37<00:04,  3.46it/s]
batch:  87%|████████▋ | 110/126 [02:37<00:04,  3.46it/s]
batch:  88%|████████▊ | 111/126 [02:37<00:04,  3.46it/s]
batch:  89%|████████▉ | 112/126 [02:38<00:04,  3.44it/s]
batch:  90%|████████▉ | 113/126 [02:38<00:03,  3.41it/s]
batch:  90%|█████████ | 114/126 [02:38<00:03,  3.42it/s]
batch:  91%|█████████▏| 115/126 [02:38<00:03,  3.43it/s]
batch:  92%|█████████▏| 116/126 [02:39<00:02,  3.44it/s]
batch:  93%|█████████▎| 117/126 [02:39<00:02,  3.42it/s]
batch:  94%|█████████▎| 118/126 [02:39<00:02,  3.43it/s]
batch:  94%|█████████▍| 119/126 [02:40<00:02,  3.43it/s]
batch:  95%|█████████▌| 120/126 [02:40<00:01,  3.44it/s]
batch:  96%|█████████▌| 121/126 [02:40<00:01,  3.45it/s]
batch:  97%|█████████▋| 122/126 [02:40<00:01,  3.46it/s]
batch:  98%|█████████▊| 123/126 [02:41<00:00,  3.89it/s]
batch:  98%|█████████▊| 124/126 [02:41<00:00,  3.75it/s]
batch:  99%|█████████▉| 125/126 [02:41<00:00,  3.63it/s]
batch: 100%|██████████| 126/126 [02:41<00:00,  1.28s/it]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 103/7971 [00:00<00:07, 1011.00it/s]
  3%|▎         | 208/7971 [00:00<00:07, 1028.54it/s]
  4%|▍         | 326/7971 [00:00<00:06, 1092.46it/s]
  5%|▌         | 436/7971 [00:00<00:07, 1070.48it/s]
  7%|▋         | 547/7971 [00:00<00:06, 1079.66it/s]
  8%|▊         | 656/7971 [00:00<00:06, 1082.23it/s]
 10%|▉         | 766/7971 [00:00<00:06, 1081.17it/s]
 11%|█         | 875/7971 [00:00<00:06, 1080.48it/s]
 12%|█▏        | 984/7971 [00:00<00:06, 1079.77it/s]
 14%|█▎        | 1092/7971 [00:01<00:06, 1069.54it/s]
 15%|█▌        | 1199/7971 [00:01<00:06, 1050.01it/s]
 16%|█▋        | 1305/7971 [00:01<00:06, 1028.40it/s]
 18%|█▊        | 1410/7971 [00:01<00:06, 1033.83it/s]
 19%|█▉        | 1520/7971 [00:01<00:06, 1050.51it/s]
 20%|██        | 1629/7971 [00:01<00:05, 1060.30it/s]
 22%|██▏       | 1738/7971 [00:01<00:05, 1068.27it/s]
 23%|██▎       | 1851/7971 [00:01<00:05, 1085.83it/s]
 25%|██▍       | 1960/7971 [00:01<00:05, 1080.27it/s]
 26%|██▌       | 2071/7971 [00:01<00:05, 1083.78it/s]
 27%|██▋       | 2183/7971 [00:02<00:05, 1092.92it/s]
 29%|██▉       | 2293/7971 [00:02<00:05, 1084.85it/s]
 30%|███       | 2402/7971 [00:02<00:05, 1066.49it/s]
 31%|███▏      | 2509/7971 [00:02<00:05, 1051.52it/s]
 33%|███▎      | 2615/7971 [00:02<00:05, 1040.23it/s]
 34%|███▍      | 2720/7971 [00:02<00:05, 1033.09it/s]
 35%|███▌      | 2824/7971 [00:02<00:05, 1019.95it/s]
 37%|███▋      | 2927/7971 [00:02<00:04, 1011.98it/s]
 38%|███▊      | 3033/7971 [00:02<00:04, 1023.40it/s]
 39%|███▉      | 3136/7971 [00:02<00:04, 1017.09it/s]
 41%|████      | 3238/7971 [00:03<00:04, 1010.12it/s]
 42%|████▏     | 3343/7971 [00:03<00:04, 1021.14it/s]
 43%|████▎     | 3448/7971 [00:03<00:04, 1023.30it/s]
 45%|████▍     | 3560/7971 [00:03<00:04, 1045.31it/s]
 46%|████▌     | 3666/7971 [00:03<00:04, 1049.50it/s]
 47%|████▋     | 3771/7971 [00:03<00:04, 1024.77it/s]
 49%|████▊     | 3874/7971 [00:03<00:04, 1022.36it/s]
 50%|████▉     | 3980/7971 [00:03<00:03, 1030.67it/s]
 51%|█████▏    | 4090/7971 [00:03<00:03, 1047.16it/s]
 53%|█████▎    | 4195/7971 [00:03<00:03, 1040.03it/s]
 54%|█████▍    | 4304/7971 [00:04<00:03, 1053.51it/s]
 55%|█████▌    | 4413/7971 [00:04<00:03, 1057.78it/s]
 57%|█████▋    | 4519/7971 [00:04<00:03, 1038.46it/s]
 58%|█████▊    | 4627/7971 [00:04<00:03, 1050.35it/s]
 59%|█████▉    | 4733/7971 [00:04<00:03, 1027.16it/s]
 61%|██████    | 4850/7971 [00:04<00:02, 1064.33it/s]
 62%|██████▏   | 4957/7971 [00:04<00:02, 1030.99it/s]
 64%|██████▎   | 5069/7971 [00:04<00:02, 1055.95it/s]
 65%|██████▍   | 5175/7971 [00:04<00:02, 1038.22it/s]
 66%|██████▋   | 5281/7971 [00:05<00:02, 1043.43it/s]
 68%|██████▊   | 5386/7971 [00:05<00:02, 1017.44it/s]
 69%|██████▉   | 5494/7971 [00:05<00:02, 1030.05it/s]
 70%|███████   | 5598/7971 [00:05<00:02, 1004.15it/s]
 72%|███████▏  | 5701/7971 [00:05<00:02, 1006.48it/s]
 73%|███████▎  | 5808/7971 [00:05<00:02, 1019.35it/s]
 74%|███████▍  | 5911/7971 [00:05<00:02, 1020.98it/s]
 75%|███████▌  | 6014/7971 [00:05<00:01, 1021.88it/s]
 77%|███████▋  | 6117/7971 [00:05<00:01, 1012.77it/s]
 78%|███████▊  | 6219/7971 [00:05<00:01, 998.84it/s] 
 79%|███████▉  | 6319/7971 [00:06<00:01, 980.31it/s]
 81%|████████  | 6418/7971 [00:06<00:01, 967.10it/s]
 82%|████████▏ | 6515/7971 [00:06<00:01, 965.19it/s]
 83%|████████▎ | 6612/7971 [00:06<00:01, 963.98it/s]
 84%|████████▍ | 6718/7971 [00:06<00:01, 987.63it/s]
 86%|████████▌ | 6817/7971 [00:06<00:01, 983.66it/s]
 87%|████████▋ | 6917/7971 [00:06<00:01, 986.87it/s]
 88%|████████▊ | 7016/7971 [00:06<00:00, 985.35it/s]
 89%|████████▉ | 7115/7971 [00:06<00:00, 978.94it/s]
 91%|█████████ | 7217/7971 [00:06<00:00, 990.92it/s]
 92%|█████████▏| 7317/7971 [00:07<00:00, 967.78it/s]
 93%|█████████▎| 7414/7971 [00:07<00:00, 949.42it/s]
 94%|█████████▍| 7510/7971 [00:07<00:00, 939.23it/s]
 95%|█████████▌| 7610/7971 [00:07<00:00, 955.97it/s]
 97%|█████████▋| 7706/7971 [00:07<00:00, 953.16it/s]
 98%|█████████▊| 7811/7971 [00:07<00:00, 981.22it/s]
 99%|█████████▉| 7910/7971 [00:07<00:00, 956.94it/s]
100%|██████████| 7971/7971 [00:07<00:00, 1023.19it/s]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 103/7971 [00:00<00:07, 1015.03it/s]
  3%|▎         | 208/7971 [00:00<00:07, 1030.47it/s]
  4%|▍         | 321/7971 [00:00<00:07, 1075.35it/s]
  5%|▌         | 429/7971 [00:00<00:07, 1028.76it/s]
  7%|▋         | 534/7971 [00:00<00:07, 1028.48it/s]
  8%|▊         | 643/7971 [00:00<00:07, 1042.26it/s]
  9%|▉         | 748/7971 [00:00<00:06, 1034.83it/s]
 11%|█         | 854/7971 [00:00<00:06, 1041.19it/s]
 12%|█▏        | 966/7971 [00:00<00:06, 1062.92it/s]
 13%|█▎        | 1074/7971 [00:01<00:06, 1060.78it/s]
 15%|█▍        | 1181/7971 [00:01<00:06, 1041.76it/s]
 16%|█▌        | 1286/7971 [00:01<00:06, 1032.07it/s]
 17%|█▋        | 1390/7971 [00:01<00:06, 1024.22it/s]
 19%|█▉        | 1504/7971 [00:01<00:06, 1052.93it/s]
 20%|██        | 1614/7971 [00:01<00:05, 1064.38it/s]
 22%|██▏       | 1723/7971 [00:01<00:05, 1071.60it/s]
 23%|██▎       | 1831/7971 [00:01<00:05, 1071.75it/s]
 24%|██▍       | 1939/7971 [00:01<00:05, 1070.73it/s]
 26%|██▌       | 2051/7971 [00:01<00:05, 1078.94it/s]
 27%|██▋       | 2159/7971 [00:02<00:05, 1074.81it/s]
 28%|██▊       | 2267/7971 [00:02<00:05, 1060.07it/s]
 30%|██▉       | 2374/7971 [00:02<00:05, 1049.09it/s]
 31%|███       | 2479/7971 [00:02<00:05, 1034.07it/s]
 32%|███▏      | 2583/7971 [00:02<00:05, 1020.19it/s]
 34%|███▎      | 2690/7971 [00:02<00:05, 1031.13it/s]
 35%|███▌      | 2794/7971 [00:02<00:05, 1019.26it/s]
 36%|███▋      | 2896/7971 [00:02<00:05, 1013.32it/s]
 38%|███▊      | 2998/7971 [00:02<00:04, 1009.96it/s]
 39%|███▉      | 3100/7971 [00:02<00:04, 1002.45it/s]
 40%|████      | 3201/7971 [00:03<00:04, 990.50it/s] 
 41%|████▏     | 3305/7971 [00:03<00:04, 1003.46it/s]
 43%|████▎     | 3409/7971 [00:03<00:04, 1010.72it/s]
 44%|████▍     | 3518/7971 [00:03<00:04, 1029.33it/s]
 45%|████▌     | 3625/7971 [00:03<00:04, 1039.00it/s]
 47%|████▋     | 3729/7971 [00:03<00:04, 1010.37it/s]
 48%|████▊     | 3831/7971 [00:03<00:04, 989.33it/s] 
 49%|████▉     | 3933/7971 [00:03<00:04, 996.98it/s]
 51%|█████     | 4041/7971 [00:03<00:03, 1014.77it/s]
 52%|█████▏    | 4143/7971 [00:04<00:03, 1013.10it/s]
 53%|█████▎    | 4248/7971 [00:04<00:03, 1020.26it/s]
 55%|█████▍    | 4359/7971 [00:04<00:03, 1041.86it/s]
 56%|█████▌    | 4464/7971 [00:04<00:03, 995.40it/s] 
 57%|█████▋    | 4564/7971 [00:04<00:03, 995.54it/s]
 59%|█████▊    | 4667/7971 [00:04<00:03, 1004.21it/s]
 60%|█████▉    | 4772/7971 [00:04<00:03, 1013.79it/s]
 61%|██████▏   | 4883/7971 [00:04<00:02, 1040.53it/s]
 63%|██████▎   | 4988/7971 [00:04<00:02, 1025.18it/s]
 64%|██████▍   | 5094/7971 [00:04<00:02, 1033.10it/s]
 65%|██████▌   | 5198/7971 [00:05<00:02, 1011.43it/s]
 67%|██████▋   | 5301/7971 [00:05<00:02, 1015.37it/s]
 68%|██████▊   | 5403/7971 [00:05<00:02, 1004.67it/s]
 69%|██████▉   | 5508/7971 [00:05<00:02, 1012.83it/s]
 70%|███████   | 5610/7971 [00:05<00:02, 1007.82it/s]
 72%|███████▏  | 5711/7971 [00:05<00:02, 1007.24it/s]
 73%|███████▎  | 5816/7971 [00:05<00:02, 1019.14it/s]
 74%|███████▍  | 5920/7971 [00:05<00:02, 1024.77it/s]
 76%|███████▌  | 6028/7971 [00:05<00:01, 1037.63it/s]
 77%|███████▋  | 6132/7971 [00:05<00:01, 1020.65it/s]
 78%|███████▊  | 6235/7971 [00:06<00:01, 1019.04it/s]
 80%|███████▉  | 6337/7971 [00:06<00:01, 1017.90it/s]
 81%|████████  | 6439/7971 [00:06<00:01, 1017.34it/s]
 82%|████████▏ | 6543/7971 [00:06<00:01, 1022.88it/s]
 83%|████████▎ | 6655/7971 [00:06<00:01, 1050.53it/s]
 85%|████████▍ | 6761/7971 [00:06<00:01, 1048.17it/s]
 86%|████████▌ | 6866/7971 [00:06<00:01, 1043.18it/s]
 88%|████████▊ | 6977/7971 [00:06<00:00, 1059.22it/s]
 89%|████████▉ | 7084/7971 [00:06<00:00, 1057.50it/s]
 90%|█████████ | 7190/7971 [00:06<00:00, 1034.05it/s]
 92%|█████████▏| 7298/7971 [00:07<00:00, 1042.75it/s]
 93%|█████████▎| 7403/7971 [00:07<00:00, 1018.70it/s]
 94%|█████████▍| 7506/7971 [00:07<00:00, 1003.05it/s]
 95%|█████████▌| 7607/7971 [00:07<00:00, 988.80it/s] 
 97%|█████████▋| 7706/7971 [00:07<00:00, 976.82it/s]
 98%|█████████▊| 7808/7971 [00:07<00:00, 986.02it/s]
 99%|█████████▉| 7907/7971 [00:07<00:00, 976.75it/s]
100%|██████████| 7971/7971 [00:07<00:00, 1023.58it/s]
2026-05-11 04:22:44.213640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:15.344784: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:25:15.346504: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:25:15.587891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:25:15.588064: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:15.608636: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:15.608780: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:15.618893: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:25:15.628753: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:25:15.642563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:25:15.651402: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:25:15.660141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:15.666560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:25:15.667171: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:25:15.667450: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:25:15.670176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:0a:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:25:15.670284: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:15.670334: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:15.670373: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:15.670409: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:25:15.670445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:25:15.670480: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:25:15.670515: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:25:15.670551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:15.675531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:25:15.675584: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:25:16.165990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:25:16.166139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:25:16.166156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:25:16.171151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:0a:00.0, compute capability: 8.0)
2026-05-11 04:25:16.217626: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:25:16.229763: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:25:23.723689: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:25:24.195803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:25:24.203208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:25:25.770927: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:25:25.905186: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:26:03.488846: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)