Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:36:38.106812: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:27.334997: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:27.351039: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:51:27.823414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:27.823517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:38.888509: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:38.888671: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:39.171193: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:39.813781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:41.748584: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.243175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.460582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.467418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.467905: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:51:42.468113: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:51:42.470586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 03:51:42.470632: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:42.470656: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:42.470672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:42.470686: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:51:42.470706: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:51:42.470721: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:51:42.470734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:51:42.470768: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:51:42.477120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:51:42.477194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:51:45.546150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:51:45.546320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:51:45.546337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:51:45.552794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:c3:00.0, compute capability: 8.0)
2026-05-11 03:51:49.542534: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:51:49.543166: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 03:51:57.194468: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:51:59.489155: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:51:59.536582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:10.696657: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:53:10.821414: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:55:12.546329: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 04:14:25.634704: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 04:16:42.437024: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:19:50.277374: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:19:50.278750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:19:50.540094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:19:50.540228: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:19:50.599187: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:19:50.599333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:19:50.629154: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:19:50.657486: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:19:50.688915: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:19:50.719526: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:19:50.748517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:19:50.754640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:19:50.755090: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:19:50.755307: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:19:50.757731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:19:50.757781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:19:50.757808: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:19:50.757826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:19:50.757843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:19:50.757860: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:19:50.757876: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:19:50.757892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:19:50.757908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:19:50.762378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:19:50.762422: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:19:51.203942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:19:51.204095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:19:51.204111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:19:51.208064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:c3:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/126 [00:00<?, ?it/s]2026-05-11 04:19:53.345024: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 04:19:53.345596: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:19:53.881067: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:19:54.362720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:19:54.364589: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:19:59.649173: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:19:59.757848: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/126 [01:32<3:13:18, 92.79s/it]
batch:   2%|▏         | 2/126 [01:33<1:19:17, 38.37s/it]
batch:   2%|▏         | 3/126 [01:33<42:59, 20.97s/it]  
batch:   3%|▎         | 4/126 [01:33<26:01, 12.80s/it]
batch:   4%|▍         | 5/126 [01:33<16:41,  8.28s/it]
batch:   5%|▍         | 6/126 [01:34<11:06,  5.56s/it]
batch:   6%|▌         | 7/126 [01:34<07:35,  3.83s/it]
batch:   6%|▋         | 8/126 [01:34<05:18,  2.70s/it]
batch:   7%|▋         | 9/126 [01:34<03:46,  1.94s/it]
batch:   8%|▊         | 10/126 [01:35<02:45,  1.42s/it]
batch:   9%|▊         | 11/126 [01:35<02:02,  1.07s/it]
batch:  10%|▉         | 12/126 [01:35<01:34,  1.21it/s]
batch:  10%|█         | 13/126 [01:36<01:14,  1.53it/s]
batch:  11%|█         | 14/126 [01:36<01:00,  1.86it/s]
batch:  12%|█▏        | 15/126 [01:36<00:50,  2.19it/s]
batch:  13%|█▎        | 16/126 [01:36<00:43,  2.51it/s]
batch:  13%|█▎        | 17/126 [01:37<00:39,  2.77it/s]
batch:  14%|█▍        | 18/126 [01:37<00:35,  3.01it/s]
batch:  15%|█▌        | 19/126 [01:37<00:33,  3.21it/s]
batch:  16%|█▌        | 20/126 [01:37<00:31,  3.36it/s]
batch:  17%|█▋        | 21/126 [01:38<00:30,  3.47it/s]
batch:  17%|█▋        | 22/126 [01:38<00:29,  3.54it/s]
batch:  18%|█▊        | 23/126 [01:38<00:28,  3.60it/s]
batch:  19%|█▉        | 24/126 [01:38<00:28,  3.63it/s]
batch:  20%|█▉        | 25/126 [01:39<00:27,  3.63it/s]
batch:  21%|██        | 26/126 [01:39<00:27,  3.66it/s]
batch:  21%|██▏       | 27/126 [01:39<00:26,  3.68it/s]
batch:  22%|██▏       | 28/126 [01:40<00:26,  3.72it/s]
batch:  23%|██▎       | 29/126 [01:40<00:26,  3.71it/s]
batch:  24%|██▍       | 30/126 [01:40<00:25,  3.71it/s]
batch:  25%|██▍       | 31/126 [01:40<00:25,  3.72it/s]
batch:  25%|██▌       | 32/126 [01:41<00:25,  3.70it/s]
batch:  26%|██▌       | 33/126 [01:41<00:24,  3.73it/s]
batch:  27%|██▋       | 34/126 [01:41<00:24,  3.72it/s]
batch:  28%|██▊       | 35/126 [01:41<00:24,  3.74it/s]
batch:  29%|██▊       | 36/126 [01:42<00:24,  3.75it/s]
batch:  29%|██▉       | 37/126 [01:42<00:23,  3.74it/s]
batch:  30%|███       | 38/126 [01:42<00:23,  3.70it/s]
batch:  31%|███       | 39/126 [01:42<00:23,  3.71it/s]
batch:  32%|███▏      | 40/126 [01:43<00:23,  3.73it/s]
batch:  33%|███▎      | 41/126 [01:43<00:22,  3.74it/s]
batch:  33%|███▎      | 42/126 [01:43<00:22,  3.74it/s]
batch:  34%|███▍      | 43/126 [01:44<00:22,  3.73it/s]
batch:  35%|███▍      | 44/126 [01:44<00:22,  3.72it/s]
batch:  36%|███▌      | 45/126 [01:44<00:21,  3.71it/s]
batch:  37%|███▋      | 46/126 [01:44<00:21,  3.72it/s]
batch:  37%|███▋      | 47/126 [01:45<00:21,  3.71it/s]
batch:  38%|███▊      | 48/126 [01:45<00:20,  3.73it/s]
batch:  39%|███▉      | 49/126 [01:45<00:20,  3.72it/s]
batch:  40%|███▉      | 50/126 [01:45<00:20,  3.72it/s]
batch:  40%|████      | 51/126 [01:46<00:20,  3.72it/s]
batch:  41%|████▏     | 52/126 [01:46<00:19,  3.73it/s]
batch:  42%|████▏     | 53/126 [01:46<00:19,  3.73it/s]
batch:  43%|████▎     | 54/126 [01:47<00:19,  3.72it/s]
batch:  44%|████▎     | 55/126 [01:47<00:19,  3.72it/s]
batch:  44%|████▍     | 56/126 [01:47<00:18,  3.71it/s]
batch:  45%|████▌     | 57/126 [01:47<00:18,  3.69it/s]
batch:  46%|████▌     | 58/126 [01:48<00:18,  3.70it/s]
batch:  47%|████▋     | 59/126 [01:48<00:18,  3.71it/s]
batch:  48%|████▊     | 60/126 [01:48<00:17,  3.72it/s]
batch:  48%|████▊     | 61/126 [01:48<00:17,  3.72it/s]
batch:  49%|████▉     | 62/126 [01:49<00:17,  3.70it/s]
batch:  50%|█████     | 63/126 [01:49<00:16,  3.71it/s]
batch:  51%|█████     | 64/126 [01:49<00:16,  3.71it/s]
batch:  52%|█████▏    | 65/126 [01:49<00:16,  3.72it/s]
batch:  52%|█████▏    | 66/126 [01:50<00:18,  3.27it/s]
batch:  53%|█████▎    | 67/126 [01:50<00:17,  3.37it/s]
batch:  54%|█████▍    | 68/126 [01:50<00:16,  3.46it/s]
batch:  55%|█████▍    | 69/126 [01:51<00:16,  3.52it/s]
batch:  56%|█████▌    | 70/126 [01:51<00:15,  3.58it/s]
batch:  56%|█████▋    | 71/126 [01:51<00:15,  3.63it/s]
batch:  57%|█████▋    | 72/126 [01:51<00:14,  3.65it/s]
batch:  58%|█████▊    | 73/126 [01:52<00:14,  3.67it/s]
batch:  59%|█████▊    | 74/126 [01:52<00:14,  3.62it/s]
batch:  60%|█████▉    | 75/126 [01:52<00:13,  3.65it/s]
batch:  60%|██████    | 76/126 [01:53<00:13,  3.66it/s]
batch:  61%|██████    | 77/126 [01:53<00:13,  3.64it/s]
batch:  62%|██████▏   | 78/126 [01:53<00:13,  3.65it/s]
batch:  63%|██████▎   | 79/126 [01:53<00:12,  3.64it/s]
batch:  63%|██████▎   | 80/126 [01:54<00:12,  3.64it/s]
batch:  64%|██████▍   | 81/126 [01:54<00:12,  3.64it/s]
batch:  65%|██████▌   | 82/126 [01:54<00:12,  3.65it/s]
batch:  66%|██████▌   | 83/126 [01:55<00:11,  3.64it/s]
batch:  67%|██████▋   | 84/126 [01:55<00:11,  3.64it/s]
batch:  67%|██████▋   | 85/126 [01:55<00:11,  3.66it/s]
batch:  68%|██████▊   | 86/126 [01:55<00:10,  3.68it/s]
batch:  69%|██████▉   | 87/126 [01:56<00:10,  3.69it/s]
batch:  70%|██████▉   | 88/126 [01:56<00:10,  3.68it/s]
batch:  71%|███████   | 89/126 [01:56<00:10,  3.67it/s]
batch:  71%|███████▏  | 90/126 [01:56<00:09,  3.66it/s]
batch:  72%|███████▏  | 91/126 [01:57<00:09,  3.67it/s]
batch:  73%|███████▎  | 92/126 [01:57<00:09,  3.65it/s]
batch:  74%|███████▍  | 93/126 [01:57<00:09,  3.63it/s]
batch:  75%|███████▍  | 94/126 [01:58<00:08,  3.67it/s]
batch:  75%|███████▌  | 95/126 [01:58<00:08,  3.68it/s]
batch:  76%|███████▌  | 96/126 [01:58<00:08,  3.68it/s]
batch:  77%|███████▋  | 97/126 [01:58<00:07,  3.69it/s]
batch:  78%|███████▊  | 98/126 [01:59<00:07,  3.68it/s]
batch:  79%|███████▊  | 99/126 [01:59<00:07,  3.71it/s]
batch:  79%|███████▉  | 100/126 [01:59<00:07,  3.69it/s]
batch:  80%|████████  | 101/126 [01:59<00:06,  3.70it/s]
batch:  81%|████████  | 102/126 [02:00<00:06,  3.69it/s]
batch:  82%|████████▏ | 103/126 [02:00<00:06,  3.69it/s]
batch:  83%|████████▎ | 104/126 [02:00<00:05,  3.70it/s]
batch:  83%|████████▎ | 105/126 [02:00<00:05,  3.68it/s]
batch:  84%|████████▍ | 106/126 [02:01<00:05,  3.70it/s]
batch:  85%|████████▍ | 107/126 [02:01<00:05,  3.71it/s]
batch:  86%|████████▌ | 108/126 [02:01<00:04,  3.69it/s]
batch:  87%|████████▋ | 109/126 [02:02<00:04,  3.69it/s]
batch:  87%|████████▋ | 110/126 [02:02<00:04,  3.66it/s]
batch:  88%|████████▊ | 111/126 [02:02<00:04,  3.68it/s]
batch:  89%|████████▉ | 112/126 [02:02<00:03,  3.70it/s]
batch:  90%|████████▉ | 113/126 [02:03<00:03,  3.68it/s]
batch:  90%|█████████ | 114/126 [02:03<00:03,  3.69it/s]
batch:  91%|█████████▏| 115/126 [02:03<00:02,  3.68it/s]
batch:  92%|█████████▏| 116/126 [02:03<00:02,  3.69it/s]
batch:  93%|█████████▎| 117/126 [02:04<00:02,  3.69it/s]
batch:  94%|█████████▎| 118/126 [02:04<00:02,  3.70it/s]
batch:  94%|█████████▍| 119/126 [02:04<00:01,  3.68it/s]
batch:  95%|█████████▌| 120/126 [02:05<00:01,  3.69it/s]
batch:  96%|█████████▌| 121/126 [02:05<00:01,  3.70it/s]
batch:  97%|█████████▋| 122/126 [02:05<00:01,  3.68it/s]
batch:  98%|█████████▊| 123/126 [02:05<00:00,  3.70it/s]
batch:  98%|█████████▊| 124/126 [02:06<00:00,  3.69it/s]
batch:  99%|█████████▉| 125/126 [02:06<00:00,  4.16it/s]
batch: 100%|██████████| 126/126 [02:06<00:00,  1.00s/it]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 111/7971 [00:00<00:07, 1108.52it/s]
  3%|▎         | 233/7971 [00:00<00:06, 1162.34it/s]
  5%|▍         | 359/7971 [00:00<00:06, 1198.75it/s]
  6%|▌         | 479/7971 [00:00<00:06, 1181.67it/s]
  8%|▊         | 601/7971 [00:00<00:06, 1194.43it/s]
  9%|▉         | 721/7971 [00:00<00:06, 1188.25it/s]
 11%|█         | 840/7971 [00:00<00:06, 1187.21it/s]
 12%|█▏        | 966/7971 [00:00<00:05, 1208.30it/s]
 14%|█▎        | 1087/7971 [00:00<00:05, 1206.85it/s]
 15%|█▌        | 1208/7971 [00:01<00:05, 1184.80it/s]
 17%|█▋        | 1327/7971 [00:01<00:05, 1164.14it/s]
 18%|█▊        | 1449/7971 [00:01<00:05, 1175.53it/s]
 20%|█▉        | 1577/7971 [00:01<00:05, 1205.98it/s]
 21%|██▏       | 1701/7971 [00:01<00:05, 1214.36it/s]
 23%|██▎       | 1823/7971 [00:01<00:05, 1194.10it/s]
 24%|██▍       | 1944/7971 [00:01<00:05, 1193.46it/s]
 26%|██▌       | 2066/7971 [00:01<00:04, 1195.97it/s]
 27%|██▋       | 2186/7971 [00:01<00:04, 1197.10it/s]
 29%|██▉       | 2306/7971 [00:01<00:04, 1187.87it/s]
 30%|███       | 2425/7971 [00:02<00:04, 1165.74it/s]
 32%|███▏      | 2542/7971 [00:02<00:04, 1160.84it/s]
 33%|███▎      | 2660/7971 [00:02<00:04, 1163.56it/s]
 35%|███▍      | 2777/7971 [00:02<00:04, 1156.62it/s]
 36%|███▋      | 2893/7971 [00:02<00:04, 1151.62it/s]
 38%|███▊      | 3009/7971 [00:02<00:04, 1149.96it/s]
 39%|███▉      | 3125/7971 [00:02<00:04, 1138.73it/s]
 41%|████      | 3239/7971 [00:02<00:04, 1116.50it/s]
 42%|████▏     | 3356/7971 [00:02<00:04, 1131.60it/s]
 44%|████▎     | 3470/7971 [00:02<00:04, 1123.68it/s]
 45%|████▌     | 3595/7971 [00:03<00:03, 1154.24it/s]
 47%|████▋     | 3711/7971 [00:03<00:03, 1132.85it/s]
 48%|████▊     | 3825/7971 [00:03<00:03, 1122.77it/s]
 49%|████▉     | 3938/7971 [00:03<00:03, 1122.92it/s]
 51%|█████     | 4053/7971 [00:03<00:03, 1130.32it/s]
 52%|█████▏    | 4167/7971 [00:03<00:03, 1127.90it/s]
 54%|█████▍    | 4287/7971 [00:03<00:03, 1142.48it/s]
 55%|█████▌    | 4408/7971 [00:03<00:03, 1162.19it/s]
 57%|█████▋    | 4525/7971 [00:03<00:03, 1132.38it/s]
 58%|█████▊    | 4641/7971 [00:03<00:02, 1137.31it/s]
 60%|█████▉    | 4755/7971 [00:04<00:02, 1126.96it/s]
 61%|██████    | 4880/7971 [00:04<00:02, 1162.54it/s]
 63%|██████▎   | 4997/7971 [00:04<00:02, 1148.11it/s]
 64%|██████▍   | 5114/7971 [00:04<00:02, 1154.34it/s]
 66%|██████▌   | 5230/7971 [00:04<00:02, 1140.11it/s]
 67%|██████▋   | 5347/7971 [00:04<00:02, 1142.88it/s]
 69%|██████▊   | 5466/7971 [00:04<00:02, 1156.67it/s]
 70%|███████   | 5582/7971 [00:04<00:02, 1132.40it/s]
 71%|███████▏  | 5698/7971 [00:04<00:02, 1134.23it/s]
 73%|███████▎  | 5816/7971 [00:05<00:01, 1146.31it/s]
 74%|███████▍  | 5932/7971 [00:05<00:01, 1146.24it/s]
 76%|███████▌  | 6048/7971 [00:05<00:01, 1146.69it/s]
 77%|███████▋  | 6163/7971 [00:05<00:01, 1137.03it/s]
 79%|███████▊  | 6277/7971 [00:05<00:01, 1116.26it/s]
 80%|████████  | 6389/7971 [00:05<00:01, 1113.74it/s]
 82%|████████▏ | 6501/7971 [00:05<00:01, 1097.03it/s]
 83%|████████▎ | 6616/7971 [00:05<00:01, 1111.99it/s]
 84%|████████▍ | 6732/7971 [00:05<00:01, 1120.88it/s]
 86%|████████▌ | 6845/7971 [00:05<00:01, 1116.06it/s]
 87%|████████▋ | 6962/7971 [00:06<00:00, 1127.30it/s]
 89%|████████▉ | 7075/7971 [00:06<00:00, 1120.91it/s]
 90%|█████████ | 7188/7971 [00:06<00:00, 1103.72it/s]
 92%|█████████▏| 7299/7971 [00:06<00:00, 1103.60it/s]
 93%|█████████▎| 7410/7971 [00:06<00:00, 1073.42it/s]
 94%|█████████▍| 7518/7971 [00:06<00:00, 1069.89it/s]
 96%|█████████▌| 7626/7971 [00:06<00:00, 1067.19it/s]
 97%|█████████▋| 7733/7971 [00:06<00:00, 1065.10it/s]
 98%|█████████▊| 7849/7971 [00:06<00:00, 1089.57it/s]
100%|█████████▉| 7959/7971 [00:06<00:00, 1070.00it/s]
100%|██████████| 7971/7971 [00:06<00:00, 1140.63it/s]

  0%|          | 0/7971 [00:00<?, ?it/s]
  1%|▏         | 117/7971 [00:00<00:06, 1149.76it/s]
  3%|▎         | 239/7971 [00:00<00:06, 1179.11it/s]
  5%|▍         | 365/7971 [00:00<00:06, 1207.62it/s]
  6%|▌         | 486/7971 [00:00<00:06, 1181.59it/s]
  8%|▊         | 609/7971 [00:00<00:06, 1196.89it/s]
  9%|▉         | 729/7971 [00:00<00:06, 1189.08it/s]
 11%|█         | 848/7971 [00:00<00:05, 1187.53it/s]
 12%|█▏        | 975/7971 [00:00<00:05, 1209.20it/s]
 14%|█▎        | 1096/7971 [00:00<00:05, 1201.40it/s]
 15%|█▌        | 1217/7971 [00:01<00:05, 1183.42it/s]
 17%|█▋        | 1336/7971 [00:01<00:05, 1168.03it/s]
 18%|█▊        | 1460/7971 [00:01<00:05, 1182.69it/s]
 20%|█▉        | 1586/7971 [00:01<00:05, 1204.63it/s]
 21%|██▏       | 1709/7971 [00:01<00:05, 1211.41it/s]
 23%|██▎       | 1831/7971 [00:01<00:05, 1199.61it/s]
 24%|██▍       | 1952/7971 [00:01<00:05, 1197.79it/s]
 26%|██▌       | 2072/7971 [00:01<00:04, 1194.27it/s]
 27%|██▋       | 2192/7971 [00:01<00:04, 1195.37it/s]
 29%|██▉       | 2312/7971 [00:01<00:04, 1189.78it/s]
 30%|███       | 2431/7971 [00:02<00:04, 1164.23it/s]
 32%|███▏      | 2548/7971 [00:02<00:04, 1158.74it/s]
 33%|███▎      | 2666/7971 [00:02<00:04, 1161.95it/s]
 35%|███▍      | 2783/7971 [00:02<00:04, 1154.97it/s]
 36%|███▋      | 2899/7971 [00:02<00:04, 1143.71it/s]
 38%|███▊      | 3017/7971 [00:02<00:04, 1147.58it/s]
 39%|███▉      | 3132/7971 [00:02<00:04, 1135.11it/s]
 41%|████      | 3246/7971 [00:02<00:04, 1120.07it/s]
 42%|████▏     | 3363/7971 [00:02<00:04, 1134.04it/s]
 44%|████▎     | 3477/7971 [00:02<00:03, 1126.64it/s]
 45%|████▌     | 3599/7971 [00:03<00:03, 1154.14it/s]
 47%|████▋     | 3715/7971 [00:03<00:03, 1131.80it/s]
 48%|████▊     | 3829/7971 [00:03<00:03, 1116.62it/s]
 49%|████▉     | 3942/7971 [00:03<00:03, 1119.99it/s]
 51%|█████     | 4060/7971 [00:03<00:03, 1135.11it/s]
 52%|█████▏    | 4174/7971 [00:03<00:03, 1131.22it/s]
 54%|█████▍    | 4294/7971 [00:03<00:03, 1145.16it/s]
 55%|█████▌    | 4414/7971 [00:03<00:03, 1161.30it/s]
 57%|█████▋    | 4531/7971 [00:03<00:03, 1136.91it/s]
 58%|█████▊    | 4647/7971 [00:03<00:02, 1138.31it/s]
 60%|█████▉    | 4761/7971 [00:04<00:02, 1134.40it/s]
 61%|██████▏   | 4885/7971 [00:04<00:02, 1159.17it/s]
 63%|██████▎   | 5001/7971 [00:04<00:02, 1143.71it/s]
 64%|██████▍   | 5119/7971 [00:04<00:02, 1154.26it/s]
 66%|██████▌   | 5235/7971 [00:04<00:02, 1147.24it/s]
 67%|██████▋   | 5350/7971 [00:04<00:02, 1142.81it/s]
 69%|██████▊   | 5470/7971 [00:04<00:02, 1159.62it/s]
 70%|███████   | 5587/7971 [00:04<00:02, 1130.63it/s]
 72%|███████▏  | 5703/7971 [00:04<00:01, 1138.93it/s]
 73%|███████▎  | 5823/7971 [00:05<00:01, 1149.57it/s]
 75%|███████▍  | 5939/7971 [00:05<00:01, 1150.33it/s]
 76%|███████▌  | 6055/7971 [00:05<00:01, 1149.60it/s]
 77%|███████▋  | 6171/7971 [00:05<00:01, 1139.68it/s]
 79%|███████▉  | 6286/7971 [00:05<00:01, 1114.13it/s]
 80%|████████  | 6398/7971 [00:05<00:01, 1106.46it/s]
 82%|████████▏ | 6509/7971 [00:05<00:01, 1082.45it/s]
 83%|████████▎ | 6628/7971 [00:05<00:01, 1107.37it/s]
 85%|████████▍ | 6742/7971 [00:05<00:01, 1115.60it/s]
 86%|████████▌ | 6855/7971 [00:05<00:01, 1114.04it/s]
 87%|████████▋ | 6971/7971 [00:06<00:00, 1121.42it/s]
 89%|████████▉ | 7084/7971 [00:06<00:00, 1115.41it/s]
 90%|█████████ | 7196/7971 [00:06<00:00, 1099.03it/s]
 92%|█████████▏| 7307/7971 [00:06<00:00, 1101.13it/s]
 93%|█████████▎| 7418/7971 [00:06<00:00, 1079.04it/s]
 94%|█████████▍| 7527/7971 [00:06<00:00, 1069.19it/s]
 96%|█████████▌| 7635/7971 [00:06<00:00, 1067.55it/s]
 97%|█████████▋| 7742/7971 [00:06<00:00, 1065.68it/s]
 99%|█████████▊| 7858/7971 [00:06<00:00, 1090.31it/s]
100%|█████████▉| 7968/7971 [00:06<00:00, 1063.92it/s]
100%|██████████| 7971/7971 [00:06<00:00, 1140.85it/s]
2026-05-11 04:25:18.704373: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:26:02.009966: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:26:02.014679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:26:02.327663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:26:02.327841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:26:02.363098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:26:02.363188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:26:02.378217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:26:02.393906: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:26:02.412075: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:26:02.428264: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:26:02.445755: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:26:02.450868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:26:02.451353: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:26:02.451622: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:26:02.456168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2026-05-11 04:26:02.456236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:26:02.456265: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:26:02.456288: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:26:02.456309: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:26:02.456330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:26:02.456350: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:26:02.456370: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:26:02.456391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:26:02.461175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:26:02.461229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:26:02.919837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:26:02.919984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:26:02.919998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:26:02.924140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:c3:00.0, compute capability: 8.0)
2026-05-11 04:26:02.968314: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:26:02.980085: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450225000 Hz
2026-05-11 04:26:09.781620: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:26:10.184172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:26:10.190560: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:26:13.181192: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:26:13.285901: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:26:31.610638: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)