Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:04:39.651424: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:45.731767: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:17:45.733500: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:17:45.895340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:17:45.895390: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:46.126545: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:17:46.126621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:17:46.244957: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:17:46.472757: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:17:46.689634: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:17:46.933734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:17:47.186303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:17:47.186809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:17:47.187115: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:17:47.187225: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:17:47.187468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:17:47.187488: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:47.187502: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:17:47.187511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:17:47.187521: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:17:47.187530: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:17:47.187538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:17:47.187547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:17:47.187567: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:17:47.188065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:17:47.188086: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:47.639234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:17:47.639317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:17:47.639327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:17:47.639986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 03:18:08.125548: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:18:08.126030: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:18:09.883189: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:18:10.488214: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:18:10.493073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:18:39.811429: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:18:39.905764: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:20:43.398479: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:41:09.337443: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:42:15.052947: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:17.925127: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:44:17.926016: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:44:17.961462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:44:17.961515: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:17.986518: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:44:17.986610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:44:17.999215: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:44:18.008755: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:44:18.020297: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:44:18.029635: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:44:18.039039: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:44:18.039416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:44:18.039660: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:44:18.039784: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:44:18.040005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:44:18.040025: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:18.040038: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:44:18.040048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:44:18.040059: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:44:18.040069: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:44:18.040078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:44:18.040088: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:44:18.040098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:44:18.040372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:44:18.040393: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:18.453655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:44:18.453744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:44:18.453754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:44:18.454416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/146 [00:00<?, ?it/s]2026-05-11 03:44:20.069105: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:44:20.069589: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:44:20.368250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:44:20.945389: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:44:20.947096: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:44:22.733043: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:44:22.822933: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/146 [00:41<1:39:56, 41.36s/it]
batch:   1%|▏         | 2/146 [00:41<41:12, 17.17s/it]  
batch:   2%|▏         | 3/146 [00:41<22:29,  9.44s/it]
batch:   3%|▎         | 4/146 [00:42<13:43,  5.80s/it]
batch:   3%|▎         | 5/146 [00:42<08:54,  3.79s/it]
batch:   4%|▍         | 6/146 [00:42<06:01,  2.58s/it]
batch:   5%|▍         | 7/146 [00:42<04:12,  1.81s/it]
batch:   5%|▌         | 8/146 [00:42<03:00,  1.31s/it]
batch:   6%|▌         | 9/146 [00:43<02:13,  1.03it/s]
batch:   7%|▋         | 10/146 [00:43<01:41,  1.34it/s]
batch:   8%|▊         | 11/146 [00:43<01:19,  1.70it/s]
batch:   8%|▊         | 12/146 [00:43<01:04,  2.09it/s]
batch:   9%|▉         | 13/146 [00:44<00:53,  2.48it/s]
batch:  10%|▉         | 14/146 [00:44<00:46,  2.85it/s]
batch:  10%|█         | 15/146 [00:44<00:41,  3.17it/s]
batch:  11%|█         | 16/146 [00:44<00:37,  3.44it/s]
batch:  12%|█▏        | 17/146 [00:45<00:35,  3.66it/s]
batch:  12%|█▏        | 18/146 [00:45<00:33,  3.84it/s]
batch:  13%|█▎        | 19/146 [00:45<00:31,  3.97it/s]
batch:  14%|█▎        | 20/146 [00:45<00:31,  4.06it/s]
batch:  14%|█▍        | 21/146 [00:45<00:30,  4.14it/s]
batch:  15%|█▌        | 22/146 [00:46<00:29,  4.18it/s]
batch:  16%|█▌        | 23/146 [00:46<00:29,  4.23it/s]
batch:  16%|█▋        | 24/146 [00:46<00:28,  4.27it/s]
batch:  17%|█▋        | 25/146 [00:46<00:28,  4.28it/s]
batch:  18%|█▊        | 26/146 [00:47<00:27,  4.30it/s]
batch:  18%|█▊        | 27/146 [00:47<00:27,  4.29it/s]
batch:  19%|█▉        | 28/146 [00:47<00:27,  4.31it/s]
batch:  20%|█▉        | 29/146 [00:47<00:27,  4.32it/s]
batch:  21%|██        | 30/146 [00:48<00:26,  4.32it/s]
batch:  21%|██        | 31/146 [00:48<00:26,  4.32it/s]
batch:  22%|██▏       | 32/146 [00:48<00:26,  4.30it/s]
batch:  23%|██▎       | 33/146 [00:48<00:26,  4.31it/s]
batch:  23%|██▎       | 34/146 [00:49<00:26,  4.31it/s]
batch:  24%|██▍       | 35/146 [00:49<00:25,  4.31it/s]
batch:  25%|██▍       | 36/146 [00:49<00:25,  4.32it/s]
batch:  25%|██▌       | 37/146 [00:49<00:25,  4.30it/s]
batch:  26%|██▌       | 38/146 [00:49<00:25,  4.32it/s]
batch:  27%|██▋       | 39/146 [00:50<00:24,  4.32it/s]
batch:  27%|██▋       | 40/146 [00:50<00:24,  4.32it/s]
batch:  28%|██▊       | 41/146 [00:50<00:24,  4.33it/s]
batch:  29%|██▉       | 42/146 [00:50<00:24,  4.31it/s]
batch:  29%|██▉       | 43/146 [00:51<00:23,  4.31it/s]
batch:  30%|███       | 44/146 [00:51<00:23,  4.31it/s]
batch:  31%|███       | 45/146 [00:51<00:23,  4.31it/s]
batch:  32%|███▏      | 46/146 [00:51<00:23,  4.30it/s]
batch:  32%|███▏      | 47/146 [00:52<00:22,  4.32it/s]
batch:  33%|███▎      | 48/146 [00:52<00:22,  4.32it/s]
batch:  34%|███▎      | 49/146 [00:52<00:22,  4.31it/s]
batch:  34%|███▍      | 50/146 [00:52<00:22,  4.32it/s]
batch:  35%|███▍      | 51/146 [00:52<00:22,  4.31it/s]
batch:  36%|███▌      | 52/146 [00:53<00:21,  4.32it/s]
batch:  36%|███▋      | 53/146 [00:53<00:21,  4.31it/s]
batch:  37%|███▋      | 54/146 [00:53<00:21,  4.30it/s]
batch:  38%|███▊      | 55/146 [00:53<00:21,  4.31it/s]
batch:  38%|███▊      | 56/146 [00:54<00:20,  4.31it/s]
batch:  39%|███▉      | 57/146 [00:54<00:20,  4.32it/s]
batch:  40%|███▉      | 58/146 [00:54<00:20,  4.32it/s]
batch:  40%|████      | 59/146 [00:54<00:20,  4.31it/s]
batch:  41%|████      | 60/146 [00:55<00:19,  4.31it/s]
batch:  42%|████▏     | 61/146 [00:55<00:19,  4.31it/s]
batch:  42%|████▏     | 62/146 [00:55<00:19,  4.32it/s]
batch:  43%|████▎     | 63/146 [00:55<00:19,  4.31it/s]
batch:  44%|████▍     | 64/146 [00:55<00:19,  4.31it/s]
batch:  45%|████▍     | 65/146 [00:56<00:18,  4.31it/s]
batch:  45%|████▌     | 66/146 [00:56<00:21,  3.74it/s]
batch:  46%|████▌     | 67/146 [00:56<00:20,  3.89it/s]
batch:  47%|████▋     | 68/146 [00:57<00:19,  4.02it/s]
batch:  47%|████▋     | 69/146 [00:57<00:18,  4.11it/s]
batch:  48%|████▊     | 70/146 [00:57<00:18,  4.17it/s]
batch:  49%|████▊     | 71/146 [00:57<00:17,  4.20it/s]
batch:  49%|████▉     | 72/146 [00:57<00:17,  4.23it/s]
batch:  50%|█████     | 73/146 [00:58<00:17,  4.25it/s]
batch:  51%|█████     | 74/146 [00:58<00:16,  4.27it/s]
batch:  51%|█████▏    | 75/146 [00:58<00:16,  4.28it/s]
batch:  52%|█████▏    | 76/146 [00:58<00:16,  4.28it/s]
batch:  53%|█████▎    | 77/146 [00:59<00:16,  4.29it/s]
batch:  53%|█████▎    | 78/146 [00:59<00:15,  4.29it/s]
batch:  54%|█████▍    | 79/146 [00:59<00:15,  4.30it/s]
batch:  55%|█████▍    | 80/146 [00:59<00:15,  4.31it/s]
batch:  55%|█████▌    | 81/146 [01:00<00:15,  4.30it/s]
batch:  56%|█████▌    | 82/146 [01:00<00:14,  4.31it/s]
batch:  57%|█████▋    | 83/146 [01:00<00:14,  4.31it/s]
batch:  58%|█████▊    | 84/146 [01:00<00:14,  4.31it/s]
batch:  58%|█████▊    | 85/146 [01:00<00:14,  4.30it/s]
batch:  59%|█████▉    | 86/146 [01:01<00:13,  4.31it/s]
batch:  60%|█████▉    | 87/146 [01:01<00:13,  4.31it/s]
batch:  60%|██████    | 88/146 [01:01<00:13,  4.30it/s]
batch:  61%|██████    | 89/146 [01:01<00:13,  4.31it/s]
batch:  62%|██████▏   | 90/146 [01:02<00:12,  4.31it/s]
batch:  62%|██████▏   | 91/146 [01:02<00:12,  4.32it/s]
batch:  63%|██████▎   | 92/146 [01:02<00:12,  4.23it/s]
batch:  64%|██████▎   | 93/146 [01:02<00:12,  4.23it/s]
batch:  64%|██████▍   | 94/146 [01:03<00:12,  4.26it/s]
batch:  65%|██████▌   | 95/146 [01:03<00:11,  4.27it/s]
batch:  66%|██████▌   | 96/146 [01:03<00:11,  4.29it/s]
batch:  66%|██████▋   | 97/146 [01:03<00:11,  4.30it/s]
batch:  67%|██████▋   | 98/146 [01:03<00:11,  4.30it/s]
batch:  68%|██████▊   | 99/146 [01:04<00:10,  4.31it/s]
batch:  68%|██████▊   | 100/146 [01:04<00:10,  4.32it/s]
batch:  69%|██████▉   | 101/146 [01:04<00:10,  4.33it/s]
batch:  70%|██████▉   | 102/146 [01:04<00:10,  4.33it/s]
batch:  71%|███████   | 103/146 [01:05<00:09,  4.33it/s]
batch:  71%|███████   | 104/146 [01:05<00:09,  4.34it/s]
batch:  72%|███████▏  | 105/146 [01:05<00:09,  4.31it/s]
batch:  73%|███████▎  | 106/146 [01:05<00:09,  4.32it/s]
batch:  73%|███████▎  | 107/146 [01:06<00:09,  4.32it/s]
batch:  74%|███████▍  | 108/146 [01:06<00:08,  4.33it/s]
batch:  75%|███████▍  | 109/146 [01:06<00:08,  4.33it/s]
batch:  75%|███████▌  | 110/146 [01:06<00:08,  4.30it/s]
batch:  76%|███████▌  | 111/146 [01:06<00:08,  4.31it/s]
batch:  77%|███████▋  | 112/146 [01:07<00:07,  4.31it/s]
batch:  77%|███████▋  | 113/146 [01:07<00:07,  4.31it/s]
batch:  78%|███████▊  | 114/146 [01:07<00:07,  4.32it/s]
batch:  79%|███████▉  | 115/146 [01:07<00:07,  4.30it/s]
batch:  79%|███████▉  | 116/146 [01:08<00:06,  4.32it/s]
batch:  80%|████████  | 117/146 [01:08<00:06,  4.31it/s]
batch:  81%|████████  | 118/146 [01:08<00:06,  4.31it/s]
batch:  82%|████████▏ | 119/146 [01:08<00:06,  4.31it/s]
batch:  82%|████████▏ | 120/146 [01:09<00:06,  4.31it/s]
batch:  83%|████████▎ | 121/146 [01:09<00:05,  4.31it/s]
batch:  84%|████████▎ | 122/146 [01:09<00:05,  4.29it/s]
batch:  84%|████████▍ | 123/146 [01:09<00:05,  4.30it/s]
batch:  85%|████████▍ | 124/146 [01:10<00:05,  4.31it/s]
batch:  86%|████████▌ | 125/146 [01:10<00:04,  4.28it/s]
batch:  86%|████████▋ | 126/146 [01:10<00:04,  4.30it/s]
batch:  87%|████████▋ | 127/146 [01:10<00:04,  4.29it/s]
batch:  88%|████████▊ | 128/146 [01:10<00:04,  4.30it/s]
batch:  88%|████████▊ | 129/146 [01:11<00:03,  4.30it/s]
batch:  89%|████████▉ | 130/146 [01:11<00:03,  4.31it/s]
batch:  90%|████████▉ | 131/146 [01:11<00:03,  4.32it/s]
batch:  90%|█████████ | 132/146 [01:11<00:03,  4.31it/s]
batch:  91%|█████████ | 133/146 [01:12<00:03,  4.32it/s]
batch:  92%|█████████▏| 134/146 [01:12<00:02,  4.33it/s]
batch:  92%|█████████▏| 135/146 [01:12<00:02,  4.32it/s]
batch:  93%|█████████▎| 136/146 [01:12<00:02,  4.31it/s]
batch:  94%|█████████▍| 137/146 [01:13<00:02,  4.33it/s]
batch:  95%|█████████▍| 138/146 [01:13<00:01,  4.34it/s]
batch:  95%|█████████▌| 139/146 [01:13<00:01,  4.33it/s]
batch:  96%|█████████▌| 140/146 [01:13<00:01,  4.34it/s]
batch:  97%|█████████▋| 141/146 [01:13<00:01,  4.34it/s]
batch:  97%|█████████▋| 142/146 [01:14<00:00,  4.34it/s]
batch:  98%|█████████▊| 143/146 [01:14<00:00,  4.35it/s]
batch:  99%|█████████▊| 144/146 [01:14<00:00,  4.32it/s]
batch:  99%|█████████▉| 145/146 [01:14<00:00,  4.96it/s]
batch: 100%|██████████| 146/146 [01:14<00:00,  1.95it/s]

  0%|          | 0/9247 [00:00<?, ?it/s]
  1%|▏         | 124/9247 [00:00<00:07, 1223.41it/s]
  3%|▎         | 248/9247 [00:00<00:07, 1225.42it/s]
  4%|▍         | 372/9247 [00:00<00:07, 1225.12it/s]
  5%|▌         | 500/9247 [00:00<00:07, 1240.15it/s]
  7%|▋         | 626/9247 [00:00<00:06, 1239.81it/s]
  8%|▊         | 751/9247 [00:00<00:06, 1236.72it/s]
  9%|▉         | 875/9247 [00:00<00:06, 1223.81it/s]
 11%|█         | 998/9247 [00:00<00:06, 1223.01it/s]
 12%|█▏        | 1124/9247 [00:00<00:06, 1234.00it/s]
 13%|█▎        | 1248/9247 [00:01<00:06, 1229.80it/s]
 15%|█▍        | 1371/9247 [00:01<00:06, 1227.90it/s]
 16%|█▌        | 1494/9247 [00:01<00:06, 1225.93it/s]
 17%|█▋        | 1617/9247 [00:01<00:06, 1209.91it/s]
 19%|█▉        | 1739/9247 [00:01<00:06, 1209.33it/s]
 20%|██        | 1860/9247 [00:01<00:06, 1203.12it/s]
 21%|██▏       | 1981/9247 [00:01<00:06, 1203.13it/s]
 23%|██▎       | 2102/9247 [00:01<00:05, 1201.95it/s]
 24%|██▍       | 2223/9247 [00:01<00:05, 1204.34it/s]
 25%|██▌       | 2344/9247 [00:01<00:05, 1201.69it/s]
 27%|██▋       | 2468/9247 [00:02<00:05, 1209.86it/s]
 28%|██▊       | 2589/9247 [00:02<00:05, 1205.98it/s]
 29%|██▉       | 2710/9247 [00:02<00:05, 1201.80it/s]
 31%|███       | 2831/9247 [00:02<00:05, 1200.72it/s]
 32%|███▏      | 2952/9247 [00:02<00:05, 1194.29it/s]
 33%|███▎      | 3072/9247 [00:02<00:05, 1189.82it/s]
 35%|███▍      | 3191/9247 [00:02<00:05, 1182.41it/s]
 36%|███▌      | 3310/9247 [00:02<00:05, 1181.97it/s]
 37%|███▋      | 3429/9247 [00:02<00:04, 1180.71it/s]
 38%|███▊      | 3548/9247 [00:02<00:04, 1179.97it/s]
 40%|███▉      | 3666/9247 [00:03<00:04, 1175.20it/s]
 41%|████      | 3784/9247 [00:03<00:04, 1171.71it/s]
 42%|████▏     | 3902/9247 [00:03<00:04, 1169.42it/s]
 43%|████▎     | 4020/9247 [00:03<00:04, 1167.52it/s]
 45%|████▍     | 4138/9247 [00:03<00:04, 1168.45it/s]
 46%|████▌     | 4257/9247 [00:03<00:04, 1174.66it/s]
 47%|████▋     | 4375/9247 [00:03<00:04, 1169.57it/s]
 49%|████▊     | 4492/9247 [00:03<00:04, 1167.80it/s]
 50%|████▉     | 4609/9247 [00:03<00:03, 1160.92it/s]
 51%|█████     | 4726/9247 [00:03<00:03, 1160.27it/s]
 52%|█████▏    | 4845/9247 [00:04<00:03, 1164.12it/s]
 54%|█████▎    | 4962/9247 [00:04<00:03, 1165.02it/s]
 55%|█████▍    | 5079/9247 [00:04<00:03, 1165.28it/s]
 56%|█████▌    | 5196/9247 [00:04<00:03, 1160.95it/s]
 57%|█████▋    | 5314/9247 [00:04<00:03, 1166.14it/s]
 59%|█████▉    | 5434/9247 [00:04<00:03, 1171.91it/s]
 60%|██████    | 5552/9247 [00:04<00:03, 1172.86it/s]
 61%|██████▏   | 5670/9247 [00:04<00:03, 1173.41it/s]
 63%|██████▎   | 5788/9247 [00:04<00:02, 1164.07it/s]
 64%|██████▍   | 5905/9247 [00:04<00:02, 1146.46it/s]
 65%|██████▌   | 6020/9247 [00:05<00:02, 1142.62it/s]
 66%|██████▋   | 6136/9247 [00:05<00:02, 1143.07it/s]
 68%|██████▊   | 6253/9247 [00:05<00:02, 1144.94it/s]
 69%|██████▉   | 6371/9247 [00:05<00:02, 1150.44it/s]
 70%|███████   | 6487/9247 [00:05<00:02, 1141.95it/s]
 71%|███████▏  | 6603/9247 [00:05<00:02, 1141.26it/s]
 73%|███████▎  | 6718/9247 [00:05<00:02, 1143.77it/s]
 74%|███████▍  | 6836/9247 [00:05<00:02, 1149.77it/s]
 75%|███████▌  | 6953/9247 [00:05<00:01, 1153.56it/s]
 76%|███████▋  | 7069/9247 [00:05<00:01, 1153.48it/s]
 78%|███████▊  | 7185/9247 [00:06<00:01, 1135.31it/s]
 79%|███████▉  | 7299/9247 [00:06<00:01, 1124.21it/s]
 80%|████████  | 7412/9247 [00:06<00:01, 1122.41it/s]
 81%|████████▏ | 7525/9247 [00:06<00:01, 1118.76it/s]
 83%|████████▎ | 7637/9247 [00:06<00:01, 1116.36it/s]
 84%|████████▍ | 7750/9247 [00:06<00:01, 1117.34it/s]
 85%|████████▌ | 7863/9247 [00:06<00:01, 1118.53it/s]
 86%|████████▋ | 7976/9247 [00:06<00:01, 1118.79it/s]
 87%|████████▋ | 8088/9247 [00:06<00:01, 1114.32it/s]
 89%|████████▊ | 8200/9247 [00:06<00:00, 1113.37it/s]
 90%|████████▉ | 8312/9247 [00:07<00:00, 1112.91it/s]
 91%|█████████ | 8426/9247 [00:07<00:00, 1117.35it/s]
 92%|█████████▏| 8541/9247 [00:07<00:00, 1120.91it/s]
 94%|█████████▎| 8656/9247 [00:07<00:00, 1129.46it/s]
 95%|█████████▍| 8769/9247 [00:07<00:00, 1124.35it/s]
 96%|█████████▌| 8882/9247 [00:07<00:00, 1116.92it/s]
 97%|█████████▋| 8994/9247 [00:07<00:00, 1106.55it/s]
 98%|█████████▊| 9105/9247 [00:07<00:00, 1098.77it/s]
100%|█████████▉| 9215/9247 [00:07<00:00, 1096.91it/s]
100%|██████████| 9247/9247 [00:07<00:00, 1164.25it/s]

  0%|          | 0/9247 [00:00<?, ?it/s]
  1%|▏         | 124/9247 [00:00<00:07, 1222.59it/s]
  3%|▎         | 248/9247 [00:00<00:07, 1226.39it/s]
  4%|▍         | 372/9247 [00:00<00:07, 1225.25it/s]
  5%|▌         | 500/9247 [00:00<00:07, 1239.89it/s]
  7%|▋         | 626/9247 [00:00<00:06, 1239.63it/s]
  8%|▊         | 751/9247 [00:00<00:06, 1236.15it/s]
  9%|▉         | 875/9247 [00:00<00:06, 1223.34it/s]
 11%|█         | 998/9247 [00:00<00:06, 1222.39it/s]
 12%|█▏        | 1124/9247 [00:00<00:06, 1233.40it/s]
 13%|█▎        | 1248/9247 [00:01<00:06, 1229.10it/s]
 15%|█▍        | 1371/9247 [00:01<00:06, 1227.39it/s]
 16%|█▌        | 1494/9247 [00:01<00:06, 1225.16it/s]
 17%|█▋        | 1617/9247 [00:01<00:06, 1209.55it/s]
 19%|█▉        | 1738/9247 [00:01<00:06, 1207.47it/s]
 20%|██        | 1859/9247 [00:01<00:06, 1198.79it/s]
 21%|██▏       | 1980/9247 [00:01<00:06, 1200.63it/s]
 23%|██▎       | 2102/9247 [00:01<00:05, 1202.83it/s]
 24%|██▍       | 2224/9247 [00:01<00:05, 1204.22it/s]
 25%|██▌       | 2347/9247 [00:01<00:05, 1207.01it/s]
 27%|██▋       | 2471/9247 [00:02<00:05, 1212.41it/s]
 28%|██▊       | 2593/9247 [00:02<00:05, 1210.78it/s]
 29%|██▉       | 2715/9247 [00:02<00:05, 1206.71it/s]
 31%|███       | 2836/9247 [00:02<00:05, 1198.80it/s]
 32%|███▏      | 2956/9247 [00:02<00:05, 1191.63it/s]
 33%|███▎      | 3076/9247 [00:02<00:05, 1193.96it/s]
 35%|███▍      | 3196/9247 [00:02<00:05, 1180.37it/s]
 36%|███▌      | 3315/9247 [00:02<00:05, 1180.31it/s]
 37%|███▋      | 3434/9247 [00:02<00:04, 1180.13it/s]
 38%|███▊      | 3553/9247 [00:02<00:04, 1179.68it/s]
 40%|███▉      | 3671/9247 [00:03<00:04, 1174.97it/s]
 41%|████      | 3789/9247 [00:03<00:04, 1171.07it/s]
 42%|████▏     | 3908/9247 [00:03<00:04, 1172.12it/s]
 44%|████▎     | 4026/9247 [00:03<00:04, 1169.29it/s]
 45%|████▍     | 4144/9247 [00:03<00:04, 1171.05it/s]
 46%|████▌     | 4263/9247 [00:03<00:04, 1175.91it/s]
 47%|████▋     | 4381/9247 [00:03<00:04, 1166.47it/s]
 49%|████▊     | 4498/9247 [00:03<00:04, 1166.54it/s]
 50%|████▉     | 4615/9247 [00:03<00:03, 1160.46it/s]
 51%|█████     | 4732/9247 [00:03<00:03, 1155.89it/s]
 52%|█████▏    | 4853/9247 [00:04<00:03, 1166.09it/s]
 54%|█████▎    | 4970/9247 [00:04<00:03, 1165.88it/s]
 55%|█████▌    | 5087/9247 [00:04<00:03, 1166.16it/s]
 56%|█████▋    | 5205/9247 [00:04<00:03, 1164.50it/s]
 58%|█████▊    | 5323/9247 [00:04<00:03, 1168.62it/s]
 59%|█████▉    | 5443/9247 [00:04<00:03, 1174.25it/s]
 60%|██████    | 5561/9247 [00:04<00:03, 1173.72it/s]
 61%|██████▏   | 5679/9247 [00:04<00:03, 1174.07it/s]
 63%|██████▎   | 5797/9247 [00:04<00:02, 1159.51it/s]
 64%|██████▍   | 5913/9247 [00:04<00:02, 1147.20it/s]
 65%|██████▌   | 6028/9247 [00:05<00:02, 1143.49it/s]
 66%|██████▋   | 6144/9247 [00:05<00:02, 1142.91it/s]
 68%|██████▊   | 6261/9247 [00:05<00:02, 1145.19it/s]
 69%|██████▉   | 6379/9247 [00:05<00:02, 1151.08it/s]
 70%|███████   | 6495/9247 [00:05<00:02, 1142.66it/s]
 72%|███████▏  | 6612/9247 [00:05<00:02, 1145.03it/s]
 73%|███████▎  | 6728/9247 [00:05<00:02, 1143.27it/s]
 74%|███████▍  | 6845/9247 [00:05<00:02, 1150.93it/s]
 75%|███████▌  | 6961/9247 [00:05<00:01, 1151.71it/s]
 77%|███████▋  | 7077/9247 [00:05<00:01, 1153.11it/s]
 78%|███████▊  | 7193/9247 [00:06<00:01, 1137.37it/s]
 79%|███████▉  | 7307/9247 [00:06<00:01, 1125.63it/s]
 80%|████████  | 7420/9247 [00:06<00:01, 1124.16it/s]
 81%|████████▏ | 7533/9247 [00:06<00:01, 1119.89it/s]
 83%|████████▎ | 7646/9247 [00:06<00:01, 1118.66it/s]
 84%|████████▍ | 7759/9247 [00:06<00:01, 1120.54it/s]
 85%|████████▌ | 7872/9247 [00:06<00:01, 1120.52it/s]
 86%|████████▋ | 7985/9247 [00:06<00:01, 1121.04it/s]
 88%|████████▊ | 8098/9247 [00:06<00:01, 1112.53it/s]
 89%|████████▉ | 8211/9247 [00:07<00:00, 1114.54it/s]
 90%|█████████ | 8323/9247 [00:07<00:00, 1114.27it/s]
 91%|█████████▏| 8438/9247 [00:07<00:00, 1122.43it/s]
 92%|█████████▏| 8553/9247 [00:07<00:00, 1123.86it/s]
 94%|█████████▎| 8668/9247 [00:07<00:00, 1131.56it/s]
 95%|█████████▍| 8782/9247 [00:07<00:00, 1128.27it/s]
 96%|█████████▌| 8895/9247 [00:07<00:00, 1113.39it/s]
 97%|█████████▋| 9007/9247 [00:07<00:00, 1104.35it/s]
 99%|█████████▊| 9118/9247 [00:07<00:00, 1104.31it/s]
100%|█████████▉| 9229/9247 [00:07<00:00, 1095.36it/s]
100%|██████████| 9247/9247 [00:07<00:00, 1164.87it/s]
2026-05-11 03:48:44.446511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:14.204633: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:52:14.217662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:52:14.282048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:52:14.282107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:14.316962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:52:14.317047: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:52:14.334765: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:52:14.353308: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:52:14.375741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:52:14.394129: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:52:14.411549: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:52:14.411958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:52:14.412296: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:52:14.412443: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:52:14.412676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:52:14.412705: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:14.412720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:52:14.412743: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:52:14.412756: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:52:14.412769: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:52:14.412781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:52:14.412794: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:52:14.412806: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:52:14.413101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:52:14.413127: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:14.837798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:52:14.837888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:52:14.837898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:52:14.838620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 03:52:14.878613: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:52:14.892338: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:52:21.699670: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:52:22.194988: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:52:22.198649: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:52:24.087203: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:52:24.172514: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:53:49.341242: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)