Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:04:39.651423: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:45.711853: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:17:45.713557: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:17:45.838615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:17:45.838666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:46.070637: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:17:46.070706: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:17:46.244836: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:17:46.472123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:17:46.689413: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:17:46.932542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:17:47.186088: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:17:47.186580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:17:47.186881: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:17:47.187043: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:17:47.187343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:17:47.187369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:47.187383: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:17:47.187392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:17:47.187402: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:17:47.187411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:17:47.187419: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:17:47.187428: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:17:47.187447: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:17:47.187907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:17:47.187929: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:47.635067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:17:47.635171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:17:47.635182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:17:47.635846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
2026-05-11 03:18:35.001051: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:18:35.001532: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:18:40.790600: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:18:42.979252: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:18:43.006746: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:18:47.494863: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:18:47.583238: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:21:05.582380: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:49:48.783509: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:50:53.108084: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:55:15.479856: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:55:15.489994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:55:15.528372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:55:15.528420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:55:15.569200: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:55:15.569279: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:55:15.590137: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:55:15.610194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:55:15.632492: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:55:15.653415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:55:15.674254: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:55:15.674635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:55:15.674858: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:55:15.674999: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:55:15.675205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:55:15.675226: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:55:15.675239: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:55:15.675250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:55:15.675260: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:55:15.675270: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:55:15.675280: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:55:15.675289: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:55:15.675299: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:55:15.675578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:55:15.675598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:55:16.081670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:55:16.081769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:55:16.081779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:55:16.082437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/146 [00:00<?, ?it/s]2026-05-11 03:55:17.943816: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:55:17.944294: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:55:18.311579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:55:18.808710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:55:18.810591: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:55:20.676646: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:55:20.771612: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/146 [01:54<4:36:13, 114.30s/it]
batch:   1%|▏         | 2/146 [01:54<1:53:17, 47.21s/it] 
batch:   2%|▏         | 3/146 [01:54<1:01:23, 25.76s/it]
batch:   3%|▎         | 4/146 [01:55<37:06, 15.68s/it]  
batch:   3%|▎         | 5/146 [01:55<23:45, 10.11s/it]
batch:   4%|▍         | 6/146 [01:55<15:45,  6.75s/it]
batch:   5%|▍         | 7/146 [01:55<10:42,  4.62s/it]
batch:   5%|▌         | 8/146 [01:55<07:24,  3.22s/it]
batch:   6%|▌         | 9/146 [01:56<05:13,  2.29s/it]
batch:   7%|▋         | 10/146 [01:56<03:45,  1.65s/it]
batch:   8%|▊         | 11/146 [01:56<02:44,  1.22s/it]
batch:   8%|▊         | 12/146 [01:56<02:03,  1.09it/s]
batch:   9%|▉         | 13/146 [01:57<01:34,  1.41it/s]
batch:  10%|▉         | 14/146 [01:57<01:14,  1.76it/s]
batch:  10%|█         | 15/146 [01:57<01:01,  2.14it/s]
batch:  11%|█         | 16/146 [01:57<00:51,  2.52it/s]
batch:  12%|█▏        | 17/146 [01:58<00:44,  2.87it/s]
batch:  12%|█▏        | 18/146 [01:58<00:40,  3.19it/s]
batch:  13%|█▎        | 19/146 [01:58<00:36,  3.46it/s]
batch:  14%|█▎        | 20/146 [01:58<00:34,  3.67it/s]
batch:  14%|█▍        | 21/146 [01:58<00:32,  3.84it/s]
batch:  15%|█▌        | 22/146 [01:59<00:31,  3.96it/s]
batch:  16%|█▌        | 23/146 [01:59<00:30,  4.06it/s]
batch:  16%|█▋        | 24/146 [01:59<00:29,  4.14it/s]
batch:  17%|█▋        | 25/146 [01:59<00:28,  4.18it/s]
batch:  18%|█▊        | 26/146 [02:00<00:28,  4.21it/s]
batch:  18%|█▊        | 27/146 [02:00<00:28,  4.21it/s]
batch:  19%|█▉        | 28/146 [02:00<00:27,  4.24it/s]
batch:  20%|█▉        | 29/146 [02:00<00:27,  4.26it/s]
batch:  21%|██        | 30/146 [02:01<00:27,  4.27it/s]
batch:  21%|██        | 31/146 [02:01<00:26,  4.28it/s]
batch:  22%|██▏       | 32/146 [02:01<00:26,  4.26it/s]
batch:  23%|██▎       | 33/146 [02:01<00:26,  4.27it/s]
batch:  23%|██▎       | 34/146 [02:02<00:26,  4.28it/s]
batch:  24%|██▍       | 35/146 [02:02<00:25,  4.28it/s]
batch:  25%|██▍       | 36/146 [02:02<00:25,  4.28it/s]
batch:  25%|██▌       | 37/146 [02:02<00:25,  4.27it/s]
batch:  26%|██▌       | 38/146 [02:02<00:25,  4.28it/s]
batch:  27%|██▋       | 39/146 [02:03<00:25,  4.27it/s]
batch:  27%|██▋       | 40/146 [02:03<00:24,  4.28it/s]
batch:  28%|██▊       | 41/146 [02:03<00:24,  4.29it/s]
batch:  29%|██▉       | 42/146 [02:03<00:24,  4.28it/s]
batch:  29%|██▉       | 43/146 [02:04<00:24,  4.28it/s]
batch:  30%|███       | 44/146 [02:04<00:23,  4.28it/s]
batch:  31%|███       | 45/146 [02:04<00:23,  4.28it/s]
batch:  32%|███▏      | 46/146 [02:04<00:23,  4.29it/s]
batch:  32%|███▏      | 47/146 [02:05<00:23,  4.29it/s]
batch:  33%|███▎      | 48/146 [02:05<00:22,  4.29it/s]
batch:  34%|███▎      | 49/146 [02:05<00:22,  4.28it/s]
batch:  34%|███▍      | 50/146 [02:05<00:22,  4.29it/s]
batch:  35%|███▍      | 51/146 [02:05<00:22,  4.29it/s]
batch:  36%|███▌      | 52/146 [02:06<00:21,  4.29it/s]
batch:  36%|███▋      | 53/146 [02:06<00:21,  4.29it/s]
batch:  37%|███▋      | 54/146 [02:06<00:21,  4.29it/s]
batch:  38%|███▊      | 55/146 [02:06<00:21,  4.30it/s]
batch:  38%|███▊      | 56/146 [02:07<00:21,  4.28it/s]
batch:  39%|███▉      | 57/146 [02:07<00:20,  4.28it/s]
batch:  40%|███▉      | 58/146 [02:07<00:20,  4.29it/s]
batch:  40%|████      | 59/146 [02:07<00:20,  4.28it/s]
batch:  41%|████      | 60/146 [02:08<00:20,  4.30it/s]
batch:  42%|████▏     | 61/146 [02:08<00:19,  4.28it/s]
batch:  42%|████▏     | 62/146 [02:08<00:19,  4.29it/s]
batch:  43%|████▎     | 63/146 [02:08<00:19,  4.29it/s]
batch:  44%|████▍     | 64/146 [02:09<00:19,  4.29it/s]
batch:  45%|████▍     | 65/146 [02:09<00:18,  4.29it/s]
batch:  45%|████▌     | 66/146 [02:09<00:21,  3.73it/s]
batch:  46%|████▌     | 67/146 [02:09<00:20,  3.87it/s]
batch:  47%|████▋     | 68/146 [02:10<00:19,  3.98it/s]
batch:  47%|████▋     | 69/146 [02:10<00:18,  4.08it/s]
batch:  48%|████▊     | 70/146 [02:10<00:18,  4.15it/s]
batch:  49%|████▊     | 71/146 [02:10<00:17,  4.18it/s]
batch:  49%|████▉     | 72/146 [02:10<00:17,  4.21it/s]
batch:  50%|█████     | 73/146 [02:11<00:17,  4.23it/s]
batch:  51%|█████     | 74/146 [02:11<00:16,  4.24it/s]
batch:  51%|█████▏    | 75/146 [02:11<00:16,  4.25it/s]
batch:  52%|█████▏    | 76/146 [02:11<00:16,  4.24it/s]
batch:  53%|█████▎    | 77/146 [02:12<00:16,  4.26it/s]
batch:  53%|█████▎    | 78/146 [02:12<00:15,  4.26it/s]
batch:  54%|█████▍    | 79/146 [02:12<00:15,  4.26it/s]
batch:  55%|█████▍    | 80/146 [02:12<00:15,  4.28it/s]
batch:  55%|█████▌    | 81/146 [02:13<00:15,  4.27it/s]
batch:  56%|█████▌    | 82/146 [02:13<00:14,  4.29it/s]
batch:  57%|█████▋    | 83/146 [02:13<00:14,  4.29it/s]
batch:  58%|█████▊    | 84/146 [02:13<00:14,  4.29it/s]
batch:  58%|█████▊    | 85/146 [02:14<00:14,  4.29it/s]
batch:  59%|█████▉    | 86/146 [02:14<00:14,  4.28it/s]
batch:  60%|█████▉    | 87/146 [02:14<00:13,  4.30it/s]
batch:  60%|██████    | 88/146 [02:14<00:13,  4.28it/s]
batch:  61%|██████    | 89/146 [02:14<00:13,  4.29it/s]
batch:  62%|██████▏   | 90/146 [02:15<00:13,  4.29it/s]
batch:  62%|██████▏   | 91/146 [02:15<00:12,  4.29it/s]
batch:  63%|██████▎   | 92/146 [02:15<00:12,  4.30it/s]
batch:  64%|██████▎   | 93/146 [02:15<00:12,  4.28it/s]
batch:  64%|██████▍   | 94/146 [02:16<00:12,  4.29it/s]
batch:  65%|██████▌   | 95/146 [02:16<00:11,  4.29it/s]
batch:  66%|██████▌   | 96/146 [02:16<00:11,  4.29it/s]
batch:  66%|██████▋   | 97/146 [02:16<00:11,  4.30it/s]
batch:  67%|██████▋   | 98/146 [02:17<00:11,  4.29it/s]
batch:  68%|██████▊   | 99/146 [02:17<00:10,  4.30it/s]
batch:  68%|██████▊   | 100/146 [02:17<00:10,  4.30it/s]
batch:  69%|██████▉   | 101/146 [02:17<00:10,  4.31it/s]
batch:  70%|██████▉   | 102/146 [02:17<00:10,  4.31it/s]
batch:  71%|███████   | 103/146 [02:18<00:09,  4.32it/s]
batch:  71%|███████   | 104/146 [02:18<00:09,  4.32it/s]
batch:  72%|███████▏  | 105/146 [02:18<00:09,  4.30it/s]
batch:  73%|███████▎  | 106/146 [02:18<00:09,  4.30it/s]
batch:  73%|███████▎  | 107/146 [02:19<00:09,  4.30it/s]
batch:  74%|███████▍  | 108/146 [02:19<00:08,  4.30it/s]
batch:  75%|███████▍  | 109/146 [02:19<00:08,  4.32it/s]
batch:  75%|███████▌  | 110/146 [02:19<00:08,  4.29it/s]
batch:  76%|███████▌  | 111/146 [02:20<00:08,  4.30it/s]
batch:  77%|███████▋  | 112/146 [02:20<00:07,  4.30it/s]
batch:  77%|███████▋  | 113/146 [02:20<00:07,  4.31it/s]
batch:  78%|███████▊  | 114/146 [02:20<00:07,  4.32it/s]
batch:  79%|███████▉  | 115/146 [02:21<00:07,  4.30it/s]
batch:  79%|███████▉  | 116/146 [02:21<00:06,  4.31it/s]
batch:  80%|████████  | 117/146 [02:21<00:06,  4.31it/s]
batch:  81%|████████  | 118/146 [02:21<00:06,  4.05it/s]
batch:  82%|████████▏ | 119/146 [02:21<00:06,  4.10it/s]
batch:  82%|████████▏ | 120/146 [02:22<00:06,  4.16it/s]
batch:  83%|████████▎ | 121/146 [02:22<00:05,  4.20it/s]
batch:  84%|████████▎ | 122/146 [02:22<00:05,  4.22it/s]
batch:  84%|████████▍ | 123/146 [02:22<00:05,  4.25it/s]
batch:  85%|████████▍ | 124/146 [02:23<00:05,  4.26it/s]
batch:  86%|████████▌ | 125/146 [02:23<00:04,  4.26it/s]
batch:  86%|████████▋ | 126/146 [02:23<00:04,  4.28it/s]
batch:  87%|████████▋ | 127/146 [02:23<00:04,  4.27it/s]
batch:  88%|████████▊ | 128/146 [02:24<00:04,  4.28it/s]
batch:  88%|████████▊ | 129/146 [02:24<00:03,  4.28it/s]
batch:  89%|████████▉ | 130/146 [02:24<00:03,  4.29it/s]
batch:  90%|████████▉ | 131/146 [02:24<00:03,  4.30it/s]
batch:  90%|█████████ | 132/146 [02:25<00:03,  4.28it/s]
batch:  91%|█████████ | 133/146 [02:25<00:03,  4.29it/s]
batch:  92%|█████████▏| 134/146 [02:25<00:02,  4.30it/s]
batch:  92%|█████████▏| 135/146 [02:25<00:02,  4.29it/s]
batch:  93%|█████████▎| 136/146 [02:25<00:02,  4.29it/s]
batch:  94%|█████████▍| 137/146 [02:26<00:02,  4.31it/s]
batch:  95%|█████████▍| 138/146 [02:26<00:01,  4.31it/s]
batch:  95%|█████████▌| 139/146 [02:26<00:01,  4.31it/s]
batch:  96%|█████████▌| 140/146 [02:26<00:01,  4.31it/s]
batch:  97%|█████████▋| 141/146 [02:27<00:01,  4.31it/s]
batch:  97%|█████████▋| 142/146 [02:27<00:00,  4.30it/s]
batch:  98%|█████████▊| 143/146 [02:27<00:00,  4.31it/s]
batch:  99%|█████████▊| 144/146 [02:27<00:00,  4.28it/s]
batch:  99%|█████████▉| 145/146 [02:27<00:00,  4.93it/s]
batch: 100%|██████████| 146/146 [02:27<00:00,  1.01s/it]

  0%|          | 0/9247 [00:00<?, ?it/s]
  1%|▏         | 124/9247 [00:00<00:07, 1225.59it/s]
  3%|▎         | 248/9247 [00:00<00:07, 1227.04it/s]
  4%|▍         | 372/9247 [00:00<00:07, 1226.86it/s]
  5%|▌         | 500/9247 [00:00<00:07, 1240.05it/s]
  7%|▋         | 625/9247 [00:00<00:06, 1243.36it/s]
  8%|▊         | 750/9247 [00:00<00:06, 1238.97it/s]
  9%|▉         | 874/9247 [00:00<00:06, 1224.51it/s]
 11%|█         | 997/9247 [00:00<00:06, 1217.45it/s]
 12%|█▏        | 1123/9247 [00:00<00:06, 1230.56it/s]
 13%|█▎        | 1247/9247 [00:01<00:06, 1226.97it/s]
 15%|█▍        | 1370/9247 [00:01<00:06, 1226.21it/s]
 16%|█▌        | 1493/9247 [00:01<00:06, 1224.42it/s]
 17%|█▋        | 1616/9247 [00:01<00:06, 1215.11it/s]
 19%|█▉        | 1738/9247 [00:01<00:06, 1208.04it/s]
 20%|██        | 1859/9247 [00:01<00:06, 1201.91it/s]
 21%|██▏       | 1980/9247 [00:01<00:06, 1202.49it/s]
 23%|██▎       | 2102/9247 [00:01<00:05, 1203.86it/s]
 24%|██▍       | 2224/9247 [00:01<00:05, 1205.13it/s]
 25%|██▌       | 2347/9247 [00:01<00:05, 1207.29it/s]
 27%|██▋       | 2471/9247 [00:02<00:05, 1212.38it/s]
 28%|██▊       | 2593/9247 [00:02<00:05, 1211.05it/s]
 29%|██▉       | 2715/9247 [00:02<00:05, 1206.46it/s]
 31%|███       | 2836/9247 [00:02<00:05, 1198.67it/s]
 32%|███▏      | 2956/9247 [00:02<00:05, 1191.71it/s]
 33%|███▎      | 3076/9247 [00:02<00:05, 1194.13it/s]
 35%|███▍      | 3196/9247 [00:02<00:05, 1180.72it/s]
 36%|███▌      | 3315/9247 [00:02<00:05, 1180.44it/s]
 37%|███▋      | 3434/9247 [00:02<00:04, 1180.13it/s]
 38%|███▊      | 3553/9247 [00:02<00:04, 1179.77it/s]
 40%|███▉      | 3671/9247 [00:03<00:04, 1175.63it/s]
 41%|████      | 3789/9247 [00:03<00:04, 1171.91it/s]
 42%|████▏     | 3908/9247 [00:03<00:04, 1172.90it/s]
 44%|████▎     | 4026/9247 [00:03<00:04, 1169.66it/s]
 45%|████▍     | 4144/9247 [00:03<00:04, 1171.71it/s]
 46%|████▌     | 4263/9247 [00:03<00:04, 1176.25it/s]
 47%|████▋     | 4381/9247 [00:03<00:04, 1170.59it/s]
 49%|████▊     | 4499/9247 [00:03<00:04, 1166.03it/s]
 50%|████▉     | 4616/9247 [00:03<00:03, 1158.19it/s]
 51%|█████     | 4732/9247 [00:03<00:03, 1158.45it/s]
 52%|█████▏    | 4853/9247 [00:04<00:03, 1167.86it/s]
 54%|█████▎    | 4970/9247 [00:04<00:03, 1167.38it/s]
 55%|█████▌    | 5087/9247 [00:04<00:03, 1164.08it/s]
 56%|█████▋    | 5205/9247 [00:04<00:03, 1162.65it/s]
 58%|█████▊    | 5322/9247 [00:04<00:03, 1164.81it/s]
 59%|█████▉    | 5443/9247 [00:04<00:03, 1173.29it/s]
 60%|██████    | 5561/9247 [00:04<00:03, 1173.24it/s]
 61%|██████▏   | 5679/9247 [00:04<00:03, 1172.83it/s]
 63%|██████▎   | 5797/9247 [00:04<00:02, 1157.71it/s]
 64%|██████▍   | 5913/9247 [00:04<00:02, 1145.46it/s]
 65%|██████▌   | 6028/9247 [00:05<00:02, 1142.34it/s]
 66%|██████▋   | 6144/9247 [00:05<00:02, 1141.98it/s]
 68%|██████▊   | 6261/9247 [00:05<00:02, 1144.18it/s]
 69%|██████▉   | 6379/9247 [00:05<00:02, 1149.91it/s]
 70%|███████   | 6495/9247 [00:05<00:02, 1141.80it/s]
 72%|███████▏  | 6612/9247 [00:05<00:02, 1144.38it/s]
 73%|███████▎  | 6727/9247 [00:05<00:02, 1145.94it/s]
 74%|███████▍  | 6845/9247 [00:05<00:02, 1151.77it/s]
 75%|███████▌  | 6961/9247 [00:05<00:01, 1152.46it/s]
 77%|███████▋  | 7077/9247 [00:05<00:01, 1153.28it/s]
 78%|███████▊  | 7193/9247 [00:06<00:01, 1137.96it/s]
 79%|███████▉  | 7307/9247 [00:06<00:01, 1125.75it/s]
 80%|████████  | 7420/9247 [00:06<00:01, 1123.96it/s]
 81%|████████▏ | 7533/9247 [00:06<00:01, 1116.57it/s]
 83%|████████▎ | 7646/9247 [00:06<00:01, 1117.38it/s]
 84%|████████▍ | 7759/9247 [00:06<00:01, 1117.88it/s]
 85%|████████▌ | 7872/9247 [00:06<00:01, 1119.09it/s]
 86%|████████▋ | 7984/9247 [00:06<00:01, 1117.44it/s]
 88%|████████▊ | 8096/9247 [00:06<00:01, 1112.83it/s]
 89%|████████▉ | 8208/9247 [00:07<00:00, 1112.42it/s]
 90%|████████▉ | 8320/9247 [00:07<00:00, 1113.01it/s]
 91%|█████████ | 8435/9247 [00:07<00:00, 1120.72it/s]
 92%|█████████▏| 8549/9247 [00:07<00:00, 1121.20it/s]
 94%|█████████▎| 8664/9247 [00:07<00:00, 1128.76it/s]
 95%|█████████▍| 8777/9247 [00:07<00:00, 1124.81it/s]
 96%|█████████▌| 8890/9247 [00:07<00:00, 1117.11it/s]
 97%|█████████▋| 9002/9247 [00:07<00:00, 1107.28it/s]
 99%|█████████▊| 9113/9247 [00:07<00:00, 1105.95it/s]
100%|█████████▉| 9224/9247 [00:07<00:00, 1099.29it/s]
100%|██████████| 9247/9247 [00:07<00:00, 1164.82it/s]

  0%|          | 0/9247 [00:00<?, ?it/s]
  1%|▏         | 124/9247 [00:00<00:07, 1224.07it/s]
  3%|▎         | 248/9247 [00:00<00:07, 1225.44it/s]
  4%|▍         | 372/9247 [00:00<00:07, 1225.39it/s]
  5%|▌         | 500/9247 [00:00<00:07, 1240.06it/s]
  7%|▋         | 626/9247 [00:00<00:06, 1239.16it/s]
  8%|▊         | 751/9247 [00:00<00:06, 1236.29it/s]
  9%|▉         | 875/9247 [00:00<00:06, 1223.03it/s]
 11%|█         | 998/9247 [00:00<00:06, 1222.50it/s]
 12%|█▏        | 1123/9247 [00:00<00:06, 1230.79it/s]
 13%|█▎        | 1247/9247 [00:01<00:06, 1228.06it/s]
 15%|█▍        | 1370/9247 [00:01<00:06, 1227.22it/s]
 16%|█▌        | 1493/9247 [00:01<00:06, 1225.35it/s]
 17%|█▋        | 1616/9247 [00:01<00:06, 1215.05it/s]
 19%|█▉        | 1738/9247 [00:01<00:06, 1207.92it/s]
 20%|██        | 1859/9247 [00:01<00:06, 1202.11it/s]
 21%|██▏       | 1980/9247 [00:01<00:06, 1200.42it/s]
 23%|██▎       | 2102/9247 [00:01<00:05, 1202.66it/s]
 24%|██▍       | 2224/9247 [00:01<00:05, 1204.24it/s]
 25%|██▌       | 2347/9247 [00:01<00:05, 1207.38it/s]
 27%|██▋       | 2471/9247 [00:02<00:05, 1212.41it/s]
 28%|██▊       | 2593/9247 [00:02<00:05, 1210.99it/s]
 29%|██▉       | 2715/9247 [00:02<00:05, 1206.58it/s]
 31%|███       | 2836/9247 [00:02<00:05, 1198.79it/s]
 32%|███▏      | 2956/9247 [00:02<00:05, 1191.94it/s]
 33%|███▎      | 3077/9247 [00:02<00:05, 1196.53it/s]
 35%|███▍      | 3197/9247 [00:02<00:05, 1182.41it/s]
 36%|███▌      | 3316/9247 [00:02<00:05, 1175.82it/s]
 37%|███▋      | 3435/9247 [00:02<00:04, 1177.01it/s]
 38%|███▊      | 3555/9247 [00:02<00:04, 1179.32it/s]
 40%|███▉      | 3673/9247 [00:03<00:04, 1174.41it/s]
 41%|████      | 3792/9247 [00:03<00:04, 1173.32it/s]
 42%|████▏     | 3911/9247 [00:03<00:04, 1173.60it/s]
 44%|████▎     | 4029/9247 [00:03<00:04, 1169.67it/s]
 45%|████▍     | 4147/9247 [00:03<00:04, 1170.97it/s]
 46%|████▌     | 4266/9247 [00:03<00:04, 1175.78it/s]
 47%|████▋     | 4384/9247 [00:03<00:04, 1170.11it/s]
 49%|████▊     | 4502/9247 [00:03<00:04, 1162.02it/s]
 50%|████▉     | 4619/9247 [00:03<00:03, 1157.30it/s]
 51%|█████     | 4735/9247 [00:03<00:03, 1157.05it/s]
 53%|█████▎    | 4856/9247 [00:04<00:03, 1167.00it/s]
 54%|█████▍    | 4973/9247 [00:04<00:03, 1166.33it/s]
 55%|█████▌    | 5090/9247 [00:04<00:03, 1167.29it/s]
 56%|█████▋    | 5207/9247 [00:04<00:03, 1162.62it/s]
 58%|█████▊    | 5325/9247 [00:04<00:03, 1166.90it/s]
 59%|█████▉    | 5446/9247 [00:04<00:03, 1175.22it/s]
 60%|██████    | 5564/9247 [00:04<00:03, 1174.90it/s]
 61%|██████▏   | 5682/9247 [00:04<00:03, 1174.73it/s]
 63%|██████▎   | 5800/9247 [00:04<00:02, 1159.53it/s]
 64%|██████▍   | 5916/9247 [00:04<00:02, 1146.91it/s]
 65%|██████▌   | 6031/9247 [00:05<00:02, 1143.32it/s]
 66%|██████▋   | 6147/9247 [00:05<00:02, 1142.33it/s]
 68%|██████▊   | 6265/9247 [00:05<00:02, 1147.86it/s]
 69%|██████▉   | 6382/9247 [00:05<00:02, 1149.69it/s]
 70%|███████   | 6497/9247 [00:05<00:02, 1139.04it/s]
 72%|███████▏  | 6615/9247 [00:05<00:02, 1144.87it/s]
 73%|███████▎  | 6731/9247 [00:05<00:02, 1149.19it/s]
 74%|███████▍  | 6848/9247 [00:05<00:02, 1151.48it/s]
 75%|███████▌  | 6964/9247 [00:05<00:01, 1152.16it/s]
 77%|███████▋  | 7080/9247 [00:05<00:01, 1150.78it/s]
 78%|███████▊  | 7196/9247 [00:06<00:01, 1136.26it/s]
 79%|███████▉  | 7310/9247 [00:06<00:01, 1125.17it/s]
 80%|████████  | 7423/9247 [00:06<00:01, 1122.94it/s]
 81%|████████▏ | 7536/9247 [00:06<00:01, 1120.03it/s]
 83%|████████▎ | 7649/9247 [00:06<00:01, 1119.00it/s]
 84%|████████▍ | 7762/9247 [00:06<00:01, 1119.42it/s]
 85%|████████▌ | 7875/9247 [00:06<00:01, 1120.31it/s]
 86%|████████▋ | 7988/9247 [00:06<00:01, 1120.24it/s]
 88%|████████▊ | 8101/9247 [00:06<00:01, 1111.28it/s]
 89%|████████▉ | 8215/9247 [00:07<00:00, 1116.12it/s]
 90%|█████████ | 8327/9247 [00:07<00:00, 1115.88it/s]
 91%|█████████▏| 8441/9247 [00:07<00:00, 1120.53it/s]
 93%|█████████▎| 8555/9247 [00:07<00:00, 1120.42it/s]
 94%|█████████▍| 8671/9247 [00:07<00:00, 1131.59it/s]
 95%|█████████▌| 8785/9247 [00:07<00:00, 1128.49it/s]
 96%|█████████▌| 8898/9247 [00:07<00:00, 1113.02it/s]
 97%|█████████▋| 9010/9247 [00:07<00:00, 1103.64it/s]
 99%|█████████▊| 9121/9247 [00:07<00:00, 1103.31it/s]
100%|█████████▉| 9232/9247 [00:07<00:00, 1097.09it/s]
100%|██████████| 9247/9247 [00:07<00:00, 1164.90it/s]
2026-05-11 04:01:40.429646: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:05:13.685084: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:05:13.697926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:05:13.763380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:05:13.763439: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:05:13.860479: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:05:13.860566: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:05:13.911700: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:05:13.962630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:05:14.016988: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:05:14.055548: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:05:14.097504: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:05:14.097910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:05:14.098239: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:05:14.098365: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:05:14.098599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:05:14.098630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:05:14.098646: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:05:14.098671: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:05:14.098684: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:05:14.098697: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:05:14.098710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:05:14.098722: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:05:14.098735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:05:14.099028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:05:14.099053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:05:14.500936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:05:14.501024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:05:14.501034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:05:14.501727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
2026-05-11 04:05:14.541995: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:05:14.555498: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 04:05:21.304206: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:05:21.785229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:05:21.788981: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:05:23.529853: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:05:23.610600: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:07:09.766976: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)