Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:35:07.328823: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:36:50.401842: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:36:50.431395: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:36:50.509943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:36:50.509991: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:36:50.556066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:36:50.556133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:36:50.578862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:36:50.600296: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:36:50.624353: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:36:50.646251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:36:50.668017: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:36:50.668384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:36:50.668697: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:36:50.668855: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:36:50.669184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:36:50.669210: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:36:50.669224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:36:50.669233: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:36:50.669242: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:36:50.669251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:36:50.669260: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:36:50.669268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:36:50.669287: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:36:50.669568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:36:50.669587: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:36:51.081354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:36:51.081443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:36:51.081454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:36:51.082116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 03:36:53.491911: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:36:53.492364: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:36:58.738575: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:37:00.870940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:37:00.895440: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:37:08.459707: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:37:08.564773: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:38:43.935346: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:51:09.880210: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:52:39.724368: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:57:15.455866: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:57:15.487259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:57:15.557047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:57:15.557094: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:57:15.607224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:57:15.607311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:57:15.631666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:57:15.655249: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:57:15.681468: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:57:15.705727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:57:15.732525: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:57:15.732909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:57:15.733136: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:57:15.733272: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:57:15.733471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:57:15.733491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:57:15.733505: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:57:15.733516: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:57:15.733526: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:57:15.733535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:57:15.733545: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:57:15.733555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:57:15.733565: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:57:15.733848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:57:15.733869: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:57:16.142676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:57:16.142768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:57:16.142777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:57:16.143447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/126 [00:00<?, ?it/s]2026-05-11 03:57:18.182856: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:57:18.183332: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:57:18.652490: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:57:19.173333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:57:19.175026: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:57:21.049629: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:57:21.147542: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/126 [02:04<4:19:28, 124.55s/it]
batch:   2%|▏         | 2/126 [02:04<1:46:16, 51.43s/it] 
batch:   2%|▏         | 3/126 [02:05<57:30, 28.05s/it]  
batch:   3%|▎         | 4/126 [02:05<34:42, 17.07s/it]
batch:   4%|▍         | 5/126 [02:05<22:10, 11.00s/it]
batch:   5%|▍         | 6/126 [02:05<14:40,  7.34s/it]
batch:   6%|▌         | 7/126 [02:05<09:56,  5.02s/it]
batch:   6%|▋         | 8/126 [02:06<06:52,  3.50s/it]
batch:   7%|▋         | 9/126 [02:06<04:49,  2.48s/it]
batch:   8%|▊         | 10/126 [02:06<03:26,  1.78s/it]
batch:   9%|▊         | 11/126 [02:06<02:30,  1.31s/it]
batch:  10%|▉         | 12/126 [02:07<01:51,  1.02it/s]
batch:  10%|█         | 13/126 [02:07<01:25,  1.33it/s]
batch:  11%|█         | 14/126 [02:07<01:06,  1.68it/s]
batch:  12%|█▏        | 15/126 [02:07<00:54,  2.06it/s]
batch:  13%|█▎        | 16/126 [02:08<00:45,  2.44it/s]
batch:  13%|█▎        | 17/126 [02:08<00:39,  2.79it/s]
batch:  14%|█▍        | 18/126 [02:08<00:34,  3.13it/s]
batch:  15%|█▌        | 19/126 [02:08<00:31,  3.41it/s]
batch:  16%|█▌        | 20/126 [02:08<00:29,  3.63it/s]
batch:  17%|█▋        | 21/126 [02:09<00:27,  3.81it/s]
batch:  17%|█▋        | 22/126 [02:09<00:26,  3.94it/s]
batch:  18%|█▊        | 23/126 [02:09<00:25,  4.04it/s]
batch:  19%|█▉        | 24/126 [02:09<00:24,  4.10it/s]
batch:  20%|█▉        | 25/126 [02:10<00:24,  4.14it/s]
batch:  21%|██        | 26/126 [02:10<00:23,  4.18it/s]
batch:  21%|██▏       | 27/126 [02:10<00:23,  4.20it/s]
batch:  22%|██▏       | 28/126 [02:10<00:23,  4.24it/s]
batch:  23%|██▎       | 29/126 [02:11<00:22,  4.23it/s]
batch:  24%|██▍       | 30/126 [02:11<00:22,  4.24it/s]
batch:  25%|██▍       | 31/126 [02:11<00:22,  4.26it/s]
batch:  25%|██▌       | 32/126 [02:11<00:22,  4.25it/s]
batch:  26%|██▌       | 33/126 [02:12<00:21,  4.27it/s]
batch:  27%|██▋       | 34/126 [02:12<00:21,  4.27it/s]
batch:  28%|██▊       | 35/126 [02:12<00:21,  4.29it/s]
batch:  29%|██▊       | 36/126 [02:12<00:20,  4.29it/s]
batch:  29%|██▉       | 37/126 [02:12<00:20,  4.28it/s]
batch:  30%|███       | 38/126 [02:13<00:20,  4.26it/s]
batch:  31%|███       | 39/126 [02:13<00:20,  4.28it/s]
batch:  32%|███▏      | 40/126 [02:13<00:20,  4.28it/s]
batch:  33%|███▎      | 41/126 [02:13<00:19,  4.30it/s]
batch:  33%|███▎      | 42/126 [02:14<00:19,  4.29it/s]
batch:  34%|███▍      | 43/126 [02:14<00:19,  4.28it/s]
batch:  35%|███▍      | 44/126 [02:14<00:19,  4.27it/s]
batch:  36%|███▌      | 45/126 [02:14<00:18,  4.26it/s]
batch:  37%|███▋      | 46/126 [02:15<00:18,  4.27it/s]
batch:  37%|███▋      | 47/126 [02:15<00:18,  4.26it/s]
batch:  38%|███▊      | 48/126 [02:15<00:18,  4.28it/s]
batch:  39%|███▉      | 49/126 [02:15<00:18,  4.27it/s]
batch:  40%|███▉      | 50/126 [02:16<00:17,  4.27it/s]
batch:  40%|████      | 51/126 [02:16<00:17,  4.26it/s]
batch:  41%|████▏     | 52/126 [02:16<00:17,  4.28it/s]
batch:  42%|████▏     | 53/126 [02:16<00:17,  4.28it/s]
batch:  43%|████▎     | 54/126 [02:16<00:16,  4.26it/s]
batch:  44%|████▎     | 55/126 [02:17<00:16,  4.27it/s]
batch:  44%|████▍     | 56/126 [02:17<00:16,  4.26it/s]
batch:  45%|████▌     | 57/126 [02:17<00:16,  4.25it/s]
batch:  46%|████▌     | 58/126 [02:17<00:16,  4.25it/s]
batch:  47%|████▋     | 59/126 [02:18<00:15,  4.26it/s]
batch:  48%|████▊     | 60/126 [02:18<00:15,  4.27it/s]
batch:  48%|████▊     | 61/126 [02:18<00:15,  4.28it/s]
batch:  49%|████▉     | 62/126 [02:18<00:15,  4.26it/s]
batch:  50%|█████     | 63/126 [02:19<00:14,  4.26it/s]
batch:  51%|█████     | 64/126 [02:19<00:14,  4.28it/s]
batch:  52%|█████▏    | 65/126 [02:19<00:14,  4.29it/s]
batch:  52%|█████▏    | 66/126 [02:19<00:16,  3.73it/s]
batch:  53%|█████▎    | 67/126 [02:20<00:15,  3.88it/s]
batch:  54%|█████▍    | 68/126 [02:20<00:14,  3.97it/s]
batch:  55%|█████▍    | 69/126 [02:20<00:14,  4.07it/s]
batch:  56%|█████▌    | 70/126 [02:20<00:13,  4.14it/s]
batch:  56%|█████▋    | 71/126 [02:21<00:13,  4.19it/s]
batch:  57%|█████▋    | 72/126 [02:21<00:12,  4.22it/s]
batch:  58%|█████▊    | 73/126 [02:21<00:12,  4.25it/s]
batch:  59%|█████▊    | 74/126 [02:21<00:12,  4.26it/s]
batch:  60%|█████▉    | 75/126 [02:21<00:11,  4.28it/s]
batch:  60%|██████    | 76/126 [02:22<00:11,  4.27it/s]
batch:  61%|██████    | 77/126 [02:22<00:11,  4.26it/s]
batch:  62%|██████▏   | 78/126 [02:22<00:11,  4.25it/s]
batch:  63%|██████▎   | 79/126 [02:22<00:11,  4.24it/s]
batch:  63%|██████▎   | 80/126 [02:23<00:10,  4.25it/s]
batch:  64%|██████▍   | 81/126 [02:23<00:10,  4.23it/s]
batch:  65%|██████▌   | 82/126 [02:23<00:10,  4.24it/s]
batch:  66%|██████▌   | 83/126 [02:23<00:10,  4.23it/s]
batch:  67%|██████▋   | 84/126 [02:24<00:09,  4.24it/s]
batch:  67%|██████▋   | 85/126 [02:24<00:09,  4.26it/s]
batch:  68%|██████▊   | 86/126 [02:24<00:09,  4.27it/s]
batch:  69%|██████▉   | 87/126 [02:24<00:09,  4.27it/s]
batch:  70%|██████▉   | 88/126 [02:25<00:08,  4.28it/s]
batch:  71%|███████   | 89/126 [02:25<00:08,  4.30it/s]
batch:  71%|███████▏  | 90/126 [02:25<00:08,  4.29it/s]
batch:  72%|███████▏  | 91/126 [02:25<00:08,  4.29it/s]
batch:  73%|███████▎  | 92/126 [02:25<00:07,  4.29it/s]
batch:  74%|███████▍  | 93/126 [02:26<00:07,  4.27it/s]
batch:  75%|███████▍  | 94/126 [02:26<00:07,  4.28it/s]
batch:  75%|███████▌  | 95/126 [02:26<00:07,  4.30it/s]
batch:  76%|███████▌  | 96/126 [02:26<00:06,  4.31it/s]
batch:  77%|███████▋  | 97/126 [02:27<00:06,  4.32it/s]
batch:  78%|███████▊  | 98/126 [02:27<00:06,  4.29it/s]
batch:  79%|███████▊  | 99/126 [02:27<00:06,  4.32it/s]
batch:  79%|███████▉  | 100/126 [02:27<00:06,  4.30it/s]
batch:  80%|████████  | 101/126 [02:28<00:05,  4.31it/s]
batch:  81%|████████  | 102/126 [02:28<00:05,  4.31it/s]
batch:  82%|████████▏ | 103/126 [02:28<00:05,  4.31it/s]
batch:  83%|████████▎ | 104/126 [02:28<00:05,  4.33it/s]
batch:  83%|████████▎ | 105/126 [02:28<00:04,  4.31it/s]
batch:  84%|████████▍ | 106/126 [02:29<00:04,  4.32it/s]
batch:  85%|████████▍ | 107/126 [02:29<00:04,  4.33it/s]
batch:  86%|████████▌ | 108/126 [02:29<00:04,  4.32it/s]
batch:  87%|████████▋ | 109/126 [02:29<00:03,  4.29it/s]
batch:  87%|████████▋ | 110/126 [02:30<00:03,  4.27it/s]
batch:  88%|████████▊ | 111/126 [02:30<00:03,  4.27it/s]
batch:  89%|████████▉ | 112/126 [02:30<00:03,  4.26it/s]
batch:  90%|████████▉ | 113/126 [02:30<00:03,  4.29it/s]
batch:  90%|█████████ | 114/126 [02:31<00:02,  4.29it/s]
batch:  91%|█████████▏| 115/126 [02:31<00:02,  4.30it/s]
batch:  92%|█████████▏| 116/126 [02:31<00:02,  4.31it/s]
batch:  93%|█████████▎| 117/126 [02:31<00:02,  4.31it/s]
batch:  94%|█████████▎| 118/126 [02:31<00:01,  4.30it/s]
batch:  94%|█████████▍| 119/126 [02:32<00:01,  4.27it/s]
batch:  95%|█████████▌| 120/126 [02:32<00:01,  4.29it/s]
batch:  96%|█████████▌| 121/126 [02:32<00:01,  4.30it/s]
batch:  97%|█████████▋| 122/126 [02:32<00:00,  4.29it/s]
batch:  98%|█████████▊| 123/126 [02:33<00:00,  4.29it/s]
batch:  98%|█████████▊| 124/126 [02:33<00:00,  4.84it/s]
batch:  99%|█████████▉| 125/126 [02:33<00:00,  4.65it/s]
batch: 100%|██████████| 126/126 [02:33<00:00,  1.22s/it]

  0%|          | 0/7971 [00:00<?, ?it/s]
  2%|▏         | 125/7971 [00:00<00:06, 1248.92it/s]
  3%|▎         | 254/7971 [00:00<00:06, 1270.56it/s]
  5%|▍         | 382/7971 [00:00<00:05, 1274.56it/s]
  6%|▋         | 510/7971 [00:00<00:05, 1254.02it/s]
  8%|▊         | 640/7971 [00:00<00:05, 1264.59it/s]
 10%|▉         | 767/7971 [00:00<00:05, 1258.84it/s]
 11%|█▏        | 901/7971 [00:00<00:05, 1277.63it/s]
 13%|█▎        | 1032/7971 [00:00<00:05, 1285.80it/s]
 15%|█▍        | 1161/7971 [00:00<00:05, 1261.71it/s]
 16%|█▌        | 1288/7971 [00:01<00:05, 1243.67it/s]
 18%|█▊        | 1413/7971 [00:01<00:05, 1243.86it/s]
 19%|█▉        | 1548/7971 [00:01<00:05, 1268.61it/s]
 21%|██        | 1681/7971 [00:01<00:04, 1285.84it/s]
 23%|██▎       | 1810/7971 [00:01<00:04, 1259.32it/s]
 24%|██▍       | 1938/7971 [00:01<00:04, 1263.46it/s]
 26%|██▌       | 2066/7971 [00:01<00:04, 1263.70it/s]
 28%|██▊       | 2193/7971 [00:01<00:04, 1265.17it/s]
 29%|██▉       | 2320/7971 [00:01<00:04, 1251.15it/s]
 31%|███       | 2446/7971 [00:01<00:04, 1236.54it/s]
 32%|███▏      | 2570/7971 [00:02<00:04, 1222.86it/s]
 34%|███▍      | 2693/7971 [00:02<00:04, 1221.79it/s]
 35%|███▌      | 2816/7971 [00:02<00:04, 1209.33it/s]
 37%|███▋      | 2937/7971 [00:02<00:04, 1199.93it/s]
 38%|███▊      | 3058/7971 [00:02<00:04, 1199.42it/s]
 40%|███▉      | 3178/7971 [00:02<00:04, 1186.43it/s]
 41%|████▏     | 3297/7971 [00:02<00:03, 1184.03it/s]
 43%|████▎     | 3416/7971 [00:02<00:03, 1185.78it/s]
 44%|████▍     | 3543/7971 [00:02<00:03, 1210.00it/s]
 46%|████▌     | 3665/7971 [00:02<00:03, 1196.50it/s]
 47%|████▋     | 3785/7971 [00:03<00:03, 1176.52it/s]
 49%|████▉     | 3903/7971 [00:03<00:03, 1167.77it/s]
 50%|█████     | 4025/7971 [00:03<00:03, 1180.95it/s]
 52%|█████▏    | 4144/7971 [00:03<00:03, 1182.82it/s]
 54%|█████▎    | 4266/7971 [00:03<00:03, 1186.95it/s]
 55%|█████▌    | 4395/7971 [00:03<00:02, 1214.55it/s]
 57%|█████▋    | 4517/7971 [00:03<00:02, 1189.78it/s]
 58%|█████▊    | 4639/7971 [00:03<00:02, 1196.16it/s]
 60%|█████▉    | 4759/7971 [00:03<00:02, 1183.22it/s]
 61%|██████▏   | 4889/7971 [00:03<00:02, 1213.90it/s]
 63%|██████▎   | 5011/7971 [00:04<00:02, 1192.33it/s]
 64%|██████▍   | 5133/7971 [00:04<00:02, 1198.31it/s]
 66%|██████▌   | 5255/7971 [00:04<00:02, 1200.81it/s]
 67%|██████▋   | 5376/7971 [00:04<00:02, 1185.65it/s]
 69%|██████▉   | 5499/7971 [00:04<00:02, 1194.56it/s]
 71%|███████   | 5621/7971 [00:04<00:01, 1196.85it/s]
 72%|███████▏  | 5741/7971 [00:04<00:01, 1181.07it/s]
 74%|███████▎  | 5866/7971 [00:04<00:01, 1200.67it/s]
 75%|███████▌  | 5987/7971 [00:04<00:01, 1195.17it/s]
 77%|███████▋  | 6107/7971 [00:05<00:01, 1188.44it/s]
 78%|███████▊  | 6226/7971 [00:05<00:01, 1172.94it/s]
 80%|███████▉  | 6344/7971 [00:05<00:01, 1166.37it/s]
 81%|████████  | 6461/7971 [00:05<00:01, 1145.24it/s]
 82%|████████▏ | 6576/7971 [00:05<00:01, 1146.37it/s]
 84%|████████▍ | 6701/7971 [00:05<00:01, 1174.73it/s]
 86%|████████▌ | 6819/7971 [00:05<00:00, 1159.01it/s]
 87%|████████▋ | 6940/7971 [00:05<00:00, 1170.71it/s]
 89%|████████▊ | 7058/7971 [00:05<00:00, 1173.32it/s]
 90%|█████████ | 7176/7971 [00:05<00:00, 1151.54it/s]
 92%|█████████▏| 7295/7971 [00:06<00:00, 1160.92it/s]
 93%|█████████▎| 7412/7971 [00:06<00:00, 1136.22it/s]
 94%|█████████▍| 7526/7971 [00:06<00:00, 1125.31it/s]
 96%|█████████▌| 7639/7971 [00:06<00:00, 1125.00it/s]
 97%|█████████▋| 7752/7971 [00:06<00:00, 1123.91it/s]
 99%|█████████▉| 7872/7971 [00:06<00:00, 1143.07it/s]
100%|██████████| 7971/7971 [00:06<00:00, 1198.25it/s]

  0%|          | 0/7971 [00:00<?, ?it/s]
  2%|▏         | 125/7971 [00:00<00:06, 1248.28it/s]
  3%|▎         | 254/7971 [00:00<00:06, 1268.70it/s]
  5%|▍         | 382/7971 [00:00<00:05, 1273.33it/s]
  6%|▋         | 510/7971 [00:00<00:05, 1253.79it/s]
  8%|▊         | 640/7971 [00:00<00:05, 1264.56it/s]
 10%|▉         | 767/7971 [00:00<00:05, 1258.95it/s]
 11%|█▏        | 901/7971 [00:00<00:05, 1277.14it/s]
 13%|█▎        | 1032/7971 [00:00<00:05, 1285.77it/s]
 15%|█▍        | 1161/7971 [00:00<00:05, 1260.90it/s]
 16%|█▌        | 1288/7971 [00:01<00:05, 1243.63it/s]
 18%|█▊        | 1413/7971 [00:01<00:05, 1243.54it/s]
 19%|█▉        | 1548/7971 [00:01<00:05, 1268.35it/s]
 21%|██        | 1681/7971 [00:01<00:04, 1286.08it/s]
 23%|██▎       | 1810/7971 [00:01<00:04, 1259.86it/s]
 24%|██▍       | 1938/7971 [00:01<00:04, 1259.70it/s]
 26%|██▌       | 2066/7971 [00:01<00:04, 1260.95it/s]
 28%|██▊       | 2193/7971 [00:01<00:04, 1263.55it/s]
 29%|██▉       | 2320/7971 [00:01<00:04, 1253.56it/s]
 31%|███       | 2446/7971 [00:01<00:04, 1238.53it/s]
 32%|███▏      | 2570/7971 [00:02<00:04, 1224.37it/s]
 34%|███▍      | 2693/7971 [00:02<00:04, 1223.50it/s]
 35%|███▌      | 2816/7971 [00:02<00:04, 1208.93it/s]
 37%|███▋      | 2937/7971 [00:02<00:04, 1200.87it/s]
 38%|███▊      | 3058/7971 [00:02<00:04, 1199.84it/s]
 40%|███▉      | 3179/7971 [00:02<00:04, 1188.56it/s]
 41%|████▏     | 3298/7971 [00:02<00:03, 1179.40it/s]
 43%|████▎     | 3418/7971 [00:02<00:03, 1184.81it/s]
 44%|████▍     | 3544/7971 [00:02<00:03, 1206.64it/s]
 46%|████▌     | 3665/7971 [00:02<00:03, 1198.03it/s]
 47%|████▋     | 3785/7971 [00:03<00:03, 1177.81it/s]
 49%|████▉     | 3903/7971 [00:03<00:03, 1168.92it/s]
 50%|█████     | 4025/7971 [00:03<00:03, 1182.49it/s]
 52%|█████▏    | 4144/7971 [00:03<00:03, 1184.55it/s]
 54%|█████▎    | 4266/7971 [00:03<00:03, 1188.35it/s]
 55%|█████▌    | 4395/7971 [00:03<00:02, 1215.71it/s]
 57%|█████▋    | 4517/7971 [00:03<00:02, 1187.09it/s]
 58%|█████▊    | 4639/7971 [00:03<00:02, 1195.24it/s]
 60%|█████▉    | 4759/7971 [00:03<00:02, 1182.39it/s]
 61%|██████▏   | 4889/7971 [00:03<00:02, 1212.35it/s]
 63%|██████▎   | 5011/7971 [00:04<00:02, 1194.67it/s]
 64%|██████▍   | 5133/7971 [00:04<00:02, 1199.57it/s]
 66%|██████▌   | 5255/7971 [00:04<00:02, 1201.61it/s]
 67%|██████▋   | 5376/7971 [00:04<00:02, 1186.16it/s]
 69%|██████▉   | 5499/7971 [00:04<00:02, 1194.59it/s]
 71%|███████   | 5621/7971 [00:04<00:01, 1197.00it/s]
 72%|███████▏  | 5741/7971 [00:04<00:01, 1180.96it/s]
 74%|███████▎  | 5866/7971 [00:04<00:01, 1200.45it/s]
 75%|███████▌  | 5987/7971 [00:04<00:01, 1193.37it/s]
 77%|███████▋  | 6107/7971 [00:05<00:01, 1186.63it/s]
 78%|███████▊  | 6226/7971 [00:05<00:01, 1171.16it/s]
 80%|███████▉  | 6344/7971 [00:05<00:01, 1165.20it/s]
 81%|████████  | 6461/7971 [00:05<00:01, 1144.95it/s]
 82%|████████▏ | 6576/7971 [00:05<00:01, 1145.20it/s]
 84%|████████▍ | 6701/7971 [00:05<00:01, 1173.69it/s]
 86%|████████▌ | 6819/7971 [00:05<00:00, 1157.71it/s]
 87%|████████▋ | 6940/7971 [00:05<00:00, 1170.21it/s]
 89%|████████▊ | 7059/7971 [00:05<00:00, 1175.65it/s]
 90%|█████████ | 7177/7971 [00:05<00:00, 1148.95it/s]
 92%|█████████▏| 7295/7971 [00:06<00:00, 1156.47it/s]
 93%|█████████▎| 7411/7971 [00:06<00:00, 1131.71it/s]
 94%|█████████▍| 7525/7971 [00:06<00:00, 1121.98it/s]
 96%|█████████▌| 7640/7971 [00:06<00:00, 1129.36it/s]
 97%|█████████▋| 7754/7971 [00:06<00:00, 1124.12it/s]
 99%|█████████▉| 7875/7971 [00:06<00:00, 1143.17it/s]
100%|██████████| 7971/7971 [00:06<00:00, 1198.04it/s]
2026-05-11 04:03:45.124115: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:07:48.259486: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:07:48.264213: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:07:48.307998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:07:48.308055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:07:48.361796: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:07:48.361884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:07:48.389534: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:07:48.416809: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:07:48.445410: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:07:48.473811: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:07:48.501146: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:07:48.501565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:07:48.501908: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:07:48.502068: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:07:48.502294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:07:48.502325: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:07:48.502340: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:07:48.502353: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:07:48.502365: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:07:48.502378: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:07:48.502390: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:07:48.502403: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:07:48.502416: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:07:48.502710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:07:48.502738: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:07:48.914051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:07:48.914142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:07:48.914152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:07:48.914865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 04:07:48.954176: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:07:48.967724: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 04:07:54.951076: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:07:55.437266: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:07:55.441029: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:07:57.750913: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:07:57.847294: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:09:14.396241: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)