Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:23:38.732347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:46.961559: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:28:46.967338: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:28:47.049300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:28:47.049392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:47.094664: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:28:47.094738: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:28:47.117257: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:28:47.138999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:28:47.162792: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:28:47.186211: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:28:47.209993: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:28:47.210408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:28:47.210785: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:28:47.210940: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:28:47.211291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:28:47.211317: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:47.211331: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:28:47.211341: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:28:47.211351: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:28:47.211360: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:28:47.211368: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:28:47.211377: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:28:47.211402: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:28:47.211686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:28:47.211706: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:28:47.629313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:28:47.629407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:28:47.629417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:28:47.630077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
2026-05-11 03:28:49.458469: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:28:49.458947: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:28:54.039569: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:28:54.528595: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:28:54.533358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:29:07.511031: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:29:07.608826: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:30:20.110126: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:46:15.605028: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:48:45.109710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:54:38.606352: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:54:38.611586: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:54:38.631780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:54:38.631830: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:54:38.637057: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:54:38.637137: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:54:38.639781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:54:38.641535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:54:38.645061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:54:38.647217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:54:38.648833: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:54:38.649224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:54:38.649482: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:54:38.649618: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:54:38.649820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:54:38.649838: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:54:38.649852: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:54:38.649862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:54:38.649872: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:54:38.649883: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:54:38.649893: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:54:38.649903: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:54:38.649913: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:54:38.650197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:54:38.650217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:54:39.065083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:54:39.065174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:54:39.065184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:54:39.065842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/68 [00:00<?, ?it/s]2026-05-11 03:54:40.839084: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:54:40.839556: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:54:41.188120: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:54:41.734357: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:54:41.741970: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:54:43.455367: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:54:43.554064: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|▏         | 1/68 [02:59<3:20:21, 179.42s/it]
batch:   3%|▎         | 2/68 [02:59<1:21:25, 74.02s/it] 
batch:   4%|▍         | 3/68 [02:59<43:41, 40.33s/it]  
batch:   6%|▌         | 4/68 [03:00<26:08, 24.50s/it]
batch:   7%|▋         | 5/68 [03:00<16:32, 15.75s/it]
batch:   9%|▉         | 6/68 [03:00<10:49, 10.48s/it]
batch:  10%|█         | 7/68 [03:00<07:14,  7.13s/it]
batch:  12%|█▏        | 8/68 [03:01<04:56,  4.94s/it]
batch:  13%|█▎        | 9/68 [03:01<03:24,  3.47s/it]
batch:  15%|█▍        | 10/68 [03:01<02:23,  2.47s/it]
batch:  16%|█▌        | 11/68 [03:01<01:41,  1.79s/it]
batch:  18%|█▊        | 12/68 [03:02<01:13,  1.32s/it]
batch:  19%|█▉        | 13/68 [03:02<00:54,  1.01it/s]
batch:  21%|██        | 14/68 [03:02<00:41,  1.31it/s]
batch:  22%|██▏       | 15/68 [03:02<00:32,  1.65it/s]
batch:  24%|██▎       | 16/68 [03:03<00:25,  2.03it/s]
batch:  25%|██▌       | 17/68 [03:03<00:21,  2.40it/s]
batch:  26%|██▋       | 18/68 [03:03<00:18,  2.76it/s]
batch:  28%|██▊       | 19/68 [03:03<00:15,  3.07it/s]
batch:  29%|██▉       | 20/68 [03:03<00:14,  3.33it/s]
batch:  31%|███       | 21/68 [03:04<00:13,  3.56it/s]
batch:  32%|███▏      | 22/68 [03:04<00:12,  3.73it/s]
batch:  34%|███▍      | 23/68 [03:04<00:11,  3.86it/s]
batch:  35%|███▌      | 24/68 [03:04<00:11,  3.97it/s]
batch:  37%|███▋      | 25/68 [03:05<00:10,  4.03it/s]
batch:  38%|███▊      | 26/68 [03:05<00:10,  4.09it/s]
batch:  40%|███▉      | 27/68 [03:05<00:09,  4.11it/s]
batch:  41%|████      | 28/68 [03:05<00:09,  4.14it/s]
batch:  43%|████▎     | 29/68 [03:06<00:09,  4.15it/s]
batch:  44%|████▍     | 30/68 [03:06<00:09,  4.17it/s]
batch:  46%|████▌     | 31/68 [03:06<00:08,  4.18it/s]
batch:  47%|████▋     | 32/68 [03:06<00:08,  4.18it/s]
batch:  49%|████▊     | 33/68 [03:07<00:08,  4.19it/s]
batch:  50%|█████     | 34/68 [03:07<00:08,  4.18it/s]
batch:  51%|█████▏    | 35/68 [03:07<00:07,  4.19it/s]
batch:  53%|█████▎    | 36/68 [03:07<00:07,  4.18it/s]
batch:  54%|█████▍    | 37/68 [03:08<00:07,  4.17it/s]
batch:  56%|█████▌    | 38/68 [03:08<00:07,  4.19it/s]
batch:  57%|█████▋    | 39/68 [03:08<00:06,  4.16it/s]
batch:  59%|█████▉    | 40/68 [03:08<00:06,  4.17it/s]
batch:  60%|██████    | 41/68 [03:08<00:06,  4.19it/s]
batch:  62%|██████▏   | 42/68 [03:09<00:06,  4.18it/s]
batch:  63%|██████▎   | 43/68 [03:09<00:05,  4.19it/s]
batch:  65%|██████▍   | 44/68 [03:09<00:05,  4.19it/s]
batch:  66%|██████▌   | 45/68 [03:09<00:05,  4.20it/s]
batch:  68%|██████▊   | 46/68 [03:10<00:05,  4.19it/s]
batch:  69%|██████▉   | 47/68 [03:10<00:04,  4.20it/s]
batch:  71%|███████   | 48/68 [03:10<00:04,  4.21it/s]
batch:  72%|███████▏  | 49/68 [03:10<00:04,  4.20it/s]
batch:  74%|███████▎  | 50/68 [03:11<00:04,  4.19it/s]
batch:  75%|███████▌  | 51/68 [03:11<00:04,  4.20it/s]
batch:  76%|███████▋  | 52/68 [03:11<00:03,  4.20it/s]
batch:  78%|███████▊  | 53/68 [03:11<00:03,  4.21it/s]
batch:  79%|███████▉  | 54/68 [03:12<00:03,  4.20it/s]
batch:  81%|████████  | 55/68 [03:12<00:03,  4.20it/s]
batch:  82%|████████▏ | 56/68 [03:12<00:02,  4.21it/s]
batch:  84%|████████▍ | 57/68 [03:12<00:02,  4.21it/s]
batch:  85%|████████▌ | 58/68 [03:13<00:02,  4.21it/s]
batch:  87%|████████▋ | 59/68 [03:13<00:02,  4.20it/s]
batch:  88%|████████▊ | 60/68 [03:13<00:01,  4.21it/s]
batch:  90%|████████▉ | 61/68 [03:13<00:01,  4.20it/s]
batch:  91%|█████████ | 62/68 [03:13<00:01,  4.22it/s]
batch:  93%|█████████▎| 63/68 [03:14<00:01,  4.21it/s]
batch:  94%|█████████▍| 64/68 [03:14<00:00,  4.23it/s]
batch:  96%|█████████▌| 65/68 [03:14<00:00,  4.24it/s]
batch:  97%|█████████▋| 66/68 [03:15<00:00,  3.68it/s]
batch:  99%|█████████▊| 67/68 [03:15<00:00,  3.83it/s]
batch: 100%|██████████| 68/68 [03:15<00:00,  4.61it/s]
batch: 100%|██████████| 68/68 [03:15<00:00,  2.87s/it]

  0%|          | 0/4312 [00:00<?, ?it/s]
  3%|▎         | 126/4312 [00:00<00:03, 1242.24it/s]
  6%|▌         | 251/4312 [00:00<00:03, 1238.68it/s]
  9%|▊         | 375/4312 [00:00<00:03, 1234.44it/s]
 12%|█▏        | 501/4312 [00:00<00:03, 1236.94it/s]
 15%|█▍        | 626/4312 [00:00<00:02, 1240.76it/s]
 17%|█▋        | 751/4312 [00:00<00:02, 1229.41it/s]
 20%|██        | 874/4312 [00:00<00:02, 1225.04it/s]
 23%|██▎       | 997/4312 [00:00<00:02, 1216.62it/s]
 26%|██▌       | 1120/4312 [00:00<00:02, 1217.00it/s]
 29%|██▉       | 1244/4312 [00:01<00:02, 1223.33it/s]
 32%|███▏      | 1367/4312 [00:01<00:02, 1221.79it/s]
 35%|███▍      | 1490/4312 [00:01<00:02, 1217.39it/s]
 37%|███▋      | 1612/4312 [00:01<00:02, 1198.56it/s]
 40%|████      | 1732/4312 [00:01<00:02, 1193.27it/s]
 43%|████▎     | 1852/4312 [00:01<00:02, 1191.55it/s]
 46%|████▌     | 1972/4312 [00:01<00:01, 1192.21it/s]
 49%|████▊     | 2092/4312 [00:01<00:01, 1178.86it/s]
 51%|█████▏    | 2210/4312 [00:01<00:01, 1175.65it/s]
 54%|█████▍    | 2328/4312 [00:01<00:01, 1176.77it/s]
 57%|█████▋    | 2447/4312 [00:02<00:01, 1180.25it/s]
 60%|█████▉    | 2566/4312 [00:02<00:01, 1164.65it/s]
 62%|██████▏   | 2683/4312 [00:02<00:01, 1160.57it/s]
 65%|██████▍   | 2800/4312 [00:02<00:01, 1144.52it/s]
 68%|██████▊   | 2915/4312 [00:02<00:01, 1138.78it/s]
 70%|███████   | 3029/4312 [00:02<00:01, 1136.36it/s]
 73%|███████▎  | 3144/4312 [00:02<00:01, 1140.22it/s]
 76%|███████▌  | 3259/4312 [00:02<00:00, 1138.19it/s]
 78%|███████▊  | 3374/4312 [00:02<00:00, 1139.76it/s]
 81%|████████  | 3488/4312 [00:02<00:00, 1136.40it/s]
 84%|████████▎ | 3603/4312 [00:03<00:00, 1136.78it/s]
 86%|████████▌ | 3718/4312 [00:03<00:00, 1134.99it/s]
 89%|████████▉ | 3832/4312 [00:03<00:00, 1129.99it/s]
 92%|█████████▏| 3946/4312 [00:03<00:00, 1116.84it/s]
 94%|█████████▍| 4064/4312 [00:03<00:00, 1129.27it/s]
 97%|█████████▋| 4177/4312 [00:03<00:00, 1120.13it/s]
 99%|█████████▉| 4290/4312 [00:03<00:00, 1107.57it/s]
100%|██████████| 4312/4312 [00:03<00:00, 1168.83it/s]

  0%|          | 0/4312 [00:00<?, ?it/s]
  3%|▎         | 126/4312 [00:00<00:03, 1242.88it/s]
  6%|▌         | 251/4312 [00:00<00:03, 1239.00it/s]
  9%|▊         | 375/4312 [00:00<00:03, 1235.61it/s]
 12%|█▏        | 501/4312 [00:00<00:03, 1237.95it/s]
 14%|█▍        | 625/4312 [00:00<00:02, 1238.42it/s]
 17%|█▋        | 749/4312 [00:00<00:02, 1226.04it/s]
 20%|██        | 872/4312 [00:00<00:02, 1222.40it/s]
 23%|██▎       | 995/4312 [00:00<00:02, 1221.52it/s]
 26%|██▌       | 1118/4312 [00:00<00:02, 1220.84it/s]
 29%|██▉       | 1242/4312 [00:01<00:02, 1225.87it/s]
 32%|███▏      | 1365/4312 [00:01<00:02, 1224.78it/s]
 35%|███▍      | 1488/4312 [00:01<00:02, 1219.41it/s]
 37%|███▋      | 1610/4312 [00:01<00:02, 1206.37it/s]
 40%|████      | 1731/4312 [00:01<00:02, 1191.74it/s]
 43%|████▎     | 1852/4312 [00:01<00:02, 1193.49it/s]
 46%|████▌     | 1973/4312 [00:01<00:01, 1197.81it/s]
 49%|████▊     | 2093/4312 [00:01<00:01, 1184.41it/s]
 51%|█████▏    | 2212/4312 [00:01<00:01, 1177.11it/s]
 54%|█████▍    | 2332/4312 [00:01<00:01, 1183.45it/s]
 57%|█████▋    | 2451/4312 [00:02<00:01, 1180.56it/s]
 60%|█████▉    | 2570/4312 [00:02<00:01, 1171.75it/s]
 62%|██████▏   | 2688/4312 [00:02<00:01, 1163.28it/s]
 65%|██████▌   | 2805/4312 [00:02<00:01, 1152.76it/s]
 68%|██████▊   | 2921/4312 [00:02<00:01, 1143.02it/s]
 70%|███████   | 3036/4312 [00:02<00:01, 1136.84it/s]
 73%|███████▎  | 3154/4312 [00:02<00:01, 1148.74it/s]
 76%|███████▌  | 3269/4312 [00:02<00:00, 1145.04it/s]
 78%|███████▊  | 3384/4312 [00:02<00:00, 1145.90it/s]
 81%|████████  | 3499/4312 [00:02<00:00, 1144.02it/s]
 84%|████████▍ | 3614/4312 [00:03<00:00, 1142.80it/s]
 86%|████████▋ | 3729/4312 [00:03<00:00, 1138.99it/s]
 89%|████████▉ | 3843/4312 [00:03<00:00, 1128.73it/s]
 92%|█████████▏| 3956/4312 [00:03<00:00, 1120.91it/s]
 94%|█████████▍| 4072/4312 [00:03<00:00, 1128.52it/s]
 97%|█████████▋| 4185/4312 [00:03<00:00, 1122.01it/s]
100%|█████████▉| 4298/4312 [00:03<00:00, 1109.29it/s]
100%|██████████| 4312/4312 [00:03<00:00, 1172.13it/s]
2026-05-11 04:04:46.062080: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:12:15.116228: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:12:15.147236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:12:15.206755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:12:15.206843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:12:15.251206: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:12:15.251322: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:12:15.274365: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:12:15.296079: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:12:15.319926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:12:15.342748: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:12:15.365162: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:12:15.365604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:12:15.365982: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:12:15.366135: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:12:15.366369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:12:15.366399: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:12:15.366414: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:12:15.366427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:12:15.366440: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:12:15.366452: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:12:15.366464: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:12:15.366477: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:12:15.366489: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:12:15.366783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:12:15.366808: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:12:15.782033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:12:15.782126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:12:15.782135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:12:15.782830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
2026-05-11 04:12:15.822060: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:12:15.835931: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 04:12:19.584764: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:12:20.070342: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:12:20.073962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:12:21.967445: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:12:22.064911: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:14:15.236779: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)