Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-11-17 18:04:13.178747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:04:27.972370: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 18:04:27.973808: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 18:04:28.000316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 18:04:28.000366: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:04:28.005303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:04:28.005368: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:04:28.007739: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 18:04:28.009313: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 18:04:28.012914: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 18:04:28.014827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 18:04:28.016272: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:04:28.016648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 18:04:28.017019: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 18:04:28.018714: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 18:04:28.018974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 18:04:28.018995: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:04:28.019010: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:04:28.019020: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:04:28.019030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 18:04:28.019039: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 18:04:28.019048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 18:04:28.019057: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 18:04:28.019077: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:04:28.019364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 18:04:28.019384: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:04:28.451898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 18:04:28.451981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 18:04:28.451992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 18:04:28.452689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-11-17 18:04:41.459807: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 18:04:41.460528: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 18:04:42.612085: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:04:43.154562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:04:43.161892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:04:44.716287: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 18:04:44.849249: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 18:05:11.065672: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-11-17 18:12:38.764899: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-11-17 18:12:43.080619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:12:50.502704: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 18:12:50.503591: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 18:12:50.530668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 18:12:50.530716: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:12:50.536072: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:12:50.536159: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:12:50.538877: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 18:12:50.540870: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 18:12:50.544951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 18:12:50.546944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 18:12:50.548473: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:12:50.548847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 18:12:50.549088: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 18:12:50.550601: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 18:12:50.550859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 18:12:50.550883: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:12:50.550897: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:12:50.550908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:12:50.550918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 18:12:50.550928: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 18:12:50.550938: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 18:12:50.550948: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 18:12:50.550958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:12:50.551245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 18:12:50.551266: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:12:50.988939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 18:12:50.989023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 18:12:50.989033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 18:12:50.989699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/92 [00:00<?, ?it/s]2025-11-17 18:12:52.600889: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 18:12:52.601355: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 18:12:52.801780: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:12:53.307492: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:12:53.309560: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:12:54.974406: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 18:12:55.094831: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|          | 1/92 [00:32<49:15, 32.48s/it]
batch:   2%|▏         | 2/92 [00:32<20:16, 13.52s/it]
batch:   3%|▎         | 3/92 [00:32<11:03,  7.46s/it]
batch:   4%|▍         | 4/92 [00:33<06:45,  4.61s/it]
batch:   5%|▌         | 5/92 [00:33<04:23,  3.03s/it]
batch:   7%|▋         | 6/92 [00:33<02:59,  2.08s/it]
batch:   8%|▊         | 7/92 [00:33<02:05,  1.48s/it]
batch:   9%|▊         | 8/92 [00:34<01:31,  1.08s/it]
batch:  10%|▉         | 9/92 [00:34<01:07,  1.22it/s]
batch:  11%|█         | 10/92 [00:34<00:52,  1.56it/s]
batch:  12%|█▏        | 11/92 [00:34<00:41,  1.94it/s]
batch:  13%|█▎        | 12/92 [00:35<00:34,  2.32it/s]
batch:  14%|█▍        | 13/92 [00:35<00:29,  2.68it/s]
batch:  15%|█▌        | 14/92 [00:35<00:25,  3.02it/s]
batch:  16%|█▋        | 15/92 [00:35<00:23,  3.30it/s]
batch:  17%|█▋        | 16/92 [00:36<00:21,  3.54it/s]
batch:  18%|█▊        | 17/92 [00:36<00:20,  3.73it/s]
batch:  20%|█▉        | 18/92 [00:36<00:19,  3.86it/s]
batch:  21%|██        | 19/92 [00:36<00:18,  3.99it/s]
batch:  22%|██▏       | 20/92 [00:36<00:17,  4.04it/s]
batch:  23%|██▎       | 21/92 [00:37<00:17,  4.09it/s]
batch:  24%|██▍       | 22/92 [00:37<00:16,  4.13it/s]
batch:  25%|██▌       | 23/92 [00:37<00:16,  4.13it/s]
batch:  26%|██▌       | 24/92 [00:37<00:16,  4.15it/s]
batch:  27%|██▋       | 25/92 [00:38<00:16,  4.12it/s]
batch:  28%|██▊       | 26/92 [00:38<00:15,  4.16it/s]
batch:  29%|██▉       | 27/92 [00:38<00:15,  4.17it/s]
batch:  30%|███       | 28/92 [00:38<00:15,  4.18it/s]
batch:  32%|███▏      | 29/92 [00:39<00:15,  4.15it/s]
batch:  33%|███▎      | 30/92 [00:39<00:14,  4.16it/s]
batch:  34%|███▎      | 31/92 [00:39<00:14,  4.17it/s]
batch:  35%|███▍      | 32/92 [00:39<00:14,  4.16it/s]
batch:  36%|███▌      | 33/92 [00:40<00:14,  4.16it/s]
batch:  37%|███▋      | 34/92 [00:40<00:13,  4.16it/s]
batch:  38%|███▊      | 35/92 [00:40<00:13,  4.19it/s]
batch:  39%|███▉      | 36/92 [00:40<00:13,  4.19it/s]
batch:  40%|████      | 37/92 [00:41<00:13,  4.20it/s]
batch:  41%|████▏     | 38/92 [00:41<00:12,  4.20it/s]
batch:  42%|████▏     | 39/92 [00:41<00:12,  4.20it/s]
batch:  43%|████▎     | 40/92 [00:41<00:12,  4.22it/s]
batch:  45%|████▍     | 41/92 [00:42<00:12,  4.24it/s]
batch:  46%|████▌     | 42/92 [00:42<00:11,  4.24it/s]
batch:  47%|████▋     | 43/92 [00:42<00:11,  4.25it/s]
batch:  48%|████▊     | 44/92 [00:42<00:11,  4.25it/s]
batch:  49%|████▉     | 45/92 [00:42<00:11,  4.25it/s]
batch:  50%|█████     | 46/92 [00:43<00:10,  4.24it/s]
batch:  51%|█████     | 47/92 [00:43<00:10,  4.22it/s]
batch:  52%|█████▏    | 48/92 [00:43<00:10,  4.22it/s]
batch:  53%|█████▎    | 49/92 [00:43<00:10,  4.20it/s]
batch:  54%|█████▍    | 50/92 [00:44<00:09,  4.20it/s]
batch:  55%|█████▌    | 51/92 [00:44<00:09,  4.20it/s]
batch:  57%|█████▋    | 52/92 [00:44<00:09,  4.21it/s]
batch:  58%|█████▊    | 53/92 [00:44<00:09,  4.20it/s]
batch:  59%|█████▊    | 54/92 [00:45<00:09,  4.17it/s]
batch:  60%|█████▉    | 55/92 [00:45<00:08,  4.19it/s]
batch:  61%|██████    | 56/92 [00:45<00:08,  4.20it/s]
batch:  62%|██████▏   | 57/92 [00:45<00:08,  4.20it/s]
batch:  63%|██████▎   | 58/92 [00:46<00:08,  4.21it/s]
batch:  64%|██████▍   | 59/92 [00:46<00:07,  4.19it/s]
batch:  65%|██████▌   | 60/92 [00:46<00:07,  4.21it/s]
batch:  66%|██████▋   | 61/92 [00:46<00:07,  4.20it/s]
batch:  67%|██████▋   | 62/92 [00:46<00:07,  4.21it/s]
batch:  68%|██████▊   | 63/92 [00:47<00:06,  4.21it/s]
batch:  70%|██████▉   | 64/92 [00:47<00:06,  4.23it/s]
batch:  71%|███████   | 65/92 [00:47<00:06,  4.22it/s]
batch:  72%|███████▏  | 66/92 [00:48<00:07,  3.64it/s]
batch:  73%|███████▎  | 67/92 [00:48<00:06,  3.78it/s]
batch:  74%|███████▍  | 68/92 [00:48<00:06,  3.89it/s]
batch:  75%|███████▌  | 69/92 [00:48<00:05,  3.99it/s]
batch:  76%|███████▌  | 70/92 [00:49<00:05,  4.08it/s]
batch:  77%|███████▋  | 71/92 [00:49<00:05,  4.11it/s]
batch:  78%|███████▊  | 72/92 [00:49<00:04,  4.16it/s]
batch:  79%|███████▉  | 73/92 [00:49<00:04,  4.17it/s]
batch:  80%|████████  | 74/92 [00:49<00:04,  4.18it/s]
batch:  82%|████████▏ | 75/92 [00:50<00:04,  4.21it/s]
batch:  83%|████████▎ | 76/92 [00:50<00:03,  4.20it/s]
batch:  84%|████████▎ | 77/92 [00:50<00:03,  4.22it/s]
batch:  85%|████████▍ | 78/92 [00:50<00:03,  4.22it/s]
batch:  86%|████████▌ | 79/92 [00:51<00:03,  4.22it/s]
batch:  87%|████████▋ | 80/92 [00:51<00:02,  4.22it/s]
batch:  88%|████████▊ | 81/92 [00:51<00:02,  4.21it/s]
batch:  89%|████████▉ | 82/92 [00:51<00:02,  4.21it/s]
batch:  90%|█████████ | 83/92 [00:52<00:02,  4.21it/s]
batch:  91%|█████████▏| 84/92 [00:52<00:01,  4.21it/s]
batch:  92%|█████████▏| 85/92 [00:52<00:01,  4.22it/s]
batch:  93%|█████████▎| 86/92 [00:52<00:01,  4.23it/s]
batch:  95%|█████████▍| 87/92 [00:53<00:01,  4.23it/s]
batch:  96%|█████████▌| 88/92 [00:53<00:00,  4.21it/s]
batch:  97%|█████████▋| 89/92 [00:53<00:00,  3.83it/s]
batch:  98%|█████████▊| 90/92 [00:53<00:00,  3.95it/s]
batch:  99%|█████████▉| 91/92 [00:54<00:00,  4.03it/s]
batch: 100%|██████████| 92/92 [00:54<00:00,  1.70it/s]

  0%|          | 0/5811 [00:00<?, ?it/s]
  2%|▏         | 119/5811 [00:00<00:04, 1181.89it/s]
  4%|▍         | 240/5811 [00:00<00:04, 1197.53it/s]
  6%|▌         | 360/5811 [00:00<00:04, 1189.51it/s]
  8%|▊         | 481/5811 [00:00<00:04, 1188.10it/s]
 10%|█         | 603/5811 [00:00<00:04, 1193.81it/s]
 12%|█▏        | 723/5811 [00:00<00:04, 1193.99it/s]
 15%|█▍        | 843/5811 [00:00<00:04, 1194.06it/s]
 17%|█▋        | 963/5811 [00:00<00:04, 1183.48it/s]
 19%|█▊        | 1082/5811 [00:00<00:04, 1181.79it/s]
 21%|██        | 1201/5811 [00:01<00:03, 1178.36it/s]
 23%|██▎       | 1319/5811 [00:01<00:03, 1172.24it/s]
 25%|██▍       | 1437/5811 [00:01<00:03, 1168.65it/s]
 27%|██▋       | 1554/5811 [00:01<00:03, 1167.62it/s]
 29%|██▉       | 1671/5811 [00:01<00:03, 1166.73it/s]
 31%|███       | 1788/5811 [00:01<00:03, 1159.78it/s]
 33%|███▎      | 1904/5811 [00:01<00:03, 1150.00it/s]
 35%|███▍      | 2020/5811 [00:01<00:03, 1144.77it/s]
 37%|███▋      | 2136/5811 [00:01<00:03, 1143.34it/s]
 39%|███▊      | 2251/5811 [00:01<00:03, 1144.19it/s]
 41%|████      | 2366/5811 [00:02<00:03, 1143.99it/s]
 43%|████▎     | 2481/5811 [00:02<00:02, 1137.24it/s]
 45%|████▍     | 2595/5811 [00:02<00:02, 1136.45it/s]
 47%|████▋     | 2709/5811 [00:02<00:02, 1134.06it/s]
 49%|████▊     | 2823/5811 [00:02<00:02, 1125.55it/s]
 51%|█████     | 2936/5811 [00:02<00:02, 1122.22it/s]
 52%|█████▏    | 3049/5811 [00:02<00:02, 1117.00it/s]
 54%|█████▍    | 3162/5811 [00:02<00:02, 1115.07it/s]
 56%|█████▋    | 3275/5811 [00:02<00:02, 1118.77it/s]
 58%|█████▊    | 3387/5811 [00:02<00:02, 1113.94it/s]
 60%|██████    | 3501/5811 [00:03<00:02, 1116.52it/s]
 62%|██████▏   | 3613/5811 [00:03<00:01, 1111.19it/s]
 64%|██████▍   | 3725/5811 [00:03<00:01, 1113.55it/s]
 66%|██████▋   | 3855/5811 [00:03<00:01, 1165.29it/s]
 68%|██████▊   | 3972/5811 [00:03<00:01, 1144.22it/s]
 70%|███████   | 4087/5811 [00:03<00:01, 1125.81it/s]
 72%|███████▏  | 4200/5811 [00:03<00:01, 1120.17it/s]
 74%|███████▍  | 4313/5811 [00:03<00:01, 1110.61it/s]
 76%|███████▌  | 4425/5811 [00:03<00:01, 1102.01it/s]
 78%|███████▊  | 4536/5811 [00:03<00:01, 1103.04it/s]
 80%|███████▉  | 4647/5811 [00:04<00:01, 1089.40it/s]
 82%|████████▏ | 4756/5811 [00:04<00:00, 1086.78it/s]
 84%|████████▎ | 4865/5811 [00:04<00:00, 1080.09it/s]
 86%|████████▌ | 4974/5811 [00:04<00:00, 1073.49it/s]
 87%|████████▋ | 5082/5811 [00:04<00:00, 1070.86it/s]
 89%|████████▉ | 5190/5811 [00:04<00:00, 1066.86it/s]
 91%|█████████ | 5297/5811 [00:04<00:00, 1065.45it/s]
 93%|█████████▎| 5404/5811 [00:04<00:00, 1060.02it/s]
 95%|█████████▍| 5511/5811 [00:04<00:00, 1062.70it/s]
 97%|█████████▋| 5618/5811 [00:04<00:00, 1054.33it/s]
 99%|█████████▊| 5724/5811 [00:05<00:00, 1055.26it/s]
100%|██████████| 5811/5811 [00:05<00:00, 1123.08it/s]

  0%|          | 0/5811 [00:00<?, ?it/s]
  2%|▏         | 119/5811 [00:00<00:04, 1183.38it/s]
  4%|▍         | 240/5811 [00:00<00:04, 1198.71it/s]
  6%|▌         | 360/5811 [00:00<00:04, 1191.41it/s]
  8%|▊         | 480/5811 [00:00<00:04, 1194.02it/s]
 10%|█         | 601/5811 [00:00<00:04, 1194.49it/s]
 12%|█▏        | 721/5811 [00:00<00:04, 1195.06it/s]
 14%|█▍        | 841/5811 [00:00<00:04, 1188.14it/s]
 17%|█▋        | 960/5811 [00:00<00:04, 1183.78it/s]
 19%|█▊        | 1079/5811 [00:00<00:04, 1175.39it/s]
 21%|██        | 1198/5811 [00:01<00:03, 1173.70it/s]
 23%|██▎       | 1316/5811 [00:01<00:03, 1168.91it/s]
 25%|██▍       | 1434/5811 [00:01<00:03, 1166.73it/s]
 27%|██▋       | 1551/5811 [00:01<00:03, 1165.90it/s]
 29%|██▊       | 1668/5811 [00:01<00:03, 1165.48it/s]
 31%|███       | 1785/5811 [00:01<00:03, 1163.91it/s]
 33%|███▎      | 1902/5811 [00:01<00:03, 1155.32it/s]
 35%|███▍      | 2018/5811 [00:01<00:03, 1148.97it/s]
 37%|███▋      | 2134/5811 [00:01<00:03, 1146.28it/s]
 39%|███▊      | 2249/5811 [00:01<00:03, 1146.79it/s]
 41%|████      | 2364/5811 [00:02<00:03, 1146.25it/s]
 43%|████▎     | 2479/5811 [00:02<00:02, 1138.92it/s]
 45%|████▍     | 2593/5811 [00:02<00:02, 1137.60it/s]
 47%|████▋     | 2707/5811 [00:02<00:02, 1130.09it/s]
 49%|████▊     | 2821/5811 [00:02<00:02, 1128.84it/s]
 50%|█████     | 2934/5811 [00:02<00:02, 1119.35it/s]
 52%|█████▏    | 3047/5811 [00:02<00:02, 1116.17it/s]
 54%|█████▍    | 3159/5811 [00:02<00:02, 1112.71it/s]
 56%|█████▋    | 3272/5811 [00:02<00:02, 1117.52it/s]
 58%|█████▊    | 3384/5811 [00:02<00:02, 1112.89it/s]
 60%|██████    | 3499/5811 [00:03<00:02, 1119.05it/s]
 62%|██████▏   | 3611/5811 [00:03<00:01, 1117.71it/s]
 64%|██████▍   | 3723/5811 [00:03<00:01, 1116.00it/s]
 66%|██████▋   | 3852/5811 [00:03<00:01, 1166.92it/s]
 68%|██████▊   | 3969/5811 [00:03<00:01, 1144.59it/s]
 70%|███████   | 4084/5811 [00:03<00:01, 1126.12it/s]
 72%|███████▏  | 4197/5811 [00:03<00:01, 1120.35it/s]
 74%|███████▍  | 4310/5811 [00:03<00:01, 1109.87it/s]
 76%|███████▌  | 4422/5811 [00:03<00:01, 1106.62it/s]
 78%|███████▊  | 4533/5811 [00:03<00:01, 1101.32it/s]
 80%|███████▉  | 4644/5811 [00:04<00:01, 1088.22it/s]
 82%|████████▏ | 4753/5811 [00:04<00:00, 1081.58it/s]
 84%|████████▎ | 4862/5811 [00:04<00:00, 1076.57it/s]
 86%|████████▌ | 4970/5811 [00:04<00:00, 1074.61it/s]
 87%|████████▋ | 5078/5811 [00:04<00:00, 1066.25it/s]
 89%|████████▉ | 5186/5811 [00:04<00:00, 1070.17it/s]
 91%|█████████ | 5294/5811 [00:04<00:00, 1069.33it/s]
 93%|█████████▎| 5401/5811 [00:04<00:00, 1063.43it/s]
 95%|█████████▍| 5508/5811 [00:04<00:00, 1065.34it/s]
 97%|█████████▋| 5615/5811 [00:04<00:00, 1056.24it/s]
 98%|█████████▊| 5721/5811 [00:05<00:00, 1056.95it/s]
100%|██████████| 5811/5811 [00:05<00:00, 1123.52it/s]
2025-11-17 18:14:13.571836: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:14:22.166918: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 18:14:22.167831: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 18:14:22.195561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 18:14:22.195619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:14:22.200892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:14:22.200977: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:14:22.203672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 18:14:22.205520: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 18:14:22.209382: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 18:14:22.211358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 18:14:22.212808: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:14:22.213207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 18:14:22.213473: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 18:14:22.215045: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 18:14:22.215320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 18:14:22.215352: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:14:22.215367: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:14:22.215381: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:14:22.215394: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 18:14:22.215407: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 18:14:22.215420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 18:14:22.215432: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 18:14:22.215446: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:14:22.215738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 18:14:22.215764: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 18:14:22.649284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 18:14:22.649376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 18:14:22.649386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 18:14:22.650770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-11-17 18:14:22.712456: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-11-17 18:14:22.726653: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 18:14:27.302939: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 18:14:27.822849: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 18:14:27.826732: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 18:14:29.397993: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 18:14:29.531097: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 18:14:52.508492: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)