Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-01-05 00:02:45.669084: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:05:04.382189: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-01-05 00:05:04.429330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-01-05 00:05:04.531384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-01-05 00:05:04.531431: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:05:05.546087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:05:05.546154: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:05:05.994988: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-01-05 00:05:06.716001: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-01-05 00:05:07.492445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-01-05 00:05:07.913538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-01-05 00:05:08.018683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:05:08.019077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-01-05 00:05:08.019449: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-01-05 00:05:08.021072: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-01-05 00:05:08.021328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-01-05 00:05:08.021348: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:05:08.021361: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:05:08.021370: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:05:08.021379: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-01-05 00:05:08.021388: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-01-05 00:05:08.021397: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-01-05 00:05:08.021406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-01-05 00:05:08.021427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:05:08.021710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-01-05 00:05:08.021730: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:05:10.842685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-01-05 00:05:10.842787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-01-05 00:05:10.842800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-01-05 00:05:10.843493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-01-05 00:05:23.207928: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-01-05 00:05:23.208503: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-01-05 00:05:24.352773: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:05:26.528947: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:05:26.534483: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:05:41.592021: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-01-05 00:05:41.663210: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-01-05 00:06:01.397046: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-01-05 00:12:23.793843: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-01-05 00:12:28.074684: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:12:42.187672: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-01-05 00:12:42.188535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-01-05 00:12:42.215656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-01-05 00:12:42.215705: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:12:42.220401: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:12:42.220478: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:12:42.222666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-01-05 00:12:42.224147: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-01-05 00:12:42.227612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-01-05 00:12:42.229271: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-01-05 00:12:42.230580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:12:42.230952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-01-05 00:12:42.231206: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-01-05 00:12:42.232778: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-01-05 00:12:42.233022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-01-05 00:12:42.233044: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:12:42.233058: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:12:42.233073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:12:42.233083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-01-05 00:12:42.233093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-01-05 00:12:42.233103: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-01-05 00:12:42.233113: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-01-05 00:12:42.233123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:12:42.233404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-01-05 00:12:42.233426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:12:42.643410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-01-05 00:12:42.643507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-01-05 00:12:42.643516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-01-05 00:12:42.644195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/78 [00:00<?, ?it/s]2026-01-05 00:12:44.200351: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-01-05 00:12:44.200899: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-01-05 00:12:44.401863: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:12:44.911744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:12:44.914440: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:12:46.466149: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-01-05 00:12:46.537193: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   1%|▏         | 1/78 [00:22<28:21, 22.10s/it]
batch:   3%|▎         | 2/78 [00:22<11:42,  9.25s/it]
batch:   4%|▍         | 3/78 [00:22<06:25,  5.13s/it]
batch:   5%|▌         | 4/78 [00:22<03:56,  3.20s/it]
batch:   6%|▋         | 5/78 [00:23<02:35,  2.13s/it]
batch:   8%|▊         | 6/78 [00:23<01:47,  1.49s/it]
batch:   9%|▉         | 7/78 [00:23<01:16,  1.08s/it]
batch:  10%|█         | 8/78 [00:23<00:56,  1.23it/s]
batch:  12%|█▏        | 9/78 [00:24<00:43,  1.58it/s]
batch:  13%|█▎        | 10/78 [00:24<00:34,  1.96it/s]
batch:  14%|█▍        | 11/78 [00:24<00:28,  2.35it/s]
batch:  15%|█▌        | 12/78 [00:24<00:24,  2.71it/s]
batch:  17%|█▋        | 13/78 [00:24<00:21,  3.04it/s]
batch:  18%|█▊        | 14/78 [00:25<00:19,  3.33it/s]
batch:  19%|█▉        | 15/78 [00:25<00:17,  3.55it/s]
batch:  21%|██        | 16/78 [00:25<00:16,  3.73it/s]
batch:  22%|██▏       | 17/78 [00:25<00:15,  3.86it/s]
batch:  23%|██▎       | 18/78 [00:26<00:15,  3.96it/s]
batch:  24%|██▍       | 19/78 [00:26<00:14,  4.03it/s]
batch:  26%|██▌       | 20/78 [00:26<00:14,  4.07it/s]
batch:  27%|██▋       | 21/78 [00:26<00:13,  4.11it/s]
batch:  28%|██▊       | 22/78 [00:27<00:13,  4.13it/s]
batch:  29%|██▉       | 23/78 [00:27<00:13,  4.15it/s]
batch:  31%|███       | 24/78 [00:27<00:12,  4.17it/s]
batch:  32%|███▏      | 25/78 [00:27<00:12,  4.16it/s]
batch:  33%|███▎      | 26/78 [00:28<00:12,  4.17it/s]
batch:  35%|███▍      | 27/78 [00:28<00:12,  4.13it/s]
batch:  36%|███▌      | 28/78 [00:28<00:12,  4.13it/s]
batch:  37%|███▋      | 29/78 [00:28<00:11,  4.11it/s]
batch:  38%|███▊      | 30/78 [00:29<00:11,  4.15it/s]
batch:  40%|███▉      | 31/78 [00:29<00:11,  4.13it/s]
batch:  41%|████      | 32/78 [00:29<00:11,  4.10it/s]
batch:  42%|████▏     | 33/78 [00:29<00:10,  4.13it/s]
batch:  44%|████▎     | 34/78 [00:30<00:10,  4.13it/s]
batch:  45%|████▍     | 35/78 [00:30<00:10,  4.16it/s]
batch:  46%|████▌     | 36/78 [00:30<00:10,  4.15it/s]
batch:  47%|████▋     | 37/78 [00:30<00:09,  4.15it/s]
batch:  49%|████▊     | 38/78 [00:30<00:09,  4.18it/s]
batch:  50%|█████     | 39/78 [00:31<00:09,  4.20it/s]
batch:  51%|█████▏    | 40/78 [00:31<00:09,  4.19it/s]
batch:  53%|█████▎    | 41/78 [00:31<00:08,  4.21it/s]
batch:  54%|█████▍    | 42/78 [00:31<00:08,  4.19it/s]
batch:  55%|█████▌    | 43/78 [00:32<00:08,  4.18it/s]
batch:  56%|█████▋    | 44/78 [00:32<00:08,  4.18it/s]
batch:  58%|█████▊    | 45/78 [00:32<00:07,  4.22it/s]
batch:  59%|█████▉    | 46/78 [00:32<00:07,  4.22it/s]
batch:  60%|██████    | 47/78 [00:33<00:07,  4.25it/s]
batch:  62%|██████▏   | 48/78 [00:33<00:07,  4.24it/s]
batch:  63%|██████▎   | 49/78 [00:33<00:06,  4.25it/s]
batch:  64%|██████▍   | 50/78 [00:33<00:06,  4.26it/s]
batch:  65%|██████▌   | 51/78 [00:34<00:06,  4.25it/s]
batch:  67%|██████▋   | 52/78 [00:34<00:06,  4.26it/s]
batch:  68%|██████▊   | 53/78 [00:34<00:05,  4.27it/s]
batch:  69%|██████▉   | 54/78 [00:34<00:05,  4.25it/s]
batch:  71%|███████   | 55/78 [00:34<00:05,  4.25it/s]
batch:  72%|███████▏  | 56/78 [00:35<00:05,  4.26it/s]
batch:  73%|███████▎  | 57/78 [00:35<00:04,  4.24it/s]
batch:  74%|███████▍  | 58/78 [00:35<00:04,  4.25it/s]
batch:  76%|███████▌  | 59/78 [00:35<00:04,  4.25it/s]
batch:  77%|███████▋  | 60/78 [00:36<00:04,  4.25it/s]
batch:  78%|███████▊  | 61/78 [00:36<00:04,  4.22it/s]
batch:  79%|███████▉  | 62/78 [00:36<00:03,  4.25it/s]
batch:  81%|████████  | 63/78 [00:36<00:03,  4.24it/s]
batch:  82%|████████▏ | 64/78 [00:37<00:03,  4.21it/s]
batch:  83%|████████▎ | 65/78 [00:37<00:03,  4.22it/s]
batch:  85%|████████▍ | 66/78 [00:37<00:03,  3.53it/s]
batch:  86%|████████▌ | 67/78 [00:37<00:02,  3.71it/s]
batch:  87%|████████▋ | 68/78 [00:38<00:02,  3.84it/s]
batch:  88%|████████▊ | 69/78 [00:38<00:02,  3.97it/s]
batch:  90%|████████▉ | 70/78 [00:38<00:01,  4.07it/s]
batch:  91%|█████████ | 71/78 [00:38<00:01,  4.10it/s]
batch:  92%|█████████▏| 72/78 [00:39<00:01,  4.13it/s]
batch:  94%|█████████▎| 73/78 [00:39<00:01,  4.14it/s]
batch:  95%|█████████▍| 74/78 [00:39<00:00,  4.17it/s]
batch:  96%|█████████▌| 75/78 [00:39<00:00,  4.21it/s]
batch:  97%|█████████▋| 76/78 [00:40<00:00,  4.18it/s]
batch:  99%|█████████▊| 77/78 [00:40<00:00,  4.21it/s]
batch: 100%|██████████| 78/78 [00:40<00:00,  1.93it/s]

  0%|          | 0/4945 [00:00<?, ?it/s]
  2%|▏         | 120/4945 [00:00<00:04, 1190.78it/s]
  5%|▍         | 241/4945 [00:00<00:03, 1190.14it/s]
  7%|▋         | 361/4945 [00:00<00:03, 1166.51it/s]
 10%|▉         | 483/4945 [00:00<00:03, 1179.22it/s]
 12%|█▏        | 601/4945 [00:00<00:03, 1175.52it/s]
 15%|█▍        | 720/4945 [00:00<00:03, 1178.12it/s]
 17%|█▋        | 838/4945 [00:00<00:03, 1167.65it/s]
 19%|█▉        | 955/4945 [00:00<00:03, 1167.30it/s]
 22%|██▏       | 1072/4945 [00:00<00:03, 1165.78it/s]
 24%|██▍       | 1190/4945 [00:01<00:03, 1164.13it/s]
 26%|██▋       | 1307/4945 [00:01<00:03, 1159.86it/s]
 29%|██▉       | 1423/4945 [00:01<00:03, 1152.48it/s]
 31%|███       | 1539/4945 [00:01<00:02, 1148.70it/s]
 33%|███▎      | 1654/4945 [00:01<00:02, 1143.85it/s]
 36%|███▌      | 1769/4945 [00:01<00:02, 1132.34it/s]
 38%|███▊      | 1883/4945 [00:01<00:02, 1131.93it/s]
 40%|████      | 1997/4945 [00:01<00:02, 1118.39it/s]
 43%|████▎     | 2110/4945 [00:01<00:02, 1119.60it/s]
 45%|████▍     | 2223/4945 [00:01<00:02, 1119.19it/s]
 47%|████▋     | 2336/4945 [00:02<00:02, 1117.85it/s]
 50%|████▉     | 2449/4945 [00:02<00:02, 1115.90it/s]
 52%|█████▏    | 2561/4945 [00:02<00:02, 1105.90it/s]
 54%|█████▍    | 2674/4945 [00:02<00:02, 1107.75it/s]
 56%|█████▋    | 2785/4945 [00:02<00:01, 1107.73it/s]
 59%|█████▊    | 2898/4945 [00:02<00:01, 1110.03it/s]
 61%|██████    | 3010/4945 [00:02<00:01, 1108.51it/s]
 63%|██████▎   | 3121/4945 [00:02<00:01, 1107.30it/s]
 65%|██████▌   | 3232/4945 [00:02<00:01, 1105.64it/s]
 68%|██████▊   | 3343/4945 [00:02<00:01, 1097.24it/s]
 70%|██████▉   | 3453/4945 [00:03<00:01, 1092.01it/s]
 72%|███████▏  | 3563/4945 [00:03<00:01, 1087.44it/s]
 74%|███████▍  | 3672/4945 [00:03<00:01, 1081.92it/s]
 76%|███████▋  | 3781/4945 [00:03<00:01, 1077.74it/s]
 79%|███████▊  | 3889/4945 [00:03<00:00, 1073.82it/s]
 81%|████████  | 3997/4945 [00:03<00:00, 1073.46it/s]
 83%|████████▎ | 4105/4945 [00:03<00:00, 1059.42it/s]
 85%|████████▌ | 4212/4945 [00:03<00:00, 1061.86it/s]
 87%|████████▋ | 4319/4945 [00:03<00:00, 1060.60it/s]
 90%|████████▉ | 4426/4945 [00:03<00:00, 1049.50it/s]
 92%|█████████▏| 4534/4945 [00:04<00:00, 1053.68it/s]
 94%|█████████▍| 4643/4945 [00:04<00:00, 1057.61it/s]
 96%|█████████▌| 4749/4945 [00:04<00:00, 1048.78it/s]
 98%|█████████▊| 4854/4945 [00:04<00:00, 1041.96it/s]
100%|██████████| 4945/4945 [00:04<00:00, 1106.79it/s]

  0%|          | 0/4945 [00:00<?, ?it/s]
  2%|▏         | 122/4945 [00:00<00:04, 1199.32it/s]
  5%|▍         | 244/4945 [00:00<00:03, 1204.09it/s]
  7%|▋         | 365/4945 [00:00<00:03, 1176.49it/s]
 10%|▉         | 486/4945 [00:00<00:03, 1184.73it/s]
 12%|█▏        | 607/4945 [00:00<00:03, 1186.31it/s]
 15%|█▍        | 726/4945 [00:00<00:03, 1186.80it/s]
 17%|█▋        | 845/4945 [00:00<00:03, 1186.33it/s]
 19%|█▉        | 964/4945 [00:00<00:03, 1177.00it/s]
 22%|██▏       | 1082/4945 [00:00<00:03, 1173.05it/s]
 24%|██▍       | 1201/4945 [00:01<00:03, 1172.62it/s]
 27%|██▋       | 1319/4945 [00:01<00:03, 1167.29it/s]
 29%|██▉       | 1436/4945 [00:01<00:03, 1151.69it/s]
 31%|███▏      | 1552/4945 [00:01<00:02, 1147.67it/s]
 34%|███▎      | 1668/4945 [00:01<00:02, 1146.71it/s]
 36%|███▌      | 1783/4945 [00:01<00:02, 1141.62it/s]
 38%|███▊      | 1898/4945 [00:01<00:02, 1140.46it/s]
 41%|████      | 2013/4945 [00:01<00:02, 1131.53it/s]
 43%|████▎     | 2127/4945 [00:01<00:02, 1133.87it/s]
 45%|████▌     | 2241/4945 [00:01<00:02, 1129.72it/s]
 48%|████▊     | 2355/4945 [00:02<00:02, 1130.80it/s]
 50%|████▉     | 2469/4945 [00:02<00:02, 1128.24it/s]
 52%|█████▏    | 2582/4945 [00:02<00:02, 1119.96it/s]
 54%|█████▍    | 2695/4945 [00:02<00:02, 1122.51it/s]
 57%|█████▋    | 2808/4945 [00:02<00:01, 1118.95it/s]
 59%|█████▉    | 2920/4945 [00:02<00:01, 1116.48it/s]
 61%|██████▏   | 3033/4945 [00:02<00:01, 1119.53it/s]
 64%|██████▎   | 3145/4945 [00:02<00:01, 1104.66it/s]
 66%|██████▌   | 3256/4945 [00:02<00:01, 1104.29it/s]
 68%|██████▊   | 3367/4945 [00:02<00:01, 1101.41it/s]
 70%|███████   | 3478/4945 [00:03<00:01, 1096.30it/s]
 73%|███████▎  | 3588/4945 [00:03<00:01, 1087.23it/s]
 75%|███████▍  | 3697/4945 [00:03<00:01, 1087.33it/s]
 77%|███████▋  | 3806/4945 [00:03<00:01, 1080.14it/s]
 79%|███████▉  | 3915/4945 [00:03<00:00, 1081.76it/s]
 81%|████████▏ | 4024/4945 [00:03<00:00, 1075.67it/s]
 84%|████████▎ | 4132/4945 [00:03<00:00, 1074.35it/s]
 86%|████████▌ | 4240/4945 [00:03<00:00, 1074.39it/s]
 88%|████████▊ | 4348/4945 [00:03<00:00, 1065.52it/s]
 90%|█████████ | 4455/4945 [00:03<00:00, 1062.99it/s]
 92%|█████████▏| 4562/4945 [00:04<00:00, 1060.92it/s]
 94%|█████████▍| 4670/4945 [00:04<00:00, 1064.37it/s]
 97%|█████████▋| 4777/4945 [00:04<00:00, 1058.55it/s]
 99%|█████████▊| 4883/4945 [00:04<00:00, 1056.29it/s]
100%|██████████| 4945/4945 [00:04<00:00, 1114.48it/s]
2026-01-05 00:14:19.675490: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:14:28.463000: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-01-05 00:14:28.464101: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-01-05 00:14:28.492736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-01-05 00:14:28.492828: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:14:28.497786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:14:28.497897: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:14:28.500225: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-01-05 00:14:28.501692: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-01-05 00:14:28.505285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-01-05 00:14:28.507136: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-01-05 00:14:28.508579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:14:28.508995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-01-05 00:14:28.509664: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-01-05 00:14:28.511319: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-01-05 00:14:28.511633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-01-05 00:14:28.511671: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:14:28.511688: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:14:28.511702: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:14:28.511715: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-01-05 00:14:28.511728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-01-05 00:14:28.511741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-01-05 00:14:28.511753: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-01-05 00:14:28.511766: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:14:28.512062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-01-05 00:14:28.512114: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-01-05 00:14:28.921013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-01-05 00:14:28.921115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-01-05 00:14:28.921125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-01-05 00:14:28.921888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-01-05 00:14:28.964258: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-01-05 00:14:28.978789: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-01-05 00:14:33.033243: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-01-05 00:14:33.539974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-01-05 00:14:33.544110: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-01-05 00:14:35.074111: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-01-05 00:14:35.147157: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-01-05 00:14:49.913478: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)