Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:18:38.128273: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:23:15.886698: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:23:15.889772: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:23:16.071738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:23:16.071840: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:23:16.180505: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:23:16.180589: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:23:16.236303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:23:16.302788: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:23:16.365879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:23:16.425224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:23:16.485623: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:23:16.486099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:23:16.486471: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:23:16.486637: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:23:16.486848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:23:16.486865: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:23:16.486878: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:23:16.486887: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:23:16.486896: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:23:16.486905: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:23:16.486913: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:23:16.486922: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:23:16.486951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:23:16.487229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:23:16.487251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:23:16.903778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:23:16.903879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:23:16.903889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:23:16.904552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:23:19.073272: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:23:19.073757: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:23:21.961891: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:23:22.452453: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:23:22.456952: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:23:36.945541: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:23:37.044907: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:25:44.680920: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:39:08.758517: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:40:37.712055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:47.811084: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:44:47.812090: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:44:47.949353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:44:47.949452: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:48.060681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:44:48.060783: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:44:48.114822: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:44:48.168582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:44:48.226763: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:44:48.284412: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:44:48.342666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:44:48.343142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:44:48.343438: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:44:48.343595: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:44:48.343813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:44:48.343834: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:48.343848: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:44:48.343859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:44:48.343869: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:44:48.343879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:44:48.343889: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:44:48.343898: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:44:48.343908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:44:48.344192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:44:48.344215: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:44:48.768058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:44:48.768159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:44:48.768168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:44:48.768845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/42 [00:00<?, ?it/s]2026-05-11 03:44:51.864297: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:44:51.864791: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:44:53.227968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:44:53.775634: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:44:53.777435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:45:17.883138: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:45:17.983567: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/42 [02:51<1:57:24, 171.82s/it]
batch:   5%|▍         | 2/42 [02:52<47:15, 70.89s/it]   
batch:   7%|▋         | 3/42 [02:52<25:06, 38.63s/it]
batch:  10%|▉         | 4/42 [02:52<14:52, 23.47s/it]
batch:  12%|█▏        | 5/42 [02:52<09:18, 15.10s/it]
batch:  14%|█▍        | 6/42 [02:53<06:01, 10.05s/it]
batch:  17%|█▋        | 7/42 [02:53<03:59,  6.84s/it]
batch:  19%|█▉        | 8/42 [02:53<02:41,  4.74s/it]
batch:  21%|██▏       | 9/42 [02:53<01:49,  3.33s/it]
batch:  24%|██▍       | 10/42 [02:53<01:16,  2.38s/it]
batch:  26%|██▌       | 11/42 [02:54<00:53,  1.72s/it]
batch:  29%|██▊       | 12/42 [02:54<00:38,  1.27s/it]
batch:  31%|███       | 13/42 [02:54<00:27,  1.04it/s]
batch:  33%|███▎      | 14/42 [02:54<00:20,  1.35it/s]
batch:  36%|███▌      | 15/42 [02:55<00:15,  1.70it/s]
batch:  38%|███▊      | 16/42 [02:55<00:12,  2.07it/s]
batch:  40%|████      | 17/42 [02:55<00:10,  2.43it/s]
batch:  43%|████▎     | 18/42 [02:55<00:08,  2.79it/s]
batch:  45%|████▌     | 19/42 [02:56<00:07,  3.10it/s]
batch:  48%|████▊     | 20/42 [02:56<00:06,  3.35it/s]
batch:  50%|█████     | 21/42 [02:56<00:05,  3.57it/s]
batch:  52%|█████▏    | 22/42 [02:56<00:05,  3.73it/s]
batch:  55%|█████▍    | 23/42 [02:57<00:05,  3.62it/s]
batch:  57%|█████▋    | 24/42 [02:57<00:04,  3.77it/s]
batch:  60%|█████▉    | 25/42 [02:57<00:04,  3.89it/s]
batch:  62%|██████▏   | 26/42 [02:57<00:04,  3.97it/s]
batch:  64%|██████▍   | 27/42 [02:58<00:03,  4.02it/s]
batch:  67%|██████▋   | 28/42 [02:58<00:03,  4.06it/s]
batch:  69%|██████▉   | 29/42 [02:58<00:03,  4.10it/s]
batch:  71%|███████▏  | 30/42 [02:58<00:02,  4.12it/s]
batch:  74%|███████▍  | 31/42 [02:59<00:02,  4.15it/s]
batch:  76%|███████▌  | 32/42 [02:59<00:02,  4.15it/s]
batch:  79%|███████▊  | 33/42 [02:59<00:02,  4.16it/s]
batch:  81%|████████  | 34/42 [02:59<00:01,  4.16it/s]
batch:  83%|████████▎ | 35/42 [03:00<00:01,  4.17it/s]
batch:  86%|████████▌ | 36/42 [03:00<00:01,  4.16it/s]
batch:  88%|████████▊ | 37/42 [03:00<00:01,  4.15it/s]
batch:  90%|█████████ | 38/42 [03:00<00:00,  4.17it/s]
batch:  93%|█████████▎| 39/42 [03:00<00:00,  4.18it/s]
batch:  95%|█████████▌| 40/42 [03:01<00:00,  4.18it/s]
batch:  98%|█████████▊| 41/42 [03:01<00:00,  4.18it/s]
batch: 100%|██████████| 42/42 [03:01<00:00,  4.70it/s]
batch: 100%|██████████| 42/42 [03:01<00:00,  4.32s/it]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1204.15it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1210.22it/s]
 14%|█▍        | 366/2660 [00:00<00:01, 1209.49it/s]
 18%|█▊        | 487/2660 [00:00<00:01, 1198.18it/s]
 23%|██▎       | 608/2660 [00:00<00:01, 1194.99it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1192.94it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1185.87it/s]
 36%|███▋      | 968/2660 [00:00<00:01, 1170.83it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1160.98it/s]
 45%|████▌     | 1203/2660 [00:01<00:01, 1155.06it/s]
 50%|████▉     | 1319/2660 [00:01<00:01, 1149.23it/s]
 54%|█████▍    | 1434/2660 [00:01<00:01, 1143.94it/s]
 58%|█████▊    | 1549/2660 [00:01<00:00, 1136.34it/s]
 63%|██████▎   | 1663/2660 [00:01<00:00, 1132.89it/s]
 67%|██████▋   | 1777/2660 [00:01<00:00, 1123.85it/s]
 71%|███████   | 1890/2660 [00:01<00:00, 1109.97it/s]
 75%|███████▌  | 2002/2660 [00:01<00:00, 1109.31it/s]
 79%|███████▉  | 2113/2660 [00:01<00:00, 1097.10it/s]
 84%|████████▎ | 2223/2660 [00:01<00:00, 1089.37it/s]
 88%|████████▊ | 2334/2660 [00:02<00:00, 1089.71it/s]
 92%|█████████▏| 2443/2660 [00:02<00:00, 1087.47it/s]
 96%|█████████▌| 2552/2660 [00:02<00:00, 1078.37it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1067.97it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1129.80it/s]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1204.72it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1210.73it/s]
 14%|█▎        | 365/2660 [00:00<00:01, 1210.85it/s]
 18%|█▊        | 487/2660 [00:00<00:01, 1194.76it/s]
 23%|██▎       | 608/2660 [00:00<00:01, 1192.97it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1191.44it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1184.95it/s]
 36%|███▋      | 968/2660 [00:00<00:01, 1170.68it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1165.54it/s]
 45%|████▌     | 1203/2660 [00:01<00:01, 1157.80it/s]
 50%|████▉     | 1319/2660 [00:01<00:01, 1151.08it/s]
 54%|█████▍    | 1435/2660 [00:01<00:01, 1147.80it/s]
 58%|█████▊    | 1550/2660 [00:01<00:00, 1139.18it/s]
 63%|██████▎   | 1664/2660 [00:01<00:00, 1129.56it/s]
 67%|██████▋   | 1777/2660 [00:01<00:00, 1124.90it/s]
 71%|███████   | 1890/2660 [00:01<00:00, 1110.46it/s]
 75%|███████▌  | 2002/2660 [00:01<00:00, 1109.87it/s]
 79%|███████▉  | 2114/2660 [00:01<00:00, 1099.74it/s]
 84%|████████▎ | 2224/2660 [00:01<00:00, 1091.87it/s]
 88%|████████▊ | 2334/2660 [00:02<00:00, 1089.12it/s]
 92%|█████████▏| 2443/2660 [00:02<00:00, 1086.68it/s]
 96%|█████████▌| 2552/2660 [00:02<00:00, 1077.69it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1067.56it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1130.27it/s]
2026-05-11 03:52:11.384132: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:58:40.411323: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:58:40.445355: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:58:40.511071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:58:40.511196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:58:40.547950: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:58:40.548068: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:58:40.567232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:58:40.586466: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:58:40.606919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:58:40.629333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:58:40.654262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:58:40.654753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:58:40.655195: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:58:40.655339: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:58:40.655573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:58:40.655601: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:58:40.655616: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:58:40.655629: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:58:40.655642: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:58:40.655655: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:58:40.655667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:58:40.655679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:58:40.655692: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:58:40.655979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:58:40.656005: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:58:41.080423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:58:41.080523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:58:41.080533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:58:41.081243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:58:41.121178: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:58:41.135406: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:58:43.787121: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:58:44.278667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:58:44.282510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:58:46.065808: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:58:46.163892: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:01:43.793618: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)