Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:20:06.341003: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:24:51.686086: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:24:51.704658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:24:52.209774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:24:52.209868: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:24:52.835413: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:24:52.835516: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:24:53.344310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:24:54.101807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:24:56.044084: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:25:05.047023: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:25:05.550366: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:25:05.550851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:25:05.551299: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:25:05.551463: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:25:05.551691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:25:05.551710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:25:05.551724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:25:05.551733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:25:05.551743: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:25:05.551752: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:25:05.551761: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:25:05.551770: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:25:05.551793: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:25:05.552074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:25:05.552094: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:25:05.968259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:25:05.968353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:25:05.968364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:25:05.969058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 03:25:07.737498: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:25:07.738071: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:25:11.049827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:25:11.536537: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:25:11.541760: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:25:13.187150: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:25:13.287847: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:27:40.108016: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:40:38.785585: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:41:45.005912: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:47:39.078533: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:47:39.089635: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:47:39.132072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:47:39.132163: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:47:39.153718: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:47:39.153816: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:47:39.164289: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:47:39.174089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:47:39.186187: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:47:39.196567: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:47:39.206862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:47:39.207340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:47:39.207685: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:47:39.207827: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:47:39.208034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:47:39.208062: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:47:39.208076: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:47:39.208086: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:47:39.208096: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:47:39.208107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:47:39.208116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:47:39.208126: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:47:39.208136: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:47:39.208413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:47:39.208434: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:47:39.630668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:47:39.630761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:47:39.630771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:47:39.631477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/42 [00:00<?, ?it/s]2026-05-11 03:47:41.448142: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:47:41.448673: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:47:41.799403: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:47:42.341301: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:47:42.343277: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:47:44.179879: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:47:44.722057: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/42 [02:39<1:49:16, 159.92s/it]
batch:   5%|▍         | 2/42 [02:40<43:59, 65.99s/it]   
batch:   7%|▋         | 3/42 [02:40<23:22, 35.97s/it]
batch:  10%|▉         | 4/42 [02:40<13:50, 21.86s/it]
batch:  12%|█▏        | 5/42 [02:40<08:40, 14.07s/it]
batch:  14%|█▍        | 6/42 [02:41<05:37,  9.36s/it]
batch:  17%|█▋        | 7/42 [02:41<03:43,  6.38s/it]
batch:  19%|█▉        | 8/42 [02:41<02:30,  4.43s/it]
batch:  21%|██▏       | 9/42 [02:41<01:42,  3.12s/it]
batch:  24%|██▍       | 10/42 [02:42<01:11,  2.23s/it]
batch:  26%|██▌       | 11/42 [02:42<00:50,  1.62s/it]
batch:  29%|██▊       | 12/42 [02:42<00:35,  1.20s/it]
batch:  31%|███       | 13/42 [02:42<00:26,  1.10it/s]
batch:  33%|███▎      | 14/42 [02:43<00:19,  1.42it/s]
batch:  36%|███▌      | 15/42 [02:43<00:15,  1.77it/s]
batch:  38%|███▊      | 16/42 [02:43<00:12,  2.15it/s]
batch:  40%|████      | 17/42 [02:43<00:09,  2.51it/s]
batch:  43%|████▎     | 18/42 [02:43<00:08,  2.86it/s]
batch:  45%|████▌     | 19/42 [02:44<00:07,  3.16it/s]
batch:  48%|████▊     | 20/42 [02:44<00:06,  3.41it/s]
batch:  50%|█████     | 21/42 [02:44<00:05,  3.62it/s]
batch:  52%|█████▏    | 22/42 [02:44<00:05,  3.77it/s]
batch:  55%|█████▍    | 23/42 [02:45<00:04,  3.89it/s]
batch:  57%|█████▋    | 24/42 [02:45<00:04,  3.98it/s]
batch:  60%|█████▉    | 25/42 [02:45<00:04,  4.04it/s]
batch:  62%|██████▏   | 26/42 [02:45<00:03,  4.09it/s]
batch:  64%|██████▍   | 27/42 [02:46<00:03,  4.12it/s]
batch:  67%|██████▋   | 28/42 [02:46<00:03,  4.15it/s]
batch:  69%|██████▉   | 29/42 [02:46<00:03,  4.16it/s]
batch:  71%|███████▏  | 30/42 [02:46<00:02,  4.18it/s]
batch:  74%|███████▍  | 31/42 [02:47<00:02,  4.19it/s]
batch:  76%|███████▌  | 32/42 [02:47<00:02,  4.18it/s]
batch:  79%|███████▊  | 33/42 [02:47<00:02,  4.20it/s]
batch:  81%|████████  | 34/42 [02:47<00:01,  4.20it/s]
batch:  83%|████████▎ | 35/42 [02:48<00:01,  4.20it/s]
batch:  86%|████████▌ | 36/42 [02:48<00:01,  4.19it/s]
batch:  88%|████████▊ | 37/42 [02:48<00:01,  4.18it/s]
batch:  90%|█████████ | 38/42 [02:48<00:00,  4.20it/s]
batch:  93%|█████████▎| 39/42 [02:48<00:00,  4.20it/s]
batch:  95%|█████████▌| 40/42 [02:49<00:00,  4.20it/s]
batch:  98%|█████████▊| 41/42 [02:49<00:00,  4.21it/s]
batch: 100%|██████████| 42/42 [02:49<00:00,  4.71it/s]
batch: 100%|██████████| 42/42 [02:49<00:00,  4.04s/it]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1203.77it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1209.97it/s]
 14%|█▍        | 366/2660 [00:00<00:01, 1210.08it/s]
 18%|█▊        | 488/2660 [00:00<00:01, 1201.54it/s]
 23%|██▎       | 609/2660 [00:00<00:01, 1198.44it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1194.02it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1189.09it/s]
 36%|███▋      | 968/2660 [00:00<00:01, 1176.64it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1168.87it/s]
 45%|████▌     | 1203/2660 [00:01<00:01, 1163.72it/s]
 50%|████▉     | 1320/2660 [00:01<00:01, 1156.17it/s]
 54%|█████▍    | 1436/2660 [00:01<00:01, 1156.25it/s]
 58%|█████▊    | 1552/2660 [00:01<00:00, 1152.73it/s]
 63%|██████▎   | 1668/2660 [00:01<00:00, 1148.78it/s]
 67%|██████▋   | 1783/2660 [00:01<00:00, 1137.57it/s]
 71%|███████▏  | 1897/2660 [00:01<00:00, 1132.50it/s]
 76%|███████▌  | 2011/2660 [00:01<00:00, 1125.79it/s]
 80%|███████▉  | 2124/2660 [00:01<00:00, 1119.65it/s]
 84%|████████▍ | 2236/2660 [00:01<00:00, 1117.40it/s]
 88%|████████▊ | 2348/2660 [00:02<00:00, 1111.57it/s]
 92%|█████████▏| 2460/2660 [00:02<00:00, 1105.25it/s]
 97%|█████████▋| 2571/2660 [00:02<00:00, 1102.42it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1145.03it/s]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1203.92it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1210.16it/s]
 14%|█▍        | 366/2660 [00:00<00:01, 1209.21it/s]
 18%|█▊        | 487/2660 [00:00<00:01, 1192.39it/s]
 23%|██▎       | 608/2660 [00:00<00:01, 1193.37it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1192.80it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1188.27it/s]
 36%|███▋      | 968/2660 [00:00<00:01, 1176.53it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1172.96it/s]
 45%|████▌     | 1204/2660 [00:01<00:01, 1169.48it/s]
 50%|████▉     | 1321/2660 [00:01<00:01, 1160.19it/s]
 54%|█████▍    | 1438/2660 [00:01<00:01, 1160.93it/s]
 58%|█████▊    | 1555/2660 [00:01<00:00, 1151.30it/s]
 63%|██████▎   | 1671/2660 [00:01<00:00, 1147.07it/s]
 67%|██████▋   | 1786/2660 [00:01<00:00, 1136.22it/s]
 71%|███████▏  | 1900/2660 [00:01<00:00, 1131.79it/s]
 76%|███████▌  | 2014/2660 [00:01<00:00, 1124.63it/s]
 80%|███████▉  | 2127/2660 [00:01<00:00, 1118.39it/s]
 84%|████████▍ | 2239/2660 [00:01<00:00, 1116.83it/s]
 88%|████████▊ | 2351/2660 [00:02<00:00, 1112.46it/s]
 93%|█████████▎| 2463/2660 [00:02<00:00, 1106.46it/s]
 97%|█████████▋| 2574/2660 [00:02<00:00, 1103.14it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1145.11it/s]
2026-05-11 03:54:45.362172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:01:43.869377: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:01:43.881279: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:01:43.996647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:01:43.996754: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:01:44.054838: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:01:44.054950: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:01:44.080891: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:01:44.106549: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:01:44.134728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:01:44.161255: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:01:44.187981: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:01:44.188501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:01:44.188995: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:01:44.189174: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:01:44.189445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:e1:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:01:44.189475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:01:44.189491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:01:44.189504: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:01:44.189517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:01:44.189530: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:01:44.189542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:01:44.189555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:01:44.189568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:01:44.189860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:01:44.189886: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:01:44.617306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:01:44.617412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:01:44.617422: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:01:44.618166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:e1:00.0, compute capability: 8.9)
2026-05-11 04:01:44.658500: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:01:44.672804: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 04:01:47.306972: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:01:47.785889: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:01:48.559578: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:01:50.795776: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:01:50.893408: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:04:14.133509: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)