Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:20:07.060902: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:25:44.343586: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:25:44.450481: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:25:44.512096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:25:44.512191: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:25:44.577895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:25:44.577972: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:25:44.610083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:25:44.657856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:25:44.696254: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:25:44.740011: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:25:44.773908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:25:44.774387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:25:44.774777: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:25:44.774937: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:25:44.775154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:25:44.775172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:25:44.775186: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:25:44.775196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:25:44.775205: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:25:44.775213: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:25:44.775222: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:25:44.775231: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:25:44.775259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:25:44.775536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:25:44.775557: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:25:45.183682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:25:45.183786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:25:45.183797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:25:45.184467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
2026-05-11 03:25:46.920164: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:25:46.920670: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:25:50.247420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:25:50.736652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:25:50.741582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:25:54.949941: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:25:55.043022: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:28:37.369676: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:41:37.724922: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:43:07.706445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:48:12.134529: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:48:12.152211: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:48:12.195666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:48:12.195757: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:48:12.234493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:48:12.234599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:48:12.253930: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:48:12.273371: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:48:12.294158: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:48:12.313761: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:48:12.333345: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:48:12.333791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:48:12.334098: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:48:12.334247: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:48:12.334483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:48:12.334502: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:48:12.334515: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:48:12.334525: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:48:12.334535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:48:12.334545: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:48:12.334555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:48:12.334565: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:48:12.334574: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:48:12.334857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:48:12.334879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:48:12.749057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:48:12.749157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:48:12.749167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:48:12.749838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/42 [00:00<?, ?it/s]2026-05-11 03:48:14.565291: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:48:14.565789: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:48:14.949819: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:48:15.462470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:48:15.464427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:48:17.396632: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:48:17.490955: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/42 [02:26<1:39:50, 146.12s/it]
batch:   5%|▍         | 2/42 [02:26<40:12, 60.31s/it]   
batch:   7%|▋         | 3/42 [02:26<21:22, 32.88s/it]
batch:  10%|▉         | 4/42 [02:26<12:39, 20.00s/it]
batch:  12%|█▏        | 5/42 [02:27<07:56, 12.87s/it]
batch:  14%|█▍        | 6/42 [02:27<05:08,  8.58s/it]
batch:  17%|█▋        | 7/42 [02:27<03:24,  5.85s/it]
batch:  19%|█▉        | 8/42 [02:27<02:18,  4.07s/it]
batch:  21%|██▏       | 9/42 [02:28<01:34,  2.87s/it]
batch:  24%|██▍       | 10/42 [02:28<01:05,  2.06s/it]
batch:  26%|██▌       | 11/42 [02:28<00:46,  1.50s/it]
batch:  29%|██▊       | 12/42 [02:28<00:33,  1.12s/it]
batch:  31%|███       | 13/42 [02:28<00:24,  1.18it/s]
batch:  33%|███▎      | 14/42 [02:29<00:18,  1.50it/s]
batch:  36%|███▌      | 15/42 [02:29<00:14,  1.86it/s]
batch:  38%|███▊      | 16/42 [02:29<00:11,  2.24it/s]
batch:  40%|████      | 17/42 [02:29<00:09,  2.60it/s]
batch:  43%|████▎     | 18/42 [02:30<00:08,  2.93it/s]
batch:  45%|████▌     | 19/42 [02:30<00:07,  3.22it/s]
batch:  48%|████▊     | 20/42 [02:30<00:06,  3.45it/s]
batch:  50%|█████     | 21/42 [02:30<00:05,  3.65it/s]
batch:  52%|█████▏    | 22/42 [02:31<00:05,  3.79it/s]
batch:  55%|█████▍    | 23/42 [02:31<00:04,  3.83it/s]
batch:  57%|█████▋    | 24/42 [02:31<00:04,  3.92it/s]
batch:  60%|█████▉    | 25/42 [02:31<00:04,  3.98it/s]
batch:  62%|██████▏   | 26/42 [02:32<00:03,  4.04it/s]
batch:  64%|██████▍   | 27/42 [02:32<00:03,  4.08it/s]
batch:  67%|██████▋   | 28/42 [02:32<00:03,  4.11it/s]
batch:  69%|██████▉   | 29/42 [02:32<00:03,  4.13it/s]
batch:  71%|███████▏  | 30/42 [02:33<00:02,  4.15it/s]
batch:  74%|███████▍  | 31/42 [02:33<00:02,  4.16it/s]
batch:  76%|███████▌  | 32/42 [02:33<00:02,  4.16it/s]
batch:  79%|███████▊  | 33/42 [02:33<00:02,  4.16it/s]
batch:  81%|████████  | 34/42 [02:34<00:01,  4.17it/s]
batch:  83%|████████▎ | 35/42 [02:34<00:01,  4.18it/s]
batch:  86%|████████▌ | 36/42 [02:34<00:01,  4.17it/s]
batch:  88%|████████▊ | 37/42 [02:34<00:01,  4.16it/s]
batch:  90%|█████████ | 38/42 [02:35<00:00,  4.18it/s]
batch:  93%|█████████▎| 39/42 [02:35<00:00,  4.17it/s]
batch:  95%|█████████▌| 40/42 [02:35<00:00,  4.18it/s]
batch:  98%|█████████▊| 41/42 [02:35<00:00,  4.19it/s]
batch: 100%|██████████| 42/42 [02:35<00:00,  4.69it/s]
batch: 100%|██████████| 42/42 [02:35<00:00,  3.71s/it]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1203.72it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1211.23it/s]
 14%|█▍        | 366/2660 [00:00<00:01, 1210.54it/s]
 18%|█▊        | 488/2660 [00:00<00:01, 1201.66it/s]
 23%|██▎       | 609/2660 [00:00<00:01, 1199.28it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1194.84it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1190.32it/s]
 36%|███▋      | 969/2660 [00:00<00:01, 1179.22it/s]
 41%|████      | 1087/2660 [00:00<00:01, 1172.06it/s]
 45%|████▌     | 1205/2660 [00:01<00:01, 1162.21it/s]
 50%|████▉     | 1322/2660 [00:01<00:01, 1160.81it/s]
 54%|█████▍    | 1439/2660 [00:01<00:01, 1155.93it/s]
 58%|█████▊    | 1555/2660 [00:01<00:00, 1152.47it/s]
 63%|██████▎   | 1671/2660 [00:01<00:00, 1148.10it/s]
 67%|██████▋   | 1786/2660 [00:01<00:00, 1137.21it/s]
 71%|███████▏  | 1900/2660 [00:01<00:00, 1132.51it/s]
 76%|███████▌  | 2014/2660 [00:01<00:00, 1125.76it/s]
 80%|███████▉  | 2127/2660 [00:01<00:00, 1119.54it/s]
 84%|████████▍ | 2239/2660 [00:01<00:00, 1117.78it/s]
 88%|████████▊ | 2351/2660 [00:02<00:00, 1110.98it/s]
 93%|█████████▎| 2463/2660 [00:02<00:00, 1104.75it/s]
 97%|█████████▋| 2574/2660 [00:02<00:00, 1102.67it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1145.68it/s]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1203.08it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1210.00it/s]
 14%|█▎        | 365/2660 [00:00<00:01, 1210.81it/s]
 18%|█▊        | 487/2660 [00:00<00:01, 1194.43it/s]
 23%|██▎       | 608/2660 [00:00<00:01, 1194.22it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1193.54it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1189.35it/s]
 36%|███▋      | 968/2660 [00:00<00:01, 1176.49it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1173.76it/s]
 45%|████▌     | 1204/2660 [00:01<00:01, 1170.23it/s]
 50%|████▉     | 1322/2660 [00:01<00:01, 1162.76it/s]
 54%|█████▍    | 1439/2660 [00:01<00:01, 1157.91it/s]
 58%|█████▊    | 1555/2660 [00:01<00:00, 1153.37it/s]
 63%|██████▎   | 1671/2660 [00:01<00:00, 1148.74it/s]
 67%|██████▋   | 1786/2660 [00:01<00:00, 1137.44it/s]
 71%|███████▏  | 1900/2660 [00:01<00:00, 1132.95it/s]
 76%|███████▌  | 2014/2660 [00:01<00:00, 1126.29it/s]
 80%|███████▉  | 2127/2660 [00:01<00:00, 1119.66it/s]
 84%|████████▍ | 2239/2660 [00:01<00:00, 1118.48it/s]
 88%|████████▊ | 2351/2660 [00:02<00:00, 1114.02it/s]
 93%|█████████▎| 2463/2660 [00:02<00:00, 1107.21it/s]
 97%|█████████▋| 2574/2660 [00:02<00:00, 1103.99it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1146.16it/s]
2026-05-11 03:55:38.746019: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:04:10.964928: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:04:10.995155: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 04:04:11.046390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:04:11.046500: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:04:11.077081: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:04:11.077177: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:04:11.091841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:04:11.105792: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:04:11.122244: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:04:11.137426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:04:11.152335: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:04:11.152824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:04:11.153240: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 04:04:11.153373: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 04:04:11.153623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4a:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 04:04:11.153651: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:04:11.153666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:04:11.153679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:04:11.153691: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 04:04:11.153704: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 04:04:11.153716: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 04:04:11.153728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 04:04:11.153741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:04:11.154037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 04:04:11.154064: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 04:04:11.574957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 04:04:11.575061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 04:04:11.575070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 04:04:11.575767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:4a:00.0, compute capability: 8.9)
2026-05-11 04:04:11.615533: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 04:04:11.629735: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 04:04:14.255539: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 04:04:14.737321: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 04:04:14.740999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 04:04:16.673254: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 04:04:16.765627: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 04:07:12.730411: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)