Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:19:38.004892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:24:45.216460: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:24:45.229384: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:24:45.319074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:24:45.319135: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:24:45.373205: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:24:45.373269: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:24:45.400460: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:24:45.426783: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:24:45.455234: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:24:45.481977: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:24:45.508083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:24:45.508479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:24:45.508774: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:24:45.508932: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:24:45.509243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:24:45.509269: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:24:45.509283: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:24:45.509293: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:24:45.509303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:24:45.509312: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:24:45.509322: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:24:45.509330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:24:45.509355: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:24:45.509636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:24:45.509656: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:24:45.923637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:24:45.950186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:24:45.950202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:24:45.950860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:24:47.712221: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:24:47.712693: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:24:50.482149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:24:50.962323: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:24:50.966791: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:24:55.882292: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:24:55.980135: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:26:42.587582: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:36:19.048005: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:36:50.837906: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:40:11.188429: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:40:11.222957: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:40:11.272979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:40:11.273057: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:40:11.301612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:40:11.301710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:40:11.316036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:40:11.330238: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:40:11.346246: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:40:11.360304: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:40:11.374159: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:40:11.374592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:40:11.374854: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:40:11.375009: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:40:11.375223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:40:11.375244: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:40:11.375258: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:40:11.375268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:40:11.375278: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:40:11.375288: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:40:11.375298: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:40:11.375308: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:40:11.375318: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:40:11.375588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:40:11.375608: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:40:11.790247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:40:11.790338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:40:11.790347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:40:11.790990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/42 [00:00<?, ?it/s]2026-05-11 03:40:13.510613: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:40:13.511099: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:40:13.858612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:40:14.422087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:40:14.423802: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:40:16.270283: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:40:16.372347: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/42 [01:55<1:19:02, 115.67s/it]
batch:   5%|▍         | 2/42 [01:55<31:50, 47.77s/it]   
batch:   7%|▋         | 3/42 [01:56<16:56, 26.07s/it]
batch:  10%|▉         | 4/42 [01:56<10:03, 15.87s/it]
batch:  12%|█▏        | 5/42 [01:56<06:18, 10.23s/it]
batch:  14%|█▍        | 6/42 [01:56<04:06,  6.84s/it]
batch:  17%|█▋        | 7/42 [01:57<02:43,  4.68s/it]
batch:  19%|█▉        | 8/42 [01:57<01:51,  3.27s/it]
batch:  21%|██▏       | 9/42 [01:57<01:16,  2.32s/it]
batch:  24%|██▍       | 10/42 [01:57<00:53,  1.68s/it]
batch:  26%|██▌       | 11/42 [01:58<00:38,  1.24s/it]
batch:  29%|██▊       | 12/42 [01:58<00:27,  1.07it/s]
batch:  31%|███       | 13/42 [01:58<00:20,  1.39it/s]
batch:  33%|███▎      | 14/42 [01:58<00:16,  1.74it/s]
batch:  36%|███▌      | 15/42 [01:59<00:12,  2.11it/s]
batch:  38%|███▊      | 16/42 [01:59<00:10,  2.48it/s]
batch:  40%|████      | 17/42 [01:59<00:08,  2.83it/s]
batch:  43%|████▎     | 18/42 [01:59<00:07,  3.14it/s]
batch:  45%|████▌     | 19/42 [01:59<00:06,  3.40it/s]
batch:  48%|████▊     | 20/42 [02:00<00:06,  3.60it/s]
batch:  50%|█████     | 21/42 [02:00<00:05,  3.77it/s]
batch:  52%|█████▏    | 22/42 [02:00<00:05,  3.88it/s]
batch:  55%|█████▍    | 23/42 [02:00<00:04,  3.98it/s]
batch:  57%|█████▋    | 24/42 [02:01<00:04,  4.05it/s]
batch:  60%|█████▉    | 25/42 [02:01<00:04,  4.09it/s]
batch:  62%|██████▏   | 26/42 [02:01<00:03,  4.14it/s]
batch:  64%|██████▍   | 27/42 [02:01<00:03,  4.16it/s]
batch:  67%|██████▋   | 28/42 [02:02<00:03,  4.19it/s]
batch:  69%|██████▉   | 29/42 [02:02<00:03,  4.20it/s]
batch:  71%|███████▏  | 30/42 [02:02<00:02,  4.21it/s]
batch:  74%|███████▍  | 31/42 [02:02<00:02,  4.22it/s]
batch:  76%|███████▌  | 32/42 [02:03<00:02,  4.20it/s]
batch:  79%|███████▊  | 33/42 [02:03<00:02,  4.21it/s]
batch:  81%|████████  | 34/42 [02:03<00:01,  4.21it/s]
batch:  83%|████████▎ | 35/42 [02:03<00:01,  4.21it/s]
batch:  86%|████████▌ | 36/42 [02:03<00:01,  4.20it/s]
batch:  88%|████████▊ | 37/42 [02:04<00:01,  4.18it/s]
batch:  90%|█████████ | 38/42 [02:04<00:00,  4.19it/s]
batch:  93%|█████████▎| 39/42 [02:04<00:00,  4.19it/s]
batch:  95%|█████████▌| 40/42 [02:04<00:00,  4.20it/s]
batch:  98%|█████████▊| 41/42 [02:05<00:00,  4.20it/s]
batch: 100%|██████████| 42/42 [02:05<00:00,  4.70it/s]
batch: 100%|██████████| 42/42 [02:05<00:00,  2.98s/it]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1202.50it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1209.40it/s]
 14%|█▍        | 366/2660 [00:00<00:01, 1210.12it/s]
 18%|█▊        | 488/2660 [00:00<00:01, 1200.55it/s]
 23%|██▎       | 609/2660 [00:00<00:01, 1198.04it/s]
 27%|██▋       | 729/2660 [00:00<00:01, 1193.89it/s]
 32%|███▏      | 849/2660 [00:00<00:01, 1188.63it/s]
 36%|███▋      | 968/2660 [00:00<00:01, 1175.81it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1173.01it/s]
 45%|████▌     | 1204/2660 [00:01<00:01, 1169.05it/s]
 50%|████▉     | 1321/2660 [00:01<00:01, 1155.19it/s]
 54%|█████▍    | 1437/2660 [00:01<00:01, 1154.96it/s]
 58%|█████▊    | 1553/2660 [00:01<00:00, 1151.34it/s]
 63%|██████▎   | 1669/2660 [00:01<00:00, 1147.30it/s]
 67%|██████▋   | 1784/2660 [00:01<00:00, 1136.43it/s]
 71%|███████▏  | 1898/2660 [00:01<00:00, 1131.55it/s]
 76%|███████▌  | 2012/2660 [00:01<00:00, 1124.85it/s]
 80%|███████▉  | 2125/2660 [00:01<00:00, 1118.57it/s]
 84%|████████▍ | 2237/2660 [00:01<00:00, 1116.81it/s]
 88%|████████▊ | 2349/2660 [00:02<00:00, 1110.16it/s]
 93%|█████████▎| 2461/2660 [00:02<00:00, 1103.79it/s]
 97%|█████████▋| 2572/2660 [00:02<00:00, 1101.34it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1144.33it/s]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1204.19it/s]
  9%|▉         | 243/2660 [00:00<00:01, 1209.15it/s]
 14%|█▍        | 366/2660 [00:00<00:01, 1209.95it/s]
 18%|█▊        | 487/2660 [00:00<00:01, 1198.90it/s]
 23%|██▎       | 608/2660 [00:00<00:01, 1197.29it/s]
 27%|██▋       | 728/2660 [00:00<00:01, 1196.56it/s]
 32%|███▏      | 848/2660 [00:00<00:01, 1185.03it/s]
 36%|███▋      | 967/2660 [00:00<00:01, 1180.66it/s]
 41%|████      | 1086/2660 [00:00<00:01, 1172.44it/s]
 45%|████▌     | 1204/2660 [00:01<00:01, 1168.88it/s]
 50%|████▉     | 1321/2660 [00:01<00:01, 1160.30it/s]
 54%|█████▍    | 1438/2660 [00:01<00:01, 1160.42it/s]
 58%|█████▊    | 1555/2660 [00:01<00:00, 1152.19it/s]
 63%|██████▎   | 1671/2660 [00:01<00:00, 1148.84it/s]
 67%|██████▋   | 1786/2660 [00:01<00:00, 1137.75it/s]
 71%|███████▏  | 1900/2660 [00:01<00:00, 1132.55it/s]
 76%|███████▌  | 2014/2660 [00:01<00:00, 1125.47it/s]
 80%|███████▉  | 2127/2660 [00:01<00:00, 1119.46it/s]
 84%|████████▍ | 2239/2660 [00:01<00:00, 1117.31it/s]
 88%|████████▊ | 2351/2660 [00:02<00:00, 1112.63it/s]
 93%|█████████▎| 2463/2660 [00:02<00:00, 1105.83it/s]
 97%|█████████▋| 2574/2660 [00:02<00:00, 1101.73it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1145.64it/s]
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Matplotlib is building the font cache; this may take a moment.
2026-05-11 03:46:40.214909: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:09.375285: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:52:09.385139: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:52:09.421663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:52:09.421736: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:09.443071: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:52:09.443150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:52:09.453784: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:52:09.463483: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:52:09.475287: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:52:09.485940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:52:09.496054: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:52:09.496469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:52:09.496806: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:52:09.496959: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:52:09.497172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:52:09.497197: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:09.497212: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:52:09.497225: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:52:09.497238: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:52:09.497250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:52:09.497262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:52:09.497275: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:52:09.497288: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:52:09.497570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:52:09.497594: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:52:09.907119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:52:09.907216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:52:09.907225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:52:09.907908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:52:09.946037: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:52:09.959518: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:52:12.622006: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:52:13.097932: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:52:13.101491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:52:14.699208: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:52:14.796227: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:54:41.171144: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)