Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-28 11:40:50.655418: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:42:21.513513: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:42:21.539050: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:42:21.679486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:42:21.679548: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:42:22.232793: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:42:22.232934: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:42:22.654562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:42:23.231252: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:42:23.801920: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:42:24.086865: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:42:24.247774: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:42:24.248512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:42:24.248884: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:42:24.249040: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:42:24.249395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:42:24.249415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:42:24.249430: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:42:24.249441: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:42:24.249450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:42:24.249460: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:42:24.249470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:42:24.249480: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:42:24.249502: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:42:24.250032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:42:24.250054: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:42:26.633753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:42:26.633858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:42:26.633869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:42:26.634506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-28 11:42:28.692874: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:42:28.693355: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-28 11:42:30.469190: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:42:32.096639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:42:32.101123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:42:45.342670: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:42:45.402850: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 11:42:57.016111: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-28 11:47:33.287362: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-28 11:47:37.947982: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:47:53.526903: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:47:53.527874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:47:53.557925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:47:53.558014: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:47:53.565141: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:47:53.565240: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:47:53.569466: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:47:53.573225: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:47:53.578734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:47:53.582261: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:47:53.584826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:47:53.585275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:47:53.585573: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:47:53.585715: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:47:53.585927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:47:53.585947: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:47:53.585962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:47:53.585972: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:47:53.585982: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:47:53.585992: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:47:53.586002: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:47:53.586012: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:47:53.586022: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:47:53.586292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:47:53.586313: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:47:54.007739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:47:54.007847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:47:54.007857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:47:54.008513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/32 [00:00<?, ?it/s]2026-05-28 11:47:55.576159: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:47:55.576668: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-28 11:47:55.791908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:47:56.322213: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:47:56.324171: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:47:57.916838: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:47:58.003247: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   3%|▎         | 1/32 [00:20<10:22, 20.10s/it]
batch:   6%|▋         | 2/32 [00:20<04:12,  8.42s/it]
batch:   9%|▉         | 3/32 [00:20<02:15,  4.69s/it]
batch:  12%|█▎        | 4/32 [00:20<01:22,  2.93s/it]
batch:  16%|█▌        | 5/32 [00:21<00:52,  1.96s/it]
batch:  19%|█▉        | 6/32 [00:21<00:35,  1.38s/it]
batch:  22%|██▏       | 7/32 [00:21<00:25,  1.00s/it]
batch:  25%|██▌       | 8/32 [00:21<00:18,  1.31it/s]
batch:  28%|██▊       | 9/32 [00:22<00:13,  1.67it/s]
batch:  31%|███▏      | 10/32 [00:22<00:10,  2.05it/s]
batch:  34%|███▍      | 11/32 [00:22<00:08,  2.42it/s]
batch:  38%|███▊      | 12/32 [00:22<00:07,  2.76it/s]
batch:  41%|████      | 13/32 [00:22<00:06,  3.07it/s]
batch:  44%|████▍     | 14/32 [00:23<00:05,  3.33it/s]
batch:  47%|████▋     | 15/32 [00:23<00:04,  3.53it/s]
batch:  50%|█████     | 16/32 [00:23<00:04,  3.69it/s]
batch:  53%|█████▎    | 17/32 [00:23<00:03,  3.79it/s]
batch:  56%|█████▋    | 18/32 [00:24<00:03,  3.88it/s]
batch:  59%|█████▉    | 19/32 [00:24<00:03,  3.95it/s]
batch:  62%|██████▎   | 20/32 [00:24<00:03,  3.99it/s]
batch:  66%|██████▌   | 21/32 [00:24<00:02,  4.03it/s]
batch:  69%|██████▉   | 22/32 [00:25<00:02,  4.05it/s]
batch:  72%|███████▏  | 23/32 [00:25<00:02,  4.08it/s]
batch:  75%|███████▌  | 24/32 [00:25<00:01,  4.10it/s]
batch:  78%|███████▊  | 25/32 [00:25<00:01,  4.09it/s]
batch:  81%|████████▏ | 26/32 [00:26<00:01,  4.10it/s]
batch:  84%|████████▍ | 27/32 [00:26<00:01,  4.10it/s]
batch:  88%|████████▊ | 28/32 [00:26<00:00,  4.11it/s]
batch:  91%|█████████ | 29/32 [00:26<00:00,  4.11it/s]
batch:  94%|█████████▍| 30/32 [00:27<00:00,  4.11it/s]
batch: 100%|██████████| 32/32 [00:27<00:00,  1.18it/s]

  0%|          | 0/1924 [00:00<?, ?it/s]
  6%|▋         | 121/1924 [00:00<00:01, 1188.77it/s]
 13%|█▎        | 241/1924 [00:00<00:01, 1185.21it/s]
 19%|█▊        | 360/1924 [00:00<00:01, 1183.63it/s]
 25%|██▍       | 479/1924 [00:00<00:01, 1178.91it/s]
 31%|███       | 597/1924 [00:00<00:01, 1167.05it/s]
 37%|███▋      | 714/1924 [00:00<00:01, 1157.25it/s]
 43%|████▎     | 830/1924 [00:00<00:00, 1149.14it/s]
 49%|████▉     | 945/1924 [00:00<00:00, 1145.41it/s]
 55%|█████▌    | 1060/1924 [00:00<00:00, 1131.73it/s]
 61%|██████    | 1174/1924 [00:01<00:00, 1130.43it/s]
 67%|██████▋   | 1288/1924 [00:01<00:00, 1115.43it/s]
 73%|███████▎  | 1400/1924 [00:01<00:00, 1113.07it/s]
 79%|███████▊  | 1512/1924 [00:01<00:00, 1104.17it/s]
 84%|████████▍ | 1623/1924 [00:01<00:00, 1102.37it/s]
 90%|█████████ | 1734/1924 [00:01<00:00, 1090.52it/s]
 96%|█████████▌| 1844/1924 [00:01<00:00, 1089.74it/s]
100%|██████████| 1924/1924 [00:01<00:00, 1122.66it/s]

  0%|          | 0/1924 [00:00<?, ?it/s]
  6%|▌         | 119/1924 [00:00<00:01, 1188.71it/s]
 12%|█▏        | 238/1924 [00:00<00:01, 1176.01it/s]
 19%|█▊        | 357/1924 [00:00<00:01, 1175.56it/s]
 25%|██▍       | 475/1924 [00:00<00:01, 1167.29it/s]
 31%|███       | 592/1924 [00:00<00:01, 1160.70it/s]
 37%|███▋      | 709/1924 [00:00<00:01, 1151.07it/s]
 43%|████▎     | 825/1924 [00:00<00:00, 1141.27it/s]
 49%|████▉     | 940/1924 [00:00<00:00, 1132.65it/s]
 55%|█████▍    | 1054/1924 [00:00<00:00, 1131.02it/s]
 61%|██████    | 1168/1924 [00:01<00:00, 1122.76it/s]
 67%|██████▋   | 1281/1924 [00:01<00:00, 1112.89it/s]
 72%|███████▏  | 1393/1924 [00:01<00:00, 1104.81it/s]
 78%|███████▊  | 1504/1924 [00:01<00:00, 1099.61it/s]
 84%|████████▍ | 1614/1924 [00:01<00:00, 1096.36it/s]
 90%|████████▉ | 1724/1924 [00:01<00:00, 1091.52it/s]
 95%|█████████▌| 1834/1924 [00:01<00:00, 1086.37it/s]
100%|██████████| 1924/1924 [00:01<00:00, 1118.45it/s]
2026-05-28 11:48:38.401789: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:48:51.255659: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:48:51.269220: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:48:51.296673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:48:51.296780: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:48:51.303862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:48:51.303978: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:48:51.306652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:48:51.308547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:48:51.312392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:48:51.315364: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:48:51.317357: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:48:51.317804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:48:51.318226: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:48:51.318387: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:48:51.318647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:48:51.318677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:48:51.318693: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:48:51.318706: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:48:51.318720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:48:51.318733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:48:51.318745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:48:51.318758: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:48:51.318771: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:48:51.319053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:48:51.319078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:48:51.728396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:48:51.728498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:48:51.728507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:48:51.729204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-28 11:48:51.768011: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-28 11:48:51.783462: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-28 11:48:53.922661: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:48:54.403047: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:48:54.406807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:48:55.862572: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:48:55.945853: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 11:49:09.900470: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)