Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-11-17 17:54:02.541677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:54:15.835113: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:54:15.836462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 17:54:15.855632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:54:15.855692: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:54:15.859680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:54:15.859748: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:54:15.861570: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:54:15.862768: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:54:15.866034: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:54:15.867507: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:54:15.868507: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:54:15.868892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:54:15.869305: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 17:54:15.871061: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:54:15.871313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:54:15.871332: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:54:15.871346: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:54:15.871355: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:54:15.871364: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:54:15.871373: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:54:15.871382: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:54:15.871391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:54:15.871411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:54:15.871703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:54:15.871722: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:54:16.279635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 17:54:16.279733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 17:54:16.279745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 17:54:16.280442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-11-17 17:54:17.306549: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 17:54:17.307175: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 17:54:18.481961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:54:18.998090: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:54:19.003382: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:54:20.556357: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 17:54:20.678985: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 17:54:41.991838: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-11-17 17:57:19.368540: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-11-17 17:57:22.984999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:57:28.494243: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:57:28.495064: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 17:57:28.521256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:57:28.521305: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:57:28.525317: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:57:28.525399: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:57:28.527292: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:57:28.528358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:57:28.531621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:57:28.533124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:57:28.534159: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:57:28.534539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:57:28.534778: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 17:57:28.536309: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:57:28.536563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:57:28.536592: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:57:28.536607: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:57:28.536617: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:57:28.536627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:57:28.536637: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:57:28.536647: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:57:28.536656: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:57:28.536666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:57:28.536943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:57:28.536965: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:57:28.952631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 17:57:28.952723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 17:57:28.952733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 17:57:28.953425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/30 [00:00<?, ?it/s]2025-11-17 17:57:30.489565: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 17:57:30.490149: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 17:57:30.690532: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:57:31.205794: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:57:31.207850: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:57:32.802838: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 17:57:32.922914: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   3%|▎         | 1/30 [00:26<12:56, 26.79s/it]
batch:   7%|▋         | 2/30 [00:27<05:13, 11.18s/it]
batch:  10%|█         | 3/30 [00:27<02:47,  6.19s/it]
batch:  13%|█▎        | 4/30 [00:27<01:39,  3.84s/it]
batch:  17%|█▋        | 5/30 [00:27<01:03,  2.54s/it]
batch:  20%|██        | 6/30 [00:28<00:42,  1.76s/it]
batch:  23%|██▎       | 7/30 [00:28<00:29,  1.26s/it]
batch:  27%|██▋       | 8/30 [00:28<00:20,  1.07it/s]
batch:  30%|███       | 9/30 [00:28<00:15,  1.39it/s]
batch:  33%|███▎      | 10/30 [00:28<00:11,  1.75it/s]
batch:  37%|███▋      | 11/30 [00:29<00:08,  2.14it/s]
batch:  40%|████      | 12/30 [00:29<00:07,  2.51it/s]
batch:  43%|████▎     | 13/30 [00:29<00:05,  2.87it/s]
batch:  47%|████▋     | 14/30 [00:29<00:05,  3.17it/s]
batch:  50%|█████     | 15/30 [00:30<00:04,  3.41it/s]
batch:  53%|█████▎    | 16/30 [00:30<00:03,  3.60it/s]
batch:  57%|█████▋    | 17/30 [00:30<00:03,  3.75it/s]
batch:  60%|██████    | 18/30 [00:30<00:03,  3.88it/s]
batch:  63%|██████▎   | 19/30 [00:31<00:02,  3.95it/s]
batch:  67%|██████▋   | 20/30 [00:31<00:02,  3.99it/s]
batch:  70%|███████   | 21/30 [00:31<00:02,  3.99it/s]
batch:  73%|███████▎  | 22/30 [00:31<00:01,  4.02it/s]
batch:  77%|███████▋  | 23/30 [00:32<00:01,  4.05it/s]
batch:  80%|████████  | 24/30 [00:32<00:01,  4.03it/s]
batch:  83%|████████▎ | 25/30 [00:32<00:01,  4.03it/s]
batch:  87%|████████▋ | 26/30 [00:32<00:00,  4.07it/s]
batch:  90%|█████████ | 27/30 [00:33<00:00,  4.08it/s]
batch:  93%|█████████▎| 28/30 [00:33<00:00,  4.09it/s]
batch: 100%|██████████| 30/30 [00:33<00:00,  1.11s/it]

  0%|          | 0/1794 [00:00<?, ?it/s]
  6%|▌         | 109/1794 [00:00<00:01, 1087.95it/s]
 13%|█▎        | 227/1794 [00:00<00:01, 1138.44it/s]
 19%|█▉        | 343/1794 [00:00<00:01, 1148.00it/s]
 26%|██▌       | 458/1794 [00:00<00:01, 1145.69it/s]
 32%|███▏      | 574/1794 [00:00<00:01, 1140.41it/s]
 38%|███▊      | 689/1794 [00:00<00:01, 1017.59it/s]
 45%|████▍     | 799/1794 [00:00<00:00, 1040.29it/s]
 51%|█████     | 911/1794 [00:00<00:00, 1063.45it/s]
 57%|█████▋    | 1021/1794 [00:00<00:00, 1072.45it/s]
 63%|██████▎   | 1130/1794 [00:01<00:00, 1073.38it/s]
 69%|██████▉   | 1238/1794 [00:01<00:00, 1070.60it/s]
 75%|███████▌  | 1346/1794 [00:01<00:00, 1069.82it/s]
 81%|████████  | 1454/1794 [00:01<00:00, 1068.61it/s]
 87%|████████▋ | 1562/1794 [00:01<00:00, 1058.16it/s]
 93%|█████████▎| 1668/1794 [00:01<00:00, 1053.38it/s]
 99%|█████████▉| 1774/1794 [00:01<00:00, 1042.32it/s]
100%|██████████| 1794/1794 [00:01<00:00, 1069.85it/s]

  0%|          | 0/1794 [00:00<?, ?it/s]
  6%|▌         | 108/1794 [00:00<00:01, 1078.39it/s]
 13%|█▎        | 227/1794 [00:00<00:01, 1135.11it/s]
 19%|█▉        | 343/1794 [00:00<00:01, 1145.15it/s]
 26%|██▌       | 458/1794 [00:00<00:01, 1144.22it/s]
 32%|███▏      | 573/1794 [00:00<00:01, 1140.93it/s]
 38%|███▊      | 688/1794 [00:00<00:00, 1129.83it/s]
 45%|████▍     | 802/1794 [00:00<00:00, 1109.42it/s]
 51%|█████     | 914/1794 [00:00<00:00, 1105.91it/s]
 57%|█████▋    | 1025/1794 [00:00<00:00, 1092.88it/s]
 63%|██████▎   | 1135/1794 [00:01<00:00, 1088.83it/s]
 69%|██████▉   | 1244/1794 [00:01<00:00, 1058.45it/s]
 75%|███████▌  | 1351/1794 [00:01<00:00, 1060.94it/s]
 81%|████████▏ | 1460/1794 [00:01<00:00, 1064.83it/s]
 87%|████████▋ | 1568/1794 [00:01<00:00, 1063.04it/s]
 93%|█████████▎| 1675/1794 [00:01<00:00, 1026.58it/s]
 99%|█████████▉| 1778/1794 [00:01<00:00, 1027.39it/s]
100%|██████████| 1794/1794 [00:01<00:00, 1077.80it/s]
2025-11-17 17:58:15.149177: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:58:20.313780: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:58:20.314992: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 17:58:20.341183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:58:20.341245: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:58:20.344979: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:58:20.345068: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:58:20.346767: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:58:20.347712: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:58:20.350625: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:58:20.351895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:58:20.352765: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:58:20.353177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:58:20.353608: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 17:58:20.355145: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:58:20.355445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:58:20.355494: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:58:20.355510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:58:20.355523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:58:20.355537: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:58:20.355550: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:58:20.355562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:58:20.355575: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:58:20.355595: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:58:20.355902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:58:20.355929: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:58:20.766655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 17:58:20.766749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 17:58:20.766759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 17:58:20.767487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-11-17 17:58:20.818860: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-11-17 17:58:20.833072: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 17:58:22.859372: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:58:23.371383: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:58:23.375311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:58:24.906064: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 17:58:25.026318: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 17:58:44.414999: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)