Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-28 11:42:05.839462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:43:24.968163: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:43:24.978880: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:43:25.090886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:43:25.090964: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:43:25.760526: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:43:25.760640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:43:26.145259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:43:26.654596: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:43:27.231802: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:43:27.471448: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:43:27.701294: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:43:27.701731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:43:27.702080: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:43:27.702243: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:43:27.702470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:43:27.702490: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:43:27.702507: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:43:27.702516: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:43:27.702526: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:43:27.702535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:43:27.702544: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:43:27.702553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:43:27.702578: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:43:27.702855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:43:27.702877: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:43:30.047785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:43:30.047886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:43:30.047897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:43:30.048569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-28 11:43:32.155253: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:43:32.155774: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-28 11:43:34.856356: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:43:36.730057: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:43:36.735001: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:43:50.341430: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:43:50.411457: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 11:44:13.867376: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-28 11:49:59.161429: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-28 11:50:03.278511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:50:13.418253: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:50:13.419172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:50:13.448020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:50:13.448117: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:50:13.453523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:50:13.453645: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:50:13.456167: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:50:13.458089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:50:13.461930: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:50:13.464240: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:50:13.465998: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:50:13.466471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:50:13.466768: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:50:13.466938: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:50:13.467288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:50:13.467319: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:50:13.467336: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:50:13.467347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:50:13.467358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:50:13.467368: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:50:13.467379: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:50:13.467389: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:50:13.467400: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:50:13.467698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:50:13.467730: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:50:13.914889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:50:13.914967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:50:13.914976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:50:13.915665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/32 [00:00<?, ?it/s]2026-05-28 11:50:15.540300: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-28 11:50:15.540796: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-28 11:50:15.766849: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:50:16.290074: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:50:16.292074: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:50:18.030511: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:50:18.119239: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   3%|▎         | 1/32 [00:21<10:51, 21.02s/it]
batch:   6%|▋         | 2/32 [00:21<04:23,  8.80s/it]
batch:   9%|▉         | 3/32 [00:21<02:21,  4.89s/it]
batch:  12%|█▎        | 4/32 [00:21<01:25,  3.05s/it]
batch:  16%|█▌        | 5/32 [00:21<00:55,  2.04s/it]
batch:  19%|█▉        | 6/32 [00:22<00:37,  1.43s/it]
batch:  22%|██▏       | 7/32 [00:22<00:25,  1.04s/it]
batch:  25%|██▌       | 8/32 [00:22<00:18,  1.28it/s]
batch:  28%|██▊       | 9/32 [00:22<00:14,  1.63it/s]
batch:  31%|███▏      | 10/32 [00:23<00:10,  2.00it/s]
batch:  34%|███▍      | 11/32 [00:23<00:08,  2.38it/s]
batch:  38%|███▊      | 12/32 [00:23<00:07,  2.74it/s]
batch:  41%|████      | 13/32 [00:23<00:06,  3.06it/s]
batch:  44%|████▍     | 14/32 [00:24<00:05,  3.33it/s]
batch:  47%|████▋     | 15/32 [00:24<00:04,  3.45it/s]
batch:  50%|█████     | 16/32 [00:24<00:04,  3.63it/s]
batch:  53%|█████▎    | 17/32 [00:24<00:03,  3.77it/s]
batch:  56%|█████▋    | 18/32 [00:25<00:03,  3.88it/s]
batch:  59%|█████▉    | 19/32 [00:25<00:03,  3.96it/s]
batch:  62%|██████▎   | 20/32 [00:25<00:02,  4.01it/s]
batch:  66%|██████▌   | 21/32 [00:25<00:02,  4.05it/s]
batch:  69%|██████▉   | 22/32 [00:26<00:02,  4.08it/s]
batch:  72%|███████▏  | 23/32 [00:26<00:02,  4.11it/s]
batch:  75%|███████▌  | 24/32 [00:26<00:01,  4.12it/s]
batch:  78%|███████▊  | 25/32 [00:26<00:01,  4.12it/s]
batch:  81%|████████▏ | 26/32 [00:27<00:01,  4.13it/s]
batch:  84%|████████▍ | 27/32 [00:27<00:01,  4.12it/s]
batch:  88%|████████▊ | 28/32 [00:27<00:00,  4.14it/s]
batch:  91%|█████████ | 29/32 [00:27<00:00,  4.14it/s]
batch:  97%|█████████▋| 31/32 [00:28<00:00,  5.04it/s]
batch: 100%|██████████| 32/32 [00:28<00:00,  1.14it/s]

  0%|          | 0/1924 [00:00<?, ?it/s]
  6%|▋         | 121/1924 [00:00<00:01, 1189.56it/s]
 13%|█▎        | 241/1924 [00:00<00:01, 1183.54it/s]
 19%|█▊        | 360/1924 [00:00<00:01, 1181.72it/s]
 25%|██▍       | 479/1924 [00:00<00:01, 1175.51it/s]
 31%|███       | 597/1924 [00:00<00:01, 1162.64it/s]
 37%|███▋      | 714/1924 [00:00<00:01, 1151.25it/s]
 43%|████▎     | 830/1924 [00:00<00:00, 1140.40it/s]
 49%|████▉     | 945/1924 [00:00<00:00, 1133.98it/s]
 55%|█████▌    | 1059/1924 [00:00<00:00, 1123.13it/s]
 61%|██████    | 1172/1924 [00:01<00:00, 1110.63it/s]
 67%|██████▋   | 1284/1924 [00:01<00:00, 1102.93it/s]
 73%|███████▎  | 1395/1924 [00:01<00:00, 1090.36it/s]
 78%|███████▊  | 1505/1924 [00:01<00:00, 1088.48it/s]
 84%|████████▍ | 1614/1924 [00:01<00:00, 1079.51it/s]
 90%|████████▉ | 1722/1924 [00:01<00:00, 1071.81it/s]
 95%|█████████▌| 1830/1924 [00:01<00:00, 1060.98it/s]
100%|██████████| 1924/1924 [00:01<00:00, 1106.69it/s]

  0%|          | 0/1924 [00:00<?, ?it/s]
  6%|▌         | 120/1924 [00:00<00:01, 1199.15it/s]
 12%|█▏        | 240/1924 [00:00<00:01, 1189.97it/s]
 19%|█▊        | 360/1924 [00:00<00:01, 1179.76it/s]
 25%|██▍       | 478/1924 [00:00<00:01, 1172.45it/s]
 31%|███       | 596/1924 [00:00<00:01, 1160.46it/s]
 37%|███▋      | 713/1924 [00:00<00:01, 1156.84it/s]
 43%|████▎     | 829/1924 [00:00<00:00, 1145.45it/s]
 49%|████▉     | 944/1924 [00:00<00:00, 1131.92it/s]
 55%|█████▍    | 1058/1924 [00:00<00:00, 1121.79it/s]
 61%|██████    | 1171/1924 [00:01<00:00, 1117.51it/s]
 67%|██████▋   | 1283/1924 [00:01<00:00, 1103.22it/s]
 72%|███████▏  | 1394/1924 [00:01<00:00, 1097.06it/s]
 78%|███████▊  | 1504/1924 [00:01<00:00, 1088.35it/s]
 84%|████████▍ | 1613/1924 [00:01<00:00, 1079.80it/s]
 89%|████████▉ | 1721/1924 [00:01<00:00, 1073.00it/s]
 95%|█████████▌| 1829/1924 [00:01<00:00, 1065.64it/s]
100%|██████████| 1924/1924 [00:01<00:00, 1109.28it/s]
2026-05-28 11:50:59.323136: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:51:07.976575: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:51:07.986551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-28 11:51:08.016672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:51:08.016781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:51:08.023243: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:51:08.023377: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:51:08.026997: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:51:08.029842: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:51:08.034445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:51:08.037840: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:51:08.040381: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:51:08.040887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:51:08.041322: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-28 11:51:08.041542: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-28 11:51:08.041934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-28 11:51:08.041981: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:51:08.041999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:51:08.042013: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:51:08.042027: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-28 11:51:08.042041: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-28 11:51:08.042053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-28 11:51:08.042067: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-28 11:51:08.042080: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:51:08.042394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-28 11:51:08.042426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-28 11:51:08.475460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-28 11:51:08.475551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-28 11:51:08.475560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-28 11:51:08.476278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-28 11:51:08.519562: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-28 11:51:08.538421: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-28 11:51:10.718958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-28 11:51:11.226825: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-28 11:51:11.230674: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-28 11:51:12.766980: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-28 11:51:12.856864: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-28 11:51:28.103105: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)