Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:04:39.651424: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:39.540734: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:17:39.605495: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:17:39.816695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:17:39.816760: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:40.605449: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:17:40.605541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:17:41.103292: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:17:41.792899: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:17:42.539920: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:17:42.914601: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:17:43.304491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:17:43.304950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:17:43.305326: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:17:43.305477: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:17:43.305707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:17:43.305724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:43.305738: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:17:43.305747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:17:43.305756: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:17:43.305765: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:17:43.305774: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:17:43.305783: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:17:43.305806: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:17:43.306082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:17:43.306102: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:17:45.794743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:17:45.794812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:17:45.794823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:17:45.795578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:17:54.505896: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:17:54.506357: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:18:06.054704: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:18:07.989041: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:18:07.993680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:18:39.797687: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:18:39.867103: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:20:39.098952: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:32:37.931109: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:33:18.230133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:35:41.920882: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:35:41.942199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:35:41.991756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:35:41.991862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:35:42.027511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:35:42.027654: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:35:42.042306: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:35:42.061217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:35:42.075775: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:35:42.089133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:35:42.104731: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:35:42.105231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:35:42.105530: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:35:42.105691: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:35:42.105908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:35:42.105935: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:35:42.105950: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:35:42.105961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:35:42.105972: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:35:42.105981: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:35:42.105991: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:35:42.106001: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:35:42.106012: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:35:42.106281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:35:42.106303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:35:42.523030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:35:42.523136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:35:42.523146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:35:42.523823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/42 [00:00<?, ?it/s]2026-05-11 03:35:44.284745: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:35:44.285256: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:35:44.564530: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:35:45.079021: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:35:45.080850: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:35:46.825863: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:35:46.914258: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/42 [01:00<41:12, 60.29s/it]
batch:   5%|▍         | 2/42 [01:00<16:38, 24.97s/it]
batch:   7%|▋         | 3/42 [01:00<08:53, 13.68s/it]
batch:  10%|▉         | 4/42 [01:01<05:18,  8.37s/it]
batch:  12%|█▏        | 5/42 [01:01<03:21,  5.44s/it]
batch:  14%|█▍        | 6/42 [01:01<02:12,  3.67s/it]
batch:  17%|█▋        | 7/42 [01:01<01:29,  2.55s/it]
batch:  19%|█▉        | 8/42 [01:01<01:01,  1.82s/it]
batch:  21%|██▏       | 9/42 [01:02<00:43,  1.32s/it]
batch:  24%|██▍       | 10/42 [01:02<00:31,  1.01it/s]
batch:  26%|██▌       | 11/42 [01:02<00:23,  1.32it/s]
batch:  29%|██▊       | 12/42 [01:02<00:18,  1.66it/s]
batch:  31%|███       | 13/42 [01:03<00:14,  2.04it/s]
batch:  33%|███▎      | 14/42 [01:03<00:11,  2.41it/s]
batch:  36%|███▌      | 15/42 [01:03<00:09,  2.76it/s]
batch:  38%|███▊      | 16/42 [01:03<00:08,  3.07it/s]
batch:  40%|████      | 17/42 [01:04<00:07,  3.32it/s]
batch:  43%|████▎     | 18/42 [01:04<00:06,  3.54it/s]
batch:  45%|████▌     | 19/42 [01:04<00:06,  3.71it/s]
batch:  48%|████▊     | 20/42 [01:04<00:05,  3.84it/s]
batch:  50%|█████     | 21/42 [01:05<00:05,  3.93it/s]
batch:  52%|█████▏    | 22/42 [01:05<00:05,  3.99it/s]
batch:  55%|█████▍    | 23/42 [01:05<00:04,  4.06it/s]
batch:  57%|█████▋    | 24/42 [01:05<00:04,  4.12it/s]
batch:  60%|█████▉    | 25/42 [01:06<00:04,  4.16it/s]
batch:  62%|██████▏   | 26/42 [01:06<00:03,  4.19it/s]
batch:  64%|██████▍   | 27/42 [01:06<00:03,  4.22it/s]
batch:  67%|██████▋   | 28/42 [01:06<00:03,  4.24it/s]
batch:  69%|██████▉   | 29/42 [01:06<00:03,  4.25it/s]
batch:  71%|███████▏  | 30/42 [01:07<00:02,  4.23it/s]
batch:  74%|███████▍  | 31/42 [01:07<00:02,  4.23it/s]
batch:  76%|███████▌  | 32/42 [01:07<00:02,  4.21it/s]
batch:  79%|███████▊  | 33/42 [01:07<00:02,  4.23it/s]
batch:  81%|████████  | 34/42 [01:08<00:01,  4.19it/s]
batch:  83%|████████▎ | 35/42 [01:08<00:01,  4.19it/s]
batch:  86%|████████▌ | 36/42 [01:08<00:01,  4.17it/s]
batch:  88%|████████▊ | 37/42 [01:08<00:01,  4.15it/s]
batch:  90%|█████████ | 38/42 [01:09<00:00,  4.17it/s]
batch:  93%|█████████▎| 39/42 [01:09<00:00,  4.16it/s]
batch:  95%|█████████▌| 40/42 [01:09<00:00,  4.56it/s]
batch:  98%|█████████▊| 41/42 [01:09<00:00,  4.44it/s]
batch: 100%|██████████| 42/42 [01:09<00:00,  1.66s/it]

  0%|          | 0/2601 [00:00<?, ?it/s]
  5%|▍         | 119/2601 [00:00<00:02, 1184.70it/s]
  9%|▉         | 238/2601 [00:00<00:01, 1184.40it/s]
 14%|█▍        | 361/2601 [00:00<00:01, 1201.68it/s]
 19%|█▊        | 482/2601 [00:00<00:01, 1196.92it/s]
 23%|██▎       | 604/2601 [00:00<00:01, 1200.84it/s]
 28%|██▊       | 725/2601 [00:00<00:01, 1195.56it/s]
 32%|███▏      | 845/2601 [00:00<00:01, 1167.42it/s]
 37%|███▋      | 962/2601 [00:00<00:01, 1162.66it/s]
 41%|████▏     | 1079/2601 [00:00<00:01, 1157.06it/s]
 46%|████▌     | 1195/2601 [00:01<00:01, 1137.63it/s]
 50%|█████     | 1312/2601 [00:01<00:01, 1141.37it/s]
 55%|█████▌    | 1431/2601 [00:01<00:01, 1149.93it/s]
 60%|█████▉    | 1550/2601 [00:01<00:00, 1155.70it/s]
 64%|██████▍   | 1668/2601 [00:01<00:00, 1160.63it/s]
 69%|██████▊   | 1785/2601 [00:01<00:00, 1148.57it/s]
 73%|███████▎  | 1900/2601 [00:01<00:00, 1140.17it/s]
 77%|███████▋  | 2015/2601 [00:01<00:00, 1126.07it/s]
 82%|████████▏ | 2131/2601 [00:01<00:00, 1135.73it/s]
 86%|████████▋ | 2245/2601 [00:01<00:00, 1123.90it/s]
 91%|█████████ | 2358/2601 [00:02<00:00, 1121.95it/s]
 95%|█████████▌| 2471/2601 [00:02<00:00, 1122.90it/s]
 99%|█████████▉| 2586/2601 [00:02<00:00, 1124.93it/s]
100%|██████████| 2601/2601 [00:02<00:00, 1149.27it/s]

  0%|          | 0/2601 [00:00<?, ?it/s]
  5%|▍         | 119/2601 [00:00<00:02, 1186.21it/s]
  9%|▉         | 238/2601 [00:00<00:01, 1185.86it/s]
 14%|█▍        | 361/2601 [00:00<00:01, 1202.72it/s]
 19%|█▊        | 482/2601 [00:00<00:01, 1197.38it/s]
 23%|██▎       | 604/2601 [00:00<00:01, 1201.64it/s]
 28%|██▊       | 725/2601 [00:00<00:01, 1192.90it/s]
 32%|███▏      | 845/2601 [00:00<00:01, 1165.35it/s]
 37%|███▋      | 962/2601 [00:00<00:01, 1161.24it/s]
 41%|████▏     | 1079/2601 [00:00<00:01, 1156.40it/s]
 46%|████▌     | 1195/2601 [00:01<00:01, 1139.89it/s]
 50%|█████     | 1312/2601 [00:01<00:01, 1143.06it/s]
 55%|█████▌    | 1431/2601 [00:01<00:01, 1151.15it/s]
 60%|█████▉    | 1550/2601 [00:01<00:00, 1156.70it/s]
 64%|██████▍   | 1668/2601 [00:01<00:00, 1160.78it/s]
 69%|██████▊   | 1785/2601 [00:01<00:00, 1148.79it/s]
 73%|███████▎  | 1900/2601 [00:01<00:00, 1140.56it/s]
 77%|███████▋  | 2015/2601 [00:01<00:00, 1126.51it/s]
 82%|████████▏ | 2131/2601 [00:01<00:00, 1136.30it/s]
 86%|████████▋ | 2245/2601 [00:01<00:00, 1124.60it/s]
 91%|█████████ | 2358/2601 [00:02<00:00, 1122.98it/s]
 95%|█████████▌| 2471/2601 [00:02<00:00, 1123.54it/s]
 99%|█████████▉| 2586/2601 [00:02<00:00, 1125.17it/s]
100%|██████████| 2601/2601 [00:02<00:00, 1149.64it/s]
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Matplotlib is building the font cache; this may take a moment.
2026-05-11 03:40:43.383800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:43:18.130077: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:43:18.140523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:43:18.183288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:43:18.183392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:43:18.220741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:43:18.220892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:43:18.238847: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:43:18.255990: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:43:18.275257: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:43:18.293251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:43:18.311128: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:43:18.311643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:43:18.312073: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:43:18.312230: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:43:18.312464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:43:18.312496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:43:18.312511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:43:18.312525: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:43:18.312538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:43:18.312551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:43:18.312563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:43:18.312576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:43:18.312588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:43:18.312880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:43:18.312906: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:43:18.726743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:43:18.726840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:43:18.726851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:43:18.727544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:43:18.766634: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:43:18.801966: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:43:21.446966: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:43:21.924140: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:43:21.927871: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:43:23.494493: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:43:23.562118: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:44:08.642831: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)