Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 03:21:39.456828: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:27:16.173780: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:27:16.188215: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:27:16.257297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:27:16.257344: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:27:16.365165: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:27:16.365232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:27:16.421132: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:27:16.478375: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:27:16.538292: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:27:16.597680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:27:16.662743: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:27:16.663112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:27:16.663428: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:27:16.663566: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:27:16.663776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:27:16.663794: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:27:16.663807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:27:16.663816: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:27:16.663825: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:27:16.663834: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:27:16.663842: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:27:16.663851: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:27:16.663872: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:27:16.664146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:27:16.664164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:27:17.084107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:27:17.084202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:27:17.084214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:27:17.084855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:27:19.438323: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:27:19.438857: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:27:22.434807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:27:22.986441: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:27:22.990915: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:27:37.031096: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:27:37.125439: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:28:49.416802: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 03:36:49.556931: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 03:37:48.246685: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:41:08.545342: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:41:08.553445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:41:08.619853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:41:08.619940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:41:08.645142: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:41:08.645271: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:41:08.657602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:41:08.669424: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:41:08.683484: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:41:08.695898: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:41:08.708005: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:41:08.708480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:41:08.708767: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:41:08.708911: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:41:08.709117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:41:08.709143: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:41:08.709157: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:41:08.709167: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:41:08.709177: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:41:08.709187: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:41:08.709197: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:41:08.709207: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:41:08.709217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:41:08.709484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:41:08.709505: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:41:09.132709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:41:09.132799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:41:09.132808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:41:09.133472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/42 [00:00<?, ?it/s]2026-05-11 03:41:10.893381: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:41:10.893873: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:41:11.186134: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:41:11.721672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:41:11.723339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:41:13.531962: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:41:13.632323: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/42 [01:59<1:21:58, 119.97s/it]
batch:   5%|▍         | 2/42 [02:00<33:01, 49.54s/it]   
batch:   7%|▋         | 3/42 [02:00<17:34, 27.03s/it]
batch:  10%|▉         | 4/42 [02:00<10:25, 16.46s/it]
batch:  12%|█▏        | 5/42 [02:00<06:32, 10.61s/it]
batch:  14%|█▍        | 6/42 [02:01<04:15,  7.08s/it]
batch:  17%|█▋        | 7/42 [02:01<02:49,  4.85s/it]
batch:  19%|█▉        | 8/42 [02:01<01:54,  3.38s/it]
batch:  21%|██▏       | 9/42 [02:01<01:19,  2.40s/it]
batch:  24%|██▍       | 10/42 [02:02<00:55,  1.73s/it]
batch:  26%|██▌       | 11/42 [02:02<00:39,  1.28s/it]
batch:  29%|██▊       | 12/42 [02:02<00:28,  1.04it/s]
batch:  31%|███       | 13/42 [02:02<00:21,  1.35it/s]
batch:  33%|███▎      | 14/42 [02:03<00:16,  1.69it/s]
batch:  36%|███▌      | 15/42 [02:03<00:13,  2.06it/s]
batch:  38%|███▊      | 16/42 [02:03<00:10,  2.43it/s]
batch:  40%|████      | 17/42 [02:03<00:09,  2.77it/s]
batch:  43%|████▎     | 18/42 [02:04<00:07,  3.08it/s]
batch:  45%|████▌     | 19/42 [02:04<00:06,  3.34it/s]
batch:  48%|████▊     | 20/42 [02:04<00:06,  3.54it/s]
batch:  50%|█████     | 21/42 [02:04<00:05,  3.71it/s]
batch:  52%|█████▏    | 22/42 [02:05<00:05,  3.83it/s]
batch:  55%|█████▍    | 23/42 [02:05<00:04,  3.92it/s]
batch:  57%|█████▋    | 24/42 [02:05<00:04,  3.98it/s]
batch:  60%|█████▉    | 25/42 [02:05<00:04,  4.03it/s]
batch:  62%|██████▏   | 26/42 [02:05<00:03,  4.07it/s]
batch:  64%|██████▍   | 27/42 [02:06<00:03,  4.09it/s]
batch:  67%|██████▋   | 28/42 [02:06<00:03,  4.12it/s]
batch:  69%|██████▉   | 29/42 [02:06<00:03,  4.12it/s]
batch:  71%|███████▏  | 30/42 [02:06<00:02,  4.14it/s]
batch:  74%|███████▍  | 31/42 [02:07<00:02,  4.14it/s]
batch:  76%|███████▌  | 32/42 [02:07<00:02,  4.13it/s]
batch:  79%|███████▊  | 33/42 [02:07<00:02,  4.15it/s]
batch:  81%|████████  | 34/42 [02:07<00:01,  4.15it/s]
batch:  83%|████████▎ | 35/42 [02:08<00:01,  4.15it/s]
batch:  86%|████████▌ | 36/42 [02:08<00:01,  4.14it/s]
batch:  88%|████████▊ | 37/42 [02:08<00:01,  4.13it/s]
batch:  90%|█████████ | 38/42 [02:08<00:00,  4.15it/s]
batch:  93%|█████████▎| 39/42 [02:09<00:00,  4.15it/s]
batch:  95%|█████████▌| 40/42 [02:09<00:00,  4.15it/s]
batch:  98%|█████████▊| 41/42 [02:09<00:00,  4.16it/s]
batch: 100%|██████████| 42/42 [02:09<00:00,  4.66it/s]
batch: 100%|██████████| 42/42 [02:09<00:00,  3.09s/it]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1198.51it/s]
  9%|▉         | 243/2660 [00:00<00:02, 1205.45it/s]
 14%|█▎        | 365/2660 [00:00<00:01, 1210.92it/s]
 18%|█▊        | 487/2660 [00:00<00:01, 1191.34it/s]
 23%|██▎       | 607/2660 [00:00<00:01, 1193.13it/s]
 27%|██▋       | 727/2660 [00:00<00:01, 1187.36it/s]
 32%|███▏      | 846/2660 [00:00<00:01, 1177.86it/s]
 36%|███▌      | 964/2660 [00:00<00:01, 1168.98it/s]
 41%|████      | 1081/2660 [00:00<00:01, 1158.07it/s]
 45%|████▌     | 1197/2660 [00:01<00:01, 1149.62it/s]
 49%|████▉     | 1312/2660 [00:01<00:01, 1142.40it/s]
 54%|█████▎    | 1427/2660 [00:01<00:01, 1143.75it/s]
 58%|█████▊    | 1542/2660 [00:01<00:00, 1135.33it/s]
 62%|██████▏   | 1656/2660 [00:01<00:00, 1125.93it/s]
 67%|██████▋   | 1769/2660 [00:01<00:00, 1114.93it/s]
 71%|███████   | 1881/2660 [00:01<00:00, 1111.30it/s]
 75%|███████▍  | 1993/2660 [00:01<00:00, 1101.24it/s]
 79%|███████▉  | 2104/2660 [00:01<00:00, 1093.26it/s]
 83%|████████▎ | 2214/2660 [00:01<00:00, 1091.20it/s]
 87%|████████▋ | 2324/2660 [00:02<00:00, 1086.40it/s]
 91%|█████████▏| 2433/2660 [00:02<00:00, 1077.91it/s]
 96%|█████████▌| 2541/2660 [00:02<00:00, 1074.48it/s]
100%|█████████▉| 2649/2660 [00:02<00:00, 1064.83it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1125.41it/s]

  0%|          | 0/2660 [00:00<?, ?it/s]
  5%|▍         | 121/2660 [00:00<00:02, 1200.41it/s]
  9%|▉         | 243/2660 [00:00<00:02, 1207.53it/s]
 14%|█▎        | 364/2660 [00:00<00:01, 1204.39it/s]
 18%|█▊        | 485/2660 [00:00<00:01, 1194.62it/s]
 23%|██▎       | 605/2660 [00:00<00:01, 1189.70it/s]
 27%|██▋       | 726/2660 [00:00<00:01, 1188.90it/s]
 32%|███▏      | 845/2660 [00:00<00:01, 1181.11it/s]
 36%|███▌      | 964/2660 [00:00<00:01, 1174.33it/s]
 41%|████      | 1082/2660 [00:00<00:01, 1162.02it/s]
 45%|████▌     | 1199/2660 [00:01<00:01, 1155.98it/s]
 49%|████▉     | 1315/2660 [00:01<00:01, 1149.64it/s]
 54%|█████▍    | 1431/2660 [00:01<00:01, 1146.67it/s]
 58%|█████▊    | 1546/2660 [00:01<00:00, 1138.23it/s]
 62%|██████▏   | 1660/2660 [00:01<00:00, 1134.05it/s]
 67%|██████▋   | 1774/2660 [00:01<00:00, 1124.26it/s]
 71%|███████   | 1887/2660 [00:01<00:00, 1110.13it/s]
 75%|███████▌  | 1999/2660 [00:01<00:00, 1109.64it/s]
 79%|███████▉  | 2110/2660 [00:01<00:00, 1098.46it/s]
 83%|████████▎ | 2220/2660 [00:01<00:00, 1095.00it/s]
 88%|████████▊ | 2330/2660 [00:02<00:00, 1091.19it/s]
 92%|█████████▏| 2440/2660 [00:02<00:00, 1084.52it/s]
 96%|█████████▌| 2549/2660 [00:02<00:00, 1075.65it/s]
100%|█████████▉| 2657/2660 [00:02<00:00, 1066.82it/s]
100%|██████████| 2660/2660 [00:02<00:00, 1129.31it/s]
2026-05-11 03:48:08.404011: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:53:13.987858: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:53:14.019507: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:53:14.100399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:53:14.100521: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:53:14.145100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:53:14.145252: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:53:14.167608: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:53:14.189443: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:53:14.213296: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:53:14.236706: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:53:14.301996: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:14.302487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:53:14.302911: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:53:14.303048: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:53:14.303328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2026-05-11 03:53:14.303363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:53:14.303379: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:53:14.303392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:53:14.303405: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:53:14.303418: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:53:14.303430: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:53:14.303443: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:53:14.303456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:14.303742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:53:14.303767: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:53:14.723157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:53:14.723249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:53:14.723258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:53:14.723959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2026-05-11 03:53:14.765271: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:53:14.781805: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2026-05-11 03:53:17.416976: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:53:17.897687: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:53:17.901436: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:53:19.869272: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:53:19.966685: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2026-05-11 03:55:17.874567: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)