Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-11-17 17:39:03.049208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:39:18.290839: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:39:18.292107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 17:39:18.318228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:39:18.318277: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:39:18.322235: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:39:18.322301: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:39:18.324185: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:39:18.325424: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:39:18.328517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:39:18.329969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:39:18.331027: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:39:18.331406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:39:18.331812: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 17:39:18.333461: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:39:18.333727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:39:18.333749: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:39:18.333763: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:39:18.333772: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:39:18.333781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:39:18.333790: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:39:18.333800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:39:18.333809: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:39:18.333830: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:39:18.334113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:39:18.334133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:39:18.741366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 17:39:18.741461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 17:39:18.741473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 17:39:18.742171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-11-17 17:39:31.529803: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 17:39:31.530455: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 17:39:32.603545: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:39:33.102750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:39:33.108602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:39:34.620729: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 17:39:34.724096: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 17:39:54.568654: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-11-17 17:44:57.396426: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-11-17 17:45:00.998819: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:45:06.945736: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:45:06.946541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 17:45:06.973527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:45:06.973575: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:45:06.980181: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:45:06.980258: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:45:06.983445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:45:06.986455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:45:06.990814: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:45:06.993163: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:45:06.995677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:45:06.996065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:45:06.996296: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 17:45:06.997810: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:45:06.998065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:45:06.998087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:45:06.998102: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:45:06.998113: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:45:06.998123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:45:06.998133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:45:06.998143: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:45:06.998153: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:45:06.998163: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:45:06.998450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:45:06.998472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:45:07.414457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 17:45:07.414553: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 17:45:07.414564: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 17:45:07.415267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/58 [00:00<?, ?it/s]2025-11-17 17:45:08.989075: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 17:45:08.989776: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 17:45:09.197993: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:45:09.711555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:45:09.713589: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:45:11.281974: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 17:45:11.405242: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/58 [00:27<26:18, 27.69s/it]
batch:   3%|▎         | 2/58 [00:27<10:46, 11.55s/it]
batch:   5%|▌         | 3/58 [00:28<05:51,  6.39s/it]
batch:   7%|▋         | 4/58 [00:28<03:33,  3.96s/it]
batch:   9%|▊         | 5/58 [00:28<02:18,  2.62s/it]
batch:  10%|█         | 6/58 [00:28<01:34,  1.81s/it]
batch:  12%|█▏        | 7/58 [00:29<01:06,  1.29s/it]
batch:  14%|█▍        | 8/58 [00:29<00:48,  1.04it/s]
batch:  16%|█▌        | 9/58 [00:29<00:36,  1.36it/s]
batch:  17%|█▋        | 10/58 [00:29<00:27,  1.72it/s]
batch:  19%|█▉        | 11/58 [00:30<00:22,  2.10it/s]
batch:  21%|██        | 12/58 [00:30<00:18,  2.47it/s]
batch:  22%|██▏       | 13/58 [00:30<00:15,  2.82it/s]
batch:  24%|██▍       | 14/58 [00:30<00:14,  3.13it/s]
batch:  26%|██▌       | 15/58 [00:31<00:12,  3.38it/s]
batch:  28%|██▊       | 16/58 [00:31<00:11,  3.59it/s]
batch:  29%|██▉       | 17/58 [00:31<00:10,  3.75it/s]
batch:  31%|███       | 18/58 [00:31<00:10,  3.87it/s]
batch:  33%|███▎      | 19/58 [00:32<00:09,  3.96it/s]
batch:  34%|███▍      | 20/58 [00:32<00:09,  4.01it/s]
batch:  36%|███▌      | 21/58 [00:32<00:09,  4.07it/s]
batch:  38%|███▊      | 22/58 [00:32<00:08,  4.10it/s]
batch:  40%|███▉      | 23/58 [00:32<00:08,  4.10it/s]
batch:  41%|████▏     | 24/58 [00:33<00:08,  4.10it/s]
batch:  43%|████▎     | 25/58 [00:33<00:08,  4.11it/s]
batch:  45%|████▍     | 26/58 [00:33<00:07,  4.14it/s]
batch:  47%|████▋     | 27/58 [00:33<00:07,  4.15it/s]
batch:  48%|████▊     | 28/58 [00:34<00:07,  4.17it/s]
batch:  50%|█████     | 29/58 [00:34<00:06,  4.18it/s]
batch:  52%|█████▏    | 30/58 [00:34<00:06,  4.18it/s]
batch:  53%|█████▎    | 31/58 [00:34<00:06,  4.19it/s]
batch:  55%|█████▌    | 32/58 [00:35<00:06,  4.17it/s]
batch:  57%|█████▋    | 33/58 [00:35<00:05,  4.20it/s]
batch:  59%|█████▊    | 34/58 [00:35<00:05,  4.17it/s]
batch:  60%|██████    | 35/58 [00:35<00:05,  4.18it/s]
batch:  62%|██████▏   | 36/58 [00:36<00:05,  4.20it/s]
batch:  64%|██████▍   | 37/58 [00:36<00:04,  4.21it/s]
batch:  66%|██████▌   | 38/58 [00:36<00:04,  4.22it/s]
batch:  67%|██████▋   | 39/58 [00:36<00:04,  4.22it/s]
batch:  69%|██████▉   | 40/58 [00:37<00:04,  4.20it/s]
batch:  71%|███████   | 41/58 [00:37<00:04,  4.19it/s]
batch:  72%|███████▏  | 42/58 [00:37<00:03,  4.21it/s]
batch:  74%|███████▍  | 43/58 [00:37<00:03,  4.18it/s]
batch:  76%|███████▌  | 44/58 [00:38<00:03,  3.92it/s]
batch:  78%|███████▊  | 45/58 [00:38<00:03,  4.00it/s]
batch:  79%|███████▉  | 46/58 [00:38<00:02,  4.06it/s]
batch:  81%|████████  | 47/58 [00:38<00:02,  4.10it/s]
batch:  83%|████████▎ | 48/58 [00:39<00:02,  4.11it/s]
batch:  84%|████████▍ | 49/58 [00:39<00:02,  4.12it/s]
batch:  86%|████████▌ | 50/58 [00:39<00:01,  4.14it/s]
batch:  88%|████████▊ | 51/58 [00:39<00:01,  4.16it/s]
batch:  90%|████████▉ | 52/58 [00:39<00:01,  4.17it/s]
batch:  91%|█████████▏| 53/58 [00:40<00:01,  4.18it/s]
batch:  93%|█████████▎| 54/58 [00:40<00:00,  4.15it/s]
batch:  95%|█████████▍| 55/58 [00:40<00:00,  4.15it/s]
batch:  97%|█████████▋| 56/58 [00:40<00:00,  4.18it/s]
batch:  98%|█████████▊| 57/58 [00:41<00:00,  4.18it/s]
batch: 100%|██████████| 58/58 [00:41<00:00,  1.41it/s]

  0%|          | 0/3656 [00:00<?, ?it/s]
  3%|▎         | 119/3656 [00:00<00:02, 1189.31it/s]
  7%|▋         | 240/3656 [00:00<00:02, 1201.01it/s]
 10%|▉         | 361/3656 [00:00<00:02, 1194.26it/s]
 13%|█▎        | 481/3656 [00:00<00:02, 1184.83it/s]
 16%|█▋        | 600/3656 [00:00<00:02, 1178.76it/s]
 20%|█▉        | 718/3656 [00:00<00:02, 1174.15it/s]
 23%|██▎       | 836/3656 [00:00<00:02, 1167.14it/s]
 26%|██▌       | 953/3656 [00:00<00:02, 1162.63it/s]
 29%|██▉       | 1070/3656 [00:00<00:02, 1162.46it/s]
 32%|███▏      | 1187/3656 [00:01<00:02, 1153.47it/s]
 36%|███▌      | 1303/3656 [00:01<00:02, 1143.55it/s]
 39%|███▉      | 1418/3656 [00:01<00:01, 1140.21it/s]
 42%|████▏     | 1533/3656 [00:01<00:01, 1134.70it/s]
 45%|████▌     | 1647/3656 [00:01<00:01, 1127.16it/s]
 48%|████▊     | 1760/3656 [00:01<00:01, 1126.56it/s]
 51%|█████     | 1873/3656 [00:01<00:01, 1120.29it/s]
 54%|█████▍    | 1986/3656 [00:01<00:01, 1115.84it/s]
 57%|█████▋    | 2098/3656 [00:01<00:01, 1111.92it/s]
 60%|██████    | 2210/3656 [00:01<00:01, 1113.98it/s]
 64%|██████▎   | 2322/3656 [00:02<00:01, 1100.99it/s]
 67%|██████▋   | 2433/3656 [00:02<00:01, 1096.97it/s]
 70%|██████▉   | 2543/3656 [00:02<00:01, 1095.78it/s]
 73%|███████▎  | 2653/3656 [00:02<00:00, 1087.64it/s]
 76%|███████▌  | 2762/3656 [00:02<00:00, 1085.82it/s]
 79%|███████▊  | 2871/3656 [00:02<00:00, 1075.17it/s]
 81%|████████▏ | 2979/3656 [00:02<00:00, 1072.94it/s]
 84%|████████▍ | 3087/3656 [00:02<00:00, 1066.72it/s]
 87%|████████▋ | 3194/3656 [00:02<00:00, 1063.17it/s]
 90%|█████████ | 3301/3656 [00:02<00:00, 1057.09it/s]
 93%|█████████▎| 3407/3656 [00:03<00:00, 1057.71it/s]
 96%|█████████▌| 3513/3656 [00:03<00:00, 1052.46it/s]
 99%|█████████▉| 3619/3656 [00:03<00:00, 1047.23it/s]
100%|██████████| 3656/3656 [00:03<00:00, 1108.95it/s]

  0%|          | 0/3656 [00:00<?, ?it/s]
  3%|▎         | 120/3656 [00:00<00:02, 1197.71it/s]
  7%|▋         | 241/3656 [00:00<00:02, 1193.19it/s]
 10%|▉         | 361/3656 [00:00<00:02, 1187.04it/s]
 13%|█▎        | 480/3656 [00:00<00:02, 1183.63it/s]
 16%|█▋        | 599/3656 [00:00<00:02, 1178.58it/s]
 20%|█▉        | 717/3656 [00:00<00:02, 1174.64it/s]
 23%|██▎       | 835/3656 [00:00<00:02, 1173.21it/s]
 26%|██▌       | 953/3656 [00:00<00:02, 1163.52it/s]
 29%|██▉       | 1070/3656 [00:00<00:02, 1163.82it/s]
 32%|███▏      | 1187/3656 [00:01<00:02, 1150.40it/s]
 36%|███▌      | 1303/3656 [00:01<00:02, 1145.73it/s]
 39%|███▉      | 1418/3656 [00:01<00:01, 1141.86it/s]
 42%|████▏     | 1533/3656 [00:01<00:01, 1135.98it/s]
 45%|████▌     | 1647/3656 [00:01<00:01, 1128.66it/s]
 48%|████▊     | 1760/3656 [00:01<00:01, 1127.89it/s]
 51%|█████     | 1873/3656 [00:01<00:01, 1123.87it/s]
 54%|█████▍    | 1986/3656 [00:01<00:01, 1118.99it/s]
 57%|█████▋    | 2098/3656 [00:01<00:01, 1116.19it/s]
 60%|██████    | 2211/3656 [00:01<00:01, 1114.39it/s]
 64%|██████▎   | 2323/3656 [00:02<00:01, 1107.84it/s]
 67%|██████▋   | 2434/3656 [00:02<00:01, 1102.43it/s]
 70%|██████▉   | 2545/3656 [00:02<00:01, 1096.19it/s]
 73%|███████▎  | 2655/3656 [00:02<00:00, 1093.61it/s]
 76%|███████▌  | 2765/3656 [00:02<00:00, 1087.08it/s]
 79%|███████▊  | 2874/3656 [00:02<00:00, 1072.75it/s]
 82%|████████▏ | 2982/3656 [00:02<00:00, 1066.00it/s]
 84%|████████▍ | 3089/3656 [00:02<00:00, 1064.24it/s]
 87%|████████▋ | 3196/3656 [00:02<00:00, 1059.16it/s]
 90%|█████████ | 3302/3656 [00:02<00:00, 1057.23it/s]
 93%|█████████▎| 3408/3656 [00:03<00:00, 1052.52it/s]
 96%|█████████▌| 3514/3656 [00:03<00:00, 1049.29it/s]
 99%|█████████▉| 3619/3656 [00:03<00:00, 1041.27it/s]
100%|██████████| 3656/3656 [00:03<00:00, 1109.03it/s]
2025-11-17 17:46:08.090361: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:46:16.074688: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:46:16.075870: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 17:46:16.105058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:46:16.105156: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:46:16.109289: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:46:16.109369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:46:16.111335: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:46:16.112502: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:46:16.115770: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:46:16.117328: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:46:16.118298: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:46:16.118714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:46:16.119403: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 17:46:16.121045: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 17:46:16.121323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-11-17 17:46:16.121370: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:46:16.121386: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:46:16.121400: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:46:16.121412: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 17:46:16.121425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 17:46:16.121438: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 17:46:16.121450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 17:46:16.121463: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:46:16.121771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 17:46:16.121798: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 17:46:16.538380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 17:46:16.538473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 17:46:16.538484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 17:46:16.539278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-11-17 17:46:16.582496: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-11-17 17:46:16.596844: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-11-17 17:46:19.819735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 17:46:20.331293: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 17:46:20.337098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 17:46:21.856274: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 17:46:21.953258: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-11-17 17:46:38.602056: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)