Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 20:54:07.103456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:25.769188: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 20:54:25.770568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 20:54:25.797157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 20:54:25.797207: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:25.801824: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 20:54:25.801890: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 20:54:25.804232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 20:54:25.805683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 20:54:25.809101: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 20:54:25.810892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 20:54:25.812312: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 20:54:25.812695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 20:54:25.813055: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 20:54:25.814518: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 20:54:25.814795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 20:54:25.814816: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:25.814831: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 20:54:25.814840: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 20:54:25.814849: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 20:54:25.814858: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 20:54:25.814867: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 20:54:25.814876: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 20:54:25.814898: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 20:54:25.815182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 20:54:25.815202: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:26.251051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 20:54:26.251154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 20:54:26.251165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 20:54:26.251876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 20:54:38.350631: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 20:54:38.351203: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 20:54:39.433134: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 20:54:39.931231: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 20:54:39.936105: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 20:54:41.445472: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 20:54:41.564319: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 20:55:07.042092: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 21:01:20.477048: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 21:01:25.834389: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:32.619304: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:01:32.620206: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:01:32.647131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:01:32.647181: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:32.651944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:01:32.652033: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:01:32.654337: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:01:32.655832: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:01:32.659419: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:01:32.661268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:01:32.662597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:01:32.662991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:01:32.663231: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:01:32.664821: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:01:32.665083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:01:32.665107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:32.665121: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:01:32.665132: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:01:32.665142: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:01:32.665152: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:01:32.665162: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:01:32.665172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:01:32.665183: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:01:32.665476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:01:32.665497: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:33.097453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:01:33.097563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:01:33.097573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:01:33.098277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/56 [00:00<?, ?it/s]2025-12-29 21:01:34.746021: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:01:34.746543: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:01:34.946918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:01:35.444540: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:01:35.446411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:01:37.107753: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:01:37.250770: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/56 [00:33<30:20, 33.11s/it]
batch:   4%|▎         | 2/56 [00:33<12:24, 13.78s/it]
batch:   5%|▌         | 3/56 [00:33<06:42,  7.60s/it]
batch:   7%|▋         | 4/56 [00:33<04:04,  4.69s/it]
batch:   9%|▉         | 5/56 [00:34<02:37,  3.09s/it]
batch:  11%|█         | 6/56 [00:34<01:45,  2.12s/it]
batch:  12%|█▎        | 7/56 [00:34<01:13,  1.50s/it]
batch:  14%|█▍        | 8/56 [00:34<00:52,  1.10s/it]
batch:  16%|█▌        | 9/56 [00:35<00:39,  1.20it/s]
batch:  18%|█▊        | 10/56 [00:35<00:29,  1.54it/s]
batch:  20%|█▉        | 11/56 [00:35<00:23,  1.91it/s]
batch:  21%|██▏       | 12/56 [00:35<00:19,  2.29it/s]
batch:  23%|██▎       | 13/56 [00:35<00:16,  2.67it/s]
batch:  25%|██▌       | 14/56 [00:36<00:13,  3.01it/s]
batch:  27%|██▋       | 15/56 [00:36<00:12,  3.30it/s]
batch:  29%|██▊       | 16/56 [00:36<00:11,  3.54it/s]
batch:  30%|███       | 17/56 [00:36<00:10,  3.73it/s]
batch:  32%|███▏      | 18/56 [00:37<00:09,  3.89it/s]
batch:  34%|███▍      | 19/56 [00:37<00:09,  4.00it/s]
batch:  36%|███▌      | 20/56 [00:37<00:08,  4.06it/s]
batch:  38%|███▊      | 21/56 [00:37<00:08,  4.12it/s]
batch:  39%|███▉      | 22/56 [00:38<00:08,  4.16it/s]
batch:  41%|████      | 23/56 [00:38<00:07,  4.17it/s]
batch:  43%|████▎     | 24/56 [00:38<00:07,  4.18it/s]
batch:  45%|████▍     | 25/56 [00:38<00:07,  4.18it/s]
batch:  46%|████▋     | 26/56 [00:39<00:07,  4.19it/s]
batch:  48%|████▊     | 27/56 [00:39<00:06,  4.18it/s]
batch:  50%|█████     | 28/56 [00:39<00:06,  4.20it/s]
batch:  52%|█████▏    | 29/56 [00:39<00:06,  4.18it/s]
batch:  54%|█████▎    | 30/56 [00:40<00:06,  4.20it/s]
batch:  55%|█████▌    | 31/56 [00:40<00:05,  4.19it/s]
batch:  57%|█████▋    | 32/56 [00:40<00:05,  4.20it/s]
batch:  59%|█████▉    | 33/56 [00:40<00:05,  4.20it/s]
batch:  61%|██████    | 34/56 [00:40<00:05,  4.22it/s]
batch:  62%|██████▎   | 35/56 [00:41<00:04,  4.24it/s]
batch:  64%|██████▍   | 36/56 [00:41<00:04,  4.23it/s]
batch:  66%|██████▌   | 37/56 [00:41<00:04,  4.20it/s]
batch:  68%|██████▊   | 38/56 [00:41<00:04,  4.20it/s]
batch:  70%|██████▉   | 39/56 [00:42<00:04,  4.20it/s]
batch:  71%|███████▏  | 40/56 [00:42<00:03,  4.21it/s]
batch:  73%|███████▎  | 41/56 [00:42<00:03,  4.20it/s]
batch:  75%|███████▌  | 42/56 [00:42<00:03,  4.18it/s]
batch:  77%|███████▋  | 43/56 [00:43<00:03,  4.18it/s]
batch:  79%|███████▊  | 44/56 [00:43<00:02,  4.17it/s]
batch:  80%|████████  | 45/56 [00:43<00:02,  4.17it/s]
batch:  82%|████████▏ | 46/56 [00:43<00:02,  4.18it/s]
batch:  84%|████████▍ | 47/56 [00:44<00:02,  4.18it/s]
batch:  86%|████████▌ | 48/56 [00:44<00:01,  4.18it/s]
batch:  88%|████████▊ | 49/56 [00:44<00:01,  4.19it/s]
batch:  89%|████████▉ | 50/56 [00:44<00:01,  4.19it/s]
batch:  91%|█████████ | 51/56 [00:45<00:01,  4.21it/s]
batch:  93%|█████████▎| 52/56 [00:45<00:00,  4.23it/s]
batch:  95%|█████████▍| 53/56 [00:45<00:00,  4.21it/s]
batch:  96%|█████████▋| 54/56 [00:45<00:00,  4.19it/s]
batch:  98%|█████████▊| 55/56 [00:45<00:00,  4.22it/s]
batch: 100%|██████████| 56/56 [00:46<00:00,  4.49it/s]
batch: 100%|██████████| 56/56 [00:46<00:00,  1.21it/s]

  0%|          | 0/3568 [00:00<?, ?it/s]
  3%|▎         | 122/3568 [00:00<00:02, 1200.54it/s]
  7%|▋         | 243/3568 [00:00<00:02, 1196.20it/s]
 10%|█         | 363/3568 [00:00<00:02, 1189.12it/s]
 14%|█▎        | 483/3568 [00:00<00:02, 1193.10it/s]
 17%|█▋        | 603/3568 [00:00<00:02, 1184.92it/s]
 20%|██        | 722/3568 [00:00<00:02, 1177.94it/s]
 24%|██▎       | 840/3568 [00:00<00:02, 1178.32it/s]
 27%|██▋       | 958/3568 [00:00<00:02, 1168.19it/s]
 30%|███       | 1075/3568 [00:00<00:02, 1161.34it/s]
 33%|███▎      | 1192/3568 [00:01<00:02, 1154.59it/s]
 37%|███▋      | 1308/3568 [00:01<00:01, 1149.52it/s]
 40%|███▉      | 1423/3568 [00:01<00:01, 1139.81it/s]
 43%|████▎     | 1537/3568 [00:01<00:01, 1136.65it/s]
 46%|████▋     | 1651/3568 [00:01<00:01, 1129.93it/s]
 49%|████▉     | 1764/3568 [00:01<00:01, 1129.24it/s]
 53%|█████▎    | 1877/3568 [00:01<00:01, 1118.13it/s]
 56%|█████▌    | 1989/3568 [00:01<00:01, 1115.50it/s]
 59%|█████▉    | 2101/3568 [00:01<00:01, 1113.09it/s]
 62%|██████▏   | 2213/3568 [00:01<00:01, 1109.19it/s]
 65%|██████▌   | 2324/3568 [00:02<00:01, 1094.25it/s]
 68%|██████▊   | 2434/3568 [00:02<00:01, 1090.85it/s]
 71%|███████▏  | 2544/3568 [00:02<00:00, 1088.48it/s]
 74%|███████▍  | 2653/3568 [00:02<00:00, 1086.16it/s]
 77%|███████▋  | 2762/3568 [00:02<00:00, 1084.48it/s]
 80%|████████  | 2871/3568 [00:02<00:00, 1078.21it/s]
 83%|████████▎ | 2979/3568 [00:02<00:00, 1070.55it/s]
 87%|████████▋ | 3087/3568 [00:02<00:00, 1068.34it/s]
 90%|████████▉ | 3194/3568 [00:02<00:00, 1061.70it/s]
 93%|█████████▎| 3301/3568 [00:02<00:00, 1057.79it/s]
 95%|█████████▌| 3407/3568 [00:03<00:00, 1055.81it/s]
 98%|█████████▊| 3513/3568 [00:03<00:00, 1048.18it/s]
100%|██████████| 3568/3568 [00:03<00:00, 1111.51it/s]

  0%|          | 0/3568 [00:00<?, ?it/s]
  3%|▎         | 122/3568 [00:00<00:02, 1200.02it/s]
  7%|▋         | 243/3568 [00:00<00:02, 1196.66it/s]
 10%|█         | 363/3568 [00:00<00:02, 1189.27it/s]
 14%|█▎        | 484/3568 [00:00<00:02, 1189.22it/s]
 17%|█▋        | 603/3568 [00:00<00:02, 1186.97it/s]
 20%|██        | 722/3568 [00:00<00:02, 1179.55it/s]
 24%|██▎       | 841/3568 [00:00<00:02, 1176.41it/s]
 27%|██▋       | 959/3568 [00:00<00:02, 1171.98it/s]
 30%|███       | 1077/3568 [00:00<00:02, 1165.67it/s]
 33%|███▎      | 1194/3568 [00:01<00:02, 1151.98it/s]
 37%|███▋      | 1310/3568 [00:01<00:01, 1147.93it/s]
 40%|███▉      | 1425/3568 [00:01<00:01, 1138.86it/s]
 43%|████▎     | 1539/3568 [00:01<00:01, 1136.83it/s]
 46%|████▋     | 1653/3568 [00:01<00:01, 1136.13it/s]
 50%|████▉     | 1767/3568 [00:01<00:01, 1130.16it/s]
 53%|█████▎    | 1881/3568 [00:01<00:01, 1121.02it/s]
 56%|█████▌    | 1994/3568 [00:01<00:01, 1119.87it/s]
 59%|█████▉    | 2106/3568 [00:01<00:01, 1116.46it/s]
 62%|██████▏   | 2218/3568 [00:01<00:01, 1099.17it/s]
 65%|██████▌   | 2328/3568 [00:02<00:01, 1096.80it/s]
 68%|██████▊   | 2438/3568 [00:02<00:01, 1093.51it/s]
 71%|███████▏  | 2548/3568 [00:02<00:00, 1091.52it/s]
 74%|███████▍  | 2658/3568 [00:02<00:00, 1085.21it/s]
 78%|███████▊  | 2767/3568 [00:02<00:00, 1084.64it/s]
 81%|████████  | 2876/3568 [00:02<00:00, 1078.13it/s]
 84%|████████▎ | 2984/3568 [00:02<00:00, 1070.01it/s]
 87%|████████▋ | 3092/3568 [00:02<00:00, 1067.95it/s]
 90%|████████▉ | 3199/3568 [00:02<00:00, 1062.43it/s]
 93%|█████████▎| 3306/3568 [00:02<00:00, 1058.71it/s]
 96%|█████████▌| 3412/3568 [00:03<00:00, 1056.83it/s]
 99%|█████████▊| 3518/3568 [00:03<00:00, 1051.96it/s]
100%|██████████| 3568/3568 [00:03<00:00, 1112.15it/s]
2025-12-29 21:02:39.583397: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:02:47.517507: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:02:47.518429: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:02:47.546677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:02:47.546737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:02:47.551511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:02:47.551598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:02:47.553921: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:02:47.555435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:02:47.558959: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:02:47.560670: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:02:47.561973: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:02:47.562384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:02:47.562677: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:02:47.564328: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:02:47.564617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:61:00.0 name: NVIDIA L40S computeCapability: 8.9
coreClock: 2.52GHz coreCount: 142 deviceMemorySize: 44.52GiB deviceMemoryBandwidth: 804.75GiB/s
2025-12-29 21:02:47.564650: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:02:47.564681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:02:47.564695: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:02:47.564708: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:02:47.564720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:02:47.564733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:02:47.564745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:02:47.564758: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:02:47.565065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:02:47.565092: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:02:47.983082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:02:47.983185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:02:47.983195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:02:47.984169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 42313 MB memory) -> physical GPU (device: 0, name: NVIDIA L40S, pci bus id: 0000:61:00.0, compute capability: 8.9)
2025-12-29 21:02:48.029447: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 21:02:48.043178: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500000000 Hz
2025-12-29 21:02:51.164140: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:02:51.672592: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:02:51.676605: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:02:53.205977: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:02:53.324258: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:03:15.953126: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)