Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 21:52:45.685815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:53:07.181751: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:53:07.190184: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:53:07.501079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 21:53:07.501250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:53:08.054143: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:53:08.054251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:53:08.428823: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:53:08.949690: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:53:09.535864: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:53:09.756572: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:53:09.923304: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:53:09.928977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:53:09.929929: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:53:09.934204: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:53:09.936868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 21:53:09.936899: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:53:09.936917: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:53:09.936933: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:53:09.936948: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:53:09.936962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:53:09.936977: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:53:09.936991: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:53:09.937030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:53:09.944535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:53:09.944573: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:53:12.569294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:53:12.569482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:53:12.569501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:53:12.574825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37373 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:4d:00.0, compute capability: 8.0)
2025-12-29 21:53:26.205364: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:53:26.206630: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000035000 Hz
2025-12-29 21:53:27.739205: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:53:29.518886: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:53:29.527563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:53:43.102324: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:53:43.207630: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:54:08.845945: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 21:56:34.381595: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 21:56:39.153817: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:48.894109: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:56:48.895298: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:56:49.219984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 21:56:49.220128: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:49.226089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:56:49.226212: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:56:49.229105: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:56:49.230562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:56:49.235441: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:56:49.237403: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:56:49.238741: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:56:49.244699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:56:49.245215: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:56:49.248209: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:56:49.250626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 21:56:49.250684: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:49.250719: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:56:49.250749: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:56:49.250778: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:56:49.250807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:56:49.250845: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:56:49.250876: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:56:49.250905: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:56:49.255449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:56:49.255488: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:56:49.798514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:56:49.798673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:56:49.798689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:56:49.806409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37373 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:4d:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/54 [00:00<?, ?it/s]2025-12-29 21:56:51.644576: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:56:51.645156: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000035000 Hz
2025-12-29 21:56:51.909523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:56:52.412969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:56:52.415681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:56:53.939903: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:56:54.044900: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/54 [00:30<27:05, 30.66s/it]
batch:   4%|▎         | 2/54 [00:30<11:06, 12.81s/it]
batch:   6%|▌         | 3/54 [00:31<06:02,  7.10s/it]
batch:   7%|▋         | 4/54 [00:31<03:40,  4.42s/it]
batch:   9%|▉         | 5/54 [00:31<02:23,  2.93s/it]
batch:  11%|█         | 6/54 [00:32<01:37,  2.04s/it]
batch:  13%|█▎        | 7/54 [00:32<01:09,  1.47s/it]
batch:  15%|█▍        | 8/54 [00:32<00:50,  1.10s/it]
batch:  17%|█▋        | 9/54 [00:33<00:38,  1.17it/s]
batch:  19%|█▊        | 10/54 [00:33<00:30,  1.46it/s]
batch:  20%|██        | 11/54 [00:33<00:24,  1.76it/s]
batch:  22%|██▏       | 12/54 [00:34<00:20,  2.05it/s]
batch:  24%|██▍       | 13/54 [00:34<00:17,  2.31it/s]
batch:  26%|██▌       | 14/54 [00:34<00:16,  2.50it/s]
batch:  28%|██▊       | 15/54 [00:34<00:14,  2.66it/s]
batch:  30%|██▉       | 16/54 [00:35<00:13,  2.81it/s]
batch:  31%|███▏      | 17/54 [00:35<00:12,  2.93it/s]
batch:  33%|███▎      | 18/54 [00:35<00:11,  3.02it/s]
batch:  35%|███▌      | 19/54 [00:36<00:11,  3.07it/s]
batch:  37%|███▋      | 20/54 [00:36<00:10,  3.11it/s]
batch:  39%|███▉      | 21/54 [00:36<00:10,  3.11it/s]
batch:  41%|████      | 22/54 [00:37<00:10,  3.12it/s]
batch:  43%|████▎     | 23/54 [00:37<00:09,  3.14it/s]
batch:  44%|████▍     | 24/54 [00:37<00:09,  3.15it/s]
batch:  46%|████▋     | 25/54 [00:38<00:09,  3.16it/s]
batch:  48%|████▊     | 26/54 [00:38<00:08,  3.18it/s]
batch:  50%|█████     | 27/54 [00:38<00:08,  3.19it/s]
batch:  52%|█████▏    | 28/54 [00:39<00:08,  3.20it/s]
batch:  54%|█████▎    | 29/54 [00:39<00:07,  3.17it/s]
batch:  56%|█████▌    | 30/54 [00:39<00:07,  3.17it/s]
batch:  57%|█████▋    | 31/54 [00:39<00:07,  3.19it/s]
batch:  59%|█████▉    | 32/54 [00:40<00:06,  3.17it/s]
batch:  61%|██████    | 33/54 [00:40<00:06,  3.18it/s]
batch:  63%|██████▎   | 34/54 [00:40<00:06,  3.17it/s]
batch:  65%|██████▍   | 35/54 [00:41<00:05,  3.19it/s]
batch:  67%|██████▋   | 36/54 [00:41<00:05,  3.20it/s]
batch:  69%|██████▊   | 37/54 [00:41<00:05,  3.20it/s]
batch:  70%|███████   | 38/54 [00:42<00:04,  3.21it/s]
batch:  72%|███████▏  | 39/54 [00:42<00:04,  3.18it/s]
batch:  74%|███████▍  | 40/54 [00:42<00:04,  3.18it/s]
batch:  76%|███████▌  | 41/54 [00:43<00:04,  3.19it/s]
batch:  78%|███████▊  | 42/54 [00:43<00:03,  3.19it/s]
batch:  80%|███████▉  | 43/54 [00:43<00:03,  3.19it/s]
batch:  81%|████████▏ | 44/54 [00:44<00:03,  3.18it/s]
batch:  83%|████████▎ | 45/54 [00:44<00:02,  3.17it/s]
batch:  85%|████████▌ | 46/54 [00:44<00:02,  3.17it/s]
batch:  87%|████████▋ | 47/54 [00:45<00:02,  3.17it/s]
batch:  89%|████████▉ | 48/54 [00:45<00:01,  3.17it/s]
batch:  91%|█████████ | 49/54 [00:45<00:01,  3.19it/s]
batch:  93%|█████████▎| 50/54 [00:45<00:01,  3.22it/s]
batch:  94%|█████████▍| 51/54 [00:46<00:00,  3.24it/s]
batch:  96%|█████████▋| 52/54 [00:46<00:00,  3.22it/s]
batch:  98%|█████████▊| 53/54 [00:46<00:00,  3.83it/s]
batch: 100%|██████████| 54/54 [00:46<00:00,  1.15it/s]

  0%|          | 0/3350 [00:00<?, ?it/s]
  3%|▎         | 94/3350 [00:00<00:03, 934.63it/s]
  6%|▌         | 192/3350 [00:00<00:03, 955.07it/s]
  9%|▊         | 289/3350 [00:00<00:03, 961.28it/s]
 12%|█▏        | 389/3350 [00:00<00:03, 971.41it/s]
 15%|█▍        | 488/3350 [00:00<00:02, 974.55it/s]
 18%|█▊        | 587/3350 [00:00<00:02, 979.71it/s]
 20%|██        | 685/3350 [00:00<00:02, 972.40it/s]
 23%|██▎       | 783/3350 [00:00<00:02, 971.72it/s]
 26%|██▋       | 881/3350 [00:00<00:02, 967.12it/s]
 29%|██▉       | 978/3350 [00:01<00:02, 955.43it/s]
 32%|███▏      | 1074/3350 [00:01<00:02, 951.54it/s]
 35%|███▍      | 1170/3350 [00:01<00:02, 951.21it/s]
 38%|███▊      | 1266/3350 [00:01<00:02, 947.30it/s]
 41%|████      | 1361/3350 [00:01<00:02, 933.59it/s]
 43%|████▎     | 1455/3350 [00:01<00:02, 931.98it/s]
 46%|████▌     | 1549/3350 [00:01<00:01, 929.47it/s]
 49%|████▉     | 1642/3350 [00:01<00:01, 928.01it/s]
 52%|█████▏    | 1735/3350 [00:01<00:01, 927.44it/s]
 55%|█████▍    | 1828/3350 [00:01<00:01, 926.78it/s]
 57%|█████▋    | 1921/3350 [00:02<00:01, 925.96it/s]
 60%|██████    | 2014/3350 [00:02<00:01, 917.77it/s]
 63%|██████▎   | 2106/3350 [00:02<00:01, 914.20it/s]
 66%|██████▌   | 2198/3350 [00:02<00:01, 914.01it/s]
 68%|██████▊   | 2290/3350 [00:02<00:01, 908.87it/s]
 71%|███████   | 2381/3350 [00:02<00:01, 905.07it/s]
 74%|███████▍  | 2472/3350 [00:02<00:00, 903.76it/s]
 77%|███████▋  | 2563/3350 [00:02<00:00, 901.46it/s]
 79%|███████▉  | 2654/3350 [00:02<00:00, 892.37it/s]
 82%|████████▏ | 2744/3350 [00:02<00:00, 885.43it/s]
 85%|████████▍ | 2833/3350 [00:03<00:00, 879.19it/s]
 87%|████████▋ | 2921/3350 [00:03<00:00, 871.22it/s]
 90%|████████▉ | 3009/3350 [00:03<00:00, 866.30it/s]
 92%|█████████▏| 3096/3350 [00:03<00:00, 864.82it/s]
 95%|█████████▌| 3183/3350 [00:03<00:00, 853.92it/s]
 98%|█████████▊| 3269/3350 [00:03<00:00, 855.61it/s]
100%|██████████| 3350/3350 [00:03<00:00, 914.46it/s]

  0%|          | 0/3350 [00:00<?, ?it/s]
  3%|▎         | 96/3350 [00:00<00:03, 952.58it/s]
  6%|▌         | 195/3350 [00:00<00:03, 968.32it/s]
  9%|▊         | 293/3350 [00:00<00:03, 973.40it/s]
 12%|█▏        | 391/3350 [00:00<00:03, 974.99it/s]
 15%|█▍        | 491/3350 [00:00<00:02, 981.15it/s]
 18%|█▊        | 590/3350 [00:00<00:02, 978.79it/s]
 21%|██        | 688/3350 [00:00<00:02, 978.39it/s]
 23%|██▎       | 786/3350 [00:00<00:02, 971.07it/s]
 26%|██▋       | 884/3350 [00:00<00:02, 967.32it/s]
 29%|██▉       | 981/3350 [00:01<00:02, 959.71it/s]
 32%|███▏      | 1077/3350 [00:01<00:02, 955.63it/s]
 35%|███▌      | 1173/3350 [00:01<00:02, 955.33it/s]
 38%|███▊      | 1269/3350 [00:01<00:02, 950.58it/s]
 41%|████      | 1365/3350 [00:01<00:02, 951.46it/s]
 44%|████▎     | 1461/3350 [00:01<00:01, 949.79it/s]
 46%|████▋     | 1556/3350 [00:01<00:01, 944.43it/s]
 49%|████▉     | 1651/3350 [00:01<00:01, 938.14it/s]
 52%|█████▏    | 1745/3350 [00:01<00:01, 937.75it/s]
 55%|█████▍    | 1839/3350 [00:01<00:01, 931.78it/s]
 58%|█████▊    | 1933/3350 [00:02<00:01, 930.98it/s]
 61%|██████    | 2027/3350 [00:02<00:01, 923.13it/s]
 63%|██████▎   | 2120/3350 [00:02<00:01, 917.08it/s]
 66%|██████▌   | 2212/3350 [00:02<00:01, 911.47it/s]
 69%|██████▉   | 2305/3350 [00:02<00:01, 911.02it/s]
 72%|███████▏  | 2397/3350 [00:02<00:01, 909.26it/s]
 74%|███████▍  | 2488/3350 [00:02<00:00, 906.96it/s]
 77%|███████▋  | 2579/3350 [00:02<00:00, 904.73it/s]
 80%|███████▉  | 2670/3350 [00:02<00:00, 900.39it/s]
 82%|████████▏ | 2761/3350 [00:02<00:00, 890.96it/s]
 85%|████████▌ | 2851/3350 [00:03<00:00, 882.11it/s]
 88%|████████▊ | 2940/3350 [00:03<00:00, 871.71it/s]
 90%|█████████ | 3028/3350 [00:03<00:00, 863.30it/s]
 93%|█████████▎| 3115/3350 [00:03<00:00, 863.01it/s]
 96%|█████████▌| 3202/3350 [00:03<00:00, 858.17it/s]
 98%|█████████▊| 3288/3350 [00:03<00:00, 855.57it/s]
100%|██████████| 3350/3350 [00:03<00:00, 919.75it/s]
2025-12-29 21:58:01.273055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:58:09.925497: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:58:09.926859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:58:10.254520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 21:58:10.254740: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:58:10.259273: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:58:10.259338: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:58:10.261294: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:58:10.262435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:58:10.266049: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:58:10.267807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:58:10.269207: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:58:10.275419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:58:10.276033: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:58:10.279546: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:58:10.281905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:4d:00.0 name: NVIDIA A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.38GiB deviceMemoryBandwidth: 1.41TiB/s
2025-12-29 21:58:10.281972: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:58:10.281997: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:58:10.282018: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:58:10.282038: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:58:10.282059: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:58:10.282078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:58:10.282097: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:58:10.282116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:58:10.286554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:58:10.286602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:58:10.830691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:58:10.830840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:58:10.830855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:58:10.835267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37373 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:4d:00.0, compute capability: 8.0)
2025-12-29 21:58:10.890076: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 21:58:10.907704: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000035000 Hz
2025-12-29 21:58:14.130216: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:58:14.628936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:58:14.636787: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:58:16.172738: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:58:16.279963: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:58:37.605089: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)