Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-12-29 20:54:40.452648: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:55.391159: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 20:54:55.392971: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 20:54:55.693384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 20:54:55.693477: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:55.697630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 20:54:55.697686: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 20:54:55.699462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 20:54:55.700765: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 20:54:55.703733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 20:54:55.705396: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 20:54:55.706637: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 20:54:55.712450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 20:54:55.713000: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 20:54:55.715465: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 20:54:55.717500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 20:54:55.717538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:55.717561: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 20:54:55.717578: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 20:54:55.717594: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 20:54:55.717610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 20:54:55.717625: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 20:54:55.717640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 20:54:55.717676: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 20:54:55.721548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 20:54:55.721593: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 20:54:56.207129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 20:54:56.207245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 20:54:56.207261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 20:54:56.212341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:c3:00.0, compute capability: 8.0)
2025-12-29 20:55:08.735971: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 20:55:08.736769: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 20:55:10.124667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 20:55:10.593411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 20:55:10.599496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 20:55:12.141679: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 20:55:12.314011: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 20:55:44.264377: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-12-29 21:01:25.330700: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-12-29 21:01:29.820748: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:35.616356: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:01:35.617586: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:01:35.940352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:01:35.940479: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:35.945250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:01:35.945328: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:01:35.947798: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:01:35.949216: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:01:35.953339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:01:35.955261: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:01:35.956464: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:01:35.961371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:01:35.961822: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:01:35.963746: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:01:35.965716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:01:35.965759: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:35.965784: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:01:35.965802: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:01:35.965818: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:01:35.965833: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:01:35.965848: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:01:35.965863: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:01:35.965878: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:01:35.970589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:01:35.970631: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:01:36.474286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:01:36.474422: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:01:36.474440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:01:36.478453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:c3:00.0, compute capability: 8.0)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/56 [00:00<?, ?it/s]2025-12-29 21:01:38.536481: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-12-29 21:01:38.537132: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 21:01:38.790783: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:01:39.304983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:01:39.308146: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:01:40.841713: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:01:40.997650: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/56 [00:44<40:46, 44.48s/it]
batch:   4%|▎         | 2/56 [00:44<16:39, 18.51s/it]
batch:   5%|▌         | 3/56 [00:45<09:00, 10.20s/it]
batch:   7%|▋         | 4/56 [00:45<05:27,  6.30s/it]
batch:   9%|▉         | 5/56 [00:45<03:31,  4.14s/it]
batch:  11%|█         | 6/56 [00:46<02:21,  2.84s/it]
batch:  12%|█▎        | 7/56 [00:46<01:38,  2.01s/it]
batch:  14%|█▍        | 8/56 [00:46<01:10,  1.47s/it]
batch:  16%|█▌        | 9/56 [00:46<00:52,  1.11s/it]
batch:  18%|█▊        | 10/56 [00:47<00:39,  1.16it/s]
batch:  20%|█▉        | 11/56 [00:47<00:31,  1.45it/s]
batch:  21%|██▏       | 12/56 [00:47<00:25,  1.74it/s]
batch:  23%|██▎       | 13/56 [00:48<00:21,  2.03it/s]
batch:  25%|██▌       | 14/56 [00:48<00:18,  2.28it/s]
batch:  27%|██▋       | 15/56 [00:48<00:16,  2.49it/s]
batch:  29%|██▊       | 16/56 [00:49<00:15,  2.66it/s]
batch:  30%|███       | 17/56 [00:49<00:13,  2.81it/s]
batch:  32%|███▏      | 18/56 [00:49<00:13,  2.91it/s]
batch:  34%|███▍      | 19/56 [00:50<00:12,  2.98it/s]
batch:  36%|███▌      | 20/56 [00:50<00:11,  3.00it/s]
batch:  38%|███▊      | 21/56 [00:50<00:11,  3.02it/s]
batch:  39%|███▉      | 22/56 [00:51<00:11,  3.04it/s]
batch:  41%|████      | 23/56 [00:51<00:10,  3.10it/s]
batch:  43%|████▎     | 24/56 [00:51<00:10,  3.11it/s]
batch:  45%|████▍     | 25/56 [00:52<00:10,  3.08it/s]
batch:  46%|████▋     | 26/56 [00:52<00:09,  3.11it/s]
batch:  48%|████▊     | 27/56 [00:52<00:09,  3.13it/s]
batch:  50%|█████     | 28/56 [00:52<00:08,  3.15it/s]
batch:  52%|█████▏    | 29/56 [00:53<00:08,  3.17it/s]
batch:  54%|█████▎    | 30/56 [00:53<00:08,  3.18it/s]
batch:  55%|█████▌    | 31/56 [00:53<00:07,  3.19it/s]
batch:  57%|█████▋    | 32/56 [00:54<00:07,  3.20it/s]
batch:  59%|█████▉    | 33/56 [00:54<00:07,  3.21it/s]
batch:  61%|██████    | 34/56 [00:54<00:06,  3.20it/s]
batch:  62%|██████▎   | 35/56 [00:55<00:06,  3.21it/s]
batch:  64%|██████▍   | 36/56 [00:55<00:06,  3.19it/s]
batch:  66%|██████▌   | 37/56 [00:55<00:05,  3.17it/s]
batch:  68%|██████▊   | 38/56 [00:56<00:05,  3.19it/s]
batch:  70%|██████▉   | 39/56 [00:56<00:05,  3.19it/s]
batch:  71%|███████▏  | 40/56 [00:56<00:05,  3.17it/s]
batch:  73%|███████▎  | 41/56 [00:57<00:04,  3.18it/s]
batch:  75%|███████▌  | 42/56 [00:57<00:04,  3.18it/s]
batch:  77%|███████▋  | 43/56 [00:57<00:04,  3.14it/s]
batch:  79%|███████▊  | 44/56 [00:58<00:03,  3.16it/s]
batch:  80%|████████  | 45/56 [00:58<00:03,  3.19it/s]
batch:  82%|████████▏ | 46/56 [00:58<00:03,  3.18it/s]
batch:  84%|████████▍ | 47/56 [00:58<00:02,  3.19it/s]
batch:  86%|████████▌ | 48/56 [00:59<00:02,  3.19it/s]
batch:  88%|████████▊ | 49/56 [00:59<00:02,  3.20it/s]
batch:  89%|████████▉ | 50/56 [00:59<00:01,  3.18it/s]
batch:  91%|█████████ | 51/56 [01:00<00:01,  3.16it/s]
batch:  93%|█████████▎| 52/56 [01:00<00:01,  3.16it/s]
batch:  95%|█████████▍| 53/56 [01:00<00:00,  3.18it/s]
batch:  96%|█████████▋| 54/56 [01:01<00:00,  3.17it/s]
batch:  98%|█████████▊| 55/56 [01:01<00:00,  3.15it/s]
batch: 100%|██████████| 56/56 [01:01<00:00,  3.37it/s]
batch: 100%|██████████| 56/56 [01:01<00:00,  1.10s/it]

  0%|          | 0/3568 [00:00<?, ?it/s]
  3%|▎         | 94/3568 [00:00<00:03, 936.72it/s]
  5%|▌         | 193/3568 [00:00<00:03, 958.21it/s]
  8%|▊         | 292/3568 [00:00<00:03, 971.77it/s]
 11%|█         | 390/3568 [00:00<00:03, 964.67it/s]
 14%|█▎        | 490/3568 [00:00<00:03, 976.48it/s]
 16%|█▋        | 588/3568 [00:00<00:03, 960.18it/s]
 19%|█▉        | 686/3568 [00:00<00:02, 962.55it/s]
 22%|██▏       | 783/3568 [00:00<00:02, 964.08it/s]
 25%|██▍       | 882/3568 [00:00<00:02, 965.56it/s]
 27%|██▋       | 979/3568 [00:01<00:02, 958.78it/s]
 30%|███       | 1075/3568 [00:01<00:02, 954.94it/s]
 33%|███▎      | 1171/3568 [00:01<00:02, 949.85it/s]
 35%|███▌      | 1266/3568 [00:01<00:02, 948.20it/s]
 38%|███▊      | 1361/3568 [00:01<00:02, 905.40it/s]
 41%|████      | 1455/3568 [00:01<00:02, 912.75it/s]
 43%|████▎     | 1550/3568 [00:01<00:02, 923.51it/s]
 46%|████▌     | 1646/3568 [00:01<00:02, 928.78it/s]
 49%|████▉     | 1741/3568 [00:01<00:01, 934.23it/s]
 51%|█████▏    | 1835/3568 [00:01<00:01, 930.29it/s]
 54%|█████▍    | 1929/3568 [00:02<00:01, 910.04it/s]
 57%|█████▋    | 2023/3568 [00:02<00:01, 918.23it/s]
 59%|█████▉    | 2115/3568 [00:02<00:01, 917.65it/s]
 62%|██████▏   | 2207/3568 [00:02<00:01, 917.69it/s]
 64%|██████▍   | 2299/3568 [00:02<00:01, 908.66it/s]
 67%|██████▋   | 2391/3568 [00:02<00:01, 911.45it/s]
 70%|██████▉   | 2483/3568 [00:02<00:01, 899.25it/s]
 72%|███████▏  | 2574/3568 [00:02<00:01, 901.22it/s]
 75%|███████▍  | 2667/3568 [00:02<00:00, 904.45it/s]
 77%|███████▋  | 2758/3568 [00:02<00:00, 899.68it/s]
 80%|███████▉  | 2848/3568 [00:03<00:00, 899.35it/s]
 82%|████████▏ | 2938/3568 [00:03<00:00, 878.04it/s]
 85%|████████▍ | 3028/3568 [00:03<00:00, 880.62it/s]
 87%|████████▋ | 3119/3568 [00:03<00:00, 884.32it/s]
 90%|████████▉ | 3210/3568 [00:03<00:00, 887.19it/s]
 92%|█████████▏| 3299/3568 [00:03<00:00, 853.40it/s]
 95%|█████████▍| 3387/3568 [00:03<00:00, 860.74it/s]
 97%|█████████▋| 3477/3568 [00:03<00:00, 866.40it/s]
100%|█████████▉| 3565/3568 [00:03<00:00, 869.57it/s]
100%|██████████| 3568/3568 [00:03<00:00, 914.82it/s]

  0%|          | 0/3568 [00:00<?, ?it/s]
  3%|▎         | 100/3568 [00:00<00:03, 978.64it/s]
  6%|▌         | 198/3568 [00:00<00:03, 969.23it/s]
  8%|▊         | 298/3568 [00:00<00:03, 982.23it/s]
 11%|█         | 397/3568 [00:00<00:03, 982.79it/s]
 14%|█▍        | 497/3568 [00:00<00:03, 968.28it/s]
 17%|█▋        | 596/3568 [00:00<00:03, 971.52it/s]
 19%|█▉        | 694/3568 [00:00<00:02, 972.84it/s]
 22%|██▏       | 792/3568 [00:00<00:02, 971.12it/s]
 25%|██▍       | 890/3568 [00:00<00:02, 969.22it/s]
 28%|██▊       | 987/3568 [00:01<00:02, 964.86it/s]
 30%|███       | 1084/3568 [00:01<00:02, 959.83it/s]
 33%|███▎      | 1180/3568 [00:01<00:02, 954.06it/s]
 36%|███▌      | 1276/3568 [00:01<00:02, 940.63it/s]
 38%|███▊      | 1371/3568 [00:01<00:02, 942.48it/s]
 41%|████      | 1466/3568 [00:01<00:02, 941.41it/s]
 44%|████▍     | 1562/3568 [00:01<00:02, 945.98it/s]
 46%|████▋     | 1657/3568 [00:01<00:02, 908.50it/s]
 49%|████▉     | 1751/3568 [00:01<00:01, 917.19it/s]
 52%|█████▏    | 1844/3568 [00:01<00:01, 917.67it/s]
 54%|█████▍    | 1938/3568 [00:02<00:01, 923.64it/s]
 57%|█████▋    | 2033/3568 [00:02<00:01, 926.31it/s]
 60%|█████▉    | 2127/3568 [00:02<00:01, 926.78it/s]
 62%|██████▏   | 2220/3568 [00:02<00:01, 923.61it/s]
 65%|██████▍   | 2313/3568 [00:02<00:01, 887.15it/s]
 67%|██████▋   | 2406/3568 [00:02<00:01, 894.01it/s]
 70%|███████   | 2500/3568 [00:02<00:01, 903.16it/s]
 73%|███████▎  | 2592/3568 [00:02<00:01, 907.26it/s]
 75%|███████▌  | 2683/3568 [00:02<00:00, 905.87it/s]
 78%|███████▊  | 2774/3568 [00:02<00:00, 906.21it/s]
 80%|████████  | 2865/3568 [00:03<00:00, 884.46it/s]
 83%|████████▎ | 2954/3568 [00:03<00:00, 885.47it/s]
 85%|████████▌ | 3045/3568 [00:03<00:00, 887.61it/s]
 88%|████████▊ | 3135/3568 [00:03<00:00, 887.60it/s]
 90%|█████████ | 3226/3568 [00:03<00:00, 889.13it/s]
 93%|█████████▎| 3316/3568 [00:03<00:00, 887.56it/s]
 95%|█████████▌| 3406/3568 [00:03<00:00, 889.03it/s]
 98%|█████████▊| 3495/3568 [00:03<00:00, 889.18it/s]
100%|██████████| 3568/3568 [00:03<00:00, 921.71it/s]
2025-12-29 21:03:00.477873: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:03:07.066990: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:03:07.068489: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-12-29 21:03:07.356168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:03:07.356308: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:03:07.360511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:03:07.360579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:03:07.362406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:03:07.363661: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:03:07.366851: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:03:07.368487: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:03:07.369673: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:03:07.374827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:03:07.375407: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-12-29 21:03:07.378600: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-12-29 21:03:07.380660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:c3:00.0 name: NVIDIA A100-SXM4-80GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 79.14GiB deviceMemoryBandwidth: 1.85TiB/s
2025-12-29 21:03:07.380705: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:03:07.380732: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:03:07.380754: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:03:07.380775: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-12-29 21:03:07.380795: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-12-29 21:03:07.380815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-12-29 21:03:07.380835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-12-29 21:03:07.380854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:03:07.384769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-12-29 21:03:07.384816: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-12-29 21:03:07.866357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-12-29 21:03:07.866474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-12-29 21:03:07.866490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-12-29 21:03:07.871489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 75642 MB memory) -> physical GPU (device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:c3:00.0, compute capability: 8.0)
2025-12-29 21:03:07.919033: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-12-29 21:03:07.931181: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2450135000 Hz
2025-12-29 21:03:11.838608: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-12-29 21:03:12.349447: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-12-29 21:03:12.357551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-12-29 21:03:14.017780: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-12-29 21:03:14.211704: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2025-12-29 21:03:49.040062: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)