Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2025-11-17 16:46:03.461249: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:47:50.908119: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:47:50.920362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:47:50.958459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2025-11-17 16:47:50.958591: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:47:51.725463: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:47:51.725700: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:47:51.919204: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:47:52.497970: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:47:53.021342: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:47:53.172800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:47:53.379480: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:47:53.380847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:47:53.381523: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:47:53.384524: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:47:53.385173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2025-11-17 16:47:53.385235: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:47:53.385282: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:47:53.385315: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:47:53.385348: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:47:53.385380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:47:53.385412: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:47:53.385444: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:47:53.385501: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:47:53.386517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:47:53.386587: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:47:55.881865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:47:55.881965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:47:55.881980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:47:55.883214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10064 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5)
2025-11-17 16:48:08.619692: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 16:48:08.620540: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500115000 Hz
2025-11-17 16:48:10.103838: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:48:11.691070: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:48:11.699323: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:48:24.996827: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:48:25.059636: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2025-11-17 16:54:37.899800: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2025-11-17 16:54:47.652783: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:32.938082: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:55:32.939812: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:55:32.970927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2025-11-17 16:55:32.971027: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:32.978622: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:55:32.978673: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:55:32.983341: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:55:32.987026: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:55:32.993847: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:55:32.997768: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:55:33.000157: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:55:33.001214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:55:33.001622: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:55:33.004338: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:55:33.004866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2025-11-17 16:55:33.004919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:33.004947: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:55:33.004971: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:55:33.004995: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:55:33.005018: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:55:33.005041: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:55:33.005064: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:55:33.005087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:55:33.005911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:55:33.005962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:55:33.461329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:55:33.461424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:55:33.461437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:55:33.462691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10064 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/52 [00:00<?, ?it/s]2025-11-17 16:55:35.022938: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2025-11-17 16:55:35.023509: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500115000 Hz
2025-11-17 16:55:35.296588: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:55:35.663493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:55:35.665791: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:55:36.973201: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:55:37.041382: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   2%|▏         | 1/52 [00:22<18:42, 22.02s/it]
batch:   4%|▍         | 2/52 [00:22<07:42,  9.26s/it]
batch:   6%|▌         | 3/52 [00:22<04:13,  5.17s/it]
batch:   8%|▊         | 4/52 [00:22<02:36,  3.25s/it]
batch:  10%|▉         | 5/52 [00:23<01:43,  2.19s/it]
batch:  12%|█▏        | 6/52 [00:23<01:11,  1.55s/it]
batch:  13%|█▎        | 7/52 [00:23<00:51,  1.15s/it]
batch:  15%|█▌        | 8/52 [00:24<00:38,  1.13it/s]
batch:  17%|█▋        | 9/52 [00:24<00:30,  1.42it/s]
batch:  19%|█▉        | 10/52 [00:24<00:24,  1.71it/s]
batch:  21%|██        | 11/52 [00:25<00:20,  1.99it/s]
batch:  23%|██▎       | 12/52 [00:25<00:17,  2.25it/s]
batch:  25%|██▌       | 13/52 [00:25<00:15,  2.47it/s]
batch:  27%|██▋       | 14/52 [00:26<00:14,  2.66it/s]
batch:  29%|██▉       | 15/52 [00:26<00:13,  2.81it/s]
batch:  31%|███       | 16/52 [00:26<00:12,  2.93it/s]
batch:  33%|███▎      | 17/52 [00:27<00:11,  3.00it/s]
batch:  35%|███▍      | 18/52 [00:27<00:11,  3.06it/s]
batch:  37%|███▋      | 19/52 [00:27<00:10,  3.11it/s]
batch:  38%|███▊      | 20/52 [00:27<00:10,  3.14it/s]
batch:  40%|████      | 21/52 [00:28<00:09,  3.16it/s]
batch:  42%|████▏     | 22/52 [00:28<00:09,  3.15it/s]
batch:  44%|████▍     | 23/52 [00:28<00:09,  3.15it/s]
batch:  46%|████▌     | 24/52 [00:29<00:08,  3.18it/s]
batch:  48%|████▊     | 25/52 [00:29<00:08,  3.16it/s]
batch:  50%|█████     | 26/52 [00:29<00:08,  3.16it/s]
batch:  52%|█████▏    | 27/52 [00:30<00:07,  3.17it/s]
batch:  54%|█████▍    | 28/52 [00:30<00:07,  3.19it/s]
batch:  56%|█████▌    | 29/52 [00:30<00:07,  3.19it/s]
batch:  58%|█████▊    | 30/52 [00:31<00:06,  3.21it/s]
batch:  60%|█████▉    | 31/52 [00:31<00:06,  3.22it/s]
batch:  62%|██████▏   | 32/52 [00:31<00:06,  3.19it/s]
batch:  63%|██████▎   | 33/52 [00:32<00:05,  3.20it/s]
batch:  65%|██████▌   | 34/52 [00:32<00:05,  3.23it/s]
batch:  67%|██████▋   | 35/52 [00:32<00:05,  3.24it/s]
batch:  69%|██████▉   | 36/52 [00:32<00:04,  3.24it/s]
batch:  71%|███████   | 37/52 [00:33<00:04,  3.23it/s]
batch:  73%|███████▎  | 38/52 [00:33<00:04,  3.24it/s]
batch:  75%|███████▌  | 39/52 [00:33<00:03,  3.25it/s]
batch:  77%|███████▋  | 40/52 [00:34<00:03,  3.25it/s]
batch:  79%|███████▉  | 41/52 [00:34<00:03,  3.25it/s]
batch:  81%|████████  | 42/52 [00:34<00:03,  3.24it/s]
batch:  83%|████████▎ | 43/52 [00:35<00:02,  3.23it/s]
batch:  85%|████████▍ | 44/52 [00:35<00:02,  3.21it/s]
batch:  87%|████████▋ | 45/52 [00:35<00:02,  3.20it/s]
batch:  88%|████████▊ | 46/52 [00:36<00:01,  3.17it/s]
batch:  90%|█████████ | 47/52 [00:36<00:01,  3.18it/s]
batch:  92%|█████████▏| 48/52 [00:36<00:01,  3.19it/s]
batch:  94%|█████████▍| 49/52 [00:37<00:00,  3.19it/s]
batch:  96%|█████████▌| 50/52 [00:37<00:00,  3.65it/s]
batch:  98%|█████████▊| 51/52 [00:37<00:00,  3.51it/s]
batch: 100%|██████████| 52/52 [00:37<00:00,  1.38it/s]

  0%|          | 0/3229 [00:00<?, ?it/s]
  3%|▎         | 96/3229 [00:00<00:03, 950.24it/s]
  6%|▌         | 194/3229 [00:00<00:03, 959.68it/s]
  9%|▉         | 294/3229 [00:00<00:03, 977.29it/s]
 12%|█▏        | 393/3229 [00:00<00:02, 981.56it/s]
 15%|█▌        | 492/3229 [00:00<00:02, 976.70it/s]
 18%|█▊        | 590/3229 [00:00<00:02, 973.66it/s]
 21%|██▏       | 688/3229 [00:00<00:02, 971.56it/s]
 24%|██▍       | 786/3229 [00:00<00:02, 967.40it/s]
 27%|██▋       | 883/3229 [00:00<00:02, 960.75it/s]
 30%|███       | 980/3229 [00:01<00:02, 960.48it/s]
 33%|███▎      | 1077/3229 [00:01<00:02, 954.38it/s]
 36%|███▋      | 1173/3229 [00:01<00:02, 953.24it/s]
 39%|███▉      | 1269/3229 [00:01<00:02, 947.25it/s]
 42%|████▏     | 1364/3229 [00:01<00:01, 941.07it/s]
 45%|████▌     | 1459/3229 [00:01<00:01, 939.53it/s]
 48%|████▊     | 1553/3229 [00:01<00:01, 923.37it/s]
 51%|█████     | 1646/3229 [00:01<00:01, 923.96it/s]
 54%|█████▍    | 1739/3229 [00:01<00:01, 923.97it/s]
 57%|█████▋    | 1832/3229 [00:01<00:01, 905.86it/s]
 60%|█████▉    | 1924/3229 [00:02<00:01, 906.35it/s]
 62%|██████▏   | 2018/3229 [00:02<00:01, 915.21it/s]
 65%|██████▌   | 2110/3229 [00:02<00:01, 911.70it/s]
 68%|██████▊   | 2202/3229 [00:02<00:01, 902.62it/s]
 71%|███████   | 2294/3229 [00:02<00:01, 904.58it/s]
 74%|███████▍  | 2386/3229 [00:02<00:00, 903.99it/s]
 77%|███████▋  | 2477/3229 [00:02<00:00, 900.75it/s]
 80%|███████▉  | 2568/3229 [00:02<00:00, 896.73it/s]
 82%|████████▏ | 2658/3229 [00:02<00:00, 886.11it/s]
 85%|████████▌ | 2747/3229 [00:02<00:00, 881.92it/s]
 88%|████████▊ | 2836/3229 [00:03<00:00, 879.45it/s]
 91%|█████████ | 2924/3229 [00:03<00:00, 876.84it/s]
 93%|█████████▎| 3012/3229 [00:03<00:00, 869.48it/s]
 96%|█████████▌| 3099/3229 [00:03<00:00, 859.75it/s]
 99%|█████████▊| 3185/3229 [00:03<00:00, 857.97it/s]
100%|██████████| 3229/3229 [00:03<00:00, 916.44it/s]

  0%|          | 0/3229 [00:00<?, ?it/s]
  3%|▎         | 96/3229 [00:00<00:03, 949.67it/s]
  6%|▌         | 197/3229 [00:00<00:03, 975.15it/s]
  9%|▉         | 297/3229 [00:00<00:02, 985.90it/s]
 12%|█▏        | 397/3229 [00:00<00:02, 985.68it/s]
 15%|█▌        | 496/3229 [00:00<00:02, 983.41it/s]
 18%|█▊        | 595/3229 [00:00<00:02, 972.10it/s]
 21%|██▏       | 694/3229 [00:00<00:02, 973.83it/s]
 25%|██▍       | 793/3229 [00:00<00:02, 975.30it/s]
 28%|██▊       | 891/3229 [00:00<00:02, 974.36it/s]
 31%|███       | 989/3229 [00:01<00:02, 951.17it/s]
 34%|███▎      | 1085/3229 [00:01<00:02, 952.39it/s]
 37%|███▋      | 1181/3229 [00:01<00:02, 952.35it/s]
 40%|███▉      | 1277/3229 [00:01<00:02, 949.61it/s]
 42%|████▏     | 1372/3229 [00:01<00:01, 947.21it/s]
 45%|████▌     | 1467/3229 [00:01<00:01, 931.73it/s]
 48%|████▊     | 1561/3229 [00:01<00:01, 930.40it/s]
 51%|█████▏    | 1655/3229 [00:01<00:01, 932.25it/s]
 54%|█████▍    | 1749/3229 [00:01<00:01, 928.57it/s]
 57%|█████▋    | 1844/3229 [00:01<00:01, 934.33it/s]
 60%|██████    | 1938/3229 [00:02<00:01, 926.29it/s]
 63%|██████▎   | 2031/3229 [00:02<00:01, 923.86it/s]
 66%|██████▌   | 2124/3229 [00:02<00:01, 920.42it/s]
 69%|██████▊   | 2217/3229 [00:02<00:01, 919.44it/s]
 72%|███████▏  | 2309/3229 [00:02<00:01, 916.42it/s]
 74%|███████▍  | 2401/3229 [00:02<00:00, 914.26it/s]
 77%|███████▋  | 2493/3229 [00:02<00:00, 905.13it/s]
 80%|████████  | 2584/3229 [00:02<00:00, 898.77it/s]
 83%|████████▎ | 2674/3229 [00:02<00:00, 893.40it/s]
 86%|████████▌ | 2764/3229 [00:02<00:00, 889.08it/s]
 88%|████████▊ | 2853/3229 [00:03<00:00, 877.76it/s]
 91%|█████████ | 2941/3229 [00:03<00:00, 877.47it/s]
 94%|█████████▍| 3029/3229 [00:03<00:00, 875.24it/s]
 97%|█████████▋| 3117/3229 [00:03<00:00, 873.78it/s]
 99%|█████████▉| 3205/3229 [00:03<00:00, 870.64it/s]
100%|██████████| 3229/3229 [00:03<00:00, 923.82it/s]
2025-11-17 16:56:35.655017: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:56:46.777157: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:56:46.781568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2025-11-17 16:56:46.811731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2025-11-17 16:56:46.811844: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:56:46.816414: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:56:46.816471: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:56:46.818552: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:56:46.820040: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:56:46.823994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:56:46.826130: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:56:46.827661: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:56:46.828662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:56:46.829163: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-17 16:56:46.832038: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2025-11-17 16:56:46.832870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2025-11-17 16:56:46.832936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:56:46.832972: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:56:46.833024: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:56:46.833058: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2025-11-17 16:56:46.833089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2025-11-17 16:56:46.833120: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2025-11-17 16:56:46.833150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2025-11-17 16:56:46.833181: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:56:46.834032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2025-11-17 16:56:46.834096: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2025-11-17 16:56:47.282268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-11-17 16:56:47.282362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2025-11-17 16:56:47.282375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2025-11-17 16:56:47.283680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10064 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5)
2025-11-17 16:56:47.332735: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2025-11-17 16:56:47.349231: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2500115000 Hz
2025-11-17 16:56:50.630219: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2025-11-17 16:56:50.985609: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2025-11-17 16:56:50.990760: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2025-11-17 16:56:52.249267: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2025-11-17 16:56:52.314370: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)