Lmod Warning:
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: curl/8.4.0
(required by: htslib/1.16)
-------------------------------------------------------------------------------




The following have been reloaded with a version change:
  1) curl/8.4.0 => curl/8.17.0     2) openssl/3.0.7 => openssl/3.6.0

2026-05-11 02:49:28.791330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:49:51.839931: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 02:49:51.845339: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 02:49:51.865768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:82:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1
coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s
2026-05-11 02:49:51.865843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:49:52.462991: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 02:49:52.463232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 02:49:52.786956: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 02:49:53.417072: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 02:49:53.937571: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 02:49:54.099470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 02:49:54.326817: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 02:49:54.327817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 02:49:54.328559: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 02:49:54.328777: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 02:49:54.329308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:82:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1
coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s
2026-05-11 02:49:54.329388: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:49:54.329448: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 02:49:54.329499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 02:49:54.329549: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 02:49:54.329599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 02:49:54.329648: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 02:49:54.329697: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 02:49:54.329778: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 02:49:54.330530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 02:49:54.330613: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 02:49:56.937807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 02:49:57.857062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 02:49:57.857215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 02:49:57.860868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11209 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN Xp, pci bus id: 0000:82:00.0, compute capability: 6.1)
2026-05-11 02:50:06.974135: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 02:50:06.974679: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2399915000 Hz
2026-05-11 02:50:09.534157: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 02:50:11.686094: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 02:50:11.695469: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 02:50:25.344958: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 02:50:25.426310: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])
2026-05-11 02:57:15.848839: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2026-05-11 02:58:40.013234: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:05:04.829016: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:05:04.864086: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:05:04.931444: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:82:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1
coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s
2026-05-11 03:05:04.931562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:05:04.963550: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:05:04.963612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:05:04.979788: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:05:04.995003: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:05:05.013152: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:05:05.024144: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:05:05.034608: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:05:05.035158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:05:05.035502: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:05:05.035631: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:05:05.036279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:82:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1
coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s
2026-05-11 03:05:05.036350: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:05:05.036398: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:05:05.036440: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:05:05.036480: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:05:05.036520: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:05:05.036559: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:05:05.036599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:05:05.036639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:05:05.037283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:05:05.037356: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:05:05.594917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:05:05.595031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:05:05.595049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:05:05.595865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11209 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN Xp, pci bus id: 0000:82:00.0, compute capability: 6.1)
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)

batch:   0%|          | 0/30 [00:00<?, ?it/s]2026-05-11 03:05:07.922509: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2026-05-11 03:05:07.923108: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2399915000 Hz
2026-05-11 03:05:08.486153: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:05:09.001175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:05:09.019841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:05:10.804712: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:05:10.913283: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'true_profiles', 'true_logcounts', 'rev_comp'] which did not match any model input. They will be ignored by the model.
  [n for n in tensors.keys() if n not in ref_input_names])

batch:   3%|▎         | 1/30 [01:11<34:21, 71.07s/it]
batch:   7%|▋         | 2/30 [01:11<13:45, 29.50s/it]
batch:  10%|█         | 3/30 [01:11<07:18, 16.22s/it]
batch:  13%|█▎        | 4/30 [01:12<04:19,  9.97s/it]
batch:  17%|█▋        | 5/30 [01:12<02:43,  6.52s/it]
batch:  20%|██        | 6/30 [01:13<01:46,  4.44s/it]
batch:  23%|██▎       | 7/30 [01:13<01:12,  3.13s/it]
batch:  27%|██▋       | 8/30 [01:13<00:50,  2.28s/it]
batch:  30%|███       | 9/30 [01:14<00:35,  1.70s/it]
batch:  33%|███▎      | 10/30 [01:14<00:26,  1.31s/it]
batch:  37%|███▋      | 11/30 [01:15<00:19,  1.04s/it]
batch:  40%|████      | 12/30 [01:15<00:15,  1.18it/s]
batch:  43%|████▎     | 13/30 [01:16<00:12,  1.41it/s]
batch:  47%|████▋     | 14/30 [01:16<00:09,  1.64it/s]
batch:  50%|█████     | 15/30 [01:16<00:08,  1.84it/s]
batch:  53%|█████▎    | 16/30 [01:17<00:06,  2.02it/s]
batch:  57%|█████▋    | 17/30 [01:17<00:06,  2.11it/s]
batch:  60%|██████    | 18/30 [01:18<00:05,  2.18it/s]
batch:  63%|██████▎   | 19/30 [01:18<00:05,  2.19it/s]
batch:  67%|██████▋   | 20/30 [01:18<00:04,  2.29it/s]
batch:  70%|███████   | 21/30 [01:19<00:03,  2.30it/s]
batch:  73%|███████▎  | 22/30 [01:19<00:03,  2.30it/s]
batch:  77%|███████▋  | 23/30 [01:20<00:03,  2.32it/s]
batch:  80%|████████  | 24/30 [01:20<00:02,  2.40it/s]
batch:  83%|████████▎ | 25/30 [01:20<00:02,  2.44it/s]
batch:  87%|████████▋ | 26/30 [01:21<00:01,  2.48it/s]
batch:  90%|█████████ | 27/30 [01:21<00:01,  2.51it/s]
batch:  93%|█████████▎| 28/30 [01:22<00:00,  2.53it/s]
batch:  97%|█████████▋| 29/30 [01:22<00:00,  2.55it/s]
batch: 100%|██████████| 30/30 [01:22<00:00,  3.23it/s]
batch: 100%|██████████| 30/30 [01:22<00:00,  2.75s/it]

  0%|          | 0/1859 [00:00<?, ?it/s]
  3%|▎         | 59/1859 [00:00<00:03, 584.21it/s]
  8%|▊         | 158/1859 [00:00<00:02, 815.27it/s]
 14%|█▎        | 254/1859 [00:00<00:01, 880.20it/s]
 19%|█▉        | 354/1859 [00:00<00:01, 919.62it/s]
 24%|██▍       | 449/1859 [00:00<00:01, 929.93it/s]
 29%|██▉       | 545/1859 [00:00<00:01, 933.31it/s]
 34%|███▍      | 639/1859 [00:00<00:01, 931.28it/s]
 39%|███▉      | 734/1859 [00:00<00:01, 933.87it/s]
 45%|████▍     | 828/1859 [00:00<00:01, 934.67it/s]
 50%|████▉     | 922/1859 [00:01<00:01, 932.11it/s]
 55%|█████▍    | 1017/1859 [00:01<00:00, 931.43it/s]
 60%|█████▉    | 1111/1859 [00:01<00:00, 929.19it/s]
 65%|██████▍   | 1204/1859 [00:01<00:00, 925.14it/s]
 70%|██████▉   | 1297/1859 [00:01<00:00, 919.86it/s]
 75%|███████▍  | 1389/1859 [00:01<00:00, 911.09it/s]
 80%|███████▉  | 1481/1859 [00:01<00:00, 900.23it/s]
 85%|████████▍ | 1572/1859 [00:01<00:00, 897.07it/s]
 89%|████████▉ | 1662/1859 [00:01<00:00, 890.03it/s]
 94%|█████████▍| 1752/1859 [00:01<00:00, 876.71it/s]
 99%|█████████▉| 1840/1859 [00:02<00:00, 864.09it/s]
100%|██████████| 1859/1859 [00:02<00:00, 898.74it/s]

  0%|          | 0/1859 [00:00<?, ?it/s]
  5%|▍         | 85/1859 [00:00<00:02, 843.54it/s]
 10%|▉         | 183/1859 [00:00<00:01, 920.09it/s]
 15%|█▌        | 280/1859 [00:00<00:01, 942.30it/s]
 20%|██        | 379/1859 [00:00<00:01, 954.32it/s]
 26%|██▌       | 475/1859 [00:00<00:01, 953.48it/s]
 31%|███       | 571/1859 [00:00<00:01, 946.97it/s]
 36%|███▌      | 666/1859 [00:00<00:01, 942.74it/s]
 41%|████      | 761/1859 [00:00<00:01, 940.37it/s]
 46%|████▌     | 856/1859 [00:00<00:01, 936.36it/s]
 51%|█████     | 950/1859 [00:01<00:00, 933.07it/s]
 56%|█████▌    | 1044/1859 [00:01<00:00, 927.50it/s]
 61%|██████    | 1138/1859 [00:01<00:00, 926.95it/s]
 66%|██████▌   | 1231/1859 [00:01<00:00, 920.83it/s]
 71%|███████   | 1324/1859 [00:01<00:00, 916.98it/s]
 76%|███████▌  | 1416/1859 [00:01<00:00, 908.44it/s]
 81%|████████  | 1507/1859 [00:01<00:00, 896.83it/s]
 86%|████████▌ | 1597/1859 [00:01<00:00, 892.54it/s]
 91%|█████████ | 1687/1859 [00:01<00:00, 887.15it/s]
 96%|█████████▌| 1776/1859 [00:01<00:00, 872.39it/s]
100%|██████████| 1859/1859 [00:02<00:00, 910.83it/s]
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Matplotlib is building the font cache; this may take a moment.
2026-05-11 03:15:07.506202: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:38.030147: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:20:38.065570: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-05-11 03:20:38.140701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:82:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1
coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s
2026-05-11 03:20:38.140887: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:38.178021: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:20:38.178153: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:20:38.196711: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:20:38.213982: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:20:38.235458: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:20:38.253074: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:20:38.269800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:20:38.270603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:20:38.271128: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-05-11 03:20:38.271331: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2026-05-11 03:20:38.271781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:82:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1
coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s
2026-05-11 03:20:38.271862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:38.271945: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:20:38.271999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:20:38.272049: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-05-11 03:20:38.272100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-05-11 03:20:38.272150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2026-05-11 03:20:38.272199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-05-11 03:20:38.272249: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:20:38.272883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2026-05-11 03:20:38.272983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-05-11 03:20:38.875223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-05-11 03:20:38.875329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2026-05-11 03:20:38.875350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2026-05-11 03:20:38.876448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11209 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN Xp, pci bus id: 0000:82:00.0, compute capability: 6.1)
2026-05-11 03:20:38.980198: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2026-05-11 03:20:39.040550: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2399915000 Hz
2026-05-11 03:20:41.875083: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-05-11 03:20:42.155416: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2026-05-11 03:20:42.161131: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-05-11 03:20:43.490916: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256
2026-05-11 03:20:43.571330: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: 
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:1059: UserWarning: bpnet.model.arch is not loaded, but a Lambda layer uses it. It may cause errors.
  , UserWarning)