Lmod Warning: ------------------------------------------------------------------------------- The following dependent module(s) are not currently loaded: curl/8.4.0 (required by: htslib/1.16) ------------------------------------------------------------------------------- The following have been reloaded with a version change: 1) curl/8.4.0 => curl/8.17.0 2) openssl/3.0.7 => openssl/3.6.0 2026-05-11 03:47:10.102334: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2026-05-11 03:53:55.594763: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2026-05-11 03:53:55.597978: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2026-05-11 03:53:55.694593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:09:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1 coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s 2026-05-11 03:53:55.694750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2026-05-11 03:53:56.508679: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2026-05-11 03:53:56.508895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2026-05-11 03:53:56.873021: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2026-05-11 03:53:57.548533: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2026-05-11 03:53:58.247122: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2026-05-11 03:53:58.580603: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2026-05-11 03:53:58.956101: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2026-05-11 03:53:58.964700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2026-05-11 03:53:58.965449: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2026-05-11 03:53:58.965749: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2026-05-11 03:53:58.967176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:09:00.0 name: NVIDIA TITAN Xp computeCapability: 6.1 coreClock: 1.911GHz coreCount: 30 deviceMemorySize: 11.90GiB deviceMemoryBandwidth: 510.07GiB/s 2026-05-11 03:53:58.967285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2026-05-11 03:53:58.967356: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2026-05-11 03:53:58.967416: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2026-05-11 03:53:58.967464: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2026-05-11 03:53:58.967511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2026-05-11 03:53:58.967558: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2026-05-11 03:53:58.967605: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2026-05-11 03:53:58.967677: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2026-05-11 03:53:58.969776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2026-05-11 03:53:58.969876: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2026-05-11 03:54:01.569901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-05-11 03:54:01.570013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2026-05-11 03:54:01.570033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2026-05-11 03:54:01.571182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11209 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN Xp, pci bus id: 0000:09:00.0, compute capability: 6.1) 2026-05-11 03:54:13.825769: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2026-05-11 03:54:13.826448: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2400255000 Hz 2026-05-11 03:54:17.521658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2026-05-11 03:54:18.823108: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2026-05-11 03:54:18.830926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2026-05-11 03:54:54.801975: W tensorflow/stream_executor/gpu/asm_compiler.cc:63] Running ptxas --version returned 256 2026-05-11 03:54:54.879171: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: ptxas exited with non-zero error code 256, output: Relying on driver to perform ptx compilation. Modify $PATH to customize ptxas location. This message will be only logged once. /home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['coordinates', 'jitters', 'index', 'status', 'rev_comp'] which did not match any model input. They will be ignored by the model. [n for n in tensors.keys() if n not in ref_input_names]) Process Process-1: Traceback (most recent call last): File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/bpnet/train/training.py", line 408, in train_and_validate train_generator, epochs=1, steps_per_epoch=train_steps) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1105, in fit callbacks.on_train_batch_end(end_step, logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 454, in on_train_batch_end self._call_batch_hook(ModeKeys.TRAIN, 'end', batch, logs=logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 296, in _call_batch_hook self._call_batch_end_hook(mode, batch, logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 316, in _call_batch_end_hook self._call_batch_hook_helper(hook_name, batch, logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 356, in _call_batch_hook_helper hook(batch, logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 1020, in on_train_batch_end self._batch_update_progbar(batch, logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 1084, in _batch_update_progbar logs = tf_utils.to_numpy_or_python_type(logs) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 514, in to_numpy_or_python_type return nest.map_structure(_to_single_numpy_or_python_type, tensors) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 659, in map_structure structure[0], [func(*x) for x in entries], File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 659, in structure[0], [func(*x) for x in entries], File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 510, in _to_single_numpy_or_python_type x = t.numpy() File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1071, in numpy maybe_arr = self._numpy() # pylint: disable=protected-access File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1039, in _numpy six.raise_from(core._status_to_exception(e.code, e.message), None) # pylint: disable=protected-access File "", line 3, in raise_from tensorflow.python.framework.errors_impl.InternalError: Could not synchronize CUDA stream: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2026-05-11 03:57:22.213719: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 Traceback (most recent call last): File "/home/users/shouvikm/miniconda3/envs/bpnet/bin/bpnet-predict", line 8, in sys.exit(predict_main()) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/bpnet/cli/predict.py", line 918, in predict_main predict(args, pred_dir) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/bpnet/cli/predict.py", line 380, in predict model = load_model(args.model) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 211, in load_model loader_impl.parse_saved_model(filepath) File "/home/users/shouvikm/miniconda3/envs/bpnet/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 114, in parse_saved_model constants.SAVED_MODEL_FILENAME_PB)) OSError: SavedModel file does not exist at: models/AML5_KMT2A_N/fold_2/model_split000/{saved_model.pbtxt|saved_model.pb} RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd Matplotlib is building the font cache; this may take a moment. 2026-05-11 04:13:11.122826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 File models/AML5_KMT2A_N/fold_2/model_split000 does not exist