.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "user-guide/auto_tutorials/train_ctlearn_model.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_user-guide_auto_tutorials_train_ctlearn_model.py: **Tutorial 2** Train CTLearn models =================================== Train models for CTAO image and waveform data with CTLearn. The image examples use only the gamma-ray and proton ``00`` files; the ``01`` files remain untouched for future evaluation tutorials. The tutorial is intenionally kept simple and uses only a small fraction of the available statistics to keep the training time manageable. The resulting performance metrics are not representative of a production-level analysis nor should they be compared directly with published results. The main goal is to illustrate the analysis workflow and to become familiar with the command-line tools and the DL2 data format. This tutorial can be modified from the users to use larger datasets and more statistics to obtain meaningful results. Each section explains a training task followed immediately by its literal command-line example. Running this script executes every example in order. .. GENERATED FROM PYTHON SOURCE LINES 21-27 Inspect the training environment and cached input data ------------------------------------------------------ Run ``explore_public_data.py`` first to download and convert the shared tutorial data. The converted directory is keyed by the ctapipe version and the optional event limit, matching the cache layout used by that tutorial. .. GENERATED FROM PYTHON SOURCE LINES 27-79 .. code-block:: Python import importlib.metadata import os import subprocess from pathlib import Path import matplotlib.pyplot as plt import pandas as pd from ctapipe import __version__ as ctapipe_version MAX_EVENTS = os.environ.get("CTLEARN_TUTORIAL_MAX_EVENTS") if MAX_EVENTS is not None and int(MAX_EVENTS) < 1: raise ValueError("CTLEARN_TUTORIAL_MAX_EVENTS must be a positive integer") MAX_EVENTS_SUFFIX = f"-max{MAX_EVENTS}" if MAX_EVENTS is not None else "" TESTDATA_DIR = ( Path( os.environ.get( "CTLEARN_TUTORIAL_DATA_DIR", str(Path.home() / ".cache" / "ctlearn" / "tutorial-data"), ) ) .expanduser() .resolve() ) CONVERTED_DIR = TESTDATA_DIR / f"ctapipe-{ctapipe_version}{MAX_EVENTS_SUFFIX}" GAMMA_FILE = CONVERTED_DIR / "gamma-diffuse_with_images_00.dl2.h5" PROTON_FILE = CONVERTED_DIR / "proton_with_images_00.dl2.h5" WAVEFORM_FILE = TESTDATA_DIR / f"gamma_prod5{MAX_EVENTS_SUFFIX}.r1.dl1.h5" print("Training environment:") for distribution in ("ctapipe", "ctlearn", "dl1-data-handler", "keras", "tensorflow"): try: version = importlib.metadata.version(distribution) except importlib.metadata.PackageNotFoundError: version = "not installed" print(f" {distribution}: {version}") missing_files = [path for path in (GAMMA_FILE, PROTON_FILE) if not path.is_file()] if missing_files: missing = "\n".join(f" - {path}" for path in missing_files) raise FileNotFoundError( "Cached stereo training data is missing. Run " "explore_public_data.py first. Missing files:\n" f"{missing}" ) print("\nCached input files:") for path in (GAMMA_FILE, PROTON_FILE): print(f" {path.name}: {path.stat().st_size / 1024**2:.1f} MiB") print("Only the _00 image files are selected; the _01 files are reserved.") .. rst-class:: sphx-glr-script-out .. code-block:: none Training environment: ctapipe: 0.33.0 ctlearn: 0.11.2.dev2+g312165f28 dl1-data-handler: 0.14.13 keras: 3.15.1 tensorflow: 2.21.0 Cached input files: gamma-diffuse_with_images_00.dl2.h5: 12.3 MiB proton_with_images_00.dl2.h5: 12.4 MiB Only the _00 image files are selected; the _01 files are reserved. .. GENERATED FROM PYTHON SOURCE LINES 80-103 Training commands, configs, and outputs --------------------------------------- CTLearn provides ``ctlearn-train-keras-model`` and ``ctlearn-train-pytorch-model`` command-line tools. Use ``--help`` or ``--help-all`` to inspect their options. Each example has a Keras command followed immediately by the equivalent PyTorch command. In a notebook, prefix either command with ``!``. The three JSON configuration files are in ``docs/source/config`` in the repository and in ``config`` in the downloaded tutorial ZIPs: - ``dl1dh_example_config.json`` selects LST/LSTCam images, the image and relative peak-time channels, bilinear mapping, and an example 50,000 p.e. intensity cut. It also specifies NectarCam waveform settings. - ``ctlearn_train_model_example_config.json`` sets the reader, epoch and batch defaults, validation split, Adam optimizer and learning rate, checkpointing, learning-rate reduction, random seed, and early stopping. - ``ctlearn_models_example_config.json`` defines the prediction heads, attention settings, custom ``SingleCNN`` and ``ResNet`` architectures, and ``LoadedModel`` transfer-learning options. .. GENERATED FROM PYTHON SOURCE LINES 103-200 .. code-block:: Python def find_config_dir(): roots = [] script_path = globals().get("__file__") if script_path is not None: script_dir = Path(script_path).resolve().parent roots.extend((script_dir, *script_dir.parents)) roots.extend((Path.cwd(), *Path.cwd().parents)) config_names = ( "dl1dh_example_config.json", "ctlearn_train_model_example_config.json", "ctlearn_models_example_config.json", ) for root in roots: for config_dir in (root / "config", root / "docs" / "source" / "config"): if all((config_dir / name).is_file() for name in config_names): return config_dir raise FileNotFoundError( "Could not locate tutorial configs in config/ or docs/source/config/. " "Extract the complete tutorial ZIP or run from the CTLearn repository." ) CONFIG_DIR = find_config_dir() DL1DH_CONFIG = CONFIG_DIR / "dl1dh_example_config.json" TRAINING_CONFIG = CONFIG_DIR / "ctlearn_train_model_example_config.json" MODEL_CONFIG = CONFIG_DIR / "ctlearn_models_example_config.json" for config_path in (DL1DH_CONFIG, TRAINING_CONFIG, MODEL_CONFIG): if not config_path.is_file(): raise FileNotFoundError(f"Tutorial configuration is missing: {config_path}") OUTPUT_DIR = ( Path( os.environ.get( "CTLEARN_TUTORIAL_OUTPUT_DIR", str(Path.home() / ".cache" / "ctlearn" / "tutorial-training"), ) ) .expanduser() .resolve() ) TRAINING_OUTPUTS = ( "keras-energy", "pytorch-energy", "keras-cameradirection", "pytorch-cameradirection", "keras-skydirection", "pytorch-skydirection", "keras-classification", "pytorch-classification", "keras-singlecnn-energy", "pytorch-singlecnn-energy", "keras-resnet-energy", "pytorch-resnet-energy", "keras-transfer-learning-energy", "pytorch-transfer-learning-energy", "keras-stereo-energy", "pytorch-stereo-energy", "keras-waveform-energy", "pytorch-waveform-energy", ) existing_outputs = [ OUTPUT_DIR / name for name in TRAINING_OUTPUTS if (OUTPUT_DIR / name).exists() ] if existing_outputs: existing = "\n".join(f" - {path}" for path in existing_outputs) raise FileExistsError( "Training output directories already exist. Set " "CTLEARN_TUTORIAL_OUTPUT_DIR to a new directory before running all " f"examples:\n{existing}" ) os.environ["CTLEARN_TUTORIAL_DATA_DIR"] = str(TESTDATA_DIR) os.environ["CTLEARN_TUTORIAL_CONVERTED_DIR"] = str(CONVERTED_DIR) os.environ["CTLEARN_TUTORIAL_CONFIG_DIR"] = str(CONFIG_DIR.resolve()) os.environ["CTLEARN_TUTORIAL_OUTPUT_DIR"] = str(OUTPUT_DIR) os.environ["CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX"] = MAX_EVENTS_SUFFIX print(f"\nTraining outputs: {OUTPUT_DIR}") def run_command_line(command): print(f"Running command: $ {command.strip()}") result = subprocess.run( command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, ) print(result.stdout, end="") result.check_returncode() .. rst-class:: sphx-glr-script-out .. code-block:: none Training outputs: /home/runner/.cache/ctlearn/tutorial-training .. GENERATED FROM PYTHON SOURCE LINES 201-208 *Part A2* Monoscopic CNN-based model training ============================================= We first train a basic monoscopic CNN-based model using the default CTLearn model configuration. The model uses a **ThinResNet** backbone for feature extraction and a standard MLP head for the task-specific prediction. We consider three different reconstruction tasks, each trained separately: .. GENERATED FROM PYTHON SOURCE LINES 210-221 1. Primary particle energy regression ------------------------------------- The model reconstructs the energy of the primary particle as a continuous value. Internally, CTLearn performs the regression on the energy in TeV and on a logarithmic scale, since the primary-particle energy spans a very wide range. Working in logarithmic space has been found to be more efficient for the regression and is therefore a fixed, non-configurable aspect of the energy reconstruction. During inference, the prediction tool converts the predicted values back from logarithmic space to the original linear energy scale in TeV. .. GENERATED FROM PYTHON SOURCE LINES 221-248 .. code-block:: Python run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 4""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188599.092084 2414 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188599.129615 2414 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188599.982559 2414 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:30:02,520 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:30:02,520 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:30:02,520 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:30:02,521 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:30:02,521 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:30:02,522 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:30:02,522 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:30:02,523 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:30:02,523 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:30:03,410 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66 E0000 00:00:1795188603.410940 2414 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:30:03,428 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:30:03,448 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:30:03,448 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:30:03,976 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:30:03,979 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795188604.024311 2414 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795188604.043836 2414 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:30:17,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 0.5534 - mae_energy: 0.5534 - val_loss: 0.3763 - val_mae_energy: 0.3763 - time: 12.54s 2026-11-20 15:30:17,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.37629, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras 2026-11-20 15:30:17,627 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras 2026-11-20 15:30:26,714 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 0.5462 - mae_energy: 0.5462 - val_loss: 0.3572 - val_mae_energy: 0.3572 - time: 8.34s 2026-11-20 15:30:26,719 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.37629 to 0.35724, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras 2026-11-20 15:30:27,113 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras W0000 00:00:1795188635.250682 2414 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:30:35,552 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 0.5259 - mae_energy: 0.5259 - val_loss: 0.3542 - val_mae_energy: 0.3542 - time: 7.77s 2026-11-20 15:30:35,552 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss improved from 0.35724 to 0.35419, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras 2026-11-20 15:30:36,147 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras W0000 00:00:1795188644.327792 2835 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:30:44,564 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 0.5251 - mae_energy: 0.5251 - val_loss: 0.3075 - val_mae_energy: 0.3075 - time: 7.74s 2026-11-20 15:30:44,565 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss improved from 0.35419 to 0.30746, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras 2026-11-20 15:30:44,950 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras W0000 00:00:1795188653.026210 2933 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:30:53,370 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 0.4961 - mae_energy: 0.4961 - val_loss: 0.4088 - val_mae_energy: 0.4088 - time: 7.75s 2026-11-20 15:30:53,371 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss did not improve from 0.30746 2026-11-20 15:30:54,048 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:30:54,048 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/config.json 2026-11-20 15:30:54,049 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:30:54,050 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:30:54,050 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:30:54,051 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:30:54,052 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:30:54,052 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:30:54,052 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188658.182740 2976 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188658.221012 2976 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188659.272269 2976 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:31:00,618 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:31:00,618 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:31:00,618 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:31:00,619 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:31:00,619 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:31:00,620 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:31:00,620 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:31:00,621 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:31:00,621 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:31:01,471 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66 2026-11-20 15:31:01,471 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:31:01,496 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:31:01,496 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:31:01,521 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:31:15,203 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 0.5453 - mae_energy: 0.5453 - val_loss: 0.3254 - val_mae_energy: 0.3254 - time: 13.68s 2026-11-20 15:31:15,204 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32539, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth 2026-11-20 15:31:15,252 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth 2026-11-20 15:31:28,165 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 0.5495 - mae_energy: 0.5495 - val_loss: 0.3468 - val_mae_energy: 0.3468 - time: 12.91s 2026-11-20 15:31:28,166 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.32539 2026-11-20 15:31:40,665 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 0.5190 - mae_energy: 0.5190 - val_loss: 0.3251 - val_mae_energy: 0.3251 - time: 12.50s 2026-11-20 15:31:40,666 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss improved from 0.32539 to 0.32513, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth 2026-11-20 15:31:40,728 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth 2026-11-20 15:31:53,569 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 0.5059 - mae_energy: 0.5059 - val_loss: 0.3342 - val_mae_energy: 0.3342 - time: 12.84s 2026-11-20 15:31:53,570 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss did not improve from 0.32513 2026-11-20 15:32:06,759 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 0.5213 - mae_energy: 0.5213 - val_loss: 0.3362 - val_mae_energy: 0.3362 - time: 13.19s 2026-11-20 15:32:06,760 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss did not improve from 0.32513 2026-11-20 15:32:06,760 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:32:06,760 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/config.json 2026-11-20 15:32:06,761 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:32:06,762 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:32:06,762 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:32:06,763 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:32:06,763 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:32:06,764 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:32:06,764 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 249-262 2. Arrival direction regression ------------------------------- The model reconstructs the arrival direction of the primary particle. This can be done in two ways: either using the **cameradirection** task, which transforms the particle origin into the camera frame and performs the regression in camera coordinates, with distances expressed in meters; or using the **skydirection** task, which performs the regression in the sky frame, with field-of-view offsets expressed in degrees. For monoscopic CNN-based models, we generally recommend using the **cameradirection** task. During inference, the prediction tool converts the reconstructed camera coordinates back to the original Alt/Az representation, with the final direction expressed in degrees. .. GENERATED FROM PYTHON SOURCE LINES 262-317 .. code-block:: Python # Camera coordinates run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-cameradirection" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco cameradirection \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-cameradirection" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco cameradirection \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 4""", ) # Sky coordinates run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-skydirection" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco skydirection \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-skydirection" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco skydirection \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 4""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-cameradirection" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco cameradirection \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188728.472466 3010 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188728.511066 3010 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188729.380341 3010 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:32:11,740 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:32:11,740 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:32:11,740 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:32:11,741 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:32:11,741 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:32:12,635 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66 E0000 00:00:1795188732.636064 3010 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:32:12,654 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:32:12,675 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['cameradirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:32:12,676 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:32:13,209 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:32:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795188733.257518 3010 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795188733.277229 3010 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:32:26,536 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 0.9475 - mae_cameradirection: 0.9475 - val_loss: 0.8524 - val_mae_cameradirection: 0.8524 - time: 12.63s 2026-11-20 15:32:26,536 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.85238, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras 2026-11-20 15:32:26,943 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras 2026-11-20 15:32:35,495 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 0.9315 - mae_cameradirection: 0.9315 - val_loss: 0.5532 - val_mae_cameradirection: 0.5532 - time: 7.80s 2026-11-20 15:32:35,499 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.85238 to 0.55317, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras 2026-11-20 15:32:35,899 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras W0000 00:00:1795188765.569988 3010 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:32:45,875 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 0.8603 - mae_cameradirection: 0.8603 - val_loss: 0.7401 - val_mae_cameradirection: 0.7401 - time: 9.29s 2026-11-20 15:32:45,875 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss did not improve from 0.55317 2026-11-20 15:32:54,407 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 0.8207 - mae_cameradirection: 0.8207 - val_loss: 0.5144 - val_mae_cameradirection: 0.5144 - time: 7.84s 2026-11-20 15:32:54,407 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss improved from 0.55317 to 0.51438, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras 2026-11-20 15:32:55,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras 2026-11-20 15:33:04,524 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 0.8414 - mae_cameradirection: 0.8414 - val_loss: 0.6887 - val_mae_cameradirection: 0.6887 - time: 8.82s 2026-11-20 15:33:04,525 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss did not improve from 0.51438 2026-11-20 15:33:05,225 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:33:05,225 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/config.json 2026-11-20 15:33:05,226 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:33:05,228 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:33:05,228 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:33:05,229 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:33:05,229 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:33:05,229 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:33:05,229 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-cameradirection" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco cameradirection \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188789.297220 3567 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188789.334557 3567 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188790.339459 3567 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:33:11,654 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:33:11,654 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:33:11,654 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:33:11,655 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:33:11,655 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:33:11,656 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:33:11,656 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:33:11,657 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:33:11,657 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:33:12,516 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66 2026-11-20 15:33:12,516 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:33:12,542 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['cameradirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:33:12,542 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:33:12,568 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:33:27,266 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 0.9342 - mae_cameradirection: 0.9342 - val_loss: 0.7929 - val_mae_cameradirection: 0.7929 - time: 14.70s 2026-11-20 15:33:27,267 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.79288, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth 2026-11-20 15:33:27,312 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth 2026-11-20 15:33:41,992 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 0.8731 - mae_cameradirection: 0.8731 - val_loss: 0.7294 - val_mae_cameradirection: 0.7294 - time: 14.68s 2026-11-20 15:33:41,993 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.79288 to 0.72937, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth 2026-11-20 15:33:42,051 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth 2026-11-20 15:33:56,401 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 0.8694 - mae_cameradirection: 0.8694 - val_loss: 0.7512 - val_mae_cameradirection: 0.7512 - time: 14.35s 2026-11-20 15:33:56,402 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss did not improve from 0.72937 2026-11-20 15:34:11,135 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 0.8294 - mae_cameradirection: 0.8294 - val_loss: 0.7304 - val_mae_cameradirection: 0.7304 - time: 14.73s 2026-11-20 15:34:11,136 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss did not improve from 0.72937 2026-11-20 15:34:25,791 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 0.8394 - mae_cameradirection: 0.8394 - val_loss: 0.7613 - val_mae_cameradirection: 0.7613 - time: 14.65s 2026-11-20 15:34:25,791 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss did not improve from 0.72937 2026-11-20 15:34:25,791 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:34:25,792 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/config.json 2026-11-20 15:34:25,792 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:34:25,793 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:34:25,793 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:34:25,794 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:34:25,794 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:34:25,795 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:34:25,795 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-skydirection" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco skydirection \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188867.354839 3590 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188867.392988 3590 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188868.242662 3590 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:34:30,574 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:34:30,575 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:34:30,575 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:34:30,576 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:34:30,576 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:34:31,468 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66 E0000 00:00:1795188871.469037 3590 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:34:31,487 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:34:31,507 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-skydirection'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['skydirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:34:31,507 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:34:32,025 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:34:32,028 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795188872.073649 3590 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795188872.092347 3590 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:34:45,328 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 1.8731 - mae_skydirection: 1.8731 - val_loss: 1.3860 - val_mae_skydirection: 1.3860 - time: 12.60s 2026-11-20 15:34:45,328 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 1.38603, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras 2026-11-20 15:34:45,729 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras 2026-11-20 15:34:54,155 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 1.7275 - mae_skydirection: 1.7275 - val_loss: 1.1071 - val_mae_skydirection: 1.1071 - time: 7.68s 2026-11-20 15:34:54,159 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 1.38603 to 1.10714, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras 2026-11-20 15:34:54,560 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras W0000 00:00:1795188902.662085 3590 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:35:02,955 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 1.7188 - mae_skydirection: 1.7188 - val_loss: 1.3278 - val_mae_skydirection: 1.3278 - time: 7.72s 2026-11-20 15:35:02,955 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss did not improve from 1.10714 2026-11-20 15:35:11,306 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 1.6788 - mae_skydirection: 1.6788 - val_loss: 1.2819 - val_mae_skydirection: 1.2819 - time: 7.68s 2026-11-20 15:35:11,307 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss did not improve from 1.10714 2026-11-20 15:35:19,709 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 1.7069 - mae_skydirection: 1.7069 - val_loss: 1.3523 - val_mae_skydirection: 1.3523 - time: 7.73s 2026-11-20 15:35:19,709 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss did not improve from 1.10714 2026-11-20 15:35:20,369 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:35:20,369 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/config.json 2026-11-20 15:35:20,370 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:35:20,371 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:35:20,371 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:35:20,372 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:35:20,372 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:35:20,373 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:35:20,373 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-skydirection" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco skydirection \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188924.478621 4145 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188924.516704 4145 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188925.568311 4145 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:35:26,910 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:35:26,910 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:35:26,910 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:35:26,911 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:35:26,911 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:35:26,912 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:35:26,912 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:35:26,913 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:35:26,913 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:35:27,774 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66 2026-11-20 15:35:27,774 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:35:27,800 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['skydirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:35:27,800 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:35:27,825 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:35:40,583 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 1.7831 - mae_skydirection: 1.7831 - val_loss: 1.5204 - val_mae_skydirection: 1.5204 - time: 12.76s 2026-11-20 15:35:40,585 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 1.52043, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth 2026-11-20 15:35:40,631 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth 2026-11-20 15:35:53,407 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 1.7512 - mae_skydirection: 1.7512 - val_loss: 1.5101 - val_mae_skydirection: 1.5101 - time: 12.78s 2026-11-20 15:35:53,408 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 1.52043 to 1.51010, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth 2026-11-20 15:35:53,477 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth 2026-11-20 15:36:06,341 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 1.7094 - mae_skydirection: 1.7094 - val_loss: 1.5536 - val_mae_skydirection: 1.5536 - time: 12.86s 2026-11-20 15:36:06,341 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss did not improve from 1.51010 2026-11-20 15:36:19,772 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 1.7316 - mae_skydirection: 1.7316 - val_loss: 1.4704 - val_mae_skydirection: 1.4704 - time: 13.43s 2026-11-20 15:36:19,773 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss improved from 1.51010 to 1.47045, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth 2026-11-20 15:36:19,834 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth 2026-11-20 15:36:32,237 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 1.7003 - mae_skydirection: 1.7003 - val_loss: 1.4875 - val_mae_skydirection: 1.4875 - time: 12.40s 2026-11-20 15:36:32,238 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss did not improve from 1.47045 2026-11-20 15:36:32,239 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:36:32,239 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/config.json 2026-11-20 15:36:32,239 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:36:32,241 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:36:32,241 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:36:32,242 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:36:32,242 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:36:32,242 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:36:32,242 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 318-347 3. Primary particle type classification --------------------------------------- The model performs binary classification of the primary particle type, learning to distinguish between the two particle classes: signal (gamma-ray events) and background (proton events), based on the information contained in the camera image. The input directory containing the signal events can be specified using ``--TrainCTLearnModel.input_dir_signal`` or the shorthand ``--signal``. One or more file patterns can be provided using ``--TrainCTLearnModel.file_pattern_signal`` or ``--pattern-signal``. The corresponding options for the background events are ``--TrainCTLearnModel.input_dir_background`` / ``--background`` and ``--TrainCTLearnModel.file_pattern_background`` / ``--pattern-background``. It is recommended to approximately balance the training dataset before training. The training log reports the number of signal and background events remaining after the selection cuts, which can be used to check the class balance. If a significant imbalance is present, the input file patterns should be adjusted to include a more balanced selection of gamma-ray and proton events. CTLearn includes a standard, fixed mechanism that computes class weights and incorporates them into the loss calculation to compensate for differences in the number of signal and background events. However, class weighting is only effective up to a certain degree of class imbalance. It is therefore important to monitor the training carefully, particularly when the input samples are not well balanced. The training logs and learning curves should be inspected throughout the training. In particular, the evolution of metrics such as loss, accuracy, and AUC on a per-epoch basis can help identify potential issues arising from class imbalance or other training problems. .. GENERATED FROM PYTHON SOURCE LINES 347-378 .. code-block:: Python run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \\ --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-background "proton_with_images_00.dl2.h*" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-classification" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco type \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 3 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \\ --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-background "proton_with_images_00.dl2.h*" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-classification" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco type \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.n_epochs 3 \\ --TrainCTLearnModel.batch_size 4""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \ --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-background "proton_with_images_00.dl2.h*" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-classification" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco type \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 3 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188993.874845 4191 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795188993.914018 4191 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795188994.744648 4191 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:36:37,054 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:36:37,055 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:36:37,055 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:36:37,055 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:36:37,056 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:36:37,056 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:36:37,057 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:36:37,057 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:36:37,057 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:36:38,270 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 128 2026-11-20 15:36:38,271 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of signal events: 66 2026-11-20 15:36:38,271 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of background events: 62 E0000 00:00:1795188998.271826 4191 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:36:38,289 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:36:38,291 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['proton_with_images_00.dl2.h*'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h*'], 'input_dir_background': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 3, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-classification'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['type'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:36:38,292 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:36:38,822 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:36:38,824 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795188998.876173 4191 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795188998.896874 4191 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:36:59,558 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/3 - accuracy: 0.4483 - auc: 0.4579 - loss: 0.7442 - val_accuracy: 0.5833 - val_auc: 0.6597 - val_loss: 0.6686 - time: 19.99s 2026-11-20 15:36:59,558 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.66858, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras 2026-11-20 15:36:59,963 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras W0000 00:00:1795189035.029267 4191 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. W0000 00:00:1795189035.094406 4405 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:37:15,617 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/3 - accuracy: 0.4483 - auc: 0.4715 - loss: 0.6968 - val_accuracy: 0.5000 - val_auc: 0.6111 - val_loss: 0.6810 - time: 14.91s 2026-11-20 15:37:15,622 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss did not improve from 0.66858 W0000 00:00:1795189055.449961 4504 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:37:35,995 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/3 - accuracy: 0.5345 - auc: 0.4872 - loss: 0.6937 - val_accuracy: 0.5833 - val_auc: 0.6146 - val_loss: 0.6616 - time: 19.70s 2026-11-20 15:37:35,995 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss improved from 0.66858 to 0.66156, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras 2026-11-20 15:37:36,372 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras 2026-11-20 15:37:37,049 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:37:37,050 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/config.json 2026-11-20 15:37:37,050 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:37:37,052 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:37:37,052 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:37:37,053 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:37:37,053 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:37:37,053 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:37:37,053 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \ --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-background "proton_with_images_00.dl2.h*" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-classification" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco type \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.n_epochs 3 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189061.026898 4548 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189061.064465 4548 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189062.047537 4548 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:37:43,356 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:37:43,356 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:37:43,356 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:37:43,357 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:37:43,357 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:37:43,358 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:37:43,359 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:37:43,359 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:37:43,359 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:37:44,514 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 128 2026-11-20 15:37:44,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of signal events: 66 2026-11-20 15:37:44,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of background events: 62 2026-11-20 15:37:44,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:37:44,524 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['proton_with_images_00.dl2.h*'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h*'], 'input_dir_background': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 3, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-classification'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['type'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:37:44,524 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:37:44,551 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:38:13,943 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/3 - loss: 0.6957 - accuracy: 0.4741 - auc: 0.4346 - val_loss: 0.6749 - val_accuracy: 0.5833 - val_auc: 0.5143 - time: 29.39s 2026-11-20 15:38:13,944 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.67488, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth 2026-11-20 15:38:13,990 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth 2026-11-20 15:38:38,875 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/3 - loss: 0.6919 - accuracy: 0.4741 - auc: 0.5469 - val_loss: 0.6676 - val_accuracy: 0.5833 - val_auc: 0.6286 - time: 24.88s 2026-11-20 15:38:38,876 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.67488 to 0.66761, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth 2026-11-20 15:38:38,935 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth 2026-11-20 15:39:08,162 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/3 - loss: 0.6943 - accuracy: 0.4741 - auc: 0.4888 - val_loss: 0.6453 - val_accuracy: 0.5833 - val_auc: 0.6286 - time: 29.23s 2026-11-20 15:39:08,163 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss improved from 0.66761 to 0.64530, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth 2026-11-20 15:39:08,225 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth 2026-11-20 15:39:08,226 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:39:08,226 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/config.json 2026-11-20 15:39:08,226 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:39:08,228 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:39:08,228 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:39:08,229 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:39:08,229 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:39:08,229 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:39:08,229 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 379-387 *Part B2* Customizing the CTLearn model architecture ==================================================== CTLearn provides a flexible configuration system that allows users to customize the neural network architecture. Two main CNN-based architectures are currently implemented: **SingleCNN** and **ResNet**, which can be set by ``--TrainCTLearnModel.model_type``. .. GENERATED FROM PYTHON SOURCE LINES 389-402 1. SingleCNN model ------------------ The **SingleCNN** component implements a plain convolutional neural network consisting of configurable convolutional layers. It can be used to construct very simple and lightweight models, but it can also be configured to create deeper architectures, such as VGG-like networks. This architecture represents the more legacy model structure in CTLearn and was the starting point for early proof-of-concept studies exploring the applicability of CNNs to the problem. For applications closer to the camera hardware and its readout (for the trigger system), particularly where predictions need to be made at very high rates and with very low latency, computational efficiency becomes increasingly important. .. GENERATED FROM PYTHON SOURCE LINES 402-433 .. code-block:: Python run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.model_type SingleCNN \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.model_type SingleCNN \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 4""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.model_type SingleCNN \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189149.910868 4635 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189149.949887 4635 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189150.814094 4635 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:39:13,209 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]' 2026-11-20 15:39:13,209 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:13,210 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:39:13,210 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:13,210 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:39:13,211 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:13,211 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json 2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:39:14,128 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66 E0000 00:00:1795189154.129317 4635 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:39:14,148 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:39:14,172 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'SingleCNN', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:39:14,173 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:39:14,266 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:39:14,268 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795189154.505340 4635 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795189154.525850 4635 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:39:16,087 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.5540 - mae_energy: 0.5540 - val_loss: 0.4468 - val_mae_energy: 0.4468 - time: 1.39s 2026-11-20 15:39:16,087 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.44676, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras 2026-11-20 15:39:16,153 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras 2026-11-20 15:39:16,754 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.5435 - mae_energy: 0.5435 - val_loss: 0.2468 - val_mae_energy: 0.2468 - time: 0.45s 2026-11-20 15:39:16,758 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.44676 to 0.24676, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras 2026-11-20 15:39:16,811 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras 2026-11-20 15:39:16,891 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:39:16,892 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/config.json 2026-11-20 15:39:16,893 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:39:16,894 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:39:16,894 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:39:16,895 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:39:16,895 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:39:16,896 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:39:16,896 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.model_type SingleCNN \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189160.908871 4890 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189160.947486 4890 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189161.991651 4890 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:39:23,339 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]' 2026-11-20 15:39:23,340 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:23,340 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:39:23,341 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:23,341 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:39:23,341 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:23,342 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json 2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:39:24,196 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66 2026-11-20 15:39:24,197 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:39:24,224 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'SingleCNN', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:39:24,224 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:39:24,227 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:39:24,632 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.5635 - mae_energy: 0.5635 - val_loss: 0.3236 - val_mae_energy: 0.3236 - time: 0.40s 2026-11-20 15:39:24,633 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32361, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/ctlearn_model.pth 2026-11-20 15:39:24,637 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/ctlearn_model.pth 2026-11-20 15:39:25,011 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.5444 - mae_energy: 0.5444 - val_loss: 0.3252 - val_mae_energy: 0.3252 - time: 0.37s 2026-11-20 15:39:25,011 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.32361 2026-11-20 15:39:25,012 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:39:25,012 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/config.json 2026-11-20 15:39:25,012 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:39:25,013 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:39:25,013 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:39:25,014 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:39:25,014 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:39:25,015 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:39:25,015 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 434-452 2. ResNet model --------------- The **ResNet** component provides the currently preferred architecture for constructing deeper CNN models. Residual networks allow substantially deeper architectures while maintaining efficient training and inference, and they generally provide improved performance compared with the simpler plain CNN architecture. The architecture, number of layers or blocks, number of filters, and other relevant parameters can therefore be configured directly through the CTLearn configuration system, allowing the model complexity to be adapted to the requirements of a particular application. For ResNet architectures, CTLearn currently supports two types of residual blocks: - Basic blocks, which use the simpler residual block structure and are suitable for relatively lightweight networks. - Bottleneck blocks, which use a more compact sequence of convolutions to enable deeper networks while controlling the computational cost. .. GENERATED FROM PYTHON SOURCE LINES 452-483 .. code-block:: Python run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-resnet-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.model_type ResNet \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-resnet-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.model_type ResNet \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 4""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-resnet-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.model_type ResNet \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189166.652839 4907 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189166.691209 4907 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189167.581765 4907 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:39:30,012 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]' 2026-11-20 15:39:30,012 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:30,012 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:39:30,013 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:30,013 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:39:30,014 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:39:30,014 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json 2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:39:30,917 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66 E0000 00:00:1795189170.917994 4907 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:39:30,935 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:39:30,958 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:39:30,958 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:39:31,673 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:39:31,676 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795189171.723895 4907 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795189171.743516 4907 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. W0000 00:00:1795189203.126112 4907 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:40:03,143 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.5939 - mae_energy: 0.5939 - val_loss: 0.3719 - val_mae_energy: 0.3719 - time: 30.50s 2026-11-20 15:40:03,143 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.37188, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras 2026-11-20 15:40:03,989 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras 2026-11-20 15:40:29,013 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.5175 - mae_energy: 0.5175 - val_loss: 0.3100 - val_mae_energy: 0.3100 - time: 23.13s 2026-11-20 15:40:29,018 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.37188 to 0.30996, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras 2026-11-20 15:40:29,907 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras 2026-11-20 15:40:31,735 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:40:31,735 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/config.json 2026-11-20 15:40:31,736 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:40:31,737 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:40:31,737 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:40:31,738 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:40:31,738 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:40:31,739 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:40:31,739 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-resnet-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.model_type ResNet \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189236.077512 5173 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189236.114719 5173 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189237.146097 5173 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:40:38,490 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]' 2026-11-20 15:40:38,490 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:40:38,490 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:40:38,491 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:40:38,491 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:40:38,492 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:40:38,492 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json 2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:40:39,369 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66 2026-11-20 15:40:39,369 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:40:39,398 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:40:39,398 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:40:39,499 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:41:11,969 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.5461 - mae_energy: 0.5461 - val_loss: 0.3256 - val_mae_energy: 0.3256 - time: 32.47s 2026-11-20 15:41:11,970 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32561, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth 2026-11-20 15:41:12,142 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth 2026-11-20 15:41:44,087 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.5275 - mae_energy: 0.5275 - val_loss: 0.3250 - val_mae_energy: 0.3250 - time: 31.94s 2026-11-20 15:41:44,089 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.32561 to 0.32498, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth 2026-11-20 15:41:44,328 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth 2026-11-20 15:41:44,328 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:41:44,329 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/config.json 2026-11-20 15:41:44,329 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:41:44,331 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:41:44,331 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:41:44,332 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:41:44,332 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:41:44,332 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:41:44,332 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 484-503 *Part C2* Using a pre-trained model and transfer learning ========================================================= The LoadedModel component allows a previously trained model following the CTLearnModel structure to be loaded for either resuming training or transfer learning. Transfer learning reuses the learned features of an existing model and adapts them to new, but related, data or tasks. This can reduce training time compared with training a model from scratch. Two parameters are particularly important: - ``--LoadedModel.trainable_backbone`` controls whether the pre-trained backbone is frozen (false) or trainable (true). - ``--LoadedModel.overwrite_head`` controls whether the structure of the original prediction head is reused (false) or replaced with a new MLP tailored to the new task or data (true). The preceding SingleCNN command trains the model loaded below. The copied configuration keeps its prediction head and leaves the backbone trainable. .. GENERATED FROM PYTHON SOURCE LINES 503-536 .. code-block:: Python run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-transfer-learning-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.model_type LoadedModel \\ --LoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy/ctlearn_model.keras" \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 4""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-transfer-learning-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\ --reco energy \\ --DLImageReader.focal_length_choice EQUIVALENT \\ --TrainCTLearnModel.model_type LoadedModel \\ --PyTorchLoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy/ctlearn_model.pth" \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 4""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-transfer-learning-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.model_type LoadedModel \ --LoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy/ctlearn_model.keras" \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189306.139857 5194 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189306.178179 5194 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189307.061421 5194 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:41:49,494 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]' 2026-11-20 15:41:49,495 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:41:49,495 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json 2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:41:50,407 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66 E0000 00:00:1795189310.408506 5194 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:41:50,427 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:41:50,450 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'LoadedModel', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:41:50,451 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:41:50,868 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:41:50,869 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795189310.917049 5194 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795189310.937757 5194 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:41:52,467 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.5428 - mae_energy: 0.5428 - val_loss: 0.3672 - val_mae_energy: 0.3672 - time: 1.39s 2026-11-20 15:41:52,467 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.36716, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras 2026-11-20 15:41:52,534 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras 2026-11-20 15:41:53,153 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.5223 - mae_energy: 0.5223 - val_loss: 0.2474 - val_mae_energy: 0.2474 - time: 0.46s 2026-11-20 15:41:53,157 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.36716 to 0.24735, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras 2026-11-20 15:41:53,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras 2026-11-20 15:41:53,294 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:41:53,294 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/config.json 2026-11-20 15:41:53,295 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:41:53,296 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:41:53,296 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:41:53,297 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:41:53,297 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:41:53,298 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:41:53,298 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-transfer-learning-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \ --reco energy \ --DLImageReader.focal_length_choice EQUIVALENT \ --TrainCTLearnModel.model_type LoadedModel \ --PyTorchLoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy/ctlearn_model.pth" \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189317.386792 5449 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189317.425816 5449 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189318.497509 5449 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:41:59,862 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]' 2026-11-20 15:41:59,863 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:41:59,863 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:41:59,863 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:41:59,864 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:41:59,864 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:41:59,864 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json 2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:42:00,761 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66 2026-11-20 15:42:00,762 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:42:00,789 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'LoadedModel', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}} 2026-11-20 15:42:00,790 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:42:00,793 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:42:01,189 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.5515 - mae_energy: 0.5515 - val_loss: 0.3242 - val_mae_energy: 0.3242 - time: 0.40s 2026-11-20 15:42:01,190 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32422, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/ctlearn_model.pth 2026-11-20 15:42:01,194 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/ctlearn_model.pth 2026-11-20 15:42:01,571 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.5290 - mae_energy: 0.5290 - val_loss: 0.3416 - val_mae_energy: 0.3416 - time: 0.38s 2026-11-20 15:42:01,571 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.32422 2026-11-20 15:42:01,571 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:42:01,572 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/config.json 2026-11-20 15:42:01,572 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:42:01,573 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:42:01,573 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:42:01,574 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:42:01,574 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:42:01,575 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:42:01,575 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 537-565 *Part D2* Stereoscopic CNN-based model training =============================================== CTLearn currently provides a basic approach for stereoscopic CNN-based model training. The images from the different telescopes are concatenated channel-wise and then passed to a standard, monoscopic CNN backbone as above. In this way, the network can process the combined input while retaining stereoscopic information. An earlier version of CTLearn also included a CNN-RNN architecture, using recurrent neural networks to explicitly handle information from multiple telescopes dynamically. This model has not yet been migrated to the latest CTLearn version based on TensorFlow 2 and Keras 3, so this tutorial focuses on the channel-wise concatenation approach. The main configuration parameters for stereoscopic training are: - ``--DLImageReader.mode``: set to "stereo" to enable stereoscopic image reading - ``--DLImageReader.min_telescopes``: defines the minimum number of telescopes that must participate in an event after the quality cuts. - ``--TrainCTLearnModel.stack_telescope_images``: set to true to stack/concatenate the images from the telescopes. Since the resulting input contains information from multiple telescopes and is therefore larger than the corresponding monoscopic input, it is recommended to reduce the batch size using ``--TrainCTLearnModel.batch_size`` to account for the increased memory consumption. .. GENERATED FROM PYTHON SOURCE LINES 565-598 .. code-block:: Python run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-stereo-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco energy \\ --DLImageReader.mode stereo \\ --DLImageReader.min_telescopes 2 \\ --TrainCTLearnModel.stack_telescope_images True \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 2 \\ --TableQualityQuery.quality_criteria []""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-stereo-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --reco energy \\ --DLImageReader.mode stereo \\ --DLImageReader.min_telescopes 2 \\ --TrainCTLearnModel.stack_telescope_images True \\ --TrainCTLearnModel.n_epochs 5 \\ --TrainCTLearnModel.batch_size 2 \\ --TableQualityQuery.quality_criteria []""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-stereo-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco energy \ --DLImageReader.mode stereo \ --DLImageReader.min_telescopes 2 \ --TrainCTLearnModel.stack_telescope_images True \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 2 \ --TableQualityQuery.quality_criteria [] WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189323.382390 5468 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189323.421272 5468 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189324.318563 5468 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:42:06,779 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:42:06,779 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:42:06,779 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:42:06,780 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:42:06,780 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:42:07,771 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 135 E0000 00:00:1795189327.771888 5468 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:42:07,790 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:42:07,813 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': True, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 2, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'stereo', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}} 2026-11-20 15:42:07,814 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:42:08,362 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:42:08,364 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795189328.438136 5468 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795189328.461709 5468 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. W0000 00:00:1795189350.058643 5581 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:42:31,327 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 0.8730 - mae_energy: 0.8730 - val_loss: 0.9720 - val_mae_energy: 0.9720 - time: 22.17s 2026-11-20 15:42:31,327 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.97196, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:42:31,734 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras W0000 00:00:1795189368.487917 5468 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. W0000 00:00:1795189368.561814 5681 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:42:49,178 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 0.8948 - mae_energy: 0.8948 - val_loss: 0.8049 - val_mae_energy: 0.8049 - time: 16.72s 2026-11-20 15:42:49,183 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.97196 to 0.80491, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:42:49,573 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:43:07,035 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 0.8224 - mae_energy: 0.8224 - val_loss: 1.1420 - val_mae_energy: 1.1420 - time: 16.81s 2026-11-20 15:43:07,036 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss did not improve from 0.80491 W0000 00:00:1795189404.185386 5468 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. W0000 00:00:1795189404.260948 5880 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled [[{{node MultiDeviceIteratorGetNextFromShard}}]] 2026-11-20 15:43:25,079 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 0.8588 - mae_energy: 0.8588 - val_loss: 0.7744 - val_mae_energy: 0.7744 - time: 17.38s 2026-11-20 15:43:25,079 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss improved from 0.80491 to 0.77435, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:43:25,470 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:43:43,480 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 0.8052 - mae_energy: 0.8052 - val_loss: 0.7101 - val_mae_energy: 0.7101 - time: 17.34s 2026-11-20 15:43:43,481 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss improved from 0.77435 to 0.71013, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:43:43,877 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras 2026-11-20 15:43:44,576 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:43:44,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/config.json 2026-11-20 15:43:44,578 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:43:44,579 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:43:44,579 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:43:44,580 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:43:44,580 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:43:44,581 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:43:44,581 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \ --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-stereo-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --reco energy \ --DLImageReader.mode stereo \ --DLImageReader.min_telescopes 2 \ --TrainCTLearnModel.stack_telescope_images True \ --TrainCTLearnModel.n_epochs 5 \ --TrainCTLearnModel.batch_size 2 \ --TableQualityQuery.quality_criteria [] WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189429.066331 6026 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189429.105682 6026 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189430.172168 6026 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:43:51,554 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]' 2026-11-20 15:43:51,555 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:43:51,555 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:43:51,555 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:43:51,556 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types and , choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge] 2026-11-20 15:43:52,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 135 2026-11-20 15:43:52,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:43:52,523 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': True, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 2, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'stereo', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}} 2026-11-20 15:43:52,524 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:43:52,552 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:44:17,944 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 0.9399 - mae_energy: 0.9399 - val_loss: 0.7731 - val_mae_energy: 0.7731 - time: 25.39s 2026-11-20 15:44:17,945 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.77312, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth 2026-11-20 15:44:17,993 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth 2026-11-20 15:44:44,069 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 0.8843 - mae_energy: 0.8843 - val_loss: 0.8150 - val_mae_energy: 0.8150 - time: 26.08s 2026-11-20 15:44:44,070 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.77312 2026-11-20 15:45:08,514 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 0.8385 - mae_energy: 0.8385 - val_loss: 0.7800 - val_mae_energy: 0.7800 - time: 24.44s 2026-11-20 15:45:08,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss did not improve from 0.77312 2026-11-20 15:45:32,097 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 0.8770 - mae_energy: 0.8770 - val_loss: 0.7547 - val_mae_energy: 0.7547 - time: 23.58s 2026-11-20 15:45:32,099 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss improved from 0.77312 to 0.75467, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth 2026-11-20 15:45:32,158 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth 2026-11-20 15:45:55,636 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 0.8275 - mae_energy: 0.8275 - val_loss: 0.7177 - val_mae_energy: 0.7177 - time: 23.48s 2026-11-20 15:45:55,637 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss improved from 0.75467 to 0.71768, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth 2026-11-20 15:45:55,697 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth 2026-11-20 15:45:55,697 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:45:55,697 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/config.json 2026-11-20 15:45:55,698 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:45:55,699 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:45:55,699 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:45:55,700 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:45:55,700 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:45:55,701 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:45:55,701 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 599-620 *Part E2* CNN-based model training using waveform data ====================================================== The final example demonstrates training directly on waveform data. The example dataset used in this tutorial is intentionally very small, so only a few events are available for training. In practical applications, waveform datasets can be very large, often reaching several terabytes, and processing the full waveforms also requires substantial RAM and computational resources. Because the example dataset is so small, the training configuration needs to be adjusted accordingly. We use a very small batch size and increase the validation split to ensure that at least one batch is available for validation. For this example, we select the MST equipped with the NectarCam camera and disable the quality selection, leaving only 10 events. These are divided into 5 batches of 2 events, with 20% of the data reserved for validation, corresponding to one validation batch. An important configuration change is required to read waveform data: the ``--TrainCTLearnModel.dl1dh_reader_type`` option must be set to the appropriate DL1 data reader component for waveform reading (**DLWaveformReader**) rather than the default image reader. .. GENERATED FROM PYTHON SOURCE LINES 620-658 .. code-block:: Python if not WAVEFORM_FILE.is_file(): raise FileNotFoundError( f"Cached waveform data is missing: {WAVEFORM_FILE}. " "Run explore_public_data.py first." ) run_command_line( """ctlearn-train-keras-model --verbose \\ --signal "$CTLEARN_TUTORIAL_DATA_DIR" \\ --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-waveform-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --reco energy \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 2 \\ --TrainCTLearnModel.validation_split 0.2 \\ --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \\ --DLWaveformReader.focal_length_choice EQUIVALENT \\ --TableQualityQuery.quality_criteria []""", ) run_command_line( """ctlearn-train-pytorch-model --verbose \\ --signal "$CTLEARN_TUTORIAL_DATA_DIR" \\ --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \\ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-waveform-energy" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\ --reco energy \\ --TrainCTLearnModel.n_epochs 2 \\ --TrainCTLearnModel.batch_size 2 \\ --TrainCTLearnModel.validation_split 0.2 \\ --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \\ --DLWaveformReader.focal_length_choice EQUIVALENT \\ --TableQualityQuery.quality_criteria []""", ) .. rst-class:: sphx-glr-script-out .. code-block:: none Running command: $ ctlearn-train-keras-model --verbose \ --signal "$CTLEARN_TUTORIAL_DATA_DIR" \ --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-waveform-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --reco energy \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 2 \ --TrainCTLearnModel.validation_split 0.2 \ --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \ --DLWaveformReader.focal_length_choice EQUIVALENT \ --TableQualityQuery.quality_criteria [] WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189557.387919 6061 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189557.426020 6061 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189558.272849 6061 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:46:00,590 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')]' 2026-11-20 15:46:00,590 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:46:00,590 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:46:00,591 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:46:00,591 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data: 2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] 2026-11-20 15:46:01,427 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 10 E0000 00:00:1795189561.428563 6061 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) 2026-11-20 15:46:01,446 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1 2026-11-20 15:46:01,467 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')], 'dl1dh_reader_type': 'DLWaveformReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma_prod5-max500.r1.dl1.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.2, 'DLWaveformReader': {'DBSCAN_params': {'eps': 0.5, 'min_samples': 5, 'metric': 'euclidean'}, 'allowed_tel_types': None, 'allowed_tels': {109, 112, 118, 119, 121, 122, 124}, 'cleaning_type': None, 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'sequence_length': 5, 'sequence_position': 'center', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}} 2026-11-20 15:46:01,468 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model. 2026-11-20 15:46:01,992 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model. 2026-11-20 15:46:01,994 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating... I0000 00:00:1795189562.050899 6061 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set. W0000 00:00:1795189562.070053 6061 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2026-11-20 15:46:08,816 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.2179 - mae_energy: 0.2179 - val_loss: 0.1557 - val_mae_energy: 0.1557 - time: 6.12s 2026-11-20 15:46:08,816 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.15568, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras 2026-11-20 15:46:09,216 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras 2026-11-20 15:46:11,278 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.0914 - mae_energy: 0.0914 - val_loss: 0.0805 - val_mae_energy: 0.0805 - time: 1.34s 2026-11-20 15:46:11,282 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.15568 to 0.08048, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras 2026-11-20 15:46:11,678 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras 2026-11-20 15:46:12,343 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:46:12,343 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/config.json 2026-11-20 15:46:12,344 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down 2026-11-20 15:46:12,345 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:46:12,345 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:46:12,346 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output: 2026-11-20 15:46:12,346 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log' 2026-11-20 15:46:12,347 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model 2026-11-20 15:46:12,347 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model Running command: $ ctlearn-train-pytorch-model --verbose \ --signal "$CTLEARN_TUTORIAL_DATA_DIR" \ --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \ --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-waveform-energy" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \ --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \ --reco energy \ --TrainCTLearnModel.n_epochs 2 \ --TrainCTLearnModel.batch_size 2 \ --TrainCTLearnModel.validation_split 0.2 \ --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \ --DLWaveformReader.focal_length_choice EQUIVALENT \ --TableQualityQuery.quality_criteria [] WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189576.306473 6316 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. I0000 00:00:1795189576.346576 6316 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1795189577.345991 6316 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None warnings.warn(msg, NoneDefaultNotAllowedWarning) 2026-11-20 15:46:18,660 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')]' 2026-11-20 15:46:18,660 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:46:18,660 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json 2026-11-20 15:46:18,661 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials 2026-11-20 15:46:18,661 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json 2026-11-20 15:46:18,662 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0 2026-11-20 15:46:18,662 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28 2026-11-20 15:46:18,663 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data: 2026-11-20 15:46:18,663 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while... WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier). Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge] 2026-11-20 15:46:19,488 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 10 2026-11-20 15:46:19,489 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device. 2026-11-20 15:46:19,513 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')], 'dl1dh_reader_type': 'DLWaveformReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma_prod5-max500.r1.dl1.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.2, 'DLWaveformReader': {'DBSCAN_params': {'eps': 0.5, 'min_samples': 5, 'metric': 'euclidean'}, 'allowed_tel_types': None, 'allowed_tels': {109, 112, 118, 119, 121, 122, 124}, 'cleaning_type': None, 'enforce_subarray_equality': True, 'focal_length_choice': , 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'sequence_length': 5, 'sequence_position': 'center', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}} 2026-11-20 15:46:19,513 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model. 2026-11-20 15:46:19,538 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating... 2026-11-20 15:46:21,373 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.2137 - mae_energy: 0.2137 - val_loss: 0.2064 - val_mae_energy: 0.2064 - time: 1.84s 2026-11-20 15:46:21,375 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.20635, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth 2026-11-20 15:46:21,419 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth 2026-11-20 15:46:23,023 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.2015 - mae_energy: 0.2015 - val_loss: 0.1923 - val_mae_energy: 0.1923 - time: 1.60s 2026-11-20 15:46:23,024 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.20635 to 0.19226, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth 2026-11-20 15:46:23,083 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth 2026-11-20 15:46:23,084 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully! 2026-11-20 15:46:23,084 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/config.json 2026-11-20 15:46:23,085 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down 2026-11-20 15:46:23,086 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:46:23,086 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:46:23,087 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output: 2026-11-20 15:46:23,087 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log' 2026-11-20 15:46:23,087 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model 2026-11-20 15:46:23,087 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model .. GENERATED FROM PYTHON SOURCE LINES 659-675 *Part F2* Comparing training curves =================================== Each training run writes its metrics to ``training_log.csv`` together with the TensorBoard model checkpoints for training and validation, the final model, the configuration and provenance files. See Part G for instructions on how to monitor the training runs live using TensorBoard. Here, we read the training logs and plot the evolution of the metrics over the training epochs. The plots show the mean absolute error (MAE) for the regression tasks and the accuracy and AUC for the classification task. Note: The statistics of the training runs are intentionally very limited, so the training curves are not expected to be smooth or to converge to optimal values. The purpose of this tutorial is to demonstrate the training workflow and the use of CTLearn, rather than to achieve optimal performance on the example dataset. .. GENERATED FROM PYTHON SOURCE LINES 675-787 .. code-block:: Python training_logs = {} for run_name in ( "keras-energy", "pytorch-energy", "keras-cameradirection", "pytorch-cameradirection", "keras-skydirection", "pytorch-skydirection", "keras-classification", "pytorch-classification", ): log_path = OUTPUT_DIR / run_name / "training_log.csv" if not log_path.is_file(): raise FileNotFoundError(f"Training log is missing: {log_path}") training_logs[run_name] = pd.read_csv(log_path) frameworks = ( ("Keras", "keras", "tab:blue"), ("PyTorch", "pytorch", "tab:orange"), ) splits = (("", "train", "-"), ("val_", "validation", "--")) def plot_metric(ax, run_name, metric, label, color, **kwargs): log = training_logs[run_name] for prefix, split_name, split_style in splits: column = f"{prefix}{metric}" if column not in log: raise ValueError( f"Column {column!r} is missing from {run_name}/training_log.csv" ) ax.plot( log["epoch"] + 1, log[column], color=color, linestyle=split_style, label=f"{label} - {split_name}", **kwargs, ) fig, (energy_ax, camera_ax, sky_ax, type_ax) = plt.subplots( 1, 4, figsize=(24, 6), constrained_layout=True ) fig.suptitle("CTLearn training curves", fontsize=16) for display_name, framework, color in frameworks: plot_metric( energy_ax, f"{framework}-energy", "mae_energy", display_name, color, marker="o", ) energy_ax.set_title("Energy") energy_ax.set_xlabel("Epoch") energy_ax.set_ylabel("Mean absolute error") energy_ax.grid(True, alpha=0.3) energy_ax.legend(fontsize=8) for display_name, framework, color in frameworks: plot_metric( camera_ax, f"{framework}-cameradirection", "mae_cameradirection", display_name, color, marker="o", ) plot_metric( sky_ax, f"{framework}-skydirection", "mae_skydirection", display_name, color, marker="s", ) camera_ax.set_title("Camera direction") camera_ax.set_xlabel("Epoch") camera_ax.set_ylabel("Mean absolute error [m]") camera_ax.grid(True, alpha=0.3) camera_ax.legend(fontsize=8) sky_ax.set_title("Sky direction") sky_ax.set_xlabel("Epoch") sky_ax.set_ylabel("Sky direction MAE [deg]") sky_ax.grid(True, alpha=0.3) sky_ax.legend(fontsize=8) for display_name, framework, color in frameworks: for metric, marker in (("accuracy", "o"), ("auc", "s")): plot_metric( type_ax, f"{framework}-classification", metric, f"{display_name} {metric}", color, marker=marker, ) type_ax.set_title("Particle type") type_ax.set_xlabel("Epoch") type_ax.set_ylabel("Score") type_ax.set_ylim(0.4, 0.7) type_ax.grid(True, alpha=0.3) type_ax.legend(fontsize=8, ncol=2, loc="best") training_curves_path = OUTPUT_DIR / "training-curves.png" fig.savefig(training_curves_path, dpi=350) print(f"\nTraining curves saved to {training_curves_path}") plt.show() .. image-sg:: /user-guide/auto_tutorials/images/sphx_glr_train_ctlearn_model_001.png :alt: CTLearn training curves, Energy, Camera direction, Sky direction, Particle type :srcset: /user-guide/auto_tutorials/images/sphx_glr_train_ctlearn_model_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Training curves saved to /home/runner/.cache/ctlearn/tutorial-training/training-curves.png .. GENERATED FROM PYTHON SOURCE LINES 788-804 *Part G2* Monitoring training with TensorBoard ============================================== CTLearn also writes TensorBoard event files, which are useful for monitoring metrics live or exploring additional details after training. The tutorial does not launch TensorBoard; to monitor a run while it is training, start it in a separate terminal and point it at the tutorial output directory: .. code-block:: console tensorboard --logdir "$HOME/.cache/ctlearn/tutorial-training/" Open the URL printed by TensorBoard in a browser. If you set ``CTLEARN_TUTORIAL_OUTPUT_DIR`` to a different location, use that directory in the command instead. On a remote or cluster environment, use the appropriate port forwarding to access the TensorBoard web interface. .. rst-class:: sphx-glr-timing **Total running time of the script:** (16 minutes 26.651 seconds) .. _sphx_glr_download_user-guide_auto_tutorials_train_ctlearn_model.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: train_ctlearn_model.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: train_ctlearn_model.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: train_ctlearn_model.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_