Tutorial 2 Train CTLearn models#

Train models for CTAO image and waveform data with CTLearn. The image examples use only the gamma-ray and proton 00 files; the 01 files remain untouched for future evaluation tutorials. The tutorial is intenionally kept simple and uses only a small fraction of the available statistics to keep the training time manageable. The resulting performance metrics are not representative of a production-level analysis nor should they be compared directly with published results. The main goal is to illustrate the analysis workflow and to become familiar with the command-line tools and the DL2 data format. This tutorial can be modified from the users to use larger datasets and more statistics to obtain meaningful results.

Each section explains a training task followed immediately by its literal command-line example. Running this script executes every example in order.

Inspect the training environment and cached input data#

Run explore_public_data.py first to download and convert the shared tutorial data. The converted directory is keyed by the ctapipe version and the optional event limit, matching the cache layout used by that tutorial.

import importlib.metadata
import os
import subprocess
from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd
from ctapipe import __version__ as ctapipe_version

MAX_EVENTS = os.environ.get("CTLEARN_TUTORIAL_MAX_EVENTS")
if MAX_EVENTS is not None and int(MAX_EVENTS) < 1:
    raise ValueError("CTLEARN_TUTORIAL_MAX_EVENTS must be a positive integer")
MAX_EVENTS_SUFFIX = f"-max{MAX_EVENTS}" if MAX_EVENTS is not None else ""

TESTDATA_DIR = (
    Path(
        os.environ.get(
            "CTLEARN_TUTORIAL_DATA_DIR",
            str(Path.home() / ".cache" / "ctlearn" / "tutorial-data"),
        )
    )
    .expanduser()
    .resolve()
)
CONVERTED_DIR = TESTDATA_DIR / f"ctapipe-{ctapipe_version}{MAX_EVENTS_SUFFIX}"
GAMMA_FILE = CONVERTED_DIR / "gamma-diffuse_with_images_00.dl2.h5"
PROTON_FILE = CONVERTED_DIR / "proton_with_images_00.dl2.h5"
WAVEFORM_FILE = TESTDATA_DIR / f"gamma_prod5{MAX_EVENTS_SUFFIX}.r1.dl1.h5"

print("Training environment:")
for distribution in ("ctapipe", "ctlearn", "dl1-data-handler", "keras", "tensorflow"):
    try:
        version = importlib.metadata.version(distribution)
    except importlib.metadata.PackageNotFoundError:
        version = "not installed"
    print(f"  {distribution}: {version}")

missing_files = [path for path in (GAMMA_FILE, PROTON_FILE) if not path.is_file()]
if missing_files:
    missing = "\n".join(f"  - {path}" for path in missing_files)
    raise FileNotFoundError(
        "Cached stereo training data is missing. Run "
        "explore_public_data.py first. Missing files:\n"
        f"{missing}"
    )

print("\nCached input files:")
for path in (GAMMA_FILE, PROTON_FILE):
    print(f"  {path.name}: {path.stat().st_size / 1024**2:.1f} MiB")
print("Only the _00 image files are selected; the _01 files are reserved.")
Training environment:
  ctapipe: 0.33.0
  ctlearn: 0.11.2.dev2+g312165f28
  dl1-data-handler: 0.14.13
  keras: 3.15.1
  tensorflow: 2.21.0

Cached input files:
  gamma-diffuse_with_images_00.dl2.h5: 12.3 MiB
  proton_with_images_00.dl2.h5: 12.4 MiB
Only the _00 image files are selected; the _01 files are reserved.

Training commands, configs, and outputs#

CTLearn provides ctlearn-train-keras-model and ctlearn-train-pytorch-model command-line tools. Use --help or --help-all to inspect their options. Each example has a Keras command followed immediately by the equivalent PyTorch command. In a notebook, prefix either command with !.

The three JSON configuration files are in docs/source/config in the repository and in config in the downloaded tutorial ZIPs:

  • dl1dh_example_config.json selects LST/LSTCam images, the image and relative peak-time channels, bilinear mapping, and an example 50,000 p.e. intensity cut. It also specifies NectarCam waveform settings.

  • ctlearn_train_model_example_config.json sets the reader, epoch and batch defaults, validation split, Adam optimizer and learning rate, checkpointing, learning-rate reduction, random seed, and early stopping.

  • ctlearn_models_example_config.json defines the prediction heads, attention settings, custom SingleCNN and ResNet architectures, and LoadedModel transfer-learning options.

def find_config_dir():
    roots = []
    script_path = globals().get("__file__")
    if script_path is not None:
        script_dir = Path(script_path).resolve().parent
        roots.extend((script_dir, *script_dir.parents))
    roots.extend((Path.cwd(), *Path.cwd().parents))

    config_names = (
        "dl1dh_example_config.json",
        "ctlearn_train_model_example_config.json",
        "ctlearn_models_example_config.json",
    )
    for root in roots:
        for config_dir in (root / "config", root / "docs" / "source" / "config"):
            if all((config_dir / name).is_file() for name in config_names):
                return config_dir

    raise FileNotFoundError(
        "Could not locate tutorial configs in config/ or docs/source/config/. "
        "Extract the complete tutorial ZIP or run from the CTLearn repository."
    )


CONFIG_DIR = find_config_dir()
DL1DH_CONFIG = CONFIG_DIR / "dl1dh_example_config.json"
TRAINING_CONFIG = CONFIG_DIR / "ctlearn_train_model_example_config.json"
MODEL_CONFIG = CONFIG_DIR / "ctlearn_models_example_config.json"
for config_path in (DL1DH_CONFIG, TRAINING_CONFIG, MODEL_CONFIG):
    if not config_path.is_file():
        raise FileNotFoundError(f"Tutorial configuration is missing: {config_path}")

OUTPUT_DIR = (
    Path(
        os.environ.get(
            "CTLEARN_TUTORIAL_OUTPUT_DIR",
            str(Path.home() / ".cache" / "ctlearn" / "tutorial-training"),
        )
    )
    .expanduser()
    .resolve()
)
TRAINING_OUTPUTS = (
    "keras-energy",
    "pytorch-energy",
    "keras-cameradirection",
    "pytorch-cameradirection",
    "keras-skydirection",
    "pytorch-skydirection",
    "keras-classification",
    "pytorch-classification",
    "keras-singlecnn-energy",
    "pytorch-singlecnn-energy",
    "keras-resnet-energy",
    "pytorch-resnet-energy",
    "keras-transfer-learning-energy",
    "pytorch-transfer-learning-energy",
    "keras-stereo-energy",
    "pytorch-stereo-energy",
    "keras-waveform-energy",
    "pytorch-waveform-energy",
)
existing_outputs = [
    OUTPUT_DIR / name for name in TRAINING_OUTPUTS if (OUTPUT_DIR / name).exists()
]
if existing_outputs:
    existing = "\n".join(f"  - {path}" for path in existing_outputs)
    raise FileExistsError(
        "Training output directories already exist. Set "
        "CTLEARN_TUTORIAL_OUTPUT_DIR to a new directory before running all "
        f"examples:\n{existing}"
    )

os.environ["CTLEARN_TUTORIAL_DATA_DIR"] = str(TESTDATA_DIR)
os.environ["CTLEARN_TUTORIAL_CONVERTED_DIR"] = str(CONVERTED_DIR)
os.environ["CTLEARN_TUTORIAL_CONFIG_DIR"] = str(CONFIG_DIR.resolve())
os.environ["CTLEARN_TUTORIAL_OUTPUT_DIR"] = str(OUTPUT_DIR)
os.environ["CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX"] = MAX_EVENTS_SUFFIX

print(f"\nTraining outputs: {OUTPUT_DIR}")


def run_command_line(command):
    print(f"Running command: $ {command.strip()}")
    result = subprocess.run(
        command,
        shell=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
    )
    print(result.stdout, end="")
    result.check_returncode()
Training outputs: /home/runner/.cache/ctlearn/tutorial-training

Part A2 Monoscopic CNN-based model training#

We first train a basic monoscopic CNN-based model using the default CTLearn model configuration. The model uses a ThinResNet backbone for feature extraction and a standard MLP head for the task-specific prediction. We consider three different reconstruction tasks, each trained separately:

1. Primary particle energy regression#

The model reconstructs the energy of the primary particle as a continuous value. Internally, CTLearn performs the regression on the energy in TeV and on a logarithmic scale, since the primary-particle energy spans a very wide range. Working in logarithmic space has been found to be more efficient for the regression and is therefore a fixed, non-configurable aspect of the energy reconstruction. During inference, the prediction tool converts the predicted values back from logarithmic space to the original linear energy scale in TeV.

run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 4""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188599.092084    2414 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188599.129615    2414 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188599.982559    2414 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:30:02,520 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:30:02,520 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:30:02,520 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:30:02,521 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:30:02,521 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:30:02,522 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:30:02,522 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:30:02,523 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:30:02,523 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:30:03,410 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66
E0000 00:00:1795188603.410940    2414 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:30:03,428 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:30:03,448 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:30:03,448 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:30:03,976 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:30:03,979 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795188604.024311    2414 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795188604.043836    2414 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:30:17,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 0.5534 - mae_energy: 0.5534 - val_loss: 0.3763 - val_mae_energy: 0.3763 - time: 12.54s
2026-11-20 15:30:17,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.37629, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
2026-11-20 15:30:17,627 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
2026-11-20 15:30:26,714 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 0.5462 - mae_energy: 0.5462 - val_loss: 0.3572 - val_mae_energy: 0.3572 - time: 8.34s
2026-11-20 15:30:26,719 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.37629 to 0.35724, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
2026-11-20 15:30:27,113 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
W0000 00:00:1795188635.250682    2414 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:30:35,552 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 0.5259 - mae_energy: 0.5259 - val_loss: 0.3542 - val_mae_energy: 0.3542 - time: 7.77s
2026-11-20 15:30:35,552 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss improved from 0.35724 to 0.35419, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
2026-11-20 15:30:36,147 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
W0000 00:00:1795188644.327792    2835 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:30:44,564 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 0.5251 - mae_energy: 0.5251 - val_loss: 0.3075 - val_mae_energy: 0.3075 - time: 7.74s
2026-11-20 15:30:44,565 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss improved from 0.35419 to 0.30746, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
2026-11-20 15:30:44,950 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn_model.keras
W0000 00:00:1795188653.026210    2933 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:30:53,370 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 0.4961 - mae_energy: 0.4961 - val_loss: 0.4088 - val_mae_energy: 0.4088 - time: 7.75s
2026-11-20 15:30:53,371 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss did not improve from 0.30746
2026-11-20 15:30:54,048 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:30:54,048 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-energy/config.json
2026-11-20 15:30:54,049 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:30:54,050 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:30:54,050 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-energy/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:30:54,051 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:30:54,052 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:30:54,052 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:30:54,052 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188658.182740    2976 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188658.221012    2976 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188659.272269    2976 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:31:00,618 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:31:00,618 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:31:00,618 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:31:00,619 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:31:00,619 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:31:00,620 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:31:00,620 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:31:00,621 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:31:00,621 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:31:01,471 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66
2026-11-20 15:31:01,471 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:31:01,496 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:31:01,496 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:31:01,521 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:31:15,203 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 0.5453 - mae_energy: 0.5453 - val_loss: 0.3254 - val_mae_energy: 0.3254 - time: 13.68s
2026-11-20 15:31:15,204 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32539, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth
2026-11-20 15:31:15,252 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth
2026-11-20 15:31:28,165 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 0.5495 - mae_energy: 0.5495 - val_loss: 0.3468 - val_mae_energy: 0.3468 - time: 12.91s
2026-11-20 15:31:28,166 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.32539
2026-11-20 15:31:40,665 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 0.5190 - mae_energy: 0.5190 - val_loss: 0.3251 - val_mae_energy: 0.3251 - time: 12.50s
2026-11-20 15:31:40,666 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss improved from 0.32539 to 0.32513, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth
2026-11-20 15:31:40,728 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn_model.pth
2026-11-20 15:31:53,569 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 0.5059 - mae_energy: 0.5059 - val_loss: 0.3342 - val_mae_energy: 0.3342 - time: 12.84s
2026-11-20 15:31:53,570 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss did not improve from 0.32513
2026-11-20 15:32:06,759 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 0.5213 - mae_energy: 0.5213 - val_loss: 0.3362 - val_mae_energy: 0.3362 - time: 13.19s
2026-11-20 15:32:06,760 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss did not improve from 0.32513
2026-11-20 15:32:06,760 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:32:06,760 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/config.json
2026-11-20 15:32:06,761 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:32:06,762 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:32:06,762 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-energy/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:32:06,763 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:32:06,763 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:32:06,764 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:32:06,764 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

2. Arrival direction regression#

The model reconstructs the arrival direction of the primary particle. This can be done in two ways: either using the cameradirection task, which transforms the particle origin into the camera frame and performs the regression in camera coordinates, with distances expressed in meters; or using the skydirection task, which performs the regression in the sky frame, with field-of-view offsets expressed in degrees. For monoscopic CNN-based models, we generally recommend using the cameradirection task. During inference, the prediction tool converts the reconstructed camera coordinates back to the original Alt/Az representation, with the final direction expressed in degrees.

# Camera coordinates
run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-cameradirection" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco cameradirection \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-cameradirection" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco cameradirection \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 4""",
)

# Sky coordinates
run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-skydirection" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco skydirection \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-skydirection" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco skydirection \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 4""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-cameradirection" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco cameradirection \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188728.472466    3010 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188728.511066    3010 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188729.380341    3010 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:32:11,740 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:32:11,740 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:32:11,740 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:32:11,741 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:32:11,741 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:32:11,742 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:32:12,635 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66
E0000 00:00:1795188732.636064    3010 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:32:12,654 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:32:12,675 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['cameradirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:32:12,676 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:32:13,209 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:32:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795188733.257518    3010 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795188733.277229    3010 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:32:26,536 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 0.9475 - mae_cameradirection: 0.9475 - val_loss: 0.8524 - val_mae_cameradirection: 0.8524 - time: 12.63s
2026-11-20 15:32:26,536 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.85238, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras
2026-11-20 15:32:26,943 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras
2026-11-20 15:32:35,495 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 0.9315 - mae_cameradirection: 0.9315 - val_loss: 0.5532 - val_mae_cameradirection: 0.5532 - time: 7.80s
2026-11-20 15:32:35,499 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.85238 to 0.55317, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras
2026-11-20 15:32:35,899 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras
W0000 00:00:1795188765.569988    3010 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:32:45,875 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 0.8603 - mae_cameradirection: 0.8603 - val_loss: 0.7401 - val_mae_cameradirection: 0.7401 - time: 9.29s
2026-11-20 15:32:45,875 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss did not improve from 0.55317
2026-11-20 15:32:54,407 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 0.8207 - mae_cameradirection: 0.8207 - val_loss: 0.5144 - val_mae_cameradirection: 0.5144 - time: 7.84s
2026-11-20 15:32:54,407 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss improved from 0.55317 to 0.51438, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras
2026-11-20 15:32:55,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn_model.keras
2026-11-20 15:33:04,524 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 0.8414 - mae_cameradirection: 0.8414 - val_loss: 0.6887 - val_mae_cameradirection: 0.6887 - time: 8.82s
2026-11-20 15:33:04,525 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss did not improve from 0.51438
2026-11-20 15:33:05,225 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:33:05,225 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/config.json
2026-11-20 15:33:05,226 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:33:05,228 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:33:05,228 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-cameradirection/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:33:05,229 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:33:05,229 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:33:05,229 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:33:05,229 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-cameradirection" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco cameradirection \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188789.297220    3567 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188789.334557    3567 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188790.339459    3567 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:33:11,654 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:33:11,654 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:33:11,654 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:33:11,655 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:33:11,655 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:33:11,656 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:33:11,656 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:33:11,657 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:33:11,657 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:33:12,516 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66
2026-11-20 15:33:12,516 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:33:12,542 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['cameradirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:33:12,542 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:33:12,568 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:33:27,266 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 0.9342 - mae_cameradirection: 0.9342 - val_loss: 0.7929 - val_mae_cameradirection: 0.7929 - time: 14.70s
2026-11-20 15:33:27,267 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.79288, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth
2026-11-20 15:33:27,312 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth
2026-11-20 15:33:41,992 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 0.8731 - mae_cameradirection: 0.8731 - val_loss: 0.7294 - val_mae_cameradirection: 0.7294 - time: 14.68s
2026-11-20 15:33:41,993 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.79288 to 0.72937, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth
2026-11-20 15:33:42,051 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn_model.pth
2026-11-20 15:33:56,401 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 0.8694 - mae_cameradirection: 0.8694 - val_loss: 0.7512 - val_mae_cameradirection: 0.7512 - time: 14.35s
2026-11-20 15:33:56,402 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss did not improve from 0.72937
2026-11-20 15:34:11,135 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 0.8294 - mae_cameradirection: 0.8294 - val_loss: 0.7304 - val_mae_cameradirection: 0.7304 - time: 14.73s
2026-11-20 15:34:11,136 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss did not improve from 0.72937
2026-11-20 15:34:25,791 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 0.8394 - mae_cameradirection: 0.8394 - val_loss: 0.7613 - val_mae_cameradirection: 0.7613 - time: 14.65s
2026-11-20 15:34:25,791 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss did not improve from 0.72937
2026-11-20 15:34:25,791 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:34:25,792 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/config.json
2026-11-20 15:34:25,792 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:34:25,793 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:34:25,793 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-cameradirection/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:34:25,794 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:34:25,794 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:34:25,795 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:34:25,795 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-skydirection" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco skydirection \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188867.354839    3590 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188867.392988    3590 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188868.242662    3590 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:34:30,574 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:34:30,575 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:34:30,575 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:34:30,576 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:34:30,576 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:34:30,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:34:31,468 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66
E0000 00:00:1795188871.469037    3590 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:34:31,487 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:34:31,507 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-skydirection'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['skydirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:34:31,507 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:34:32,025 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:34:32,028 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795188872.073649    3590 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795188872.092347    3590 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:34:45,328 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 1.8731 - mae_skydirection: 1.8731 - val_loss: 1.3860 - val_mae_skydirection: 1.3860 - time: 12.60s
2026-11-20 15:34:45,328 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 1.38603, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras
2026-11-20 15:34:45,729 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras
2026-11-20 15:34:54,155 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 1.7275 - mae_skydirection: 1.7275 - val_loss: 1.1071 - val_mae_skydirection: 1.1071 - time: 7.68s
2026-11-20 15:34:54,159 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 1.38603 to 1.10714, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras
2026-11-20 15:34:54,560 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn_model.keras
W0000 00:00:1795188902.662085    3590 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:35:02,955 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 1.7188 - mae_skydirection: 1.7188 - val_loss: 1.3278 - val_mae_skydirection: 1.3278 - time: 7.72s
2026-11-20 15:35:02,955 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss did not improve from 1.10714
2026-11-20 15:35:11,306 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 1.6788 - mae_skydirection: 1.6788 - val_loss: 1.2819 - val_mae_skydirection: 1.2819 - time: 7.68s
2026-11-20 15:35:11,307 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss did not improve from 1.10714
2026-11-20 15:35:19,709 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 1.7069 - mae_skydirection: 1.7069 - val_loss: 1.3523 - val_mae_skydirection: 1.3523 - time: 7.73s
2026-11-20 15:35:19,709 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss did not improve from 1.10714
2026-11-20 15:35:20,369 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:35:20,369 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/config.json
2026-11-20 15:35:20,370 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:35:20,371 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:35:20,371 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-skydirection/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:35:20,372 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:35:20,372 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:35:20,373 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:35:20,373 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-skydirection" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco skydirection \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188924.478621    4145 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188924.516704    4145 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188925.568311    4145 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:35:26,910 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:35:26,910 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:35:26,910 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:35:26,911 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:35:26,911 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:35:26,912 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:35:26,912 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:35:26,913 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:35:26,913 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:35:27,774 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66
2026-11-20 15:35:27,774 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:35:27,800 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['skydirection'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:35:27,800 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:35:27,825 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:35:40,583 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 1.7831 - mae_skydirection: 1.7831 - val_loss: 1.5204 - val_mae_skydirection: 1.5204 - time: 12.76s
2026-11-20 15:35:40,585 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 1.52043, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth
2026-11-20 15:35:40,631 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth
2026-11-20 15:35:53,407 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 1.7512 - mae_skydirection: 1.7512 - val_loss: 1.5101 - val_mae_skydirection: 1.5101 - time: 12.78s
2026-11-20 15:35:53,408 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 1.52043 to 1.51010, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth
2026-11-20 15:35:53,477 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth
2026-11-20 15:36:06,341 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 1.7094 - mae_skydirection: 1.7094 - val_loss: 1.5536 - val_mae_skydirection: 1.5536 - time: 12.86s
2026-11-20 15:36:06,341 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss did not improve from 1.51010
2026-11-20 15:36:19,772 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 1.7316 - mae_skydirection: 1.7316 - val_loss: 1.4704 - val_mae_skydirection: 1.4704 - time: 13.43s
2026-11-20 15:36:19,773 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss improved from 1.51010 to 1.47045, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth
2026-11-20 15:36:19,834 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn_model.pth
2026-11-20 15:36:32,237 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 1.7003 - mae_skydirection: 1.7003 - val_loss: 1.4875 - val_mae_skydirection: 1.4875 - time: 12.40s
2026-11-20 15:36:32,238 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss did not improve from 1.47045
2026-11-20 15:36:32,239 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:36:32,239 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/config.json
2026-11-20 15:36:32,239 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:36:32,241 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:36:32,241 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-skydirection/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:36:32,242 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:36:32,242 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:36:32,242 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:36:32,242 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

3. Primary particle type classification#

The model performs binary classification of the primary particle type, learning to distinguish between the two particle classes: signal (gamma-ray events) and background (proton events), based on the information contained in the camera image. The input directory containing the signal events can be specified using --TrainCTLearnModel.input_dir_signal or the shorthand --signal. One or more file patterns can be provided using --TrainCTLearnModel.file_pattern_signal or --pattern-signal. The corresponding options for the background events are --TrainCTLearnModel.input_dir_background / --background and --TrainCTLearnModel.file_pattern_background / --pattern-background.

It is recommended to approximately balance the training dataset before training. The training log reports the number of signal and background events remaining after the selection cuts, which can be used to check the class balance. If a significant imbalance is present, the input file patterns should be adjusted to include a more balanced selection of gamma-ray and proton events. CTLearn includes a standard, fixed mechanism that computes class weights and incorporates them into the loss calculation to compensate for differences in the number of signal and background events. However, class weighting is only effective up to a certain degree of class imbalance. It is therefore important to monitor the training carefully, particularly when the input samples are not well balanced. The training logs and learning curves should be inspected throughout the training. In particular, the evolution of metrics such as loss, accuracy, and AUC on a per-epoch basis can help identify potential issues arising from class imbalance or other training problems.

run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \\
        --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-background "proton_with_images_00.dl2.h*" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-classification" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco type \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 3 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \\
        --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-background "proton_with_images_00.dl2.h*" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-classification" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco type \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.n_epochs 3 \\
        --TrainCTLearnModel.batch_size 4""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \
        --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-background "proton_with_images_00.dl2.h*" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-classification" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco type \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 3 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188993.874845    4191 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795188993.914018    4191 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795188994.744648    4191 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:36:37,054 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:36:37,055 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:36:37,055 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:36:37,055 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:36:37,056 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:36:37,056 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:36:37,057 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:36:37,057 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:36:37,057 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:36:38,270 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 128
2026-11-20 15:36:38,271 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of signal events: 66
2026-11-20 15:36:38,271 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of background events: 62
E0000 00:00:1795188998.271826    4191 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:36:38,289 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:36:38,291 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['proton_with_images_00.dl2.h*'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h*'], 'input_dir_background': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 3, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-classification'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['type'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:36:38,292 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:36:38,822 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:36:38,824 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795188998.876173    4191 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795188998.896874    4191 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:36:59,558 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/3 - accuracy: 0.4483 - auc: 0.4579 - loss: 0.7442 - val_accuracy: 0.5833 - val_auc: 0.6597 - val_loss: 0.6686 - time: 19.99s
2026-11-20 15:36:59,558 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.66858, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras
2026-11-20 15:36:59,963 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras
W0000 00:00:1795189035.029267    4191 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
W0000 00:00:1795189035.094406    4405 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:37:15,617 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/3 - accuracy: 0.4483 - auc: 0.4715 - loss: 0.6968 - val_accuracy: 0.5000 - val_auc: 0.6111 - val_loss: 0.6810 - time: 14.91s
2026-11-20 15:37:15,622 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss did not improve from 0.66858
W0000 00:00:1795189055.449961    4504 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:37:35,995 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/3 - accuracy: 0.5345 - auc: 0.4872 - loss: 0.6937 - val_accuracy: 0.5833 - val_auc: 0.6146 - val_loss: 0.6616 - time: 19.70s
2026-11-20 15:37:35,995 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss improved from 0.66858 to 0.66156, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras
2026-11-20 15:37:36,372 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn_model.keras
2026-11-20 15:37:37,049 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:37:37,050 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-classification/config.json
2026-11-20 15:37:37,050 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:37:37,052 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:37:37,052 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-classification/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:37:37,053 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:37:37,053 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:37:37,053 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:37:37,053 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h*" \
        --background "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-background "proton_with_images_00.dl2.h*" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-classification" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco type \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.n_epochs 3 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189061.026898    4548 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189061.064465    4548 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189062.047537    4548 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:37:43,356 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:37:43,356 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:37:43,356 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:37:43,357 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:37:43,357 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:37:43,358 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:37:43,359 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:37:43,359 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:37:43,359 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:37:44,514 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 128
2026-11-20 15:37:44,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of signal events: 66
2026-11-20 15:37:44,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of background events: 62
2026-11-20 15:37:44,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:37:44,524 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['proton_with_images_00.dl2.h*'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h*'], 'input_dir_background': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 3, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-classification'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['type'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:37:44,524 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:37:44,551 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:38:13,943 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/3 - loss: 0.6957 - accuracy: 0.4741 - auc: 0.4346 - val_loss: 0.6749 - val_accuracy: 0.5833 - val_auc: 0.5143 - time: 29.39s
2026-11-20 15:38:13,944 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.67488, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth
2026-11-20 15:38:13,990 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth
2026-11-20 15:38:38,875 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/3 - loss: 0.6919 - accuracy: 0.4741 - auc: 0.5469 - val_loss: 0.6676 - val_accuracy: 0.5833 - val_auc: 0.6286 - time: 24.88s
2026-11-20 15:38:38,876 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.67488 to 0.66761, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth
2026-11-20 15:38:38,935 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth
2026-11-20 15:39:08,162 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/3 - loss: 0.6943 - accuracy: 0.4741 - auc: 0.4888 - val_loss: 0.6453 - val_accuracy: 0.5833 - val_auc: 0.6286 - time: 29.23s
2026-11-20 15:39:08,163 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss improved from 0.66761 to 0.64530, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth
2026-11-20 15:39:08,225 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn_model.pth
2026-11-20 15:39:08,226 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:39:08,226 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/config.json
2026-11-20 15:39:08,226 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:39:08,228 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:39:08,228 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-classification/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:39:08,229 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:39:08,229 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:39:08,229 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:39:08,229 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

Part B2 Customizing the CTLearn model architecture#

CTLearn provides a flexible configuration system that allows users to customize the neural network architecture.

Two main CNN-based architectures are currently implemented: SingleCNN and ResNet, which can be set by --TrainCTLearnModel.model_type.

1. SingleCNN model#

The SingleCNN component implements a plain convolutional neural network consisting of configurable convolutional layers. It can be used to construct very simple and lightweight models, but it can also be configured to create deeper architectures, such as VGG-like networks. This architecture represents the more legacy model structure in CTLearn and was the starting point for early proof-of-concept studies exploring the applicability of CNNs to the problem. For applications closer to the camera hardware and its readout (for the trigger system), particularly where predictions need to be made at very high rates and with very low latency, computational efficiency becomes increasingly important.

run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.model_type SingleCNN \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.model_type SingleCNN \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 4""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.model_type SingleCNN \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189149.910868    4635 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189149.949887    4635 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189150.814094    4635 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:39:13,209 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]'
2026-11-20 15:39:13,209 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:13,210 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:39:13,210 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:13,210 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:39:13,211 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:13,211 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json
2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:39:13,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:39:14,128 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66
E0000 00:00:1795189154.129317    4635 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:39:14,148 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:39:14,172 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'SingleCNN', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:39:14,173 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:39:14,266 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:39:14,268 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795189154.505340    4635 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795189154.525850    4635 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:39:16,087 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.5540 - mae_energy: 0.5540 - val_loss: 0.4468 - val_mae_energy: 0.4468 - time: 1.39s
2026-11-20 15:39:16,087 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.44676, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras
2026-11-20 15:39:16,153 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras
2026-11-20 15:39:16,754 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.5435 - mae_energy: 0.5435 - val_loss: 0.2468 - val_mae_energy: 0.2468 - time: 0.45s
2026-11-20 15:39:16,758 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.44676 to 0.24676, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras
2026-11-20 15:39:16,811 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn_model.keras
2026-11-20 15:39:16,891 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:39:16,892 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/config.json
2026-11-20 15:39:16,893 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:39:16,894 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:39:16,894 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-singlecnn-energy/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:39:16,895 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:39:16,895 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:39:16,896 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:39:16,896 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.model_type SingleCNN \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189160.908871    4890 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189160.947486    4890 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189161.991651    4890 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:39:23,339 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]'
2026-11-20 15:39:23,340 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:23,340 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:39:23,341 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:23,341 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:39:23,341 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:23,342 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json
2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:39:23,343 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:39:24,196 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66
2026-11-20 15:39:24,197 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:39:24,224 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'SingleCNN', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:39:24,224 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:39:24,227 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:39:24,632 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.5635 - mae_energy: 0.5635 - val_loss: 0.3236 - val_mae_energy: 0.3236 - time: 0.40s
2026-11-20 15:39:24,633 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32361, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/ctlearn_model.pth
2026-11-20 15:39:24,637 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/ctlearn_model.pth
2026-11-20 15:39:25,011 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.5444 - mae_energy: 0.5444 - val_loss: 0.3252 - val_mae_energy: 0.3252 - time: 0.37s
2026-11-20 15:39:25,011 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.32361
2026-11-20 15:39:25,012 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:39:25,012 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/config.json
2026-11-20 15:39:25,012 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:39:25,013 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:39:25,013 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-singlecnn-energy/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:39:25,014 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:39:25,014 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:39:25,015 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:39:25,015 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

2. ResNet model#

The ResNet component provides the currently preferred architecture for constructing deeper CNN models. Residual networks allow substantially deeper architectures while maintaining efficient training and inference, and they generally provide improved performance compared with the simpler plain CNN architecture. The architecture, number of layers or blocks, number of filters, and other relevant parameters can therefore be configured directly through the CTLearn configuration system, allowing the model complexity to be adapted to the requirements of a particular application. For ResNet architectures, CTLearn currently supports two types of residual blocks:

  • Basic blocks, which use the simpler residual block structure and are suitable for relatively lightweight networks.

  • Bottleneck blocks, which use a more compact sequence of convolutions to enable deeper networks while controlling the computational cost.

run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-resnet-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.model_type ResNet \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-resnet-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.model_type ResNet \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 4""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-resnet-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.model_type ResNet \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189166.652839    4907 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189166.691209    4907 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189167.581765    4907 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:39:30,012 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]'
2026-11-20 15:39:30,012 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:30,012 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:39:30,013 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:30,013 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:39:30,014 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:39:30,014 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json
2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:39:30,015 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:39:30,917 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66
E0000 00:00:1795189170.917994    4907 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:39:30,935 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:39:30,958 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:39:30,958 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:39:31,673 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:39:31,676 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795189171.723895    4907 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795189171.743516    4907 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
W0000 00:00:1795189203.126112    4907 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:40:03,143 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.5939 - mae_energy: 0.5939 - val_loss: 0.3719 - val_mae_energy: 0.3719 - time: 30.50s
2026-11-20 15:40:03,143 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.37188, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras
2026-11-20 15:40:03,989 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras
2026-11-20 15:40:29,013 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.5175 - mae_energy: 0.5175 - val_loss: 0.3100 - val_mae_energy: 0.3100 - time: 23.13s
2026-11-20 15:40:29,018 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.37188 to 0.30996, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras
2026-11-20 15:40:29,907 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn_model.keras
2026-11-20 15:40:31,735 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:40:31,735 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/config.json
2026-11-20 15:40:31,736 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:40:31,737 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:40:31,737 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-resnet-energy/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:40:31,738 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:40:31,738 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:40:31,739 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:40:31,739 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-resnet-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.model_type ResNet \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189236.077512    5173 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189236.114719    5173 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189237.146097    5173 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:40:38,490 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]'
2026-11-20 15:40:38,490 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:40:38,490 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:40:38,491 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:40:38,491 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:40:38,492 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:40:38,492 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json
2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:40:38,493 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:40:39,369 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66
2026-11-20 15:40:39,369 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:40:39,398 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:40:39,398 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:40:39,499 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:41:11,969 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.5461 - mae_energy: 0.5461 - val_loss: 0.3256 - val_mae_energy: 0.3256 - time: 32.47s
2026-11-20 15:41:11,970 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32561, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth
2026-11-20 15:41:12,142 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth
2026-11-20 15:41:44,087 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.5275 - mae_energy: 0.5275 - val_loss: 0.3250 - val_mae_energy: 0.3250 - time: 31.94s
2026-11-20 15:41:44,089 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.32561 to 0.32498, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth
2026-11-20 15:41:44,328 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn_model.pth
2026-11-20 15:41:44,328 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:41:44,329 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/config.json
2026-11-20 15:41:44,329 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:41:44,331 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:41:44,331 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-resnet-energy/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:41:44,332 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:41:44,332 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:41:44,332 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:41:44,332 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

Part C2 Using a pre-trained model and transfer learning#

The LoadedModel component allows a previously trained model following the CTLearnModel structure to be loaded for either resuming training or transfer learning. Transfer learning reuses the learned features of an existing model and adapts them to new, but related, data or tasks. This can reduce training time compared with training a model from scratch.

Two parameters are particularly important:

  • --LoadedModel.trainable_backbone controls whether the pre-trained backbone is frozen (false) or trainable (true).

  • --LoadedModel.overwrite_head controls whether the structure of the original prediction head is reused (false) or replaced with a new MLP tailored to the new task or data (true).

The preceding SingleCNN command trains the model loaded below. The copied configuration keeps its prediction head and leaves the backbone trainable.

run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-transfer-learning-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.model_type LoadedModel \\
        --LoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy/ctlearn_model.keras" \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 4""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-transfer-learning-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \\
        --reco energy \\
        --DLImageReader.focal_length_choice EQUIVALENT \\
        --TrainCTLearnModel.model_type LoadedModel \\
        --PyTorchLoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy/ctlearn_model.pth" \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 4""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-transfer-learning-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.model_type LoadedModel \
        --LoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-singlecnn-energy/ctlearn_model.keras" \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189306.139857    5194 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189306.178179    5194 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189307.061421    5194 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:41:49,494 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]'
2026-11-20 15:41:49,495 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:41:49,495 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:41:49,496 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json
2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:41:49,497 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:41:50,407 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 66
E0000 00:00:1795189310.408506    5194 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:41:50,427 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:41:50,450 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'LoadedModel', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:41:50,451 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:41:50,868 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:41:50,869 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795189310.917049    5194 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795189310.937757    5194 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:41:52,467 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.5428 - mae_energy: 0.5428 - val_loss: 0.3672 - val_mae_energy: 0.3672 - time: 1.39s
2026-11-20 15:41:52,467 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.36716, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras
2026-11-20 15:41:52,534 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras
2026-11-20 15:41:53,153 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.5223 - mae_energy: 0.5223 - val_loss: 0.2474 - val_mae_energy: 0.2474 - time: 0.46s
2026-11-20 15:41:53,157 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.36716 to 0.24735, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras
2026-11-20 15:41:53,212 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn_model.keras
2026-11-20 15:41:53,294 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:41:53,294 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/config.json
2026-11-20 15:41:53,295 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:41:53,296 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:41:53,296 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-transfer-learning-energy/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:41:53,297 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:41:53,297 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:41:53,298 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:41:53,298 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-transfer-learning-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_models_example_config.json" \
        --reco energy \
        --DLImageReader.focal_length_choice EQUIVALENT \
        --TrainCTLearnModel.model_type LoadedModel \
        --PyTorchLoadedModel.load_model_from "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-singlecnn-energy/ctlearn_model.pth" \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 4
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189317.386792    5449 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189317.425816    5449 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189318.497509    5449 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:41:59,862 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')]'
2026-11-20 15:41:59,863 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:41:59,863 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:41:59,863 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:41:59,864 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:41:59,864 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:41:59,864 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json
2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:41:59,866 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:42:00,761 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 66
2026-11-20 15:42:00,762 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:42:00,789 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 4, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_models_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'LoadedModel', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': [['> 1000 phe', 'hillas_intensity > 1000']]}}}}
2026-11-20 15:42:00,790 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:42:00,793 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:42:01,189 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.5515 - mae_energy: 0.5515 - val_loss: 0.3242 - val_mae_energy: 0.3242 - time: 0.40s
2026-11-20 15:42:01,190 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.32422, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/ctlearn_model.pth
2026-11-20 15:42:01,194 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/ctlearn_model.pth
2026-11-20 15:42:01,571 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.5290 - mae_energy: 0.5290 - val_loss: 0.3416 - val_mae_energy: 0.3416 - time: 0.38s
2026-11-20 15:42:01,571 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.32422
2026-11-20 15:42:01,571 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:42:01,572 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/config.json
2026-11-20 15:42:01,572 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:42:01,573 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:42:01,573 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-transfer-learning-energy/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:42:01,574 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:42:01,574 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:42:01,575 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:42:01,575 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

Part D2 Stereoscopic CNN-based model training#

CTLearn currently provides a basic approach for stereoscopic CNN-based model training. The images from the different telescopes are concatenated channel-wise and then passed to a standard, monoscopic CNN backbone as above. In this way, the network can process the combined input while retaining stereoscopic information. An earlier version of CTLearn also included a CNN-RNN architecture, using recurrent neural networks to explicitly handle information from multiple telescopes dynamically. This model has not yet been migrated to the latest CTLearn version based on TensorFlow 2 and Keras 3, so this tutorial focuses on the channel-wise concatenation approach.

The main configuration parameters for stereoscopic training are:

  • --DLImageReader.mode: set to “stereo” to enable stereoscopic image reading

  • --DLImageReader.min_telescopes: defines the minimum number of telescopes that must participate in an event after the quality cuts.

  • --TrainCTLearnModel.stack_telescope_images: set to true to stack/concatenate the images from the telescopes.

Since the resulting input contains information from multiple telescopes and is therefore larger than the corresponding monoscopic input, it is recommended to reduce the batch size using --TrainCTLearnModel.batch_size to account for the increased memory consumption.

run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-stereo-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco energy \\
        --DLImageReader.mode stereo \\
        --DLImageReader.min_telescopes 2 \\
        --TrainCTLearnModel.stack_telescope_images True \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 2 \\
        --TableQualityQuery.quality_criteria []""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \\
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-stereo-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --reco energy \\
        --DLImageReader.mode stereo \\
        --DLImageReader.min_telescopes 2 \\
        --TrainCTLearnModel.stack_telescope_images True \\
        --TrainCTLearnModel.n_epochs 5 \\
        --TrainCTLearnModel.batch_size 2 \\
        --TableQualityQuery.quality_criteria []""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-stereo-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco energy \
        --DLImageReader.mode stereo \
        --DLImageReader.min_telescopes 2 \
        --TrainCTLearnModel.stack_telescope_images True \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 2 \
        --TableQualityQuery.quality_criteria []
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189323.382390    5468 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189323.421272    5468 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189324.318563    5468 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:42:06,779 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:42:06,779 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:42:06,779 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:42:06,780 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:42:06,780 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:42:06,781 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:42:07,771 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 135
E0000 00:00:1795189327.771888    5468 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:42:07,790 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:42:07,813 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': True, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EFFECTIVE: 1>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 2, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'stereo', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}}
2026-11-20 15:42:07,814 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:42:08,362 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:42:08,364 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795189328.438136    5468 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795189328.461709    5468 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
W0000 00:00:1795189350.058643    5581 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:42:31,327 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/5 - loss: 0.8730 - mae_energy: 0.8730 - val_loss: 0.9720 - val_mae_energy: 0.9720 - time: 22.17s
2026-11-20 15:42:31,327 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.97196, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:42:31,734 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
W0000 00:00:1795189368.487917    5468 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
W0000 00:00:1795189368.561814    5681 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:42:49,178 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/5 - loss: 0.8948 - mae_energy: 0.8948 - val_loss: 0.8049 - val_mae_energy: 0.8049 - time: 16.72s
2026-11-20 15:42:49,183 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.97196 to 0.80491, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:42:49,573 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:43:07,035 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 3/5 - loss: 0.8224 - mae_energy: 0.8224 - val_loss: 1.1420 - val_mae_energy: 1.1420 - time: 16.81s
2026-11-20 15:43:07,036 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 3: val_loss did not improve from 0.80491
W0000 00:00:1795189404.185386    5468 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
W0000 00:00:1795189404.260948    5880 local_rendezvous.cc:412] Local rendezvous is aborting with status: CANCELLED: GetNextFromShard was cancelled
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-11-20 15:43:25,079 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 4/5 - loss: 0.8588 - mae_energy: 0.8588 - val_loss: 0.7744 - val_mae_energy: 0.7744 - time: 17.38s
2026-11-20 15:43:25,079 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: val_loss improved from 0.80491 to 0.77435, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:43:25,470 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:43:43,480 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 5/5 - loss: 0.8052 - mae_energy: 0.8052 - val_loss: 0.7101 - val_mae_energy: 0.7101 - time: 17.34s
2026-11-20 15:43:43,481 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: val_loss improved from 0.77435 to 0.71013, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:43:43,877 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 5: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn_model.keras
2026-11-20 15:43:44,576 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:43:44,577 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/config.json
2026-11-20 15:43:44,578 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:43:44,579 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:43:44,579 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-stereo-energy/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:43:44,580 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:43:44,580 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:43:44,581 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:43:44,581 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_CONVERTED_DIR" \
        --pattern-signal "gamma-diffuse_with_images_00.dl2.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-stereo-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --reco energy \
        --DLImageReader.mode stereo \
        --DLImageReader.min_telescopes 2 \
        --TrainCTLearnModel.stack_telescope_images True \
        --TrainCTLearnModel.n_epochs 5 \
        --TrainCTLearnModel.batch_size 2 \
        --TableQualityQuery.quality_criteria []
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189429.066331    6026 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189429.105682    6026 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189430.172168    6026 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:43:51,554 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')]'
2026-11-20 15:43:51,555 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:43:51,555 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:43:51,555 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:43:51,556 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:43:51,557 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.33.0' [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
WARNING: MergeConflictWarning: Cannot merge meta key 'CTAPIPE_VERSION' types <class 'str'> and <class 'str'>, choosing CTAPIPE_VERSION='0.17.0' [astropy.utils.metadata.merge]
2026-11-20 15:43:52,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 135
2026-11-20 15:43:52,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:43:52,523 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json')], 'dl1dh_reader_type': 'DLImageReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma-diffuse_with_images_00.dl2.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data/ctapipe-0.33.0-max500'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 5, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': True, 'validation_split': 0.1, 'DLImageReader': {'allowed_tel_types': ['LST_LST_LSTCam'], 'allowed_tels': None, 'channels': ['image', 'relative_peak_time'], 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EFFECTIVE: 1>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 2, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'stereo', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}}
2026-11-20 15:43:52,524 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:43:52,552 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:44:17,944 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/5 - loss: 0.9399 - mae_energy: 0.9399 - val_loss: 0.7731 - val_mae_energy: 0.7731 - time: 25.39s
2026-11-20 15:44:17,945 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.77312, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth
2026-11-20 15:44:17,993 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth
2026-11-20 15:44:44,069 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/5 - loss: 0.8843 - mae_energy: 0.8843 - val_loss: 0.8150 - val_mae_energy: 0.8150 - time: 26.08s
2026-11-20 15:44:44,070 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss did not improve from 0.77312
2026-11-20 15:45:08,514 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3/5 - loss: 0.8385 - mae_energy: 0.8385 - val_loss: 0.7800 - val_mae_energy: 0.7800 - time: 24.44s
2026-11-20 15:45:08,515 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 3: val_loss did not improve from 0.77312
2026-11-20 15:45:32,097 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4/5 - loss: 0.8770 - mae_energy: 0.8770 - val_loss: 0.7547 - val_mae_energy: 0.7547 - time: 23.58s
2026-11-20 15:45:32,099 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: val_loss improved from 0.77312 to 0.75467, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth
2026-11-20 15:45:32,158 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 4: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth
2026-11-20 15:45:55,636 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5/5 - loss: 0.8275 - mae_energy: 0.8275 - val_loss: 0.7177 - val_mae_energy: 0.7177 - time: 23.48s
2026-11-20 15:45:55,637 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: val_loss improved from 0.75467 to 0.71768, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth
2026-11-20 15:45:55,697 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 5: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn_model.pth
2026-11-20 15:45:55,697 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:45:55,697 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/config.json
2026-11-20 15:45:55,698 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:45:55,699 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:45:55,699 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-stereo-energy/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:45:55,700 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:45:55,700 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:45:55,701 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:45:55,701 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

Part E2 CNN-based model training using waveform data#

The final example demonstrates training directly on waveform data. The example dataset used in this tutorial is intentionally very small, so only a few events are available for training. In practical applications, waveform datasets can be very large, often reaching several terabytes, and processing the full waveforms also requires substantial RAM and computational resources. Because the example dataset is so small, the training configuration needs to be adjusted accordingly. We use a very small batch size and increase the validation split to ensure that at least one batch is available for validation. For this example, we select the MST equipped with the NectarCam camera and disable the quality selection, leaving only 10 events. These are divided into 5 batches of 2 events, with 20% of the data reserved for validation, corresponding to one validation batch.

An important configuration change is required to read waveform data: the --TrainCTLearnModel.dl1dh_reader_type option must be set to the appropriate DL1 data reader component for waveform reading (DLWaveformReader) rather than the default image reader.

if not WAVEFORM_FILE.is_file():
    raise FileNotFoundError(
        f"Cached waveform data is missing: {WAVEFORM_FILE}. "
        "Run explore_public_data.py first."
    )
run_command_line(
    """ctlearn-train-keras-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_DATA_DIR" \\
        --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-waveform-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --reco energy \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 2 \\
        --TrainCTLearnModel.validation_split 0.2 \\
        --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \\
        --DLWaveformReader.focal_length_choice EQUIVALENT \\
        --TableQualityQuery.quality_criteria []""",
)

run_command_line(
    """ctlearn-train-pytorch-model --verbose \\
        --signal "$CTLEARN_TUTORIAL_DATA_DIR" \\
        --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \\
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-waveform-energy" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \\
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \\
        --reco energy \\
        --TrainCTLearnModel.n_epochs 2 \\
        --TrainCTLearnModel.batch_size 2 \\
        --TrainCTLearnModel.validation_split 0.2 \\
        --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \\
        --DLWaveformReader.focal_length_choice EQUIVALENT \\
        --TableQualityQuery.quality_criteria []""",
)
Running command: $ ctlearn-train-keras-model --verbose \
        --signal "$CTLEARN_TUTORIAL_DATA_DIR" \
        --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/keras-waveform-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --reco energy \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 2 \
        --TrainCTLearnModel.validation_split 0.2 \
        --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \
        --DLWaveformReader.focal_length_choice EQUIVALENT \
        --TableQualityQuery.quality_criteria []
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189557.387919    6061 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189557.426020    6061 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189558.272849    6061 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:46:00,590 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')]'
2026-11-20 15:46:00,590 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:46:00,590 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:46:00,591 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:46:00,591 DEBUG [ctlearn.ctlearn-train-keras-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Loading data:
2026-11-20 15:46:00,592 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
2026-11-20 15:46:01,427 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup): Number of events loaded: 10
E0000 00:00:1795189561.428563    6061 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
2026-11-20 15:46:01,446 INFO [ctlearn.ctlearn-train-keras-model] (train_model.setup_framework): Number of devices: 1
2026-11-20 15:46:01,467 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.run): CONFIG: {'TrainCTLearnKerasModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')], 'dl1dh_reader_type': 'DLWaveformReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma_prod5-max500.r1.dl1.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy'), 'overwrite': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.2, 'DLWaveformReader': {'DBSCAN_params': {'eps': 0.5, 'min_samples': 5, 'metric': 'euclidean'}, 'allowed_tel_types': None, 'allowed_tels': {109, 112, 118, 119, 121, 122, 124}, 'cleaning_type': None, 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'sequence_length': 5, 'sequence_position': 'center', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}}
2026-11-20 15:46:01,468 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Setting up the Keras model.
2026-11-20 15:46:01,992 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Compiling CTLearn model.
2026-11-20 15:46:01,994 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating...
I0000 00:00:1795189562.050899    6061 generator_dataset_op.cc:213] Memory patch applied: M_TRIM_THRESHOLD=128 kb was set.
W0000 00:00:1795189562.070053    6061 dataset.cc:997] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2026-11-20 15:46:08,816 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 1/2 - loss: 0.2179 - mae_energy: 0.2179 - val_loss: 0.1557 - val_mae_energy: 0.1557 - time: 6.12s
2026-11-20 15:46:08,816 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: val_loss improved from None to 0.15568, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras
2026-11-20 15:46:09,216 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras
2026-11-20 15:46:11,278 INFO [ctlearn.ctlearn-train-keras-model] (train_model._on_epoch_end): Epoch 2/2 - loss: 0.0914 - mae_energy: 0.0914 - val_loss: 0.0805 - val_mae_energy: 0.0805 - time: 1.34s
2026-11-20 15:46:11,282 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: val_loss improved from 0.15568 to 0.08048, saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras
2026-11-20 15:46:11,678 INFO [ctlearn.ctlearn-train-keras-model] (train_model.on_epoch_end): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn_model.keras
2026-11-20 15:46:12,343 INFO [ctlearn.ctlearn-train-keras-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:46:12,343 INFO [ctlearn.ctlearn-train-keras-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/config.json
2026-11-20 15:46:12,344 INFO [ctlearn.ctlearn-train-keras-model] (train_model.finish): Tool is shutting down
2026-11-20 15:46:12,345 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:46:12,345 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/keras-waveform-energy/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:46:12,346 INFO [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): Output:
2026-11-20 15:46:12,346 DEBUG [ctlearn.ctlearn-train-keras-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-keras-model.provenance.log'
2026-11-20 15:46:12,347 INFO [ctlearn.ctlearn-train-keras-model] (tool.run): Finished ctlearn-train-keras-model
2026-11-20 15:46:12,347 DEBUG [ctlearn.ctlearn-train-keras-model] (application.exit): Exiting application: ctlearn-train-keras-model
Running command: $ ctlearn-train-pytorch-model --verbose \
        --signal "$CTLEARN_TUTORIAL_DATA_DIR" \
        --pattern-signal "gamma_prod5${CTLEARN_TUTORIAL_MAX_EVENTS_SUFFIX}.r1.dl1.h5" \
        --output "$CTLEARN_TUTORIAL_OUTPUT_DIR/pytorch-waveform-energy" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/ctlearn_train_model_example_config.json" \
        --config "$CTLEARN_TUTORIAL_CONFIG_DIR/dl1dh_example_config.json" \
        --reco energy \
        --TrainCTLearnModel.n_epochs 2 \
        --TrainCTLearnModel.batch_size 2 \
        --TrainCTLearnModel.validation_split 0.2 \
        --TrainCTLearnModel.dl1dh_reader_type DLWaveformReader \
        --DLWaveformReader.focal_length_choice EQUIVALENT \
        --TableQualityQuery.quality_criteria []
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189576.306473    6316 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1795189576.346576    6316 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 AMX_FP16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1795189577.345991    6316 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/ctapipe/core/traits.py:211: NoneDefaultNotAllowedWarning: Setting default_value=None and allow_none will result in misleading error messages. To require an value, set allow_none=True, default_value=None and check later that the value is not None
  warnings.warn(msg, NoneDefaultNotAllowedWarning)
2026-11-20 15:46:18,660 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): Loading config from '[PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')]'
2026-11-20 15:46:18,660 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:46:18,660 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json
2026-11-20 15:46:18,661 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Looking for /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config in /home/runner/work/ctlearn/ctlearn/examples/tutorials
2026-11-20 15:46:18,661 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application._load_config_files): Loaded config file: /home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json
2026-11-20 15:46:18,662 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.initialize): ctapipe version 0.33.0
2026-11-20 15:46:18,662 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): ctlearn version 0.11.2.dev2+g312165f28
2026-11-20 15:46:18,663 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Loading data:
2026-11-20 15:46:18,663 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): For a large dataset, this may take a while...
WARNING: MergeConflictWarning: In merged column 'obs_id' the 'description' attribute does not match (Observation Block ID != Simulation Run Identifier).  Using Simulation Run Identifier for merged output [astropy.utils.metadata.merge]
2026-11-20 15:46:19,488 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup): Number of events loaded: 10
2026-11-20 15:46:19,489 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.setup_framework): Using CPU device.
2026-11-20 15:46:19,513 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.run): CONFIG: {'TrainCTLearnPyTorchModel': {'batch_size': 2, 'config_files': [PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/ctlearn_train_model_example_config.json'), PosixPath('/home/runner/work/ctlearn/ctlearn/docs/source/config/dl1dh_example_config.json')], 'dl1dh_reader_type': 'DLWaveformReader', 'early_stopping': None, 'file_pattern_background': ['*.h5'], 'file_pattern_signal': ['gamma_prod5-max500.r1.dl1.h5'], 'input_dir_background': None, 'input_dir_signal': PosixPath('/home/runner/.cache/ctlearn/tutorial-data'), 'log_config': {}, 'log_datefmt': '%Y-%m-%d %H:%M:%S', 'log_file': None, 'log_file_level': 'INFO', 'log_format': '[%(name)s]%(highlevel)s %(message)s', 'log_level': 10, 'logging_config': {}, 'lr_reducing': {'factor': 0.5, 'patience': 5, 'min_delta': 0.01, 'min_lr': 1e-06}, 'model_type': 'ResNet', 'n_epochs': 2, 'num_workers': 0, 'optimizer': {'name': 'Adam', 'base_learning_rate': 0.0001, 'adam_epsilon': 1e-08}, 'output_dir': PosixPath('/home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy'), 'overwrite': False, 'persistent_workers': False, 'provenance_log': PosixPath('/home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'), 'quiet': False, 'random_seed': 1234, 'reco_tasks': ['energy'], 'save_best_validation_only': True, 'show_config': False, 'show_config_json': False, 'sort_by_intensity': False, 'stack_telescope_images': False, 'validation_split': 0.2, 'DLWaveformReader': {'DBSCAN_params': {'eps': 0.5, 'min_samples': 5, 'metric': 'euclidean'}, 'allowed_tel_types': None, 'allowed_tels': {109, 112, 118, 119, 121, 122, 124}, 'cleaning_type': None, 'enforce_subarray_equality': True, 'focal_length_choice': <FocalLengthKind.EQUIVALENT: 2>, 'force_dl1_lookup': False, 'image_mapper_type': [('type', '*', 'BilinearMapper')], 'min_telescopes': 1, 'min_telescopes_of_type': [('type', '*', 0)], 'mode': 'mono', 'sequence_length': 5, 'sequence_position': 'center', 'skip_incompatible_files': False, 'TableQualityQuery': {'passed_all_column_name': 'PASSED_ALL', 'quality_criteria': []}}}}
2026-11-20 15:46:19,513 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Setting up the PyTorch model.
2026-11-20 15:46:19,538 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating...
2026-11-20 15:46:21,373 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1/2 - loss: 0.2137 - mae_energy: 0.2137 - val_loss: 0.2064 - val_mae_energy: 0.2064 - time: 1.84s
2026-11-20 15:46:21,375 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: val_loss improved from None to 0.20635, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth
2026-11-20 15:46:21,419 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 1: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth
2026-11-20 15:46:23,023 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2/2 - loss: 0.2015 - mae_energy: 0.2015 - val_loss: 0.1923 - val_mae_energy: 0.1923 - time: 1.60s
2026-11-20 15:46:23,024 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: val_loss improved from 0.20635 to 0.19226, saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth
2026-11-20 15:46:23,083 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Epoch 2: finished saving model to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn_model.pth
2026-11-20 15:46:23,084 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.start): Training and evaluating finished successfully!
2026-11-20 15:46:23,084 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model._write_config_file): Writing configuration file to /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/config.json
2026-11-20 15:46:23,085 INFO [ctlearn.ctlearn-train-pytorch-model] (train_model.finish): Tool is shutting down
2026-11-20 15:46:23,086 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:46:23,086 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/.cache/ctlearn/tutorial-training/pytorch-waveform-energy/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:46:23,087 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): Output:
2026-11-20 15:46:23,087 DEBUG [ctlearn.ctlearn-train-pytorch-model] (tool.write_provenance): PROVENANCE: 'Details about provenance is found in /home/runner/work/ctlearn/ctlearn/examples/tutorials/ctlearn-train-pytorch-model.provenance.log'
2026-11-20 15:46:23,087 INFO [ctlearn.ctlearn-train-pytorch-model] (tool.run): Finished ctlearn-train-pytorch-model
2026-11-20 15:46:23,087 DEBUG [ctlearn.ctlearn-train-pytorch-model] (application.exit): Exiting application: ctlearn-train-pytorch-model

Part F2 Comparing training curves#

Each training run writes its metrics to training_log.csv together with the TensorBoard model checkpoints for training and validation, the final model, the configuration and provenance files.

See Part G for instructions on how to monitor the training runs live using TensorBoard. Here, we read the training logs and plot the evolution of the metrics over the training epochs. The plots show the mean absolute error (MAE) for the regression tasks and the accuracy and AUC for the classification task.

Note: The statistics of the training runs are intentionally very limited, so the training curves are not expected to be smooth or to converge to optimal values. The purpose of this tutorial is to demonstrate the training workflow and the use of CTLearn, rather than to achieve optimal performance on the example dataset.

training_logs = {}
for run_name in (
    "keras-energy",
    "pytorch-energy",
    "keras-cameradirection",
    "pytorch-cameradirection",
    "keras-skydirection",
    "pytorch-skydirection",
    "keras-classification",
    "pytorch-classification",
):
    log_path = OUTPUT_DIR / run_name / "training_log.csv"
    if not log_path.is_file():
        raise FileNotFoundError(f"Training log is missing: {log_path}")
    training_logs[run_name] = pd.read_csv(log_path)

frameworks = (
    ("Keras", "keras", "tab:blue"),
    ("PyTorch", "pytorch", "tab:orange"),
)
splits = (("", "train", "-"), ("val_", "validation", "--"))


def plot_metric(ax, run_name, metric, label, color, **kwargs):
    log = training_logs[run_name]
    for prefix, split_name, split_style in splits:
        column = f"{prefix}{metric}"
        if column not in log:
            raise ValueError(
                f"Column {column!r} is missing from {run_name}/training_log.csv"
            )
        ax.plot(
            log["epoch"] + 1,
            log[column],
            color=color,
            linestyle=split_style,
            label=f"{label} - {split_name}",
            **kwargs,
        )


fig, (energy_ax, camera_ax, sky_ax, type_ax) = plt.subplots(
    1, 4, figsize=(24, 6), constrained_layout=True
)
fig.suptitle("CTLearn training curves", fontsize=16)

for display_name, framework, color in frameworks:
    plot_metric(
        energy_ax,
        f"{framework}-energy",
        "mae_energy",
        display_name,
        color,
        marker="o",
    )
energy_ax.set_title("Energy")
energy_ax.set_xlabel("Epoch")
energy_ax.set_ylabel("Mean absolute error")
energy_ax.grid(True, alpha=0.3)
energy_ax.legend(fontsize=8)

for display_name, framework, color in frameworks:
    plot_metric(
        camera_ax,
        f"{framework}-cameradirection",
        "mae_cameradirection",
        display_name,
        color,
        marker="o",
    )
    plot_metric(
        sky_ax,
        f"{framework}-skydirection",
        "mae_skydirection",
        display_name,
        color,
        marker="s",
    )
camera_ax.set_title("Camera direction")
camera_ax.set_xlabel("Epoch")
camera_ax.set_ylabel("Mean absolute error [m]")
camera_ax.grid(True, alpha=0.3)
camera_ax.legend(fontsize=8)
sky_ax.set_title("Sky direction")
sky_ax.set_xlabel("Epoch")
sky_ax.set_ylabel("Sky direction MAE [deg]")
sky_ax.grid(True, alpha=0.3)
sky_ax.legend(fontsize=8)

for display_name, framework, color in frameworks:
    for metric, marker in (("accuracy", "o"), ("auc", "s")):
        plot_metric(
            type_ax,
            f"{framework}-classification",
            metric,
            f"{display_name} {metric}",
            color,
            marker=marker,
        )
type_ax.set_title("Particle type")
type_ax.set_xlabel("Epoch")
type_ax.set_ylabel("Score")
type_ax.set_ylim(0.4, 0.7)
type_ax.grid(True, alpha=0.3)
type_ax.legend(fontsize=8, ncol=2, loc="best")

training_curves_path = OUTPUT_DIR / "training-curves.png"
fig.savefig(training_curves_path, dpi=350)
print(f"\nTraining curves saved to {training_curves_path}")
plt.show()
CTLearn training curves, Energy, Camera direction, Sky direction, Particle type
Training curves saved to /home/runner/.cache/ctlearn/tutorial-training/training-curves.png

Part G2 Monitoring training with TensorBoard#

CTLearn also writes TensorBoard event files, which are useful for monitoring metrics live or exploring additional details after training. The tutorial does not launch TensorBoard; to monitor a run while it is training, start it in a separate terminal and point it at the tutorial output directory:

tensorboard --logdir "$HOME/.cache/ctlearn/tutorial-training/"

Open the URL printed by TensorBoard in a browser. If you set CTLEARN_TUTORIAL_OUTPUT_DIR to a different location, use that directory in the command instead. On a remote or cluster environment, use the appropriate port forwarding to access the TensorBoard web interface.

Total running time of the script: (16 minutes 26.651 seconds)

Gallery generated by Sphinx-Gallery