Skip to content

Configuration

Summary of config.yaml

The config.yaml file contains configurations for various components involved in the data management, processing, and orchestration workflows related to the ALS beamlines.

Globus Endpoints

  • globus_endpoints: Defines multiple Globus endpoints used for data transfer between various systems.
  • spot832, data832, alcf832, nersc832, etc., each has specific configurations like:
    • root_path: The root directory for data storage.
    • uri: The URI for accessing the endpoint.
    • uuid: Unique identifier for each endpoint.
    • name: Descriptive name for the endpoint.
  • These endpoints represent different storage locations across systems like ALS, ALCF, and NERSC.

Globus Apps

  • globus_apps: Defines the application configurations needed for Globus transfers, such as the als_transfer app, with environment variables for client_id and client_secret.

Harbor Images

  • harbor_images832: Specifies the image for tomography reconstruction (recon_image) and multi-resolution conversion (multires_image) in the Harbor registry.

GHCR Images

  • ghcr_images832: Specifies the same tomography images as above but stored in the GitHub Container Registry (GHCR) for easier access during workflows.

Prefect

  • prefect.deployments: Defines the Prefect flow deployments, including unique identifiers (uuid) for specific flows related to file processing, like new_file_832.

SciCat

  • scicat: Configures the SciCat metadata service used for ingesting jobs and managing metadata. It includes the jobs_api_url for the API used to track and ingest jobs into the SciCat platform.

Key Configuration Highlights:

  • The YAML defines detailed configurations for each endpoint, ensuring smooth integration between systems like Globus, ALCF, NERSC, and data832.
  • Provides details for image containers used in tomography processes and how to access them.
  • Specifies the integration of SciCat for metadata tracking and the Prefect orchestration system for scheduling workflows.