Skip to content

CLI Documentation

The brcschema command-line interface provides tools for transforming data between BRC and OSTI schemas, and for interacting with the OSTI E-Link 2.0 API.

Installation

This repository uses uv instead of Poetry. In this project, uv creates the local .venv, installs dependencies from pyproject.toml and uv.lock, and runs CLI commands inside that environment.

Install uv first:

curl -LsSf https://astral.sh/uv/install.sh | sh

You can also use brew install uv, pipx install uv, or pip install uv.

Then sync the project environment:

uv sync

After syncing, the brcschema command will be available through uv run.

Precede commands with uv run, or activate the local virtual environment first:

source .venv/bin/activate

If you see brcschema: command not found, run the command with uv run or activate .venv first.

Global Options

All commands support the following global options:

  • -v, --verbose: Increase verbosity (can be used multiple times: -v for INFO, -vv for DEBUG)

Commands

transform

Transform data between OSTI format and BRC schema format.

Usage:

uv run brcschema transform -T <transformation_type> -o <output_file> <input_file>

Options:

  • -T, --tx-type (required): Type of transformation
  • osti_to_brc: Convert OSTI schema to BRC schema
  • brc_to_osti: Convert BRC schema to OSTI schema
  • -o, --output (required): Output file path (format determined by extension: .json or .yaml)

Arguments:

  • input_data: Path to input file (JSON or YAML)

Examples:

# Transform OSTI format to BRC schema (YAML output)
uv run brcschema transform -T osti_to_brc -o data_out_brc_form.yaml data_in_osti_form.yaml

# Transform OSTI format to BRC schema (JSON output)
uv run brcschema transform -T osti_to_brc -o data_out_brc_form.json data_in_osti_form.json

# Transform BRC schema to OSTI format
uv run brcschema transform -T brc_to_osti -o data_out_osti_form.yaml data_in_brc_form.yaml

Notes:

  • Input files can be in JSON or YAML format
  • JSON input is automatically converted to YAML for processing
  • Output format is determined by the file extension

retrieve-osti

Retrieve metadata records from the OSTI E-Link 2.0 API using OSTI IDs or DOIs.

Usage:

uv run brcschema retrieve-osti [OPTIONS] -o <output_file>

Options:

  • --osti-ids: OSTI ID to retrieve (can be specified multiple times)
  • --dois: DOI to retrieve (can be specified multiple times)
  • --osti-id-file: Path to file containing OSTI IDs (one per line)
  • --doi-file: Path to file containing DOIs (one per line)
  • -o, --output (required): Output JSON file path
  • --api-key: OSTI API key (alternatively set OSTI_API_KEY environment variable)
  • --no-pretty: Disable pretty-printing of JSON output

Examples:

# Retrieve by single OSTI ID
uv run brcschema retrieve-osti --osti-ids 2584700 -o records.json

# Retrieve multiple OSTI IDs
uv run brcschema retrieve-osti --osti-ids 2584700 --osti-ids 2574191 -o records.json

# Retrieve by DOI (OSTI format)
uv run brcschema retrieve-osti --dois 10.11578/2584700 -o records.json

# Retrieve from ID file
uv run brcschema retrieve-osti --osti-id-file ids.txt -o records.json

# Mix OSTI IDs and DOIs
uv run brcschema retrieve-osti --osti-ids 2584700 --dois 10.11578/2584700 -o records.json

# With API key
uv run brcschema retrieve-osti --osti-ids 2584700 -o records.json --api-key YOUR_API_KEY

Notes:

  • At least one OSTI ID or DOI must be provided
  • You can mix different input methods (command-line options and files)
  • Retrieved records are saved in OSTI schema format (JSON)
  • DOIs must be OSTI-style (10.11578/<osti_id>); other DOI formats are skipped
  • API key is optional but may be required for certain records or rate limits

retrieve-osti-site

Retrieve OSTI records for a DOE site ownership code from legacy E-Link 1, E-Link 2.0, the public OSTI.GOV records API, or an authentication-dependent combination of those sources.

Usage:

uv run brcschema retrieve-osti-site --site-code <site_code> -o <output_file>

Options:

  • --site-code (required): DOE site ownership code, such as GLBRC, CBI, CABBI, or JBEI
  • --source: Source API to query: legacy, elink2, or public; repeat to query more than one source.
  • --product-type: Optional OSTI product type filter
  • --entry-date-start: Optional lower entry-date boundary. YYYY-MM-DD is converted to OSTI's MM/DD/YYYY query format.
  • --entry-date-end: Optional upper entry-date boundary. YYYY-MM-DD is converted to OSTI's MM/DD/YYYY query format.
  • --rows: Rows requested from each source API. Defaults to 500.
  • -l, --limit: Maximum records to keep from each source API
  • -o, --output (required): Output JSON file path
  • --api-key: E-Link 2.0 API key, alternatively set OSTI_API_KEY; required when elink2 is selected
  • --api-url: Optional E-Link 2.0 API URL
  • --legacy-api-url: Optional legacy E-Link 1 XML API URL
  • --legacy-username: Legacy E-Link 1 username, alternatively set OSTI_LEGACY_USERNAME; required when legacy is selected
  • --legacy-password: Legacy E-Link 1 password, alternatively set OSTI_LEGACY_PASSWORD; required when legacy is selected
  • --public-api-url: Optional public OSTI.GOV record API URL
  • --no-pretty: Disable pretty-printing of JSON output

Default source selection depends on available credentials. With no authentication, the command warns and uses only public, which can return only public/released OSTI.GOV records. With only an E-Link 2.0 API key, the command uses public and elink2. With both legacy E-Link 1 and E-Link 2.0 credentials, it uses legacy and elink2.

Examples:

# Query the default sources for the credentials available in the environment
uv run brcschema retrieve-osti-site --site-code GLBRC -o glbrc_records.json

# Query only E-Link 2.0 for recently entered records
uv run brcschema retrieve-osti-site --site-code GLBRC --source elink2 --entry-date-start 2026-01-01 -o glbrc_elink2.json

# Query only legacy E-Link 1 records
uv run brcschema retrieve-osti-site --site-code CBI --source legacy --legacy-username "$OSTI_LEGACY_USERNAME" --legacy-password "$OSTI_LEGACY_PASSWORD" --limit 100 -o cbi_legacy.json

# Query only public OSTI.GOV records
uv run brcschema retrieve-osti-site --site-code CBI --source public --limit 100 -o cbi_public.json

Output Metadata:

The output includes a transform-compatible top-level records list and additional metadata:

  • retrieval_sources: one entry per source API, including api, origin_schema, normalized_schema, query parameters, and counts
  • record_origins: one entry per record index, identifying the record's source API and origin schema

This lets downstream scripts distinguish legacy E-Link 1 records (osti_elink1_xml), public OSTI.GOV records (osti_public_api_v1_json), and E-Link 2.0 records (osti_elink2_json) while still feeding the same records list to transform -T osti_to_brc.

transmit-osti

Transmit metadata records to OSTI E-Link 2.0 API, creating new records or updating existing ones.

Usage:

uv run brcschema transmit-osti [OPTIONS] <input_file>

Options:

  • --api-key: OSTI API key (alternatively set OSTI_API_KEY environment variable)
  • --api-url: OSTI API URL (alternatively set OSTI_API_URL environment variable)
  • Defaults to production URL if not specified
  • For development/testing: https://review.osti.gov/elink2api/
  • -d, --dry-run: Test processing and validation without making persistent changes
  • -n, --new-only: Only create new records; skip updates to existing records
  • --skip-url: Skip records containing specified string in site_url field
  • -v, --verbose: Enable INFO-level logging
  • -l, --limit: Maximum number of records to process (useful for testing)

Arguments:

  • input_data: Path to input file containing records in OSTI format (JSON or YAML)

Examples:

# Basic transmission
uv run brcschema transmit-osti data_in_osti_form.yaml

# Dry run to test without making changes
uv run brcschema transmit-osti --dry-run data_in_osti_form.json

# Only create new records (don't update existing)
uv run brcschema transmit-osti --new-only data_in_osti_form.yaml

# Test with first 10 records only
uv run brcschema transmit-osti --dry-run --limit 10 data_in_osti_form.yaml

# Skip GitHub repositories
uv run brcschema transmit-osti --skip-url github data_in_osti_form.yaml

# Use development API
uv run brcschema transmit-osti --api-url https://review.osti.gov/elink2api/ data_in_osti_form.yaml

# Verbose output with API key
uv run brcschema transmit-osti -v --api-key YOUR_API_KEY data_in_osti_form.yaml

Notes:

  • Input files must contain a records key with an array of record objects
  • The command checks for existing records using identifiers before creating new ones
  • JSON input is automatically converted to YAML for processing
  • A summary is displayed showing counts of new, updated, failed, and skipped records
  • Failed records are logged with error details

Authentication

Several commands interact with the OSTI E-Link 2.0 API and may require authentication:

Setting API Key

You can provide the API key in two ways:

  1. Command-line option: Use --api-key YOUR_API_KEY
  2. Environment variable: Set OSTI_API_KEY=YOUR_API_KEY

Example:

# Using environment variable (recommended for security)
export OSTI_API_KEY="your-api-key-here"
uv run brcschema retrieve-osti --osti-ids 2584700 -o records.json

# Using command-line option
uv run brcschema retrieve-osti --osti-ids 2584700 -o records.json --api-key YOUR_API_KEY

Setting API URL

For testing or development environments:

  1. Command-line option: Use --api-url https://review.osti.gov/elink2api/
  2. Environment variable: Set OSTI_API_URL=https://review.osti.gov/elink2api/

For Python usage examples, see OSTI E-Link Integration.

File Formats

Input Files

The CLI accepts both JSON and YAML formats for input files:

YAML Example (records.yaml):

records:
  - title: "Example Dataset"
    doi: "10.11578/1234567"
    description: "A sample dataset"
    # ... other fields

JSON Example (records.json):

{
  "records": [
    {
      "title": "Example Dataset",
      "doi": "10.11578/1234567",
      "description": "A sample dataset"
    }
  ]
}

ID Files

When using --osti-id-file or --doi-file, provide one identifier per line:

Example (ids.txt):

2584700
2574191
2573456

Workflows

Complete Workflow: Retrieve, Transform, and Update

# Step 1: Retrieve records from OSTI
brcschema retrieve-osti --osti-ids 2584700 -o osti_records.json

# Step 2: Transform to BRC schema
brcschema transform -T osti_to_brc -o brc_records.yaml osti_records.json

# Step 3: [Edit brc_records.yaml as needed]

# Step 4: Transform back to OSTI format
brcschema transform -T brc_to_osti -o updated_osti.yaml brc_records.yaml

# Step 5: Test transmission with dry-run
brcschema transmit-osti --dry-run updated_osti.yaml

# Step 6: Transmit to OSTI
brcschema transmit-osti updated_osti.yaml

Batch Processing with Dry Run

# Test with a small subset first
brcschema transmit-osti --dry-run --limit 5 large_dataset.yaml

# If successful, process all records
brcschema transmit-osti large_dataset.yaml

Error Handling

Common Issues

  1. No records found: Ensure your input file has a records key with an array
  2. Authentication errors: Verify your API key is correct and has appropriate permissions
  3. Validation errors: Check that your records conform to the OSTI schema
  4. Multiple matches: If identifiers match multiple records, the operation will fail for that record

Debugging

Use verbose mode to see detailed logging:

# Single verbose flag for INFO level
brcschema -v transmit-osti data.yaml

# Double verbose flag for DEBUG level
brcschema -vv transmit-osti data.yaml

Use dry-run mode to test without making changes:

brcschema transmit-osti --dry-run -v data.yaml

Getting Help

For help with any command:

# General help
brcschema --help

# Command-specific help
brcschema transform --help
brcschema retrieve-osti --help
brcschema transmit-osti --help