> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clinia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Basics

> Understand how Clinia pipelines process data between ingestion and persistence

export const WorkInProgress = ({children}) => {
  const workInProgressSvg = <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 30" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className="lucide lucide-construction-icon lucide-construction">
      <g transform="translate(-4,-2)">
        <rect x="2" y="6" width="20" height="8" rx="1" />
        <path d="M17 14v7" />
        <path d="M7 14v7" />
        <path d="M17 3v3" />
        <path d="M7 3v3" />
        <path d="M10 14 2.3 6.3" />
        <path d="m14 6 7.7 7.7" />
        <path d="m8 6 8 8" />
      </g>
    </svg>;
  return <Accordion title="Coming soon" icon={workInProgressSvg}>
      {children}
    </Accordion>;
};

Pipelines let you transform, validate, and enrich data between the ingestion API and the registry. They execute sequential processors, each operating on the output of previous steps, so that your collections stay consistent while adding derived insights.

## Pipeline overview

* **Single pipeline per collection** — Each collection can declare one pipeline. You can sequence as many processors as needed inside it.
* **Versioned definitions** — Updating a pipeline creates a new version (for example `clinics.2`). In-flight ingestions finish on the version that was active when they started.
* **Deterministic execution** — Steps run in order. A failure stops the pipeline and the ingestion fails.

<WorkInProgress>
  Pipeline creation APIs are rolling out. Contact Clinia support if you need early access. You can already [monitor executions](#monitoring-pipelines).
</WorkInProgress>

## Dataflow through the pipeline

1. **Ingestion request** hits a source (`bulk`, `bundle`, or single writes).
2. **Pipeline dispatch** loads the latest pipeline version configured on the target collection.
3. **Processor chain** runs sequentially. Each step can:
   * Enrich the payload (vectorizers)
   * Mutate properties (Clinia functions)
   * Validate intermediate results before continuing
4. **Default schema validation** executes after all processors to ensure the resource still complies with its [profile](/explanation/data-model/profiles-relationships).
5. **Persistence** writes the transformed data into the registry and emits receipts for observability.

Design your processor order to minimize expensive work and to fail fast when validation issues occur.

## Creating a pipeline

```bash {collectForTests=createPipeline} theme={null}
curl -X PUT "https://$CLINIA_WORKSPACE/sources/my-external-system/v1/collections/my-collection/pipeline" \
  -H "X-Clinia-API-Key: $CLINIA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "steps": [
    {
      "type": "VECTORIZER",
      "vectorizer": {
        "inputProperty": "description",
        "modelId": "text-embedding-004",
        "propertyKey": "vector",
        "provider": "google",
        "dimensions": 768
      }
    }
  ]
}'
```

Each step follows the same structure:

```json theme={null}
{
  "type": "<PROCESSOR_TYPE>",
  "<processor-type-in-lowercase>": {
    // Processor configuration
  },
  "trigger": {
    // Optional trigger configuration
  }
}
```

## Triggers

Triggers control whether a processor runs for a specific payload.

### Operator trigger

See our [operators](/explanation/operator) documentation for details on the DSL.

```json theme={null}
{
  "trigger": {
    "type": "OPERATOR_TRIGGER",
    "operator": {
      // Uses the same DSL as search queries
    }
  }
}
```

Use operators to inspect the current payload or results from previous steps before executing costly processors.

### Always trigger

```json theme={null}
{
  "trigger": {
    "type": "ALWAYS_TRIGGER",
    "onlyOnTriggeredPipeline": false
  }
}
```

This is the implicit default. Set `onlyOnTriggeredPipeline` to `true` when a processor should run only if a prior step has already executed (for example, conditional validation).

## Schema validation

* Pipelines include an automatic validation pass at the end of the chain.
* Add explicit [Schema Validator](/explanation/ingestion-pipeline/built-in-processors#schema-validator) steps earlier to fail fast before expensive processing or to validate post-mutation states.

```json theme={null}
{
  "steps": [
    {
      "type": "SCHEMA_VALIDATOR",
      "schemaValidator": {}
    }
  ]
}
```

Validation uses the rules defined in your collection profile (required fields, cardinality, vocabulary bindings, etc.).

## Monitoring pipelines

Use the pipeline execution APIs to audit and debug ingestion flows:

* [Get pipeline execution](/api-reference/pipeline-executions/get-a-pipeline-execution) for a specific execution ID.
* [Query pipeline executions](/api-reference/pipeline-executions/query-pipeline-executions) to build dashboards or human-in-the-loop queues.

```bash {collectForTests=pipelineStatus} theme={null}
curl -X GET "https://$CLINIA_WORKSPACE/sources/my-external-system/v1/pipelines/executions/{pipelineId}" \
  -H "X-Clinia-API-Key: $CLINIA_TOKEN"
```

Every endpoint accepts `withOperationBody=true` if you need to inspect the payload that triggered the execution.

Next steps:

* See [built-in processors](/explanation/ingestion-pipeline/built-in-processors) for enrichment or validation options.
* Explore [custom processors](/explanation/ingestion-pipeline/custom-processor) to extend the pipeline with bespoke logic.