Basics

​

Pipeline overview

• Single pipeline per collection — Each collection can declare one pipeline. You can sequence as many processors as needed inside it.
• Versioned definitions — Updating a pipeline creates a new version (for example clinics.2). In-flight ingestions finish on the version that was active when they started.
• Deterministic execution — Steps run in order. A failure stops the pipeline and the ingestion fails.

​

Dataflow through the pipeline

1. Ingestion request hits a source (bulk, bundle, or single writes).
2. Pipeline dispatch loads the latest pipeline version configured on the target collection.
3. Processor chain runs sequentially. Each step can:
   • Enrich the payload (vectorizers)
   • Mutate properties (Clinia functions)
   • Validate intermediate results before continuing
4. Default schema validation executes after all processors to ensure the resource still complies with its profile.
5. Persistence writes the transformed data into the registry and emits receipts for observability.

​

Creating a pipeline

curl -X PUT "https://$CLINIA_WORKSPACE/sources/my-external-system/v1/collections/my-collection/pipeline" \
  -H "X-Clinia-API-Key: $CLINIA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "steps": [
    {
      "type": "VECTORIZER",
      "vectorizer": {
        "inputProperty": "description",
        "modelId": "text-embedding-004",
        "propertyKey": "vector",
        "provider": "google",
        "dimensions": 768
      }
    }
  ]
}'

{
  "type": "<PROCESSOR_TYPE>",
  "<processor-type-in-lowercase>": {
    // Processor configuration
  },
  "trigger": {
    // Optional trigger configuration
  }
}

​

Triggers

​

Operator trigger

{
  "trigger": {
    "type": "OPERATOR_TRIGGER",
    "operator": {
      // Uses the same DSL as search queries
    }
  }
}

​

Always trigger

{
  "trigger": {
    "type": "ALWAYS_TRIGGER",
    "onlyOnTriggeredPipeline": false
  }
}

​

Schema validation

• Pipelines include an automatic validation pass at the end of the chain.
• Add explicit Schema Validator steps earlier to fail fast before expensive processing or to validate post-mutation states.

{
  "steps": [
    {
      "type": "SCHEMA_VALIDATOR",
      "schemaValidator": {}
    }
  ]
}

​

Monitoring pipelines

• Get pipeline execution for a specific execution ID.
• Query pipeline executions to build dashboards or human-in-the-loop queues.

curl -X GET "https://$CLINIA_WORKSPACE/sources/my-external-system/v1/pipelines/executions/{pipelineId}" \
  -H "X-Clinia-API-Key: $CLINIA_TOKEN"

• See built-in processors for enrichment or validation options.
• Explore custom processors to extend the pipeline with bespoke logic.