pluginsplugin-airag

IngestDocument​Ingest​Document

Ingest documents into an embedding store

Currently supports text documents (TXT, HTML, Markdown).

yaml
type: "io.kestra.plugin.ai.rag.IngestDocument"

Ingest documents into a KV embedding store. WARNING: the KV embedding store is for quick prototyping only; it stores embedding vectors in a KV store and loads them all into memory.

yaml
id: document_ingestion
namespace: company.ai

tasks:
  - id: ingest
    type: io.kestra.plugin.ai.rag.IngestDocument
    provider:
      type: io.kestra.plugin.ai.provider.GoogleGemini
      modelName: gemini-embedding-exp-03-07
      apiKey: "{{ kv('GEMINI_API_KEY') }}"
    embeddings:
      type: io.kestra.plugin.ai.embeddings.KestraKVStore
    drop: true
    fromExternalURLs:
      - https://raw.githubusercontent.com/kestra-io/docs/refs/heads/main/content/blogs/release-0-24.md
Properties

Embedding store provider

Language model provider

Must be configured with an embedding model.

Document splitter

Default false

Drop the store before ingestion (useful for testing)

List of inline documents

SubType string

List of document URLs from external sources

SubType string

List of internal storage URIs for documents

Path in the task working directory containing documents to ingest

Each document in the directory will be ingested into the embedding store. Ingestion is recursive and protected against path traversal (CWE-22).

SubType string

Additional metadata to add to all ingested documents

Additional outputs from the embedding store

Number of ingested documents

Input token count

Output token count

Total token count

API endpoint

The Azure OpenAI endpoint in the format: https://{resource}.openai.azure.com/

Model name

API Key

Client ID

Client secret

API version

Tenant ID

The database name

The database server host

The database password

The database server port

The table to store embeddings in

The database user

Default false

Whether to use use an IVFFlat index

An IVFFlat index divides vectors into lists, and then searches a subset of those lists closest to the query vector. It has faster build times and uses less memory than HNSW but has lower query performance (in terms of speed-recall tradeoff).

The API key

The collection name

The database server host

The database server port

Endpoint URL

Project location

Model name

Project ID

API Key

Model name

The collection name

The host

The index name

The scheme (e.g., mongodb+srv)

Create the index

The database

SubType string

The metadata field names

The connection string options

The password

The username

API Key

Model name

API base URL

Default {{flow.id}}-embedding-store

The name of the KV pair to use

The database base URL

The collection name

The database server host

The database server port

Default embedding-index

The index name

Basic authorization password

Basic authorization username

Token

Milvus auth token. Required if authentication is enabled; omit for local deployments without auth.

Auto flush on delete

If true, flush after delete operations.

Auto flush on insert

If true, flush after insert operations. Setting it to false can improve throughput.

Collection name

Target collection. Created automatically if it does not exist. Default: "default".

Consistency level

Read/write consistency level. Common values include STRONG, BOUNDED, or EVENTUALLY (depends on client/version).

Database name

Logical database to use. If not provided, the default database is used.

Host

Milvus host name (used when uri is not set). Default: "localhost".

ID field name

Field name for document IDs. Default depends on collection schema.

Index type

Vector index type (e.g., IVF_FLAT, IVF_SQ8, HNSW). Depends on Milvus deployment and dataset.

Metadata field name

Field name for metadata. Default depends on collection schema.

Metric type

Similarity metric (e.g., L2, IP, COSINE). Should match the embedding provider’s expected metric.

Password

Required when authentication/TLS is enabled. See https://milvus.io/docs/authenticate.md

Port

Milvus port (used when uri is not set). Typical: 19530 (gRPC) or 9091 (HTTP). Default: 19530.

Retrieve embeddings on search

If true, return stored embeddings along with matches. Default: false.

Text field name

Field name for original text. Default depends on collection schema.

URI

Connection URI. Use either uri OR host/port (not both). Examples:

  • gRPC (typical): "milvus://host: 19530"
  • HTTP: "http://host: 9091"

Username

Required when authentication/TLS is enabled. See https://milvus.io/docs/authenticate.md

Vector field name

Field name for the embedding vector. Must match the index definition and embedding dimensionality.

API Key

Model name

Default https://api.deepseek.com/v1

API base URL

The API key

The cloud provider

The index

The cloud provider region

The namespace (default will be used if not provided)

API Key

Model name

Maximum overlap size (characters). Only full sentences are considered for overlap.

Maximum segment size (characters)

Default RECURSIVE
Possible Values
RECURSIVEPARAGRAPHLINESENTENCEWORD

DocumentSplitter type

Recommended: RECURSIVE for generic text. It splits into paragraphs first and fits as many as possible into a single TextSegment. If paragraphs are too long, they are recursively split into lines, then sentences, then words, then characters until they fit into a segment.

API key

Weaviate API key. Omit for local deployments without auth.

Host

Cluster host name without protocol, e.g., "abc123.weaviate.network".

Avoid duplicates

If true (default), a hash-based ID is derived from each text segment to prevent duplicates. If false, a random ID is used.

Possible Values
ONEQUORUMALL

Consistency level

Write consistency: ONE, QUORUM (default), or ALL.

gRPC port

Port for gRPC if enabled (e.g., 50051).

Metadata field name

Field used to store metadata. Defaults to "_metadata" if not set.

SubType string

Metadata keys

The list of metadata keys to store - if not provided, it will default to an empty list.

Object class

Weaviate class to store objects in (must start with an uppercase letter). Defaults to "Default" if not set.

Port

Optional port (e.g., 443 for https, 80 for http). Leave unset to use provider defaults.

Scheme

Cluster scheme: "https" (recommended) or "http".

Secure gRPC

Whether the gRPC connection is secured (TLS).

Use gRPC for batch inserts

If true, use gRPC for batch inserts. HTTP remains required for search operations.

Model endpoint

Model name

API Key

Model name

API base URL

Document content

Document metadata

SubType string
Min items 1

List of HTTP Elasticsearch servers

Must be a URI like https://example.com: 9200 with scheme and port

Basic authorization configuration

SubType string

List of HTTP headers to be sent with every request

Each item is a key: value string, e.g., Authorization: Token XYZ

Path prefix for all HTTP requests

If set to /my/path, each client request becomes /my/path/ + endpoint. Useful when Elasticsearch is behind a proxy providing a base path; do not use otherwise.

Treat responses with deprecation warnings as failures

Trust all SSL CA certificates

Use this if the server uses a self-signed SSL certificate

The name of the index to store embeddings

AWS Access Key ID

Model name

AWS Secret Access Key

Default COHERE
Possible Values
COHERETITAN

Amazon Bedrock Embedding Model Type