Scoping using metadata

Metadata is structured data attached to Knowledge and Memories. Use it when you already know a hard constraint before retrieval runs, such as department=legal, region=us, status=published, or author=alice. HydraDB has two metadata layers:

Layer	Sent at ingest as	Query filter shape	Best for
Tenant metadata	`metadata`	Top-level keys in `metadata_filters`	Stable, high-cardinality fields you filter on often. Declare these in `database_metadata_schema`, usually with `enable_match: true`.
Additional metadata	`additional_metadata`	Nested under `metadata_filters.additional_metadata`	Free-form per-source fields for display, citations, debugging, external IDs, or occasional filters.

If a field will be scoped on every query, it belongs in metadata and the database schema. If it’s ad-hoc or unique to one document, use additional_metadata instead.

{
  "metadata_filters": {
    "department": "legal",
    "additional_metadata": {
      "author": "alice"
    }
  }
}

1. Choose the right metadata layer

I want to…	Put it in…	Filter with…	Notes
Scope most queries by a field like department, region, plan, customer, or status	`metadata`	`metadata_filters: { "department": "legal" }`	Declare the field in `database_metadata_schema`; set `enable_match: true` for the intended filter path.
Store source-specific fields like author, Slack timestamp, external ID, document version	`additional_metadata`	`metadata_filters: { "additional_metadata": { "author": "alice" } }`	No schema required. Better for occasional filters and display metadata.
Combine hard scoping with semantic search	Both	`metadata_filters` plus your natural-language `query`	Filters narrow candidates; ranking still uses the query.
Search semantically over a metadata text field	`metadata` with `enable_dense_embedding: true`	Put the desired concept in `query`	Only supported for `VARCHAR` fields. Do not put fuzzy concepts in `metadata_filters`.
Search by keyword over a metadata text field	`metadata` with `enable_sparse_embedding: true`	Use normal `/query` text/BM25 behavior	Only supported for `VARCHAR` fields.
Partition by user, workspace, or team	`collection`	Send `collection` on every request	Use metadata filters inside that partition, not as a replacement for it.

The database field was formerly tenant_id and collection was formerly sub_tenant_id; the old names still work as deprecated aliases. Follow this for when to use database and collection.

2. How metadata filters run

For user-provided metadata_filters, HydraDB uses a correctness-first pipeline:

MongoDB source-id prefilter. Safe scalar tenant/additional metadata filters are resolved to matching source_ids in MongoDB.
Scoped retrieval. Vector/BM25 retrieval searches only those source IDs.
Post-filter correctness net. Hydrated chunks are checked again against the requested metadata. This also protects graph expansion and fallback/retry paths from leaking excluded sources.

That means a valid filter with no matching sources returns an empty result set; HydraDB does not silently widen it into an unfiltered search.

Field	Type	Purpose
`metadata`	object for knowledge; JSON-encoded string for memory items	Database-schema fields. Keys must match `database_metadata_schema`. The fast scoping path.
`additional_metadata`	object	Free-form per-document or per-memory fields. No schema. The flexible, slower path.

Filter semantics

Behavior	Contract
Multiple keys	AND logic. Every provided key/value must match.
Database + additional metadata	AND logic across both layers.
Values	Exact equality for scalar values.
Lists	Exact set equality for supported list-shaped comparisons, not OR/IN. For example, `tags: ["alpha", "beta"]` matches a stored `"alpha,beta"` set, but `tags: ["alpha"]` does not.
Range / contains / fuzzy	Not supported in user `metadata_filters`. Put fuzzy concepts in `query`, use semantic metadata fields, or post-process client-side.
OR	Run multiple queries and union client-side.
Graph context	Respects metadata filters; graph paths from excluded sources are removed.

metadata_filters are hard constraints, not semantic hints. A filter like { "mood": "happy" } requires an exact stored value; it does not expand to related values like “joyful” or “cheerful”. To search metadata text semantically, declare a VARCHAR field with enable_dense_embedding and include the concept in the main query.

3. Define tenant metadata schema

Most of the time the defaults are right. When they aren’t, here’s where to start:

Plan scoping fields before first ingest. Schema is immutable, and undeclared scope keys are silently ignored at query time. If you’ll scope on it more than once, declare it.
Pick metadata for hot paths, additional_metadata for cold ones. Database-level scopes are pre-applied in the vector store; document-level scopes force a post-retrieval pass with over-fetch.
Use exact-equality only. metadata_filters are equality constraints across all categories. Range, contains, or fuzzy matching belong in the query, query mode, or downstream reranking - not here.
Keep keys stable. Renaming a metadata key requires re-ingesting affected sources. Same goes for changing enable_match flags.
Ingested metadata is immutable. Once a source is indexed, its metadata and additional_metadata values are locked. To change them, re-ingest the source with the new values (use upsert: true and the same id).
Don’t substitute metadata_filters for collection. metadata_filters scopes results inside a partition. For partitioning by user, team, or workspace, use collection.

4. Minimal working example

Two phases: set up metadata (declare the schema, then attach values at ingest), then scope at query.

Step 1 - Create metadata

The schema lives at the database level; values land on each source at ingest time. Both happen before any query.

Step 1a: Declare the schema at database creation

curl -X POST 'https://api.hydradb.com/databases' \
  -H "Authorization: Bearer $HYDRA_DB_API_KEY" \
  -H "API-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "database": "acme_corp",
    "database_metadata_schema": [
      {
        "name": "department",
        "data_type": "VARCHAR",
        "enable_match": true
      },
      {
        "name": "priority",
        "data_type": "INT64",
        "enable_match": true
      },
      {
        "name": "summary_label",
        "data_type": "VARCHAR",
        "enable_dense_embedding": true,
        "enable_sparse_embedding": true
      }
    ]
  }'

client.databases.create(
    database="acme_corp",
    database_metadata_schema=[
        {"name": "department", "data_type": "VARCHAR", "enable_match": True},
        {"name": "priority", "data_type": "INT64", "enable_match": True},
        {
            "name": "summary_label",
            "data_type": "VARCHAR",
            "enable_dense_embedding": True,
            "enable_sparse_embedding": True,
        },
    ],
)

await client.databases.create({
  database: "acme_corp",
  databaseMetadataSchema: [
    { name: "department", dataType: "VARCHAR", enableMatch: true },
    { name: "priority", dataType: "INT64", enableMatch: true },
    {
      name: "summary_label",
      dataType: "VARCHAR",
      enableDenseEmbedding: true,
      enableSparseEmbedding: true,
    },
  ],
});

Schema field options

Field	Type / values	Purpose
`name`	string	Metadata key. Must start with a letter or `_`, contain only letters/numbers/underscores, and not use reserved system names such as `source_id`, `chunk_id`, or `metadata`.
`data_type`	`VARCHAR`, `BOOL`, `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `JSON`, `ARRAY` or friendly aliases `string`, `boolean`, `integer`, `float`, `object`, `array`	Defaults to `VARCHAR`.
`max_length`	integer	Max length for `VARCHAR`. Default `1024`, maximum `65535`.
`enable_match`	boolean	Creates the intended fast exact-match/filtering path for this field. Use this for top-level `metadata_filters` keys.
`enable_dense_embedding`	boolean	Adds a dense semantic-search lane for a `VARCHAR` metadata field.
`enable_sparse_embedding`	boolean	Adds a sparse/BM25 keyword-search lane for a `VARCHAR` metadata field.
`filterable`	boolean	Backward-compatible shorthand for `enable_match: true`. Prefer `enable_match` in new docs/code.
`searchable`	boolean	Backward-compatible input accepted by the API, but do not rely on it to replace explicit `enable_dense_embedding` / `enable_sparse_embedding`. Set those flags directly.

Limits and guardrails:

Up to 32 custom tenant metadata fields.
Field names are unique case-insensitively.
Dense/sparse embedding flags are only valid on VARCHAR fields.
Runtime metadata values must match the declared type when the tenant has a schema.
Unknown metadata keys are rejected on ingest/edit when a non-empty tenant schema exists.

Add schema fields later

You can add database metadata fields after database creation with PATCH /databases/{database}/metadata-schema. This is additive only:

add new fields: yes
delete fields: no
change type/flags of existing fields: no

Adding fields updates the stored database schema and MongoDB filter indexes. Existing Milvus collections are not altered/backfilled for newly added dense/sparse metadata lanes yet. If you need a new metadata field to participate in semantic/BM25 metadata search for existing data, create a new database/schema and re-ingest, or confirm the current platform migration path with support.

5. Attach metadata at ingest

For knowledge ingestion, send metadata and additional_metadata on each document_metadata[] item or app_knowledge[] item.

curl -X POST 'https://api.hydradb.com/context/ingest' \
  -H "Authorization: Bearer $HYDRA_DB_API_KEY" \
  -H "API-Version: 2" \
  -F "type=knowledge" \
  -F "database=acme_corp" \
  -F 'app_knowledge=[
    {
      "database": "acme_corp",
      "collection": "default",
      "id": "auth-controls-001",
      "kind": "knowledge_base",
      "fields": {
        "title": "Authentication controls",
        "body": "Authentication controls are reviewed quarterly for SOC2."
      },
      "metadata": {
        "department": "security",
        "priority": 7,
        "summary_label": "quarterly access control review"
      },
      "additional_metadata": {
        "author": "alice",
        "doc_version": 3
      }
    }
  ]'

import json

client.context.ingest(
    type="knowledge",
    database="acme_corp",
    app_knowledge=json.dumps([
        {
            "database": "acme_corp",
            "collection": "default",
            "id": "auth-controls-001",
            "kind": "knowledge_base",
            "fields": {
                "title": "Authentication controls",
                "body": "Authentication controls are reviewed quarterly for SOC2.",
            },
            "metadata": {
                "department": "security",
                "priority": 7,
                "summary_label": "quarterly access control review",
            },
            "additional_metadata": {"author": "alice", "doc_version": 3},
        }
    ]),
)

await client.context.ingest({
  type: "knowledge",
  database: "acme_corp",
  appKnowledge: JSON.stringify([
    {
      database: "acme_corp",
      collection: "default",
      id: "auth-controls-001",
      kind: "knowledge_base",
      fields: {
        title: "Authentication controls",
        body: "Authentication controls are reviewed quarterly for SOC2.",
      },
      metadata: {
        department: "security",
        priority: 7,
        summary_label: "quarterly access control review",
      },
      additional_metadata: { author: "alice", doc_version: 3 },
    },
  ]),
});

For type=memory, the memories multipart field is already JSON-stringified, and each memory item’s metadata is currently validated as a JSON-encoded string. Keep additional_metadata as an object. For knowledge ingestion (document_metadata and app_knowledge), metadata is an object.

6. Query with metadata filters

Mix database-level (top-level) and document-level (nested) scopes in the same metadata_filters object:

curl -X POST 'https://api.hydradb.com/query' \
  -H "Authorization: Bearer $HYDRA_DB_API_KEY" \
  -H "API-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "database": "acme_corp",
    "collection": "default",
    "query": "How do access control reviews work?",
    "type": "knowledge",
    "query_by": "hybrid",
    "metadata_filters": {
      "department": "security",
      "priority": 7,
      "additional_metadata": {
        "author": "alice"
      }
    }
  }'

result = client.query(
    database="acme_corp",
    collection="default",
    query="How do access control reviews work?",
    type="knowledge",
    query_by="hybrid",
    metadata_filters={
        "department": "security",
        "priority": 7,
        "additional_metadata": {"author": "alice"},
    },
)

const result = await client.query({
  database: "acme_corp",
  collection: "default",
  query: "How do access control reviews work?",
  type: "knowledge",
  queryBy: "hybrid",
  metadataFilters: {
    department: "security",
    priority: 7,
    additional_metadata: { author: "alice" },
  },
});

Use the legacy alias only when maintaining older clients:

{
  "metadata_filters": {
    "document_metadata": { "author": "alice" }
  }
}

If both aliases are present and both are objects, additional_metadata wins on conflicts.

7. Update metadata without re-ingesting

Use PATCH /context/sources/{source_id}/metadata when you know the source ID and need to update metadata in place.

{
  "database": "acme_corp",
  "collection": "default",
  "tenant_metadata": {
    "department": "legal"
  },
  "additional_metadata": {
    "author": "grace"
  }
}

Behavior:

The source must already exist.
collection is required.
At least one of tenant_metadata or additional_metadata is required.
The update is a merge/upsert: sent keys are inserted or overwritten; omitted keys are preserved.
document_metadata is not accepted on this endpoint; use additional_metadata.
enable_match-only tenant metadata updates are MongoDB-only and take effect for filters/listing.
If an edited tenant metadata field has enable_dense_embedding or enable_sparse_embedding, HydraDB synchronously syncs the relevant vector store lane and reports vector_sync_required / vector_synced in the response (the milvus_sync_required / milvus_synced aliases are still emitted, deprecated).

For full document/content replacement, re-ingest with upsert: true and the same source id. Upsert replaces the source payload and metadata supplied by ingestion.

8. Listing with metadata filters

Use POST /context/list when you want to browse or page sources rather than run semantic retrieval:

{
  "database": "acme_corp",
  "collection": "default",
  "type": "knowledge",
  "filters": {
    "metadata": { "department": "legal" },
    "additional_metadata": { "author": "alice" },
    "source_fields": { "type": "slack" }
  }
}

/context/list also accepts legacy aliases tenant_metadata for metadata and document_metadata for additional_metadata.

9. Common mistakes

Symptom	Cause	Fix
`metadata_filters` doesn’t scope results	Key isn’t declared in `database_metadata_schema`, or `enable_match` is `false`	Re-create the database with the field declared (`enable_match: true`), or move the field to `additional_metadata` and nest the scope under `additional_metadata`.
Scope on an `additional_metadata` key silently ignored	Scope passed as top-level key	Nest it: `metadata_filters: { additional_metadata: { author: "alice" } }`.
Schema field changes don’t take effect	Schema is immutable after database creation	Create a new database with the corrected schema. There’s no in-place schema migration.
Metadata values on a source don’t update	Ingested metadata is immutable	Re-ingest the source with `upsert: true` and the same `id`.
Query returns 0 results after adding a scope	Over-scoping - combined constraints exclude everything	Start with the minimum hard constraints; add scopes one at a time and re-check counts.
Range / contains / fuzzy scope doesn’t work	`metadata_filters` is equality-only	Move that constraint into the `query` text, use a different `query_by`, or apply it in your own post-scoping.
Fields used in `metadata` but missing from response objects	Field declared as embedding-only without `enable_match`	Embedding-enabled doesn’t imply scoping-enabled. Add `enable_match: true` to use it in `metadata_filters`.
`additional_metadata` scope feels slow	It is - over-fetches ~3× to compensate for post-retrieval matching	Move the hot field into `metadata` and declare it in the schema.
Metadata edit returns 400	Unknown key, wrong type, over-size value, too-deep nesting, reserved key, or missing `collection`	Check schema and validation limits; send only `tenant_metadata` / `additional_metadata`.
Dense/sparse metadata edit rejects `null`	Null would leave stale semantic metadata vectors	Set a non-null value or re-ingest with the desired metadata.
New schema field does not participate in semantic metadata search	Additive schema update does not alter/backfill existing Milvus fields yet	Create the desired schema before ingesting, or migrate/re-ingest into a database with the final schema.

10. Advanced patterns

Stacked scopes with collection partitioning. Use collection for the partition (per-user, per-workspace), and use metadata_filters to scope inside that partition. They’re complementary, not interchangeable. See Multi-Tenant. Published vs draft. Add a status field with enable_match: true to your schema; tag every source with metadata.status = "draft" | "published"; pass metadata_filters: { status: "published" } on user-facing queries. Keeps work-in-progress out of customer answers automatically. Multi-language corpora. Add a language field with enable_match: true; route each query to the right language by passing metadata_filters: { language: detect_language(query) }. Schema-as-product. Treat database_metadata_schema as part of your data contract - review it like a database migration. The cost of getting it wrong (immutability + re-ingest) is real; the cost of getting it right is one extra meeting.

Knowledge - what attaches to knowledge sources
Memories - metadata fields on memory items
Multi-Tenant Support - partitioning vs scoping
Query - how metadata_filters interact with ranking
Create Database - API Reference - full database_metadata_schema reference
Query - API Reference - full metadata_filters reference
List Context - source browsing filters
Update Source Metadata - point metadata edits
Update Metadata Schema - additive database schema changes

Get Started

Usage

Concepts

Plugins

For Agents

Scoping using metadata

1. Choose the right metadata layer

2. How metadata filters run

Filter semantics

3. Define tenant metadata schema

4. Minimal working example

Step 1 - Create metadata

Step 1a: Declare the schema at database creation

Schema field options

Add schema fields later

5. Attach metadata at ingest

6. Query with metadata filters

7. Update metadata without re-ingesting

8. Listing with metadata filters

9. Common mistakes

10. Advanced patterns

​1. Choose the right metadata layer

​2. How metadata filters run

​Filter semantics

​3. Define tenant metadata schema

​4. Minimal working example

​Step 1 - Create metadata

​Step 1a: Declare the schema at database creation

​Schema field options

​Add schema fields later

​5. Attach metadata at ingest

​6. Query with metadata filters

​7. Update metadata without re-ingesting

​8. Listing with metadata filters

​9. Common mistakes

​10. Advanced patterns

​Related

1. Choose the right metadata layer

2. How metadata filters run

Filter semantics

3. Define tenant metadata schema

4. Minimal working example

Step 1 - Create metadata

Step 1a: Declare the schema at database creation

Schema field options

Add schema fields later

5. Attach metadata at ingest

6. Query with metadata filters

7. Update metadata without re-ingesting

8. Listing with metadata filters

9. Common mistakes

10. Advanced patterns

Related