Scoping using metadata

Metadata is structured data attached to Knowledge and Memories. Use it when you already know a hard constraint before retrieval runs, such as department=legal, region=us, status=published, or author=alice. HydraDB has two metadata layers:

Layer	Sent at ingest as	Query filter shape	Best for
Tenant metadata	`metadata`	Top-level keys in `metadata_filters`	Stable, high-cardinality fields you filter on often. Declare these in `database_metadata_schema`, usually with `enable_match: true`.
Additional metadata	`additional_metadata`	Nested under `metadata_filters.additional_metadata`	Free-form per-source fields for display, citations, debugging, external IDs, or occasional filters.

If a field will be scoped on every query, it belongs in metadata and the database schema. If it’s ad-hoc or unique to one document, use additional_metadata instead.

{
  "metadata_filters": {
    "department": "legal",
    "additional_metadata": {
      "author": "alice"
    }
  }
}

1. Choose the right metadata layer

I want to…	Put it in…	Filter with…	Notes
Scope most queries by a field like department, region, plan, customer, or status	`metadata`	`metadata_filters: { "department": "legal" }`	Declare the field in `database_metadata_schema`; set `enable_match: true` for the intended filter path.
Store source-specific fields like author, Slack timestamp, external ID, document version	`additional_metadata`	`metadata_filters: { "additional_metadata": { "author": "alice" } }`	No schema required. Better for occasional filters and display metadata.
Combine hard scoping with semantic search	Both	`metadata_filters` plus your natural-language `query`	Filters narrow candidates; ranking still uses the query.
Search semantically over a metadata text field	`metadata` with `enable_dense_embedding: true`	Put the desired concept in `query`	Only supported for `VARCHAR` fields. Do not put fuzzy concepts in `metadata_filters`.
Search by keyword over a metadata text field	`metadata` with `enable_sparse_embedding: true`	Use normal `/query` text/BM25 behavior	Only supported for `VARCHAR` fields.
Partition by user, workspace, or team	`collection`	Send `collection` on every request	Use metadata filters inside that partition, not as a replacement for it.

The database field was formerly tenant_id and collection was formerly sub_tenant_id; the old names still work as deprecated aliases. Follow this for when to use database and collection.

2. How metadata filters run

For user-provided metadata_filters, HydraDB uses a correctness-first pipeline:

MongoDB source-id prefilter. Safe scalar tenant/additional metadata filters are resolved to matching source_ids in MongoDB.
Scoped retrieval. Vector/BM25 retrieval searches only those source IDs.
Post-filter correctness net. Hydrated chunks are checked again against the requested metadata. This also protects graph expansion and fallback/retry paths from leaking excluded sources.

That means a valid filter with no matching sources returns an empty result set; HydraDB does not silently widen it into an unfiltered search.

Field	Type	Purpose
`metadata`	object for knowledge; JSON-encoded string for memory items	Database-schema fields. Keys must match `database_metadata_schema`. The fast scoping path.
`additional_metadata`	object	Free-form per-document or per-memory fields. No schema. The flexible, slower path.

Filter semantics

Behavior	Contract
Multiple keys	AND logic. Every provided key/value must match.
Database + additional metadata	AND logic across both layers.
Values	Exact equality for scalar values.
Lists	ANY, not ALL. A list matches a source that has any one of the listed values (OR/IN). For example, `tags: ["alpha", "beta"]` matches a source tagged `"alpha"` and a source tagged `"beta"`. There is no ALL/AND operator inside a single key. Lists are supported on `VARCHAR` fields only - a list passed for a declared field of any other type is rejected with `400 VALIDATION_ERROR`.
Range / contains / fuzzy	Not supported in user `metadata_filters`. Put fuzzy concepts in `query`, use semantic metadata fields, or post-process client-side.
OR	Within one key, pass a list of values. Across different keys, run multiple queries and union client-side.
Graph context	Respects metadata filters; graph paths from excluded sources are removed.
Size	Each list holds at most 500 values, and the whole `metadata_filters` object is capped at 64 KiB. See Filter size limits.

metadata_filters are hard constraints, not semantic hints. A filter like { "mood": "happy" } requires an exact stored value; it does not expand to related values like “joyful” or “cheerful”. To search metadata text semantically, declare a VARCHAR field with enable_dense_embedding and include the concept in the main query.

3. Define tenant metadata schema

Most of the time the defaults are right. When they aren’t, here’s where to start:

Plan scoping fields before first ingest. Schema is immutable, and undeclared scope keys are silently ignored at query time. If you’ll scope on it more than once, declare it.
Pick metadata for hot paths, additional_metadata for cold ones. Database-level scopes are pre-applied in the vector store; document-level scopes force a post-retrieval pass with over-fetch.
Use exact-equality only. metadata_filters are equality constraints across all categories: a scalar is an exact match, and a list matches any one of its values. Range, contains, or fuzzy matching belong in the query, query mode, or downstream reranking - not here.
Keep keys stable. Renaming a metadata key requires re-ingesting affected sources. Same goes for changing enable_match flags.
Ingested metadata is immutable. Once a source is indexed, its metadata and additional_metadata values are locked. To change them, re-ingest the source with the new values (use upsert: true and the same id).
Don’t substitute metadata_filters for collection. metadata_filters scopes results inside a partition. For partitioning by user, team, or workspace, use collection.

4. Minimal working example

Two phases: set up metadata (declare the schema, then attach values at ingest), then scope at query.

Step 1 - Create metadata

The schema lives at the database level; values land on each source at ingest time. Both happen before any query.

Step 1a: Declare the schema at database creation

curl -X POST 'https://api.hydradb.com/databases' \
  -H "Authorization: Bearer $HYDRA_DB_API_KEY" \
  -H "API-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "database": "acme_corp",
    "database_metadata_schema": [
      {
        "name": "department",
        "data_type": "VARCHAR",
        "enable_match": true
      },
      {
        "name": "priority",
        "data_type": "INT64",
        "enable_match": true
      },
      {
        "name": "summary_label",
        "data_type": "VARCHAR",
        "enable_dense_embedding": true,
        "enable_sparse_embedding": true
      }
    ]
  }'

client.databases.create(
    database="acme_corp",
    database_metadata_schema=[
        {"name": "department", "data_type": "VARCHAR", "enable_match": True},
        {"name": "priority", "data_type": "INT64", "enable_match": True},
        {
            "name": "summary_label",
            "data_type": "VARCHAR",
            "enable_dense_embedding": True,
            "enable_sparse_embedding": True,
        },
    ],
)

await client.databases.create({
  database: "acme_corp",
  databaseMetadataSchema: [
    { name: "department", dataType: "VARCHAR", enableMatch: true },
    { name: "priority", dataType: "INT64", enableMatch: true },
    {
      name: "summary_label",
      dataType: "VARCHAR",
      enableDenseEmbedding: true,
      enableSparseEmbedding: true,
    },
  ],
});

Schema field options

Field	Type / values	Purpose
`name`	string	Metadata key. Must start with a letter or `_`, contain only letters/numbers/underscores, and not use reserved system names such as `source_id`, `chunk_id`, or `metadata`.
`data_type`	`VARCHAR`, `BOOL`, `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `JSON`, `ARRAY` or friendly aliases `string`, `boolean`, `integer`, `float`, `object`, `array`	Defaults to `VARCHAR`.
`max_length`	integer	Max length for `VARCHAR`. Default `1024`, maximum `65535`. This sizes one declared schema field, and is not the limit on how much metadata you can send per request - for that see Size limits.
`enable_match`	boolean	Creates the intended fast exact-match/filtering path for this field. Use this for top-level `metadata_filters` keys.
`enable_dense_embedding`	boolean	Adds a dense semantic-search lane for a `VARCHAR` metadata field.
`enable_sparse_embedding`	boolean	Adds a sparse/BM25 keyword-search lane for a `VARCHAR` metadata field.
`filterable`	boolean	Backward-compatible shorthand for `enable_match: true`. Prefer `enable_match` in new docs/code.
`searchable`	boolean	Backward-compatible input accepted by the API, but do not rely on it to replace explicit `enable_dense_embedding` / `enable_sparse_embedding`. Set those flags directly.

Limits and guardrails:

Up to 32 custom tenant metadata fields.
Up to 6 embedding-enabled fields per database. enable_dense_embedding and enable_sparse_embedding each count as one, so a field with both set counts as two. enable_match does not count against this limit - only the embedding flags do. Exceeding it fails database creation with 400 before anything is provisioned.
Field names are unique case-insensitively.
Dense/sparse embedding flags are only valid on VARCHAR fields.
Runtime metadata values must match the declared type when the tenant has a schema.
Unknown metadata keys are rejected on ingest/edit when a non-empty tenant schema exists.
Each metadata layer has a byte budget per request - see Size limits.

Size limits

Every request that attaches metadata is checked against two caps, on ingest and on metadata edit alike:

Layer	Send as	Cap
Database (tenant) metadata	`metadata` at ingest, `database_metadata` on edit	16 KiB (16,384 bytes)
Document metadata	`additional_metadata`	1 KiB (1,024 bytes)

Older spellings are still accepted, but not uniformly - which one works depends on the endpoint:

Alias	On `/context/ingest`	On `PATCH /context/{id}/metadata`
`tenant_metadata`	accepted	accepted (`database_metadata` wins if both are sent)
`document_metadata`	accepted	rejected with `400` - send `additional_metadata`

Where an alias is accepted it is held to exactly the same cap as the canonical field. Use the canonical names above and this never comes up. The cap applies to the whole map, not to any one value, and it is measured on the map’s compact JSON encoding in UTF-8 bytes. Three consequences worth planning around:

Keys and punctuation count. Quotes, colons, commas and braces are all part of the payload you are billed for.
Bytes, not characters. Accented Latin characters cost 2 bytes, most CJK characters 3, and emoji 4.
Budget in bytes from the start. A 950-character summary sounds comfortably under a 1 KiB cap, but with two small sibling keys it serializes to 1,015 bytes - 65 bytes of that is structure alone. Push the summary to 1,000 characters and the request is rejected at 1,065 bytes.

Document metadata: 1,015 bytes, just inside the 1 KiB cap

{"title":"Q3 Board Deck","author":"[email protected]","summary":"<950 characters>"}

Exceeding either cap fails the whole request with 400 before anything is ingested. The message names the offending field and reports both numbers, so you can see exactly how far over you are:

{
  "success": false,
  "data": null,
  "error": {
    "code": "INVALID_INPUT",
    "message": "additional_metadata is too large (1065 bytes when serialized; the maximum is 1024). Reduce the number or size of metadata fields."
  }
}

On PATCH /context/{id}/metadata the same message is prefixed with invalid metadata edit:.

If a document needs more than 1 KiB of descriptive metadata, put the long text in the document body where it gets chunked and embedded, and keep additional_metadata for the short values you actually filter on.

Filter size limits

The caps above bound the metadata you store. metadata_filters on /query has its own, separate pair - these bound what you send at query time and are unrelated to how much metadata a source carries:

Limit	Cap
Values in any one list	500
Whole `metadata_filters` object	64 KiB (65,536 bytes)

Measured the same way - compact JSON, UTF-8 bytes, keys and punctuation counted - and the object total includes the nested additional_metadata dict. Both exist because every value in a list is expanded into the filter expression sent to the vector store. The per-list cap catches one runaway list; the object cap catches many individually-legal lists adding up. Twenty lists of 500 values are each within the element cap but total roughly 127 KiB, so the object cap is what rejects them. Over either limit returns 400 before the query runs, naming the offending key or the actual byte count:

metadata_filters.customer_id (got 743) must contain at most 500 values

metadata_filters is too large (130251 bytes when serialized; the maximum is 65536).
Reduce the number or size of filter values.

Needing far more than 500 values in one filter usually means the constraint belongs in the data rather than the query - add a metadata field that groups those values (a segment, tier, or cohort key) and filter on that instead.

Add schema fields later

You can add database metadata fields after database creation with PATCH /databases/{database}/metadata-schema. This is additive only:

add new fields: yes
delete fields: no
change type/flags of existing fields: no

Adding fields updates the stored database schema and MongoDB filter indexes. Existing Milvus collections are not altered/backfilled for newly added dense/sparse metadata lanes yet. If you need a new metadata field to participate in semantic/BM25 metadata search for existing data, create a new database/schema and re-ingest, or confirm the current platform migration path with support.

5. Attach metadata at ingest

For knowledge ingestion, send metadata and additional_metadata on each document_metadata[] item or app_knowledge[] item.

curl -X POST 'https://api.hydradb.com/context/ingest' \
  -H "Authorization: Bearer $HYDRA_DB_API_KEY" \
  -H "API-Version: 2" \
  -F "type=knowledge" \
  -F "database=acme_corp" \
  -F 'app_knowledge=[
    {
      "database": "acme_corp",
      "collection": "default",
      "id": "auth-controls-001",
      "kind": "knowledge_base",
      "fields": {
        "title": "Authentication controls",
        "body": "Authentication controls are reviewed quarterly for SOC2."
      },
      "metadata": {
        "department": "security",
        "priority": 7,
        "summary_label": "quarterly access control review"
      },
      "additional_metadata": {
        "author": "alice",
        "doc_version": 3
      }
    }
  ]'

import json

client.context.ingest(
    type="knowledge",
    database="acme_corp",
    app_knowledge=json.dumps([
        {
            "database": "acme_corp",
            "collection": "default",
            "id": "auth-controls-001",
            "kind": "knowledge_base",
            "fields": {
                "title": "Authentication controls",
                "body": "Authentication controls are reviewed quarterly for SOC2.",
            },
            "metadata": {
                "department": "security",
                "priority": 7,
                "summary_label": "quarterly access control review",
            },
            "additional_metadata": {"author": "alice", "doc_version": 3},
        }
    ]),
)

await client.context.ingest({
  type: "knowledge",
  database: "acme_corp",
  appKnowledge: JSON.stringify([
    {
      database: "acme_corp",
      collection: "default",
      id: "auth-controls-001",
      kind: "knowledge_base",
      fields: {
        title: "Authentication controls",
        body: "Authentication controls are reviewed quarterly for SOC2.",
      },
      metadata: {
        department: "security",
        priority: 7,
        summary_label: "quarterly access control review",
      },
      additional_metadata: { author: "alice", doc_version: 3 },
    },
  ]),
});

For type=memory, the memories multipart field is already JSON-stringified, and each memory item’s metadata is currently validated as a JSON-encoded string. Keep additional_metadata as an object. For knowledge ingestion (document_metadata and app_knowledge), metadata is an object.

6. Query with metadata filters

Mix database-level (top-level) and document-level (nested) scopes in the same metadata_filters object:

curl -X POST 'https://api.hydradb.com/query' \
  -H "Authorization: Bearer $HYDRA_DB_API_KEY" \
  -H "API-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "database": "acme_corp",
    "collection": "default",
    "query": "How do access control reviews work?",
    "type": "knowledge",
    "query_by": "hybrid",
    "metadata_filters": {
      "department": "security",
      "priority": 7,
      "additional_metadata": {
        "author": "alice"
      }
    }
  }'

result = client.query(
    database="acme_corp",
    collection="default",
    query="How do access control reviews work?",
    type="knowledge",
    query_by="hybrid",
    metadata_filters={
        "department": "security",
        "priority": 7,
        "additional_metadata": {"author": "alice"},
    },
)

const result = await client.query({
  database: "acme_corp",
  collection: "default",
  query: "How do access control reviews work?",
  type: "knowledge",
  queryBy: "hybrid",
  metadataFilters: {
    department: "security",
    priority: 7,
    additional_metadata: { author: "alice" },
  },
});

Use the legacy alias only when maintaining older clients:

{
  "metadata_filters": {
    "document_metadata": { "author": "alice" }
  }
}

If both aliases are present and both are objects, additional_metadata wins on conflicts.

7. Update metadata without re-ingesting

Use PATCH /context/{id}/metadata when you know the source ID and need to update metadata in place.

{
  "database": "acme_corp",
  "collection": "default",
  "tenant_metadata": {
    "department": "legal"
  },
  "additional_metadata": {
    "author": "grace"
  }
}

Behavior:

The source must already exist.
collection is required.
At least one of tenant_metadata, additional_metadata, or acl is required.
The update is a merge/upsert: sent keys are inserted or overwritten; omitted keys are preserved.
document_metadata is not accepted on this endpoint; use additional_metadata.
The same endpoint accepts acl to change who may retrieve the source. Unlike metadata, acl replaces rather than merges, and an acl-only body is a valid edit. See Access Control.
enable_match-only tenant metadata updates are MongoDB-only and take effect for filters/listing.
If an edited tenant metadata field has enable_dense_embedding or enable_sparse_embedding, HydraDB synchronously syncs the relevant vector store lane and reports vector_sync_required / vector_synced in the response (the milvus_sync_required / milvus_synced aliases are still emitted, deprecated).

For full document/content replacement, re-ingest with upsert: true and the same source id. Upsert replaces the source payload and metadata supplied by ingestion.

8. Listing with metadata filters

Use POST /context/list when you want to browse or page sources rather than run semantic retrieval:

{
  "database": "acme_corp",
  "collection": "default",
  "type": "knowledge",
  "filters": {
    "metadata": { "department": "legal" },
    "additional_metadata": { "author": "alice" },
    "source_fields": { "type": "slack" }
  }
}

/context/list also accepts legacy aliases tenant_metadata for metadata and document_metadata for additional_metadata.

9. Common mistakes

Symptom	Cause	Fix
`metadata_filters` doesn’t scope results	Key isn’t declared in `database_metadata_schema`, or `enable_match` is `false`	Re-create the database with the field declared (`enable_match: true`), or move the field to `additional_metadata` and nest the scope under `additional_metadata`.
Scope on an `additional_metadata` key silently ignored	Scope passed as top-level key	Nest it: `metadata_filters: { additional_metadata: { author: "alice" } }`.
Schema field changes don’t take effect	Schema is immutable after database creation	Create a new database with the corrected schema. There’s no in-place schema migration.
Metadata values on a source don’t update	Ingested metadata is immutable	Re-ingest the source with `upsert: true` and the same `id`.
Query returns 0 results after adding a scope	Over-scoping - combined constraints exclude everything	Start with the minimum hard constraints; add scopes one at a time and re-check counts.
Range / contains / fuzzy scope doesn’t work	`metadata_filters` is equality-only	Move that constraint into the `query` text, use a different `query_by`, or apply it in your own post-scoping.
Fields used in `metadata` but missing from response objects	Field declared as embedding-only without `enable_match`	Embedding-enabled doesn’t imply scoping-enabled. Add `enable_match: true` to use it in `metadata_filters`.
`additional_metadata` scope feels slow	It is - over-fetches ~3× to compensate for post-retrieval matching	Move the hot field into `metadata` and declare it in the schema.
Metadata edit returns 400	Unknown key, wrong type, over-size payload (16 KiB database metadata / 1 KiB document metadata), too-deep nesting, reserved key, or missing `collection`	Check schema and size limits; send only `tenant_metadata` / `additional_metadata`.
Dense/sparse metadata edit rejects `null`	Null would leave stale semantic metadata vectors	Set a non-null value or re-ingest with the desired metadata.
New schema field does not participate in semantic metadata search	Additive schema update does not alter/backfill existing Milvus fields yet	Create the desired schema before ingesting, or migrate/re-ingest into a database with the final schema.

10. Advanced patterns

Stacked scopes with collection partitioning. Use collection for the partition (per-user, per-workspace), and use metadata_filters to scope inside that partition. They’re complementary, not interchangeable. See Multi-Tenant. Published vs draft. Add a status field with enable_match: true to your schema; tag every source with metadata.status = "draft" | "published"; pass metadata_filters: { status: "published" } on user-facing queries. Keeps work-in-progress out of customer answers automatically. Multi-language corpora. Add a language field with enable_match: true; route each query to the right language by passing metadata_filters: { language: detect_language(query) }. Schema-as-product. Treat database_metadata_schema as part of your data contract - review it like a database migration. The cost of getting it wrong (immutability + re-ingest) is real; the cost of getting it right is one extra meeting.

Knowledge - what attaches to knowledge sources
Memories - metadata fields on memory items
Multi-Tenant Support - partitioning vs scoping
Query - how metadata_filters interact with ranking
Create Database - API Reference - full database_metadata_schema reference
Query - API Reference - full metadata_filters reference
List Context - source browsing filters
Access Control - restricting who may retrieve a document, which is not a metadata filter
Update Source Metadata - point metadata edits
Update Metadata Schema - additive database schema changes

Get Started

Usage

Concepts

Plugins

For Agents

Scoping using metadata

1. Choose the right metadata layer

2. How metadata filters run

Filter semantics

3. Define tenant metadata schema

4. Minimal working example

Step 1 - Create metadata

Step 1a: Declare the schema at database creation

Schema field options

Size limits

Filter size limits

Add schema fields later

5. Attach metadata at ingest

6. Query with metadata filters

7. Update metadata without re-ingesting

8. Listing with metadata filters

9. Common mistakes

10. Advanced patterns

​1. Choose the right metadata layer

​2. How metadata filters run

​Filter semantics

​3. Define tenant metadata schema

​4. Minimal working example

​Step 1 - Create metadata

​Step 1a: Declare the schema at database creation

​Schema field options

​Size limits

​Filter size limits

​Add schema fields later

​5. Attach metadata at ingest

​6. Query with metadata filters

​7. Update metadata without re-ingesting

​8. Listing with metadata filters

​9. Common mistakes

​10. Advanced patterns

​Related

1. Choose the right metadata layer

2. How metadata filters run

Filter semantics

3. Define tenant metadata schema

4. Minimal working example

Step 1 - Create metadata

Step 1a: Declare the schema at database creation

Schema field options

Size limits

Filter size limits

Add schema fields later

5. Attach metadata at ingest

6. Query with metadata filters

7. Update metadata without re-ingesting

8. Listing with metadata filters

9. Common mistakes

10. Advanced patterns

Related