> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hydradb.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Scoping using metadata
> How HydraDB uses metadata to scope query results with schema-backed fields, free-form additional metadata, source metadata edits, and exact metadata filters.
Metadata is structured data attached to [Knowledge](/essentials/v2/knowledge) and [Memories](/essentials/v2/memories). Use it when you already know a hard constraint before retrieval runs, such as `department=legal`, `region=us`, `status=published`, or `author=alice`.
HydraDB has two metadata layers:
| Layer | Sent at ingest as | Query filter shape | Best for |
| ------------------- | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Tenant metadata | `metadata` | Top-level keys in `metadata_filters` | Stable, high-cardinality fields you filter on often. Declare these in `database_metadata_schema`, usually with `enable_match: true`. |
| Additional metadata | `additional_metadata` | Nested under `metadata_filters.additional_metadata` | Free-form per-source fields for display, citations, debugging, external IDs, or occasional filters. |
If a field will be scoped on **every** query, it belongs in `metadata` and the [database schema](/api-reference/v2/endpoint/create-tenant). If it's ad-hoc or unique to one document, use `additional_metadata` instead.
```json theme={"dark"}
{
"metadata_filters": {
"department": "legal",
"additional_metadata": {
"author": "alice"
}
}
}
```
***
## 1. Choose the right metadata layer
| I want to… | Put it in… | Filter with… | Notes |
| ---------------------------------------------------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Scope most queries by a field like department, region, plan, customer, or status | `metadata` | `metadata_filters: { "department": "legal" }` | Declare the field in `database_metadata_schema`; set `enable_match: true` for the intended filter path. |
| Store source-specific fields like author, Slack timestamp, external ID, document version | `additional_metadata` | `metadata_filters: { "additional_metadata": { "author": "alice" } }` | No schema required. Better for occasional filters and display metadata. |
| Combine hard scoping with semantic search | Both | `metadata_filters` plus your natural-language `query` | Filters narrow candidates; ranking still uses the query. |
| Search semantically over a metadata text field | `metadata` with `enable_dense_embedding: true` | Put the desired concept in `query` | Only supported for `VARCHAR` fields. Do not put fuzzy concepts in `metadata_filters`. |
| Search by keyword over a metadata text field | `metadata` with `enable_sparse_embedding: true` | Use normal `/query` text/BM25 behavior | Only supported for `VARCHAR` fields. |
| Partition by user, workspace, or team | `collection` | Send `collection` on every request | Use metadata filters inside that partition, not as a replacement for it. |
The `database` field was formerly `tenant_id` and `collection` was formerly `sub_tenant_id`; the old names still work as deprecated aliases. [Follow this for when to use `database` and `collection`](./multi-tenant#2-when-to-use-each).
***
## 2. How metadata filters run
For user-provided `metadata_filters`, HydraDB uses a correctness-first pipeline:
1. **MongoDB source-id prefilter.** Safe scalar tenant/additional metadata filters are resolved to matching `source_id`s in MongoDB.
2. **Scoped retrieval.** Vector/BM25 retrieval searches only those source IDs.
3. **Post-filter correctness net.** Hydrated chunks are checked again against the requested metadata. This also protects graph expansion and fallback/retry paths from leaking excluded sources.
That means a valid filter with no matching sources returns an empty result set; HydraDB does not silently widen it into an unfiltered search.
| Field | Type | Purpose |
| --------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `metadata` | object for knowledge; JSON-encoded string for memory items | Database-schema fields. Keys must match `database_metadata_schema`. The fast scoping path. |
| `additional_metadata` | object | Free-form per-document or per-memory fields. No schema. The flexible, slower path. |
### Filter semantics
| Behavior | Contract |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Multiple keys | AND logic. Every provided key/value must match. |
| Database + additional metadata | AND logic across both layers. |
| Values | Exact equality for scalar values. |
| Lists | Exact set equality for supported list-shaped comparisons, not OR/IN. For example, `tags: ["alpha", "beta"]` matches a stored `"alpha,beta"` set, but `tags: ["alpha"]` does not. |
| Range / contains / fuzzy | Not supported in user `metadata_filters`. Put fuzzy concepts in `query`, use semantic metadata fields, or post-process client-side. |
| OR | Run multiple queries and union client-side. |
| Graph context | Respects metadata filters; graph paths from excluded sources are removed. |
`metadata_filters` are hard constraints, not semantic hints. A filter like `{ "mood": "happy" }` requires an exact stored value; it does not expand to related values like "joyful" or "cheerful". To search metadata text semantically, declare a `VARCHAR` field with `enable_dense_embedding` and include the concept in the main `query`.
***
## 3. Define tenant metadata schema
Most of the time the defaults are right. When they aren't, here's where to start:
* **Plan scoping fields before first ingest.** Schema is immutable, and undeclared scope keys are silently ignored at query time. If you'll scope on it more than once, declare it.
* **Pick `metadata` for hot paths, `additional_metadata` for cold ones.** Database-level scopes are pre-applied in the vector store; document-level scopes force a post-retrieval pass with over-fetch.
* **Use exact-equality only.** `metadata_filters` are equality constraints across all categories. Range, contains, or fuzzy matching belong in the query, query mode, or downstream reranking - not here.
* **Keep keys stable.** Renaming a metadata key requires re-ingesting affected sources. Same goes for changing `enable_match` flags.
* **Ingested metadata is immutable.** Once a source is indexed, its `metadata` and `additional_metadata` values are locked. To change them, re-ingest the source with the new values (use `upsert: true` and the same `id`).
* **Don't substitute `metadata_filters` for `collection`.** `metadata_filters` scopes results *inside* a partition. For partitioning by user, team, or workspace, use [`collection`](/essentials/v2/multi-tenant).
***
## 4. Minimal working example
Two phases: **set up metadata** (declare the schema, then attach values at ingest), then **scope at query**.
### Step 1 - Create metadata
The schema lives at the database level; values land on each source at ingest time. Both happen before any query.
#### Step 1a: Declare the schema at database creation
```bash cURL theme={"dark"}
curl -X POST 'https://api.hydradb.com/databases' \
-H "Authorization: Bearer $HYDRA_DB_API_KEY" \
-H "API-Version: 2" \
-H "Content-Type: application/json" \
-d '{
"database": "acme_corp",
"database_metadata_schema": [
{
"name": "department",
"data_type": "VARCHAR",
"enable_match": true
},
{
"name": "priority",
"data_type": "INT64",
"enable_match": true
},
{
"name": "summary_label",
"data_type": "VARCHAR",
"enable_dense_embedding": true,
"enable_sparse_embedding": true
}
]
}'
```
```python Python SDK theme={"dark"}
client.databases.create(
database="acme_corp",
database_metadata_schema=[
{"name": "department", "data_type": "VARCHAR", "enable_match": True},
{"name": "priority", "data_type": "INT64", "enable_match": True},
{
"name": "summary_label",
"data_type": "VARCHAR",
"enable_dense_embedding": True,
"enable_sparse_embedding": True,
},
],
)
```
```typescript TypeScript SDK theme={"dark"}
await client.databases.create({
database: "acme_corp",
databaseMetadataSchema: [
{ name: "department", dataType: "VARCHAR", enableMatch: true },
{ name: "priority", dataType: "INT64", enableMatch: true },
{
name: "summary_label",
dataType: "VARCHAR",
enableDenseEmbedding: true,
enableSparseEmbedding: true,
},
],
});
```
### Schema field options
| Field | Type / values | Purpose |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | string | Metadata key. Must start with a letter or `_`, contain only letters/numbers/underscores, and not use reserved system names such as `source_id`, `chunk_id`, or `metadata`. |
| `data_type` | `VARCHAR`, `BOOL`, `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `JSON`, `ARRAY` or friendly aliases `string`, `boolean`, `integer`, `float`, `object`, `array` | Defaults to `VARCHAR`. |
| `max_length` | integer | Max length for `VARCHAR`. Default `1024`, maximum `65535`. |
| `enable_match` | boolean | Creates the intended fast exact-match/filtering path for this field. Use this for top-level `metadata_filters` keys. |
| `enable_dense_embedding` | boolean | Adds a dense semantic-search lane for a `VARCHAR` metadata field. |
| `enable_sparse_embedding` | boolean | Adds a sparse/BM25 keyword-search lane for a `VARCHAR` metadata field. |
| `filterable` | boolean | Backward-compatible shorthand for `enable_match: true`. Prefer `enable_match` in new docs/code. |
| `searchable` | boolean | Backward-compatible input accepted by the API, but do not rely on it to replace explicit `enable_dense_embedding` / `enable_sparse_embedding`. Set those flags directly. |
Limits and guardrails:
* Up to 32 custom tenant metadata fields.
* Field names are unique case-insensitively.
* Dense/sparse embedding flags are only valid on `VARCHAR` fields.
* Runtime `metadata` values must match the declared type when the tenant has a schema.
* Unknown `metadata` keys are rejected on ingest/edit when a non-empty tenant schema exists.
### Add schema fields later
You can add database metadata fields after database creation with [`PATCH /databases/{database}/metadata-schema`](/api-reference/v2/endpoint/update-metadata-schema). This is additive only:
* add new fields: yes
* delete fields: no
* change type/flags of existing fields: no
Adding fields updates the stored database schema and MongoDB filter indexes. Existing Milvus collections are not altered/backfilled for newly added dense/sparse metadata lanes yet. If you need a new metadata field to participate in semantic/BM25 metadata search for existing data, create a new database/schema and re-ingest, or confirm the current platform migration path with support.
***
## 5. Attach metadata at ingest
For knowledge ingestion, send `metadata` and `additional_metadata` on each `document_metadata[]` item or `app_knowledge[]` item.
```bash cURL theme={"dark"}
curl -X POST 'https://api.hydradb.com/context/ingest' \
-H "Authorization: Bearer $HYDRA_DB_API_KEY" \
-H "API-Version: 2" \
-F "type=knowledge" \
-F "database=acme_corp" \
-F 'app_knowledge=[
{
"database": "acme_corp",
"collection": "default",
"id": "auth-controls-001",
"kind": "knowledge_base",
"fields": {
"title": "Authentication controls",
"body": "Authentication controls are reviewed quarterly for SOC2."
},
"metadata": {
"department": "security",
"priority": 7,
"summary_label": "quarterly access control review"
},
"additional_metadata": {
"author": "alice",
"doc_version": 3
}
}
]'
```
```python Python SDK theme={"dark"}
import json
client.context.ingest(
type="knowledge",
database="acme_corp",
app_knowledge=json.dumps([
{
"database": "acme_corp",
"collection": "default",
"id": "auth-controls-001",
"kind": "knowledge_base",
"fields": {
"title": "Authentication controls",
"body": "Authentication controls are reviewed quarterly for SOC2.",
},
"metadata": {
"department": "security",
"priority": 7,
"summary_label": "quarterly access control review",
},
"additional_metadata": {"author": "alice", "doc_version": 3},
}
]),
)
```
```typescript TypeScript SDK theme={"dark"}
await client.context.ingest({
type: "knowledge",
database: "acme_corp",
appKnowledge: JSON.stringify([
{
database: "acme_corp",
collection: "default",
id: "auth-controls-001",
kind: "knowledge_base",
fields: {
title: "Authentication controls",
body: "Authentication controls are reviewed quarterly for SOC2.",
},
metadata: {
department: "security",
priority: 7,
summary_label: "quarterly access control review",
},
additional_metadata: { author: "alice", doc_version: 3 },
},
]),
});
```
For `type=memory`, the `memories` multipart field is already JSON-stringified, and each memory item's `metadata` is currently validated as a JSON-encoded string. Keep `additional_metadata` as an object. For knowledge ingestion (`document_metadata` and `app_knowledge`), `metadata` is an object.
***
## 6. Query with metadata filters
Mix database-level (top-level) and document-level (nested) scopes in the same `metadata_filters` object:
```bash cURL theme={"dark"}
curl -X POST 'https://api.hydradb.com/query' \
-H "Authorization: Bearer $HYDRA_DB_API_KEY" \
-H "API-Version: 2" \
-H "Content-Type: application/json" \
-d '{
"database": "acme_corp",
"collection": "default",
"query": "How do access control reviews work?",
"type": "knowledge",
"query_by": "hybrid",
"metadata_filters": {
"department": "security",
"priority": 7,
"additional_metadata": {
"author": "alice"
}
}
}'
```
```python Python SDK theme={"dark"}
result = client.query(
database="acme_corp",
collection="default",
query="How do access control reviews work?",
type="knowledge",
query_by="hybrid",
metadata_filters={
"department": "security",
"priority": 7,
"additional_metadata": {"author": "alice"},
},
)
```
```typescript TypeScript SDK theme={"dark"}
const result = await client.query({
database: "acme_corp",
collection: "default",
query: "How do access control reviews work?",
type: "knowledge",
queryBy: "hybrid",
metadataFilters: {
department: "security",
priority: 7,
additional_metadata: { author: "alice" },
},
});
```
Use the legacy alias only when maintaining older clients:
```json theme={"dark"}
{
"metadata_filters": {
"document_metadata": { "author": "alice" }
}
}
```
If both aliases are present and both are objects, `additional_metadata` wins on conflicts.
***
## 7. Update metadata without re-ingesting
Use [`PATCH /context/sources/{source_id}/metadata`](/api-reference/v2/endpoint/update-source-metadata) when you know the source ID and need to update metadata in place.
```json theme={"dark"}
{
"database": "acme_corp",
"collection": "default",
"tenant_metadata": {
"department": "legal"
},
"additional_metadata": {
"author": "grace"
}
}
```
Behavior:
* The source must already exist.
* `collection` is required.
* At least one of `tenant_metadata` or `additional_metadata` is required.
* The update is a merge/upsert: sent keys are inserted or overwritten; omitted keys are preserved.
* `document_metadata` is not accepted on this endpoint; use `additional_metadata`.
* `enable_match`-only tenant metadata updates are MongoDB-only and take effect for filters/listing.
* If an edited tenant metadata field has `enable_dense_embedding` or `enable_sparse_embedding`, HydraDB synchronously syncs the relevant vector store lane and reports `vector_sync_required` / `vector_synced` in the response (the `milvus_sync_required` / `milvus_synced` aliases are still emitted, deprecated).
For full document/content replacement, re-ingest with `upsert: true` and the same source `id`. Upsert replaces the source payload and metadata supplied by ingestion.
***
## 8. Listing with metadata filters
Use [`POST /context/list`](/api-reference/v2/endpoint/list-documents) when you want to browse or page sources rather than run semantic retrieval:
```json theme={"dark"}
{
"database": "acme_corp",
"collection": "default",
"type": "knowledge",
"filters": {
"metadata": { "department": "legal" },
"additional_metadata": { "author": "alice" },
"source_fields": { "type": "slack" }
}
}
```
`/context/list` also accepts legacy aliases `tenant_metadata` for `metadata` and `document_metadata` for `additional_metadata`.
***
## 9. Common mistakes
| Symptom | Cause | Fix |
| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metadata_filters` doesn't scope results | Key isn't declared in `database_metadata_schema`, or `enable_match` is `false` | Re-create the database with the field declared (`enable_match: true`), or move the field to `additional_metadata` and nest the scope under `additional_metadata`. |
| Scope on an `additional_metadata` key silently ignored | Scope passed as top-level key | Nest it: `metadata_filters: { additional_metadata: { author: "alice" } }`. |
| Schema field changes don't take effect | Schema is immutable after database creation | Create a new database with the corrected schema. There's no in-place schema migration. |
| Metadata values on a source don't update | Ingested metadata is immutable | Re-ingest the source with `upsert: true` and the same `id`. |
| Query returns 0 results after adding a scope | Over-scoping - combined constraints exclude everything | Start with the minimum hard constraints; add scopes one at a time and re-check counts. |
| Range / contains / fuzzy scope doesn't work | `metadata_filters` is equality-only | Move that constraint into the `query` text, use a different `query_by`, or apply it in your own post-scoping. |
| Fields used in `metadata` but missing from response objects | Field declared as embedding-only without `enable_match` | Embedding-enabled doesn't imply scoping-enabled. Add `enable_match: true` to use it in `metadata_filters`. |
| `additional_metadata` scope feels slow | It is - over-fetches \~3× to compensate for post-retrieval matching | Move the hot field into `metadata` and declare it in the schema. |
| Metadata edit returns 400 | Unknown key, wrong type, over-size value, too-deep nesting, reserved key, or missing `collection` | Check schema and validation limits; send only `tenant_metadata` / `additional_metadata`. |
| Dense/sparse metadata edit rejects `null` | Null would leave stale semantic metadata vectors | Set a non-null value or re-ingest with the desired metadata. |
| New schema field does not participate in semantic metadata search | Additive schema update does not alter/backfill existing Milvus fields yet | Create the desired schema before ingesting, or migrate/re-ingest into a database with the final schema. |
***
## 10. Advanced patterns
**Stacked scopes with collection partitioning.** Use `collection` for the partition (per-user, per-workspace), and use `metadata_filters` to scope *inside* that partition. They're complementary, not interchangeable. See [Multi-Tenant](/essentials/v2/multi-tenant).
**Published vs draft.** Add a `status` field with `enable_match: true` to your schema; tag every source with `metadata.status = "draft" | "published"`; pass `metadata_filters: { status: "published" }` on user-facing queries. Keeps work-in-progress out of customer answers automatically.
**Multi-language corpora.** Add a `language` field with `enable_match: true`; route each query to the right language by passing `metadata_filters: { language: detect_language(query) }`.
**Schema-as-product.** Treat `database_metadata_schema` as part of your data contract - review it like a database migration. The cost of getting it wrong (immutability + re-ingest) is real; the cost of getting it right is one extra meeting.
***
## Related
* [Knowledge](/essentials/v2/knowledge) - what attaches to knowledge sources
* [Memories](/essentials/v2/memories) - metadata fields on memory items
* [Multi-Tenant Support](/essentials/v2/multi-tenant) - partitioning vs scoping
* [Query](/essentials/v2/query) - how `metadata_filters` interact with ranking
* [Create Database - API Reference](/api-reference/v2/endpoint/create-tenant) - full `database_metadata_schema` reference
* [Query - API Reference](/api-reference/v2/endpoint/query) - full `metadata_filters` reference
* [List Context](/api-reference/v2/endpoint/list-documents) - source browsing filters
* [Update Source Metadata](/api-reference/v2/endpoint/update-source-metadata) - point metadata edits
* [Update Metadata Schema](/api-reference/v2/endpoint/update-metadata-schema) - additive database schema changes