Build your own Glean with HydraDB

This guide will walk you through building an extremely powerful workplace search and AI assistant platform that rivals Glean using HydraDB APIs. You’ll learn how to create a unified retrieval experience across multiple data sources and generate answers in your application layer.

Note: All code in this guide uses the official HydraDB Python SDK. Base URL: https://api.hydradb.com. Get your API key at app.hydradb.com.

Prerequisites

Required knowledge: Python basics, REST APIs, environment variables
Required tools:

HydraDB API key
Python 3.11 or 3.12 (python --version)
pip install hydradb-sdk

What You’ll Build

By the end of this cookbook, you’ll be able to:

Ingest documents from multiple data sources (Slack, email, Google Drive, Jira) into a unified HydraDB knowledge base
Run natural language workplace search across all sources with a single /query call
Personalize search results per user by storing and retrieving AI memories
Generate grounded AI answers from retrieved context using any LLM

Overview

A Glean-like application typically includes these core features:

Universal Search: Search across multiple data sources (documents, emails, chats, etc.)
Retrieval-Assisted Answers: Generate intelligent answers from retrieved company knowledge
AI Memories for User Preferences: Remember and adapt to individual user preferences, search patterns, and behavioral patterns
Data Ingestion: Connect to various apps and services
Knowledge Graph: Build connections between information
Security & Access Control: Role-based permissions and data isolation

Architecture Overview

Step 1: Data Ingestion Strategy

1.1 App Connectors Setup

You’ll need to connect to various data sources. Here are recommended approaches:

Option A: Using Composio

Composio provides pre-built connectors for popular apps:

// Example: Setting up Composio connector
const composioConfig = {
  connectors: [
    {
      name: 'slack',
      config: {
        token: process.env.SLACK_BOT_TOKEN,
        channels: ['general', 'random', 'project-*']
      }
    },
    {
      name: 'gmail',
      config: {
        credentials: process.env.GMAIL_CREDENTIALS,
        labels: ['INBOX', 'SENT', 'IMPORTANT']
      }
    },
    {
      name: 'notion',
      config: {
        token: process.env.NOTION_TOKEN,
        databases: ['projects', 'docs', 'meetings']
      }
    }
  ]
};

Option B: Vanilla API Integration

For custom integrations, use the native APIs:

// Example: Slack API integration
class SlackConnector {
  constructor(token) {
    this.token = token;
    this.client = new WebClient(token);
  }

  async fetchMessages(channelId, limit = 100) {
    const result = await this.client.conversations.history({
      channel: channelId,
      limit: limit
    });
    
    return result.messages.map(msg => ({
      id: msg.ts,
      text: msg.text,
      user: msg.user,
      timestamp: msg.ts,
      channel: channelId,
      type: 'slack_message'
    }));
  }

  async fetchChannels() {
    const result = await this.client.conversations.list();
    return result.channels;
  }
}

// Example: Gmail API integration
class GmailConnector {
  constructor(credentials) {
    this.gmail = google.gmail({ version: 'v1', auth: credentials });
  }

  async fetchEmails(query = 'in:inbox', maxResults = 100) {
    const response = await this.gmail.users.messages.list({
      userId: 'me',
      q: query,
      maxResults: maxResults
    });

    const emails = [];
    for (const message of response.data.messages) {
      const email = await this.gmail.users.messages.get({
        userId: 'me',
        id: message.id
      });
      
      emails.push({
        id: email.data.id,
        subject: this.getHeader(email.data.payload.headers, 'Subject'),
        from: this.getHeader(email.data.payload.headers, 'From'),
        body: this.getBody(email.data.payload),
        timestamp: email.data.internalDate,
        type: 'gmail'
      });
    }
    
    return emails;
  }
}

1.2 Data Normalization

Create a unified data format for all sources:

Important: For optimal performance, limit each batch to a maximum of 20 app sources per request. Send multiple batch requests with an interval of 1 second between each request.

// Unified data structure for HydraDB app upload
const normalizedData = {
  id: 'unique_id',
  database: 'your_database',
  collection: 'your_collection',
  title: 'Document/Message Title',
  type: 'slack_message', // Source category: gmail, slack_message, notion_page, document, etc.
  timestamp: '2024-01-01T00:00:00Z', // ISO timestamp
  content: {
    text: 'Main content text',
    html_base64: 'base64_encoded_html',
    markdown: 'markdown_content'
  },
  url: 'https://app.com/item/123', // Optional: source URL
  description: 'Optional description of the source', // Optional
  metadata: {}, // Optional database-level metadata
  additional_metadata: {
    author: '[email protected]',
    id: 'original_id',
    tags: ['project-a', 'urgent', 'meeting-notes'],
    permissions: ['[email protected]', '[email protected]']
  }
};

1.3 Batch Upload to HydraDB

Use HydraDB’s batch upload capabilities for efficient data ingestion:

Best Practice: Always verify processing after upload using the /context/status endpoint to ensure your data is properly indexed.

import os
import json
from hydra_db import HydraDB

client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

# Upload a batch of knowledge sources
def upload_batch(sources: list, database: str, collection: str = None):
    app_knowledge = [
        {**source, "database": database, "collection": collection or database}
        for source in sources
    ]

    result = client.context.ingest(
        database=database,
        app_knowledge=json.dumps(app_knowledge)
    )
    return result

# Upload with verification — confirm each item is indexed before proceeding
def upload_with_verification(sources: list, database: str, collection: str = None):
    upload_result = upload_batch(sources, database, collection)

    if upload_result.data.results:
        for item in upload_result.data.results:
            id = item.id
            status = client.context.status(
                database=database,
                ids=[id]
            )
            items = status.data.statuses or []
            if items and items[0].indexing_status == "errored":
                raise Exception(f"Processing failed for source {id}")

    return upload_result

import { HydraDBClient } from "@hydradb/sdk";

const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

// Upload a batch of knowledge sources
const uploadBatch = async (sources: any[], database: string, collection?: string) => {
  const appKnowledge = sources.map(source => ({
    ...source,
    database: database,
    collection: collection || database
  }));

  const result = await client.context.ingest({
    database: database,
    appKnowledge: JSON.stringify(appKnowledge)
  });

  return result;
};

// Upload with verification — confirm each item is indexed before proceeding
const uploadWithVerification = async (sources: any[], database: string, collection?: string) => {
  const uploadResult = await uploadBatch(sources, database, collection);

  const items = uploadResult.data?.results ?? [];
  for (const item of items) {
    const id = item.id;
    const status = await client.context.status({
      database: database,
      ids: [id]
    });
    const statusItem = status.data.statuses[0];
    if (statusItem?.indexingStatus === "errored") {
      throw new Error(`Processing failed for source ${id}`);
    }
  }

  return uploadResult;
};

Step 2: Search and Answer Generation

2.1 Universal Search Interface

Create a search interface that queries across all data sources:

Note: HydraDB supports filtering by source_title and source_type using the metadata parameter. Use these for targeted searches across specific data sources.

import os
from hydra_db import HydraDB

client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

def search(query: str, database: str, collection: str = None,
           max_results: int = 10, mode: str = "fast",
           metadata: dict = None):
    return client.query(
        query=query,
        database=database,
        collection=collection,
        max_results=max_results,
        mode=mode,
        alpha=0.5,        # Balance semantic vs keyword search (0.0 to 1.0)
        recency_bias=0.3, # Recency preference (0.0 to 1.0)
        **({"metadata": metadata} if metadata else {}),
    )

# Filter search by source type
def query_by_source_type(query: str, database: str, source_type: str):
    return search(query, database, metadata={"source_type": source_type})

# Filter search by source title
def query_by_source_title(query: str, database: str, source_title: str):
    return search(query, database, metadata={"source_title": source_title})

import { HydraDBClient } from "@hydradb/sdk";

const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

interface SearchOptions {
  collection?: string;
  max_results?: number;
  mode?: string;
  metadata?: Record<string, any>;
}

const search = async (query: string, database: string, options: SearchOptions = {}) => {
  const {
    collection,
    max_results = 10,
    mode = "fast",
    metadata,
  } = options;

  return await client.query({
    query,
    database: database,
    collection: collection,
    maxResults: max_results,
    mode,
    alpha: 0.5,         // Balance semantic vs keyword search (0.0 to 1.0)
    recencyBias: 0.3,  // Recency preference (0.0 to 1.0)
    ...(metadata && { metadata }),
  });
};

// Filter search by source type
const searchBySourceType = (query: string, database: string, sourceType: string) =>
  search(query, database, { metadata: { source_type: sourceType } });

// Filter search by source title
const searchBySourceTitle = (query: string, database: string, sourceTitle: string) =>
  search(query, database, { metadata: { source_title: sourceTitle } });

2.2 Advanced Search Features

Implement advanced search capabilities:

Advanced Features:

mode: Use "thinking" for multi-query retrieval with reranking, or "fast" for single-query retrieval

alpha: Controls semantic vs keyword matching (0.0-1.0, or "auto")

recency_bias: Prioritizes recent content (0.0-1.0)

import os
import json
from hydra_db import HydraDB

client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

# Search with optional source type / title metadata filters
def search_with_filters(query: str, database: str, source_types: str = None, source_titles: str = None):
    metadata = {}
    if source_types:
        metadata["source_type"] = source_types
    if source_titles:
        metadata["source_title"] = source_titles

    return client.query(
        query=query,
        database=database,
        max_results=10,
        mode="fast",
        alpha=0.5,
        recency_bias=0.3,
        **({"metadata": metadata} if metadata else {})
    )

# Guide retrieval with additional context — prepend context to the query string
def search_with_context(query: str, database: str, context: str):
    return client.query(
        query=f"{context}\n\n{query}" if context else query,
        database=database,
        max_results=10,
        mode="fast",
        alpha=0.5,
        recency_bias=0.3,
    )

# Conversational search — fold prior conversation turns into the query
def conversational_search(query: str, database: str, conversation_history: list = None):
    history = conversation_history or []
    enriched_query = (
        f"Previous conversation context: {json.dumps(history)}\n\n{query}"
        if history else query
    )
    return client.query(
        query=enriched_query,
        database=database,
        max_results=10,
        mode="fast",
        alpha=0.5,
        recency_bias=0.3,
    )

import { HydraDBClient } from "@hydradb/sdk";

const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

// Search with optional source type / title metadata filters
const searchWithFilters = async (
  query: string,
  database: string,
  filters: { sourceTypes?: string; sourceTitles?: string } = {}
) => {
  const metadata: Record<string, any> = {};
  if (filters.sourceTypes) metadata.source_type = filters.sourceTypes;
  if (filters.sourceTitles) metadata.source_title = filters.sourceTitles;

  return await client.query({
    query,
    database: database,
    maxResults: 10,
    mode: "fast",
    alpha: 0.5,
    recencyBias: 0.3,
    ...(Object.keys(metadata).length > 0 && { metadata })
  });
};

// Guide retrieval with additional context — prepend context to the query string
const searchWithContext = async (query: string, database: string, context: string) =>
  client.query({
    query: context ? `${context}\n\n${query}` : query,
    database: database,
    maxResults: 10,
    mode: "fast",
    alpha: 0.5,
    recencyBias: 0.3,
  });

// Conversational search — fold prior conversation turns into the query
const conversationalSearch = async (
  query: string,
  database: string,
  conversationHistory: object[] = []
) =>
  client.query({
    query: conversationHistory.length
      ? `Previous conversation context: ${JSON.stringify(conversationHistory)}\n\n${query}`
      : query,
    database: database,
    maxResults: 10,
    mode: "fast",
    alpha: 0.5,
    recencyBias: 0.3,
  });

2.3 AI Memories and User Preferences

One of the most powerful features of building a Glean-like application with HydraDB is leveraging AI Memories to create truly personalized experiences. HydraDB automatically manages AI memories using collection for user-level isolation. This allows your application to remember user preferences, past interactions, and behavioral patterns, making every search and interaction more relevant and efficient.

Understanding AI Memories

HydraDB’s AI memories are dynamic, user-specific profiles that evolve over time. They capture not just what users say, but their intentions, preferences, and unique behaviors. HydraDB automatically manages these memories using collection for user-level isolation. This enables your Glean clone to:

Remember User Preferences: Format preferences, source preferences, search patterns
Understand Intent: Learn what types of information users typically seek
Adapt Responses: Tailor answers based on past interactions
Anticipate Needs: Suggest relevant information before users ask

Implementing AI Memories in Your Search

import os
from datetime import datetime
from hydra_db import HydraDB

client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

# In-memory profile store — replace with your own database in production
user_profiles: dict = {}

def get_user_profile(user_id: str) -> dict:
    if user_id not in user_profiles:
        user_profiles[user_id] = {
            "user_id": user_id,
            "preferred_mode": "fast",
            "preferred_source_types": [],
            "frequent_queries": [],
            "preferred_formats": ["bullet_points"],
            "response_style": "concise",
            "favorite_sources": [],
            "search_history": [],
            "last_interaction": None
        }
    return user_profiles[user_id]

def build_personalized_instructions(profile: dict, query: str) -> str:
    instructions = "User preferences: "
    if "bullet_points" in profile["preferred_formats"]:
        instructions += "Prefer bullet point responses. "
    if profile["response_style"] == "concise":
        instructions += "Keep responses concise and to the point. "
    if profile["favorite_sources"]:
        instructions += f"User frequently uses sources: {', '.join(profile['favorite_sources'])}. "
    if profile["frequent_queries"]:
        instructions += f"User often searches for: {', '.join(profile['frequent_queries'][:3])}. "
    instructions += f"Current query: {query}"
    return instructions

def save_user_profile(user_id: str, profile: dict):
    # Persist to your backend database
    print(f"Saving profile for user {user_id}:", profile)

# Search with personalized query enrichment derived from the user's local profile
def search_with_memory(query: str, database: str, user_id: str):
    profile = get_user_profile(user_id)
    enriched_query = f"{build_personalized_instructions(profile, query)}\n\n{query}"

    kwargs = dict(
        query=enriched_query,
        database=database,
        collection=user_id,
        max_results=10,
        mode=profile["preferred_mode"] or "fast",
        alpha=0.5,
        recency_bias=0.3,
    )
    if profile["preferred_source_types"]:
        kwargs["metadata"] = {"source_type": profile["preferred_source_types"]}

    search_results = client.query(**kwargs)

    # Update local profile history
    profile["search_history"].append({
        "query": query,
        "timestamp": datetime.utcnow().isoformat(),
        "result_count": len(search_results.data.chunks or [])
    })
    profile["search_history"] = profile["search_history"][-50:]

    if not any(query.lower() in q.lower() for q in profile["frequent_queries"]):
        profile["frequent_queries"] = ([query] + profile["frequent_queries"])[:10]

    for chunk in (search_results.data.chunks or []):
        if chunk.source_title and chunk.source_title not in profile["favorite_sources"]:
            profile["favorite_sources"].append(chunk.source_title)
    profile["favorite_sources"] = profile["favorite_sources"][:5]

    profile["last_interaction"] = datetime.utcnow().isoformat()
    save_user_profile(user_id, profile)

    return search_results

import { HydraDBClient } from "@hydradb/sdk";

const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

// In-memory profile store — replace with your own database in production
const userProfiles = new Map<string, any>();

const getUserProfile = (userId: string) => {
  if (!userProfiles.has(userId)) {
    userProfiles.set(userId, {
      userId,
      preferredMode: "fast",
      preferredSourceTypes: [] as string[],
      frequentQueries: [] as string[],
      preferredFormats: ["bullet_points"],
      responseStyle: "concise",
      favoriteSources: [] as string[],
      searchHistory: [] as object[],
      lastInteraction: null as string | null
    });
  }
  return userProfiles.get(userId)!;
};

const buildPersonalizedInstructions = (userProfile: any, query: string): string => {
  let instructions = "User preferences: ";
  if (userProfile.preferredFormats.includes("bullet_points")) instructions += "Prefer bullet point responses. ";
  if (userProfile.responseStyle === "concise") instructions += "Keep responses concise and to the point. ";
  if (userProfile.favoriteSources.length > 0) instructions += `User frequently uses sources: ${userProfile.favoriteSources.join(", ")}. `;
  if (userProfile.frequentQueries.length > 0) instructions += `User often searches for: ${userProfile.frequentQueries.slice(0, 3).join(", ")}. `;
  instructions += `Current query: ${query}`;
  return instructions;
};

const saveUserProfile = async (userId: string, profile: any) => {
  // Persist to your backend database
  console.log(`Saving profile for user ${userId}:`, profile);
};

// Search with personalized query enrichment derived from the user's local profile
const searchWithMemory = async (query: string, database: string, userId: string) => {
  const userProfile = getUserProfile(userId);
  const enrichedQuery = `${buildPersonalizedInstructions(userProfile, query)}\n\n${query}`;

  const searchResults = await client.query({
    query: enrichedQuery,
    database: database,
    collection: userId,
    maxResults: 10,
    mode: userProfile.preferredMode || "fast",
    alpha: 0.5,
    recencyBias: 0.3,
    ...(userProfile.preferredSourceTypes?.length > 0 && {
      metadata: { source_type: userProfile.preferredSourceTypes }
    })
  });

  // Update local profile history
  userProfile.searchHistory.push({ query, timestamp: new Date().toISOString(), resultCount: searchResults.data.chunks?.length || 0 });
  if (userProfile.searchHistory.length > 50) userProfile.searchHistory = userProfile.searchHistory.slice(-50);

  const queryLower = query.toLowerCase();
  if (!userProfile.frequentQueries.some((q: string) => q.toLowerCase().includes(queryLower))) {
    userProfile.frequentQueries = [query, ...userProfile.frequentQueries].slice(0, 10);
  }

  if (searchResults.data.chunks) {
    for (const chunk of searchResults.data.chunks) {
      if (chunk.source && !userProfile.favoriteSources.includes(chunk.source)) {
        userProfile.favoriteSources.push(chunk.source);
      }
    }
    userProfile.favoriteSources = userProfile.favoriteSources.slice(0, 5);
  }

  userProfile.lastInteraction = new Date().toISOString();
  await saveUserProfile(userId, userProfile);

  return searchResults;
};

User Preference Learning

class PreferenceLearner {
  constructor() {
    this.preferencePatterns = new Map();
  }

  async learnFromInteraction(userId, interaction) {
    const {
      query,
      selectedResults,
      responseFormat,
      searchFilters,
      timeSpent,
      followUpQueries
    } = interaction;

    const patterns = this.preferencePatterns.get(userId) || {
      queryPatterns: [],
      formatPreferences: {},
      sourcePreferences: {},
      timePatterns: [],
      filterPreferences: {}
    };

    // Learn query patterns
    this.learnQueryPatterns(patterns, query, followUpQueries);
    
    // Learn format preferences
    this.learnFormatPreferences(patterns, responseFormat, selectedResults);
    
    // Learn source preferences
    this.learnSourcePreferences(patterns, selectedResults);
    
    // Learn time patterns
    this.learnTimePatterns(patterns, timeSpent);
    
    // Learn filter preferences
    this.learnFilterPreferences(patterns, searchFilters);

    this.preferencePatterns.set(userId, patterns);
  }

  learnQueryPatterns(patterns, query, followUpQueries) {
    // Analyze query complexity, length, and type
    const queryAnalysis = {
      length: query.length,
      complexity: this.analyzeComplexity(query),
      type: this.classifyQueryType(query),
      hasFilters: query.includes('in:') || query.includes('from:'),
      timestamp: new Date().toISOString()
    };

    patterns.queryPatterns.push(queryAnalysis);
    
    // Keep only recent patterns
    if (patterns.queryPatterns.length > 100) {
      patterns.queryPatterns = patterns.queryPatterns.slice(-100);
    }
  }

  learnFormatPreferences(patterns, format, selectedResults) {
    if (!patterns.formatPreferences[format]) {
      patterns.formatPreferences[format] = 0;
    }
    patterns.formatPreferences[format]++;
  }

  learnSourcePreferences(patterns, selectedResults) {
    selectedResults.forEach(result => {
      const sourceType = result.source;
      if (!patterns.sourcePreferences[sourceType]) {
        patterns.sourcePreferences[sourceType] = 0;
      }
      patterns.sourcePreferences[sourceType]++;
    });
  }

  analyzeComplexity(query) {
    const words = query.split(' ').length;
    const hasQuotes = query.includes('"');
    const hasOperators = /AND|OR|NOT|in:|from:|to:/.test(query);
    
    let complexity = 'simple';
    if (words > 5 || hasQuotes || hasOperators) complexity = 'complex';
    if (words > 10 || (hasQuotes && hasOperators)) complexity = 'advanced';
    
    return complexity;
  }

  classifyQueryType(query) {
    if (query.includes('how to') || query.includes('steps')) return 'how-to';
    if (query.includes('what is') || query.includes('define')) return 'definition';
    if (query.includes('when') || query.includes('schedule')) return 'temporal';
    if (query.includes('who') || query.includes('contact')) return 'person';
    if (query.includes('where') || query.includes('location')) return 'location';
    return 'general';
  }
}

Memory-Enhanced Search Interface

const MemoryEnhancedSearch = ({ userId }) => {
  const [query, setQuery] = useState('');
  const [results, setResults] = useState(null);
  const [userPreferences, setUserPreferences] = useState(null);
  const [suggestions, setSuggestions] = useState([]);

  const searchClient = new PersonalizedSearch(API_KEY, TENANT_ID);
  const preferenceLearner = new PreferenceLearner();

  useEffect(() => {
    // Load user preferences on component mount
    loadUserPreferences();
  }, [userId]);

  const loadUserPreferences = async () => {
    const profile = await searchClient.getUserProfile(userId);
    setUserPreferences(profile);
    
    // Generate search suggestions based on user's history
    const suggestions = generateSuggestions(profile);
    setSuggestions(suggestions);
  };

  const generateSuggestions = (profile) => {
    const suggestions = [];
    
    // Suggest based on frequent queries
    if (profile.frequentQueries.length > 0) {
      suggestions.push({
        type: 'frequent',
        text: profile.frequentQueries[0],
        label: 'Frequently searched'
      });
    }
    
    // Suggest based on recent searches
    if (profile.searchHistory.length > 0) {
      const recentSearches = profile.searchHistory
        .slice(-3)
        .map(h => h.query);
      
      recentSearches.forEach(query => {
        suggestions.push({
          type: 'recent',
          text: query,
          label: 'Recent search'
        });
      });
    }
    
    return suggestions;
  };

  const handleSearch = async () => {
    const startTime = Date.now();
    
    try {
      const searchResults = await searchClient.searchWithMemory(query, userId);
      setResults(searchResults);
      
      // Learn from this interaction
      const interaction = {
        query,
        selectedResults: [], // Will be populated when user clicks results
        responseFormat: searchResults.format || 'default',
        searchFilters: {},
        timeSpent: Date.now() - startTime,
        followUpQueries: []
      };
      
      await preferenceLearner.learnFromInteraction(userId, interaction);
      
    } catch (error) {
      console.error('Search failed:', error);
    }
  };

  const handleResultClick = async (result) => {
    // Update user preferences when they click on results
    const profile = await searchClient.getUserProfile(userId);
    
    // Mark this source type as preferred
    if (result.source && !profile.preferredSourceTypes.includes(result.source)) {
      profile.preferredSourceTypes.push(result.source);
      await searchClient.saveUserProfile(userId, profile);
    }
  };

  return (
    <div className="memory-enhanced-search">
      <div className="search-header">
        <input
          type="text"
          value={query}
          onChange={(e) => setQuery(e.target.value)}
          placeholder="Search with your preferences in mind..."
          className="search-input"
        />
        <button onClick={handleSearch}>Search</button>
      </div>

      {/* Show personalized suggestions */}
      {suggestions.length > 0 && (
        <div className="search-suggestions">
          <h4>Based on your preferences:</h4>
          {suggestions.map((suggestion, index) => (
            <button
              key={index}
              className="suggestion-chip"
              onClick={() => setQuery(suggestion.text)}
            >
              {suggestion.text}
              <span className="suggestion-label">{suggestion.label}</span>
            </button>
          ))}
        </div>
      )}

      {/* Show user preferences summary */}
      {userPreferences && (
        <div className="user-preferences">
          <details>
            <summary>Your Search Preferences</summary>
            <div className="preferences-content">
              <p><strong>Preferred sources:</strong> {userPreferences.preferredSourceTypes.join(', ') || 'None set'}</p>
              <p><strong>Response style:</strong> {userPreferences.responseStyle}</p>
              <p><strong>Frequent searches:</strong> {userPreferences.frequentQueries.slice(0, 3).join(', ')}</p>
            </div>
          </details>
        </div>
      )}

      {results && (
        <SearchResults 
          results={results} 
          onResultClick={handleResultClick}
          userPreferences={userPreferences}
        />
      )}
    </div>
  );
};

Benefits of AI Memories in Your Glean Clone

Personalized Search Results: Users get results tailored to their preferences and past behavior
Faster Information Discovery: The system learns what sources and formats users prefer
Improved User Experience: Every interaction feels more personal and relevant
Reduced Cognitive Load: Users don’t need to repeat their preferences or search patterns
Adaptive Learning: The system continuously improves based on user interactions
Automatic Management: HydraDB handles memory updates automatically - no manual implementation required

Best Practices for AI Memories

Respect Privacy: Always give users control over their data and preferences
Transparency: Show users what preferences are being used and allow them to modify them
Graceful Degradation: Ensure the system works well even without user history
Collection Isolation: Use collection to isolate user data for personalized experiences
Performance: Cache user profiles to avoid repeated API calls

Step 3: Frontend Implementation

3.1 Search Interface

// React component example
import React, { useState, useEffect } from 'react';

const GleanSearchInterface = () => {
  const [query, setQuery] = useState('');
  const [results, setResults] = useState(null);
  const [loading, setLoading] = useState(false);
  const [filters, setFilters] = useState({
    sourceTypes: [],
    dateRange: null,
    authors: []
  });

  const searchClient = new AdvancedSearch(API_KEY, TENANT_ID);

  const handleSearch = async () => {
    setLoading(true);
    try {
      const searchResults = await searchClient.searchWithFilters(query, filters);
      setResults(searchResults);
    } catch (error) {
      console.error('Search failed:', error);
    } finally {
      setLoading(false);
    }
  };

  return (
    <div className="glean-search">
      <div className="search-header">
        <input
          type="text"
          value={query}
          onChange={(e) => setQuery(e.target.value)}
          placeholder="Search across all your work..."
          className="search-input"
        />
        <button onClick={handleSearch} disabled={loading}>
          {loading ? 'Searching...' : 'Search'}
        </button>
      </div>

      <div className="search-filters">
        <SourceTypeFilter
          value={filters.sourceTypes}
          onChange={(types) => setFilters({...filters, sourceTypes: types})}
        />
        <DateRangeFilter
          value={filters.dateRange}
          onChange={(range) => setFilters({...filters, dateRange: range})}
        />
      </div>

      {results && (
        <SearchResults results={results} />
      )}
    </div>
  );
};

3.2 Results Display

const SearchResults = ({ results }) => {
  const { chunks, graphContext } = results.data ?? {};

  return (
    <div className="search-results">
      {chunks && chunks.length > 0 && (
        <div className="source-results">
          <h3>Results ({chunks.length})</h3>
          {chunks.map((chunk, index) => (
            <ChunkCard key={index} chunk={chunk} />
          ))}
        </div>
      )}

      {graphContext && graphContext.queryPaths && (
        <div className="graph-context">
          <h3>Related Knowledge Graph Paths</h3>
          <pre>{JSON.stringify(graphContext.queryPaths, null, 2)}</pre>
        </div>
      )}
    </div>
  );
};

const ChunkCard = ({ chunk }) => {
  return (
    <div className="source-card">
      <div className="source-header">
        <span className="source-type">{chunk.source}</span>
        <span className="source-title">{chunk.source_title}</span>
        <span className="source-date">{formatDate(chunk.timestamp)}</span>
      </div>

      <div className="source-chunk">
        <p>{chunk.chunk_content}</p>
        {chunk.bounding_box && (
          <div className="chunk-highlight">
            Position: {chunk.bounding_box.x}, {chunk.bounding_box.y}
          </div>
        )}
      </div>
    </div>
  );
};

Step 4: Data Synchronization

4.1 Scheduled Sync Jobs

class DataSyncManager {
  constructor(connectors, cortexIngestion) {
    this.connectors = connectors;
    this.cortexIngestion = cortexIngestion;
    this.syncIntervals = {
      slack: 5 * 60 * 1000, // 5 minutes
      gmail: 10 * 60 * 1000, // 10 minutes
      notion: 30 * 60 * 1000, // 30 minutes
      documents: 60 * 60 * 1000 // 1 hour
    };
  }

  async startSync() {
    // Start sync jobs for each connector
    Object.entries(this.syncIntervals).forEach(([connector, interval]) => {
      setInterval(() => {
        this.syncConnector(connector);
      }, interval);
    });
  }

  async syncConnector(connectorName) {
    try {
      const connector = this.connectors[connectorName];
      const newData = await connector.fetchNewData();
      
      if (newData.length > 0) {
        const normalizedData = newData.map(item => 
          this.normalizeData(item, connectorName)
        );
        
        // Use batch upload with verification
        await this.cortexIngestion.uploadWithVerification(normalizedData);
        console.log(`Synced ${newData.length} items from ${connectorName}`);
      }
    } catch (error) {
      console.error(`Sync failed for ${connectorName}:`, error);
      // Implement retry logic with exponential backoff
      await this.retrySync(connectorName, error);
    }
  }

  async retrySync(connectorName, error, attempt = 1) {
    const maxAttempts = 3;
    const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
    
    if (attempt < maxAttempts) {
      console.log(`Retrying sync for ${connectorName} in ${delay}ms (attempt ${attempt + 1})`);
      setTimeout(() => {
        this.syncConnector(connectorName);
      }, delay);
    } else {
      console.error(`Max retry attempts reached for ${connectorName}:`, error);
    }
  }

  normalizeData(item, sourceType) {
    return {
      id: `${sourceType}_${item.id}`,
      title: item.title || item.subject || item.text?.substring(0, 100),
      source: sourceType,
      timestamp: item.timestamp || item.created_at || new Date().toISOString(),
      content: {
        text: item.text || item.body || item.content,
        html_base64: item.html ? btoa(item.html) : '',
        markdown: item.markdown || ''
      },
      url: item.url,
      description: item.description,
      metadata: {},
      additional_metadata: {
        id: item.id,
        author: item.author || item.user,
        tags: item.tags || [],
        created_at: item.created_at,
        updated_at: item.updated_at
      }
    };
  }
}

4.2 Webhook Integration

import os
import json
from hydra_db import HydraDB
from flask import Flask, request

app = Flask(__name__)
client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

@app.post("/webhooks/slack")
def slack_webhook():
    event = request.json.get("event", {})

    if event.get("type") == "message":
        from datetime import datetime
        normalized_data = {
            "id": f"slack_{event['ts']}",
            "database": os.environ["TENANT_ID"],
            "collection": os.environ["SUB_TENANT_ID"],
            "title": f"Message in {event['channel']}",
            "type": "slack_message",
            "content": {"text": event["text"]},
            "metadata": {},
            "additional_metadata": {
                "id": event["ts"],
                "author": event["user"],
                "created_at": datetime.utcfromtimestamp(float(event["ts"])).isoformat(),
                "channel": event["channel"]
            }
        }

        client.context.ingest(
            database=os.environ["TENANT_ID"],
            app_knowledge=json.dumps([normalized_data])
        )

    return "OK", 200

import { HydraDBClient } from "@hydradb/sdk";

const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

// Express.js webhook handler
app.post("/webhooks/slack", async (req, res) => {
  const { event } = req.body;

  if (event.type === "message") {
    const normalizedData = {
      id: `slack_${event.ts}`,
      database: process.env.TENANT_ID!,
      collection: process.env.SUB_TENANT_ID!,
      title: `Message in ${event.channel}`,
      type: "slack_message",
      content: { text: event.text },
      metadata: {},
      additional_metadata: {
        id: event.ts,
        author: event.user,
        created_at: new Date(event.ts * 1000).toISOString(),
        channel: event.channel
      }
    };

    await client.context.ingest({
      database: process.env.TENANT_ID!,
      appKnowledge: JSON.stringify([normalizedData])
    });
  }

  res.status(200).send("OK");
});

Step 5: Security and Access Control

5.1 Multi-Tenant Architecture

class TenantManager {
  constructor() {
    this.tenants = new Map();
  }

  async createTenant(database, config) {
    const tenant = {
      id: database,
      subTenants: new Map(),
      permissions: config.permissions || {},
      dataSources: config.dataSources || []
    };
    
    this.tenants.set(database, tenant);
    return tenant;
  }

  async createSubTenant(database, collection, config) {
    const tenant = this.tenants.get(database);
    if (!tenant) throw new Error('Database not found');

    const subTenant = {
      id: collection,
      parentTenant: database,
      permissions: config.permissions || {},
      dataSources: config.dataSources || []
    };

    tenant.subTenants.set(collection, subTenant);
    return subTenant;
  }

  async searchWithTenantContext(query, database, collection = null, userId = null) {
    const tenant = this.tenants.get(database);
    if (!tenant) throw new Error('Database not found');

    // Check user permissions
    if (userId && !this.hasPermission(tenant, collection, userId)) {
      throw new Error('Access denied');
    }

    const searchOptions = {
      database,
      collection,
      metadata: {
        database: database
      }
    };

    if (collection) {
      searchOptions.metadata.collection = collection;
    }

    return await searchClient.search(query, searchOptions);
  }

  hasPermission(tenant, collection, userId) {
    // Implement your permission logic here
    return true; // Simplified for example
  }
}

5.2 Data Privacy and Compliance

class DataPrivacyManager {
  constructor() {
    this.retentionPolicies = new Map();
  }

  async applyRetentionPolicy(database, policy) {
    const { retentionDays, dataTypes, autoDelete } = policy;
    
    if (autoDelete) {
      const cutoffDate = new Date();
      cutoffDate.setDate(cutoffDate.getDate() - retentionDays);
      
      // Delete old data from HydraDB
      await this.deleteOldData(database, cutoffDate);
    }
  }

  async deleteOldData(database, memoryId, collection = null) {
    // Use HydraDB's context deletion endpoint for memory deletion.
    // Raw fetch to the REST API: the JSON body uses snake_case wire keys.
    const request = { database: database, ids: [memoryId] };
    if (collection) {
      request.collection = collection;
    }

    const response = await fetch('https://api.hydradb.com/context', {
      method: 'DELETE',
      headers: {
        'Authorization': `Bearer ${API_KEY}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({
        type: 'memory',
        request
      })
    });

    return response.json();
  }

  async anonymizeData(data, anonymizationRules) {
    // Implement data anonymization based on rules
    let anonymizedData = { ...data };
    
    anonymizationRules.forEach(rule => {
      if (rule.field && anonymizedData[rule.field]) {
        anonymizedData[rule.field] = this.anonymizeValue(
          anonymizedData[rule.field], 
          rule.method
        );
      }
    });
    
    return anonymizedData;
  }

  anonymizeValue(value, method) {
    switch (method) {
      case 'hash':
        return crypto.createHash('sha256').update(value).digest('hex');
      case 'mask':
        return value.replace(/./g, '*');
      case 'redact':
        return '[REDACTED]';
      default:
        return value;
    }
  }
}

Step 6: Performance Optimization

6.1 Caching Strategy

class SearchCache {
  constructor() {
    this.cache = new Map();
    this.ttl = 5 * 60 * 1000; // 5 minutes
  }

  async getCachedResults(query, filters) {
    const key = this.generateCacheKey(query, filters);
    const cached = this.cache.get(key);
    
    if (cached && Date.now() - cached.timestamp < this.ttl) {
      return cached.results;
    }
    
    return null;
  }

  async setCachedResults(query, filters, results) {
    const key = this.generateCacheKey(query, filters);
    this.cache.set(key, {
      results,
      timestamp: Date.now()
    });
  }

  generateCacheKey(query, filters) {
    return `${query}_${JSON.stringify(filters)}`;
  }

  clearExpired() {
    const now = Date.now();
    for (const [key, value] of this.cache.entries()) {
      if (now - value.timestamp > this.ttl) {
        this.cache.delete(key);
      }
    }
  }
}

6.2 Rate Limiting

class RateLimiter {
  constructor(limit, windowMs) {
    this.limit = limit;
    this.windowMs = windowMs;
    this.requests = new Map();
  }

  async checkLimit(userId) {
    const now = Date.now();
    const userRequests = this.requests.get(userId) || [];
    
    // Remove old requests outside the window
    const validRequests = userRequests.filter(
      timestamp => now - timestamp < this.windowMs
    );
    
    if (validRequests.length >= this.limit) {
      return false; // Rate limit exceeded
    }
    
    // Add current request
    validRequests.push(now);
    this.requests.set(userId, validRequests);
    
    return true; // Request allowed
  }
}

Step 7: Monitoring and Analytics

7.1 Search Analytics

class SearchAnalytics {
  constructor() {
    this.metrics = {
      searches: 0,
      successfulSearches: 0,
      failedSearches: 0,
      averageResponseTime: 0,
      popularQueries: new Map(),
      sourceTypeUsage: new Map()
    };
  }

  async trackSearch(query, filters, results, responseTime) {
    this.metrics.searches++;
    
    if (results && results.answer) {
      this.metrics.successfulSearches++;
    } else {
      this.metrics.failedSearches++;
    }

    // Track popular queries
    const queryKey = query.toLowerCase().trim();
    this.metrics.popularQueries.set(
      queryKey, 
      (this.metrics.popularQueries.get(queryKey) || 0) + 1
    );

    // Track source type usage
    if (results && results.data?.chunks) {
      results.data.chunks.forEach(chunk => {
        const sourceType = chunk.source;
        this.metrics.sourceTypeUsage.set(
          sourceType,
          (this.metrics.sourceTypeUsage.get(sourceType) || 0) + 1
        );
      });
    }

    // Update average response time
    this.updateAverageResponseTime(responseTime);
  }

  updateAverageResponseTime(newTime) {
    const currentAvg = this.metrics.averageResponseTime;
    const totalSearches = this.metrics.searches;
    
    this.metrics.averageResponseTime = 
      (currentAvg * (totalSearches - 1) + newTime) / totalSearches;
  }

  getMetrics() {
    return {
      ...this.metrics,
      popularQueries: Array.from(this.metrics.popularQueries.entries())
        .sort((a, b) => b[1] - a[1])
        .slice(0, 10),
      sourceTypeUsage: Array.from(this.metrics.sourceTypeUsage.entries())
        .sort((a, b) => b[1] - a[1])
    };
  }
}

Best Practices and Recommendations

1. Data Ingestion Best Practices

Batch Processing: Use HydraDB’s batch upload endpoints for efficiency
Batch Limits: Limit to 20 app sources per request with 1-second intervals between batches
Incremental Sync: Only sync new/changed data to minimize API calls
Error Handling: Implement retry logic with exponential backoff
Processing Verification: Always verify upload processing using /context/status
Rate Limiting: Respect API rate limits and implement queuing

2. Search Optimization

Query Preprocessing: Clean and normalize user queries
Result Ranking: Use alpha and recency_bias for fine-tuning
Metadata Filtering: Use source_title and source_type for targeted searches
Thinking Mode: Use mode: "thinking" for complex queries that benefit from multi-query retrieval with reranking
Caching: Cache frequent queries and results

3. Security Considerations

Data Encryption: Encrypt sensitive data at rest and in transit
Access Control: Implement role-based access control (RBAC)
Audit Logging: Log all search queries and data access
Data Retention: Implement automatic data deletion policies

4. Performance Tips

Connection Pooling: Reuse HTTP connections
Async Processing: Use async/await for non-blocking operations
Memory Management: Implement proper cleanup for large datasets
Batch Optimization: Respect 20-source batch limits and 1-second intervals
Processing Verification: Verify uploads to ensure data is properly indexed
Monitoring: Track response times and error rates

5. User Experience

Autocomplete: Implement search suggestions
Faceted Search: Allow filtering by source type, date, author
Saved Searches: Let users save and share search queries
Export Results: Allow users to export search results

Deployment Checklist

Set up authentication and authorization
Configure data connectors and sync schedules
Implement error handling and monitoring
Set up caching and rate limiting
Configure backup and disaster recovery
Test with production data volumes
Set up analytics and reporting
Document API usage and troubleshooting

Conclusion

Building a Glean-like application with HydraDB APIs provides you with a powerful, scalable foundation for workplace search and AI assistance. By following this guide and implementing the best practices outlined, you can create a comprehensive solution that rivals commercial offerings while maintaining full control over your data and user experience. The key to success is starting with a solid architecture, implementing proper data synchronization, and gradually adding advanced features like multi-step reasoning, conversation memory, and personalized responses. HydraDB’s APIs provide the AI capabilities you need, while your application handles the data ingestion, user interface, and business logic. Remember to monitor performance, gather user feedback, and continuously iterate on your implementation to create the best possible user experience.

Change Log

Date	Change
2026-05-14	Replaced raw `fetch()`-based `HydraDBDataIngestion` class with official SDK calls (`client.context.ingest`, `client.context.status`)
2026-05-14	Replaced raw `fetch()`-based `GleanSearch` class with `client.query` SDK call
2026-05-14	Replaced `AdvancedSearch` class with standalone SDK helper functions (`searchWithFilters`, `searchWithContext`, `conversationalSearch`)
2026-05-14	Replaced `PersonalizedSearch` class with flat SDK-based `searchWithMemory` function
2026-05-14	Replaced webhook `cortexIngestion.uploadBatch()` call with `client.context.ingest()` SDK call
2026-05-14	Added Python SDK equivalents in `<CodeGroup>` tabs for all replaced code blocks
2026-05-14	Removed “Status: In progress” notice

​Prerequisites

​What You’ll Build

​Overview

​Architecture Overview

​Step 1: Data Ingestion Strategy

​1.1 App Connectors Setup

​Option A: Using Composio

​Option B: Vanilla API Integration

​1.2 Data Normalization

​1.3 Batch Upload to HydraDB

​Step 2: Search and Answer Generation

​2.1 Universal Search Interface

​2.2 Advanced Search Features

​2.3 AI Memories and User Preferences

​Understanding AI Memories

​Implementing AI Memories in Your Search

​User Preference Learning

​Memory-Enhanced Search Interface

​Benefits of AI Memories in Your Glean Clone

​Best Practices for AI Memories

​Step 3: Frontend Implementation

​3.1 Search Interface

​3.2 Results Display

​Step 4: Data Synchronization

​4.1 Scheduled Sync Jobs

​4.2 Webhook Integration

​Step 5: Security and Access Control

​5.1 Multi-Tenant Architecture

​5.2 Data Privacy and Compliance

​Step 6: Performance Optimization

​6.1 Caching Strategy

​6.2 Rate Limiting

​Step 7: Monitoring and Analytics

​7.1 Search Analytics

​Best Practices and Recommendations

​1. Data Ingestion Best Practices

​2. Search Optimization

​3. Security Considerations

​4. Performance Tips

​5. User Experience

​Deployment Checklist

​Conclusion

​Change Log