Elasticsearch to OpenSearch Migration Plan

Purpose

Document the feasibility and plan for migrating from Elasticsearch 8.x to OpenSearch 2.x. This migration would reduce licensing concerns and costs while maintaining feature parity.

Current State

Elasticsearch Usage in Codebase

Current Client

• Package: @elastic/elasticsearch v8.x
• Connection: Via ElasticsearchService from @nestjs/elasticsearch

Compatibility Assessment

What Works the Same

What Requires Changes

Migration Steps

Phase 1: Client Swap (Day 1)

1. Install OpenSearch client
   npm uninstall @elastic/elasticsearch @nestjs/elasticsearch
   npm install @opensearch-project/opensearch
   
2. Create OpenSearch module wrapper
   • Create opensearch.module.ts to replace ElasticsearchModule
   • Maintain same service interface
3. Update imports across codebase
   • Find/replace ElasticsearchService references

Phase 2: Vector Search Migration (Day 1-2)

1. Update index mappings for vector fields Before (Elasticsearch):
   {
     "product_embedding": {
       "type": "dense_vector",
       "dims": 1536,
       "index": true,
       "similarity": "cosine"
     }
   }
   
   After (OpenSearch):
   {
     "product_embedding": {
       "type": "knn_vector",
       "dimension": 1536,
       "method": {
         "name": "hnsw",
         "space_type": "cosinesimil",
         "engine": "nmslib"
       }
     }
   }
   
2. Update KNN query syntax Before (Elasticsearch 8.x):
   {
     knn: {
       field: 'product_embedding',
       query_vector: userEmbedding,
       k: 10,
       num_candidates: 50
     }
   }
   
   After (OpenSearch):
   {
     query: {
       knn: {
         product_embedding: {
           vector: userEmbedding,
           k: 10
         }
       }
     }
   }
   
3. Reindex data with new mappings

Phase 3: Testing & Validation (Day 2-3)

1. Unit tests - Verify all search functions work
2. Integration tests - Test against OpenSearch instance
3. Performance comparison - Benchmark KNN search latency
4. Data validation - Ensure search results quality unchanged

Phase 4: Deployment

1. Set up OpenSearch cluster (or use managed service)
2. Migrate data from Elasticsearch
3. Update environment configurations
4. Deploy updated application
5. Monitor for issues

Files to Modify

Estimated Effort

Risks & Mitigations

Decision

Status: Planned for future (not urgent) Rationale: Current Elasticsearch setup works well. Migration provides cost savings but requires dedicated effort. Plan preserved here for when the time is right.

Resources

• OpenSearch Documentation
• OpenSearch KNN Plugin
• ES to OpenSearch Migration Guide

Last Updated

2025-01-27