> ## Documentation Index
> Fetch the complete documentation index at: https://ihub.learnille.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Moderation System Architecture

> Migrated from the previous Learnille internal documentation hub

## Overview

The moderation system provides automated content moderation for user-generated content including chat messages and reviews. It uses a multi-layered approach combining OpenAI's Moderation API, custom LLM analysis, and Google's Perspective API to ensure content safety while minimizing false positives.

## System Flow

```mermaid theme={null}
flowchart TD
    A[Message Created] --> B[InstantDB Subscriber]
    B --> C{Already Moderated?}
    C -->|Yes| Z[Skip]
    C -->|No| D[Add to Moderation Queue]
    D --> E[ChatModerationProcessor]
    E --> F[ModerationService.runModeration]
    F --> G[OpenAI Moderation API]
    G --> H{Flagged?}
    
    H -->|No| I[Enqueue Notification]
    I --> J[Mark as Moderated]
    
    H -->|Yes| K[LLM Flagging Analysis]
    K --> L[Determine Action]
    L --> M{Room Type?}
    M -->|Category Room| N[Perspective API Scores]
    M -->|Group/Private| O[LLM Sentiment Analysis]
    N --> P[Update Message with Results]
    O --> P
    P --> Q[Skip Notification]
```

## Components

### 1. InstantDB Subscriber Service

**Location**: `server/src/instantdb/subscriber.service.ts`

Subscribes to real-time message updates from InstantDB and queues unmoderated messages for processing.

| Responsibility | Description |
| - | - |
| Subscription Management | Creates and maintains WebSocket subscriptions to InstantDB |
| Idempotency | Skips already-moderated messages |
| Job Deduplication | Uses `messageId` as BullMQ `jobId` to prevent duplicates |
| Health Monitoring | Cron job checks subscription health every minute |

### 2. Moderation Service

**Location**: `server/src/moderation/moderation.service.ts`

Orchestrates the moderation pipeline and determines final actions.

```typescript theme={null}
async runModeration(type: ModerationType, data: any): Promise<ModerationResultDto>
```

**Moderation Types**:

* `CHAT` - Chat messages
* `REVIEW` - Product/service reviews

### 3. Analysis Service

**Location**: `server/src/moderation/analysis.service.ts`

Provides AI/ML analysis capabilities:

| Method | Provider | Purpose |
| - | - | - |
| `checkOpenAIModeration()` | OpenAI | Binary flag detection (fast, first-pass) |
| `getFlaggingDetailsFromLLM()` | OpenRouter | Detailed severity analysis for flagged content |
| `getSentimentFromLLM()` | OpenRouter | Sentiment scoring (-1 to 1) |
| `getPerspectiveScores()` | Google Perspective API | Toxicity, profanity, and sexual content scores |

### 4. Queue Processors

**Location**: `server/src/moderation/processors/`

| Processor | Queue | Job Type |
| - | - | - |
| `ChatModerationProcessor` | `chat-moderation` | `MODERATE_CHAT` |
| `ReviewModerationProcessor` | `review-moderation` | `MODERATE_REVIEW` |
| `CommentModerationProcessor` | `comment-moderation` | `MODERATE_COMMENT` |

## Decision Flow

### Step 1: OpenAI Moderation (Gate)

All content first passes through OpenAI's Moderation API. This is a fast, binary check.

* **Not Flagged**: Immediately enqueue notification, skip detailed analysis
* **Flagged**: Proceed to detailed LLM analysis

### Step 2: LLM Flagging Analysis

For flagged content, the LLM provides:

* `reason`: Human-readable explanation
* `recommendedAction`: `RETAIN`, `WARN`, or `REMOVE`
* `needsHumanReview`: Boolean for edge cases

### Step 3: Context-Specific Analysis

Additional analysis based on content type and context:

| Context | Analysis |
| - | - |
| Category Room Chat | Perspective API (toxicity, profanity, sexual) |
| Group/Private Chat | LLM Sentiment Analysis |
| Reviews | LLM Sentiment + Perspective API |

## Data Model

### ModerationResultDto

```typescript theme={null}
{
  messageId?: string;
  reviewId?: string;
  isFlagged: boolean;
  action: ModerationAction;
  needsHumanReview: boolean;
  moderationReason?: string;
  sentimentScore?: number;
  sentimentLabel?: SentimentLabel;
  toxicityScore?: number;
  profanityScore?: number;
  sexualScore?: number;
  moderatedAt: Date;
}
```

### ModerationAction Enum

* `RETAIN` - Content is safe
* `WARN` - Content is borderline, may need review
* `REMOVE` - Content should be hidden/removed

### SentimentLabel Enum

* `POSITIVE`
* `NEUTRAL`
* `NEGATIVE`

## Configuration

Environment variables:

| Variable | Description |
| - | - |
| `OPENAI_API_KEY` | OpenAI API key for moderation |
| `OPENROUTER_API_KEY` | OpenRouter API key for LLM |
| `OPENROUTER_BASE_URL` | OpenRouter base URL |
| `PERSPECTIVE_API_KEY` | Google Perspective API key |
| `MODERATION_LLM_MODEL` | Model ID for LLM analysis |
| `MODERATION_SENTIMENT_PROMPT` | Prompt template for sentiment |
| `MODERATION_FLAGGING_PROMPT` | Prompt template for flagging |

## Moderation Flow Implementation & Cleanup

Implemented a robust, sequential moderation-to-notification flow and performed comprehensive code cleanup.

## Changes Made

### 1. Moderation→Notification Flow Restructure

* **Sequential Processing**: Modified `ModerationService` to trigger notifications *only after* OpenAI moderation passes.
* **Notification Gate**: OpenAI moderation acts as the first filter. If content is flagged, detailed LLM analysis runs but notifications are skipped.
* **Decoupled Subscriber**: Removed notification logic from `instantdb.service.ts` and `subscriber.service.ts`, making them purely responsible for queuing moderation.
* **Idempotency**: Leveraged BullMQ `jobId` for robust deduplication.

### 2. Code Cleanup & Optimization

* **Debug Logs**: Removed all `this.logger.debug` statements from moderation-related services.
* **Console Logs**: Removed `console.log` statements from `analysis.service.ts`.
* **Comment Stripping**: Removed all JSDoc and internal comments from:
  * `moderation.service.ts`
  * `analysis.service.ts`
  * `instantdb.service.ts`
* **Protocol Fix**: Verified and restored URLs corrupted during comment stripping.

## Verification Results

### Integration Flow

```mermaid theme={null}
sequenceDiagram
    participant IDB as InstantDB
    participant SUB as Subscriber
    participant MOD as ModerationService
    participant NOTIF as NotificationService

    IDB->>SUB: New Message (moderated: false)
    SUB->>MOD: Add Moderation Job
    MOD->>MOD: OpenAI Moderation Check
    alt Content is Clean
        MOD->>NOTIF: Enqueue Notification Job
        MOD->>IDB: Mark as Moderated (isFlagged: false)
    else Content is Flagged
        MOD->>MOD: LLM Severity Analysis
        MOD->>IDB: Mark as Moderated (isFlagged: true)
        Note over MOD: Skips Notification
    end
```

### Clean Code Verification

Ran `grep` checks confirming 0 instances of debug logs and comments in the modified service files.

## Queue Configuration

Moderation queues are configured with:

* `removeOnComplete: false` (for debugging/audit)
* `attempts: 3` with exponential backoff
* `jobId: chat-mod-{messageId}` for deduplication

## Integration Points

### Inbound

* **InstantDB Subscriptions**: Real-time message stream
* **Direct API Calls**: For batch moderation

### Outbound

* **Notification Service**: Enqueues notifications for clean content
* **InstantDB Updates**: Marks messages as moderated with results

## Notification Flow

Notifications are only sent for content that passes OpenAI moderation:

1. OpenAI check returns `!flagged`
2. `enqueueNotificationForChat()` fetches message details
3. Validates sender/room metadata
4. Queues notification job
5. Marks message as `notificationProcessedAt`

Flagged content skips notification entirely.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.