Learnille Impression Tracking System — PRD

1. Overview

This document defines the Product Requirements for a secure, tamper-resistant impression tracking system for Learnille’s marketplace. The system tracks course and consultation visibility, enables instructor/consultant analytics, and supports relevance ranking.

2. Goals

Primary Goals

Track true impressions: product actually appears in user viewport
Maintain tamper-resistant impression signals via HMAC signing
Provide accurate session-level deduping and per-product metrics
Support obfuscated endpoints to prevent competitor detection

Secondary Goals

Enable instructor/consultant dashboards (impressions, CTR, referrers)
Support ML/relevance ranking via impression data
Allow anti-fraud heuristics (velocity, bot detection)

3. Non-Goals

No external analytics vendor integration
No third-party fingerprinting SDK
No cookie consent requirements (privacy-first approach)
No real-time streaming infrastructure at initial stage

4. Architecture Overview

Initial Phase

Client (Public Web)
   ↓
Beacon Endpoint (NestJS API)
   ↓
BeaconGuard (HMAC + Timestamp + Nonce)
   ↓
ImpressionService
   ↓
PostgreSQL (impressions)
   ↓
Redis (nonce tracking, rate limiting)

Future Upgrade Path

Client → API → Redis Streams → Worker → PostgreSQL (TimescaleDB) → Aggregations

TimescaleDB: Convert impressions table to a hypertable for efficient time-series partitioning and faster aggregation queries (e.g., time_bucket).

5. Client-Side Requirements

5.1 Impression Trigger

Use browser IntersectionObserver to record an impression only when:

Product card enters viewport
Remains visible ≥ 150ms
Impression for (entity_id + session_id) has not been fired this session

5.2 Required Client Fields

Field	Type	Description
`entity_id`	UUID	Course or consultation ID
`entity_type`	string	”course”, “consultation”, “instructor”, “consultant”
`event_type`	string	”impression”, “click”, “bounce”
`session_id`	UUID	Browser session identifier
`referrer`	string	Document referrer URL
`device_fingerprint`	string	Browser fingerprint hash
`timestamp`	number	Unix timestamp (ms)
`nonce`	string	12-char random string
`signature`	string	HMAC-SHA256 signature

5.3 Legitimacy Verification (Handshake)

To ensure requests are absolutely legitimate and “authorized” by the server without user login: 1. Handshake Phase

Client calls GET /api/v1/beacon/h.
Server verifies client (IP reputation, standard headers).
Server returns a Signed Session Token (JWT-like) containing:
- session_id
- issued_at
- expiry (e.g., 2 hours)
Token is signed with BEACON_PRIVATE_KEY.

2. Signing Phase Client signs each impression batch using:

HMAC_SHA256(payload + timestamp + nonce + session_token, BEACON_SESSION_SECRET)

BEACON_SESSION_SECRET is derived from the session token or a shared secret.
Server verifies both the Session Token signature and the Impression HMAC.

This “double-seal” ensures:

The client was initialized by our server.
The payload hasn’t been tampered with.
Replay attacks are impossible (even if signature is stolen, it’s tied to an expired token).

5.4 Payload Obfuscation

All fields use short codes to obscure intent:

Short	Full
`t`	entity_type
`e`	entity_id
`v`	event_type
`r`	referrer
`f`	device_fingerprint
`s`	session_id

Entity types use codes: c=course, n=consultation, i=instructor, o=consultant

6. Session Requirements

6.1 Browser Session ID

Stored in sessionStorage (not cookies)
Value: UUIDv4
Expires when tab closes
Key: bkn_sid

6.2 Deduplication

Track fired impressions in sessionStorage
Key format: bkn:{entity_type}:{entity_id}
Prevents duplicate tracking per session

6.3 Device Fingerprint

Minimal fingerprint using:

User agent
Screen resolution
Timezone offset
Language

Hash via: SHA256(ua + screen + tz + lang).substring(0, 16)

6.4 Client-Side Batching

To improve performance and reduce server load:

Queue: Store impressions in a memory queue.
Flush Triggers:
- Time: Every 2 seconds.
- Size: Every 10 queued items.
- Event: pagehide / beforeunload (using navigator.sendBeacon).
Format: Batch array sent to single endpoint.

7. Server Endpoint Specification

7.1 HTTP (Handshake)

GET /api/v1/beacon/h

Returns: { "token": "signed.session.token", "key": "temporary-signing-key" }

7.2 HTTP (Batch)

POST /api/v1/beacon/batch
Content-Type: application/json

Request Body:

{
  "token": "signed.session.token",
  "batch": [
    { "d": "...", "ts": 1702..., "n": "...", "sig": "..." }
  ]
}

7.3 HTTP (Pixel Fallback)

(Unchanged: Single request via px.gif)

8. Attribution (“The Golden Thread”)

To track conversion value back to impressions:

8.1 Search Session Correlation

Generate search_session_id on search execution.
Store search_session_id in sessionStorage for persistence across the search results page.
Pass search_session_id to:
1. Impression payload (ss field).
2. Product URL (query param ?sid=...).

8.2 Checkout Association

The useBeacon hook captures sid from the URL on product detail pages and saves it to sessionStorage (key: search_session_id).
On Order creation, the frontend sends this search_session_id to the backend.
The Order entity persists searchSessionId.
Analytics: Join orders table with impressions via search_session_id to calculate “Revenue per Search” and “Conversion Rate by Rank”.

9. Dedupe Logic

8.1 Client-Side Dedupe

Per session, per entity:

sessionStorage.getItem(`bkn:${type}:${id}`)

8.2 Server-Side Dedupe

Redis-based with 1-hour TTL:

impression:{user_or_ip}:{entity_type}:{entity_id}

Uses SET NX (set-if-not-exists) for atomic deduplication.

9. Database Schema

impressions

CREATE TABLE impressions (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  entity_type VARCHAR(50) NOT NULL,
  entity_id UUID NOT NULL,
  event_type VARCHAR(50) NOT NULL,
  user_id UUID,
  session_id VARCHAR(255),
  ip_address INET,
  device_type VARCHAR(100),
  device_fingerprint VARCHAR(255),
  referrer TEXT,
  search_query TEXT,
  value DECIMAL(10,2),
  metadata JSONB,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_impressions_entity ON impressions(entity_type, entity_id, created_at);
CREATE INDEX idx_impressions_user ON impressions(user_id, created_at);
CREATE INDEX idx_impressions_session ON impressions(session_id);

Metadata JSONB

Stores owner IDs for filtering:

{
  "instructor_id": "uuid",
  "consultant_id": "uuid"
}

10. Analytics Queries

Impressions by Period

SELECT 
  DATE_TRUNC('day', created_at) AS period,
  COUNT(*) AS impressions
FROM impressions
WHERE entity_type = 'course' 
  AND entity_id = :entityId
  AND created_at >= NOW() - INTERVAL '7 days'
GROUP BY period
ORDER BY period;

CTR Calculation

SELECT 
  SUM(CASE WHEN event_type = 'click' THEN 1 ELSE 0 END)::float /
  NULLIF(SUM(CASE WHEN event_type = 'impression' THEN 1 ELSE 0 END), 0) AS ctr
FROM impressions
WHERE entity_type = 'course' AND entity_id = :entityId;

Instructor Dashboard

SELECT 
  entity_id,
  COUNT(*) FILTER (WHERE event_type = 'impression') AS impressions,
  COUNT(*) FILTER (WHERE event_type = 'click') AS clicks
FROM impressions
WHERE entity_type = 'course'
  AND metadata->>'instructor_id' = :instructorId
  AND created_at >= NOW() - INTERVAL '30 days'
GROUP BY entity_id;

11. Fraud & Abuse Detection

Early Indicators

Signal	Threshold	Action
Excessive impressions/min	>100 per session	Reject (HTTP 403 / Forbidden)
Invalid signature	Any	Reject
Expired timestamp	>30s	Reject
Duplicate nonce	Any	Reject
Same IP, many entities	>50 in 1 min	Flag

Future Upgrades

Browser behavior scoring
ML-based anomaly detection
Scroll velocity analysis

12. Operational Requirements

12.1 Performance

API handles ≥500 req/sec
≤15ms average response time
Redis for hot-path operations

12.2 Security

Rotate BEACON_SECRET every 90 days
Enforce TLS
Limit body size to 4KB
Rate-limit by IP/session

12.3 Monitoring

Track rejection rate by reason
Alert on >5% signature failures
Log suspicious patterns

13. Client SDK Requirements

useBeacon Hook

function useBeacon(entityType: string, entityId: string) {
  useEffect(() => {
    if (!entityId) return;
    const key = `bkn:${entityType}:${entityId}`;
    if (sessionStorage.getItem(key)) return;
    sessionStorage.setItem(key, '1');
    beacon(entityType, entityId, 'impression');
  }, [entityType, entityId]);
}

beacon Function

Generate/maintain session ID
Sign events with HMAC
Fallback to pixel if fetch blocked
Batch send (via BeaconQueue)

14. Success Metrics

Metric	Target
Impression delivery rate	≥98%
Server processing time	≤15ms
Signature rejection rate	<1%
Duplicate detection rate	>95%
CTR accuracy	±2%

15. Environment Variables

Variable	Location	Description
`BEACON_SECRET`	Server	HMAC signing key (32+ chars)
`NEXT_PUBLIC_BEACON_KEY`	Frontend	Public signing key
`BEACON_RATE_LIMIT`	Server	Requests per minute (default: 30)
`BEACON_NONCE_TTL`	Server	Nonce expiry seconds (default: 60)

16. File Structure

server/src/impression/
├── impression.module.ts
├── impression.service.ts
├── impression.controller.ts
├── beacon.controller.ts     # Public obfuscated endpoints
├── beacon.service.ts        # Decoding & batching logic
├── guards/
│   └── beacon.guard.ts      # HMAC/Timestamp validation
└── dto/
    └── beacon.dto.ts        # Signed payload definitions

public/src/
├── services/beacon.ts
└── hooks/useBeacon.ts

17. Implementation Phases

Phase 1: Core (Week 1)

BeaconController with endpoints
BeaconGuard with HMAC validation
Frontend beacon.ts service
useBeacon hook
Integration on course/consultation pages

Phase 2: Security (Week 2)

Timestamp validation
Nonce tracking in Redis
Rate limiting
Behavioral detection

Phase 3: Analytics (Week 3)

Dashboard API endpoints
Aggregation queries
Instructor/consultant views

18. Open Questions

Should we track scroll depth for engagement scoring?
Should impressions be versioned for ranking model training?
Should we implement A/B testing for different CTR thresholds?

19. Next Steps

✅ Approve PRD
Generate API contract
Generate NestJS implementation
Generate JS client SDK
Deploy MVP
Add monitoring & alerts