> ## Documentation Index
> Fetch the complete documentation index at: https://ihub.learnille.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbooks and Playbooks

> Migrated from the previous Learnille internal documentation hub

> The AWS ECS commands in this document are legacy examples and are not the current Learnille production procedure. The backend is deployed through Dokploy; use [the production deployment guide](https://ihub.learnille.com/engineering/deployment/guide.md) for current release and rollback operations.

## Overview

This document contains operational runbooks and playbooks for common scenarios in the Learnille platform. Runbooks provide step-by-step procedures for routine operations, while playbooks offer guidance for incident response and complex situations.

## Incident Response Playbook

### 1. Service Outage Response

#### Detection Phase

**When:** Monitoring alerts indicate service degradation or outage

**Immediate Actions:**

1. **Acknowledge Alert**
   ```bash theme={null}
   # Check current system status
   curl -f https://api.learnille.com/health || echo "API down"
   curl -f https://app.learnille.com/health || echo "App down"
   ```

2. **Assess Impact**
   * Check affected services and user impact
   * Review error rates and latency metrics
   * Identify affected user segments

3. **Notify Stakeholders**
   ```bash theme={null}
   # Send initial notification
   curl -X POST https://api.pagerduty.com/incidents \
     -H "Authorization: Token token=$PAGERDUTY_TOKEN" \
     -d '{
       "incident": {
         "type": "incident",
         "title": "Learnille API Service Outage",
         "service": {"id": "SERVICE_ID"},
         "priority": {"id": "PRIORITY_ID"}
       }
     }'
   ```

#### Investigation Phase

1. **Check Application Logs**
   ```bash theme={null}
   # View recent application logs
   aws logs tail /aws/ecs/learnille-api --since 10m

   # Check for error patterns
   aws logs filter-log-events \
     --log-group-name /aws/ecs/learnille-api \
     --filter-pattern "ERROR" \
     --start-time $(date -d '10 minutes ago' +%s)
   ```

2. **Database Health Check**
   ```bash theme={null}
   # Check PostgreSQL service status
   pg_isready -h localhost -p 5432 -U learnille

   # Query active database connections via psql
   psql -h localhost -U learnille -d learnille_prod -c "SELECT count(*) FROM pg_stat_activity;"

   # Query New Relic NRQL for database transaction duration
   # NRQL: SELECT average(databaseDuration) FROM Transaction WHERE appName = 'Learnille API (Self-Hosted)' SINCE 1 hour ago
   ```

3. **Infrastructure Status**
   ```bash theme={null}
   # Check ECS service status
   aws ecs describe-services \
     --cluster learnille-prod \
     --services learnille-api

   # Check load balancer health
   aws elbv2 describe-target-health \
     --target-group-arn $TARGET_GROUP_ARN
   ```

#### Resolution Phase

1. **Common Quick Fixes**
   ```bash theme={null}
   # Restart unhealthy tasks
   aws ecs update-service \
     --cluster learnille-prod \
     --service learnille-api \
     --force-new-deployment

   # Scale up service if overloaded
   aws ecs update-service \
     --cluster learnille-prod \
     --service learnille-api \
     --desired-count 5
   ```

2. **Database Issues**
   ```bash theme={null}
   # Check for long-running queries
   aws rds describe-db-instances --db-instance-identifier learnille-db

   # Restart database if needed
   aws rds reboot-db-instance --db-instance-identifier learnille-db
   ```

3. **Rollback if Necessary**
   ```bash theme={null}
   # Execute rollback procedure
   kubectl set image deployment/learnille-api app=learnille/api:1.2.2
   kubectl rollout status deployment/learnille-api
   ```

#### Recovery Phase

1. **Verify Service Recovery**
   ```bash theme={null}
   # Health checks
   curl -f https://api.learnille.com/health
   curl -f https://app.learnille.com/health

   # Performance validation
   ab -n 100 -c 10 https://api.learnille.com/api/v1/courses
   ```

2. **Update Status**
   ```bash theme={null}
   # Update incident status
   curl -X PUT https://api.pagerduty.com/incidents/$INCIDENT_ID \
     -H "Authorization: Token token=$PAGERDUTY_TOKEN" \
     -d '{"incident": {"status": "resolved"}}'
   ```

3. **Post-Mortem**
   * Document root cause
   * Identify improvement actions
   * Update runbooks if needed

### 2. Database Performance Issues

#### Symptoms

* Slow query response times
* High CPU utilization on database
* Connection pool exhaustion
* Increased error rates

#### Investigation Steps

1. **Check Database Metrics**
   ```bash theme={null}
   # CPU and memory usage on host
   top -b -n 1 | head -n 20

   # New Relic NRQL query for host CPU & Memory
   # NRQL: SELECT average(cpuPercent), average(memoryUsedBytes) FROM SystemSample SINCE 1 hour ago
   ```

2. **Identify Slow Queries**
   ```sql theme={null}
   -- Find slow queries
   SELECT
     query,
     calls,
     total_time,
     mean_time,
     rows
   FROM pg_stat_statements
   ORDER BY mean_time DESC
   LIMIT 10;

   -- Check active connections
   SELECT
     pid,
     usename,
     client_addr,
     query_start,
     state,
     query
   FROM pg_stat_activity
   WHERE state != 'idle';
   ```

3. **Check Index Usage**
   ```sql theme={null}
   -- Unused indexes
   SELECT
     schemaname,
     tablename,
     indexname,
     idx_scan
   FROM pg_stat_user_indexes
   WHERE idx_scan = 0
   ORDER BY tablename;

   -- Index hit rate
   SELECT
     sum(idx_blks_hit) / (sum(idx_blks_hit) + sum(idx_blks_read)) AS hit_rate
   FROM pg_statio_user_indexes;
   ```

#### Resolution Steps

1. **Optimize Queries**
   * Add missing indexes
   * Rewrite inefficient queries
   * Implement query result caching

2. **Scale Database**
   ```bash theme={null}
   # Increase instance size
   aws rds modify-db-instance \
     --db-instance-identifier learnille-db \
     --db-instance-class db.r5.large \
     --apply-immediately

   # Add read replicas
   aws rds create-db-instance-read-replica \
     --db-instance-identifier learnille-db-replica \
     --source-db-instance-identifier learnille-db
   ```

3. **Connection Pool Optimization**
   ```bash theme={null}
   # Update connection pool settings
   aws rds modify-db-parameter-group \
     --db-parameter-group-name learnille-db-params \
     --parameters "ParameterName=max_connections,ParameterValue=200,ApplyMethod=immediate"
   ```

### 3. Security Incident Response

#### Detection

* Unusual login patterns
* Unexpected data access
* Security monitoring alerts
* User reports of suspicious activity

#### Containment

1. **Isolate Affected Systems**
   ```bash theme={null}
   # Block suspicious IP addresses
   aws waf update-ip-set \
     --name suspicious-ips \
     --scope REGIONAL \
     --id $IP_SET_ID \
     --addresses $SUSPICIOUS_IP

   # Disable compromised accounts
   aws cognito-idp admin-disable-user \
     --user-pool-id $USER_POOL_ID \
     --username $COMPROMISED_USER
   ```

2. **Preserve Evidence**
   ```bash theme={null}
   # Collect logs
   aws logs create-export-task \
     --log-group-name /aws/ecs/learnille-api \
     --from $(date -d '1 hour ago' +%s) \
     --to $(date +%s) \
     --destination $S3_BUCKET \
     --destination-prefix security-incident/$(date +%Y%m%d_%H%M%S)

   # Take database snapshot
   aws rds create-db-snapshot \
     --db-instance-identifier learnille-db \
     --db-snapshot-identifier security-incident-$(date +%Y%m%d)
   ```

#### Investigation

1. **Log Analysis**
   ```bash theme={null}
   # Search for suspicious patterns
   aws logs filter-log-events \
     --log-group-name /aws/ecs/learnille-api \
     --filter-pattern "ERROR.*auth.*failed" \
     --start-time $(date -d '24 hours ago' +%s)
   ```

2. **Access Review**
   ```bash theme={null}
   # Check recent IAM activity
   aws iam list-access-keys \
     --user-name $SUSPICIOUS_USER

   # Review CloudTrail logs
   aws cloudtrail lookup-events \
     --start-time $(date -d '24 hours ago' +%s) \
     --lookup-attributes AttributeKey=Username,AttributeValue=$SUSPICIOUS_USER
   ```

#### Recovery

1. **Password Reset**
   ```bash theme={null}
   # Force password reset for affected users
   aws cognito-idp admin-set-user-password \
     --user-pool-id $USER_POOL_ID \
     --username $AFFECTED_USER \
     --password $TEMP_PASSWORD \
     --permanent
   ```

2. **Security Updates**
   ```bash theme={null}
   # Update security groups
   aws ec2 revoke-security-group-ingress \
     --group-id $SG_ID \
     --protocol tcp \
     --port 80 \
     --cidr 0.0.0.0/0

   # Rotate access keys
   aws iam create-access-key --user-name $COMPROMISED_USER
   aws iam delete-access-key --user-name $COMPROMISED_USER --access-key-id $OLD_KEY
   ```

## Operational Runbooks

### 1. Deployment Runbook

#### Pre-Deployment Checklist

* [ ] Code review completed
* [ ] Tests passing
* [ ] Security scan clean
* [ ] Documentation updated
* [ ] Rollback plan documented
* [ ] Communication plan ready

#### Deployment Steps

1. **Prepare Release**
   ```bash theme={null}
   # Create release branch
   git checkout -b release/v1.2.3 main

   # Update version
   npm version 1.2.3 --no-git-tag-version
   ```

2. **Build Artifacts**
   ```bash theme={null}
   # Build application
   npm run build

   # Create Docker image
   docker build -t learnille/api:1.2.3 .

   # Push to registry
   docker push learnille/api:1.2.3
   ```

3. **Deploy to Staging**
   ```bash theme={null}
   # Update staging environment
   kubectl set image deployment/learnille-api app=learnille/api:1.2.3 -n staging
   kubectl rollout status deployment/learnille-api -n staging
   ```

4. **Validation**
   ```bash theme={null}
   # Health checks
   curl -f https://api-staging.learnille.com/health

   # Smoke tests
   npm run test:smoke -- --env staging
   ```

5. **Production Deployment**
   ```bash theme={null}
   # Blue-green deployment
   kubectl set image deployment/learnille-api-blue app=learnille/api:1.2.3
   kubectl rollout status deployment/learnille-api-blue

   # Switch traffic
   kubectl patch service learnille-api -p '{"spec":{"selector":{"version":"blue"}}}'
   ```

#### Post-Deployment

1. **Monitor Performance**
   ```bash theme={null}
   # Check metrics
   aws cloudwatch get-metric-statistics \
     --namespace AWS/ECS \
     --metric-name CPUUtilization \
     --start-time $(date -d '1 hour ago' +%s) \
     --end-time $(date +%s) \
     --period 300 \
     --statistics Average
   ```

2. **Verify Functionality**
   * User login and registration
   * Course creation and enrollment
   * Payment processing
   * Email notifications

3. **Update Documentation**
   * Release notes published
   * API documentation updated
   * User guides updated

### 2. Backup and Recovery Runbook

#### Daily Backup Procedure

```bash theme={null}
#!/bin/bash
# daily-backup.sh

DATE=$(date +%Y%m%d)
BACKUP_DIR="/backups/$DATE"

# Database backup
pg_dump learnille_prod > $BACKUP_DIR/database.sql

# File storage backup
aws s3 sync s3://learnille-uploads $BACKUP_DIR/uploads/

# Configuration backup
tar -czf $BACKUP_DIR/config.tar.gz /etc/learnille/

# Upload to S3
aws s3 cp $BACKUP_DIR s3://learnille-backups/daily/$DATE/ --recursive

# Cleanup old backups (keep 30 days)
find /backups -name "*" -type d -mtime +30 -exec rm -rf {} +
```

#### Database Recovery

1. **Assess Damage**
   ```bash theme={null}
   # Check database status
   aws rds describe-db-instances --db-instance-identifier learnille-db

   # Verify backup integrity
   aws s3 ls s3://learnille-backups/daily/
   ```

2. **Restore Database**
   ```bash theme={null}
   # Create new instance from backup
   aws rds restore-db-instance-from-db-snapshot \
     --db-instance-identifier learnille-db-restored \
     --db-snapshot-identifier learnille-backup-20231201 \
     --db-instance-class db.r5.large

   # Update application configuration
   kubectl set env deployment/learnille-api DATABASE_URL=$NEW_DB_URL
   ```

3. **Data Validation**
   ```sql theme={null}
   -- Verify data integrity
   SELECT COUNT(*) FROM users;
   SELECT COUNT(*) FROM courses;
   SELECT COUNT(*) FROM enrollments;

   -- Check for data corruption
   SELECT * FROM users WHERE email IS NULL;
   ```

#### File Recovery

```bash theme={null}
# Restore from S3 backup
aws s3 sync s3://learnille-backups/daily/2023-12-01/uploads/ s3://learnille-uploads/

# Verify file integrity
aws s3 ls s3://learnille-uploads/ --recursive | wc -l
```

### 3. Monitoring Setup Runbook

#### Application Monitoring

1. **Install Monitoring Agent**
   ```bash theme={null}
   # Install CloudWatch agent
   wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
   sudo rpm -U amazon-cloudwatch-agent.rpm

   # Configure agent
   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
   ```

2. **Configure Metrics**
   ```json theme={null}
   {
     "metrics": {
       "namespace": "Learnille/API",
       "metrics_collected": {
         "cpu": {
           "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
           "metrics_collection_interval": 60
         },
         "mem": {
           "measurement": ["mem_used_percent"],
           "metrics_collection_interval": 60
         },
         "disk": {
           "measurement": ["disk_used_percent"],
           "metrics_collection_interval": 300
         }
       }
     }
   }
   ```

3. **Set Up Alarms**
   ```bash theme={null}
   # CPU utilization alarm
   aws cloudwatch put-metric-alarm \
     --alarm-name "HighCPUUtilization" \
     --alarm-description "CPU utilization is high" \
     --metric-name CPUUtilization \
     --namespace AWS/ECS \
     --statistic Average \
     --period 300 \
     --threshold 80 \
     --comparison-operator GreaterThanThreshold \
     --evaluation-periods 2 \
     --alarm-actions $SNS_TOPIC_ARN
   ```

#### New Relic Alert Conditions Setup

1. **Configure High Memory Alarm in New Relic**
   * Condition Type: NRQL Alert Condition
   * NRQL Query: `SELECT average(memoryUsedBytes / memoryTotalBytes * 100) FROM SystemSample`
   * Threshold: `> 85%` for 5 minutes

2. **Configure Database Connection & Response Time Alarm**
   * Condition Type: NRQL Alert Condition
   * NRQL Query: `SELECT average(databaseDuration) FROM Transaction WHERE appName = 'Learnille API (Self-Hosted)'`
   * Threshold: `> 0.1` seconds for 5 minutes
   ```
   ```

#### Database Monitoring

1. **Enable Enhanced Monitoring**
   ```bash theme={null}
   aws rds modify-db-instance \
     --db-instance-identifier learnille-db \
     --monitoring-interval 60 \
     --monitoring-role-arn $MONITORING_ROLE_ARN
   ```

2. **Configure Database Metrics**
   ```bash theme={null}
   # Database connections
   aws cloudwatch put-metric-alarm \
     --alarm-name "HighDBConnections" \
     --metric-name DatabaseConnections \
     --namespace AWS/RDS \
     --statistic Maximum \
     --period 300 \
     --threshold 80 \
     --comparison-operator GreaterThanThreshold

   # Read latency
   aws cloudwatch put-metric-alarm \
     --alarm-name "HighReadLatency" \
     --metric-name ReadLatency \
     --namespace AWS/RDS \
     --statistic Average \
     --period 300 \
     --threshold 0.010 \
     --comparison-operator GreaterThanThreshold
   ```

### 4. Capacity Planning Runbook

#### Resource Usage Analysis

1. **Current Usage Assessment**
   ```bash theme={null}
   # CPU usage trends
   aws cloudwatch get-metric-statistics \
     --namespace AWS/ECS \
     --metric-name CPUUtilization \
     --start-time $(date -d '30 days ago' +%s) \
     --end-time $(date +%s) \
     --period 3600 \
     --statistics Average

   # Memory usage trends
   aws cloudwatch get-metric-statistics \
     --namespace AWS/ECS \
     --metric-name MemoryUtilization \
     --start-time $(date -d '30 days ago' +%s) \
     --end-time $(date +%s) \
     --period 3600 \
     --statistics Average
   ```

2. **Growth Projections**
   * Analyze user growth trends
   * Project resource requirements
   * Identify scaling thresholds
   * Plan capacity upgrades

#### Scaling Procedures

1. **Horizontal Scaling**
   ```bash theme={null}
   # Scale ECS service
   aws ecs update-service \
     --cluster learnille-prod \
     --service learnille-api \
     --desired-count 10

   # Scale database read replicas
   aws rds modify-db-instance \
     --db-instance-identifier learnille-db-replica-1 \
     --db-instance-class db.r5.large \
     --apply-immediately
   ```

2. **Vertical Scaling**
   ```bash theme={null}
   # Upgrade instance type
   aws ecs update-service \
     --cluster learnille-prod \
     --service learnille-api \
     --task-definition learnille-api-v2 \
     --force-new-deployment

   # Upgrade database
   aws rds modify-db-instance \
     --db-instance-identifier learnille-db \
     --db-instance-class db.r5.xlarge \
     --apply-immediately
   ```

## Maintenance Runbooks

### 1. Security Patching

```bash theme={null}
#!/bin/bash
# security-patching.sh

# Update system packages
sudo yum update -y

# Update Docker images
docker pull learnille/api:latest

# Restart services
kubectl rollout restart deployment/learnille-api

# Verify updates
rpm -qa | grep -i security
docker images | grep learnille/api
```

### 2. Log Rotation

```bash theme={null}
#!/bin/bash
# log-rotation.sh

# Rotate application logs
logrotate -f /etc/logrotate.d/learnille

# Archive old logs to S3
aws s3 sync /var/log/learnille/archive/ s3://learnille-logs/archive/

# Clean old archives (keep 90 days)
find /var/log/learnille/archive -name "*.gz" -mtime +90 -delete

# Verify log rotation
ls -la /var/log/learnille/
df -h /var/log
```

### 3. Certificate Renewal

```bash theme={null}
#!/bin/bash
# certificate-renewal.sh

# Check certificate expiration
openssl x509 -in /etc/ssl/certs/learnille.crt -text -noout | grep "Not After"

# Request new certificate
aws acm request-certificate \
  --domain-name learnille.com \
  --validation-method DNS

# Update CloudFront distribution
aws cloudfront update-distribution \
  --id $DISTRIBUTION_ID \
  --distribution-config file://distribution-config.json

# Verify certificate
curl -I https://learnille.com
```

## Communication Templates

### Incident Notification

```markdown theme={null}
**INCIDENT ALERT**

**Service:** Learnille API
**Severity:** High
**Status:** Investigating
**Start Time:** 2024-01-15 14:30 UTC
**Description:** API service experiencing elevated error rates
**Impact:** Users may experience slow response times or temporary service unavailability
**Updates:** Investigating database performance issues
**ETA:** 15 minutes
```

### Maintenance Notification

```markdown theme={null}
**MAINTENANCE NOTICE**

**Service:** Learnille Platform
**Date:** 2024-01-20
**Time:** 02:00 - 04:00 UTC
**Description:** Database maintenance and security patching
**Impact:** Service may be unavailable for up to 10 minutes
**Contact:** infrastructure@learnille.com
```

### Status Update

```markdown theme={null}
**STATUS UPDATE**

**Incident:** API Service Outage
**Status:** Resolved
**Resolution:** Database connection pool optimized
**Timeline:**
- 14:30: Incident detected
- 14:35: Investigation started
- 14:45: Root cause identified
- 14:50: Fix deployed
- 15:00: Service fully recovered

**Next Steps:** Post-mortem analysis scheduled for tomorrow
```

## Escalation Procedures

### Level 1 Support

* Monitor alerts and basic troubleshooting
* Follow runbooks for common issues
* Escalate to Level 2 if unresolved within 15 minutes

### Level 2 Support

* Advanced troubleshooting and diagnostics
* Coordinate with development team
* Implement fixes and workarounds
* Escalate to Level 3 for critical issues

### Level 3 Support

* Executive decision making
* External vendor coordination
* Crisis management
* Customer communication

## Review and Updates

### Monthly Review

* [ ] Review incident response effectiveness
* [ ] Update runbooks based on lessons learned
* [ ] Validate monitoring and alerting
* [ ] Test backup and recovery procedures
* [ ] Update contact information

### Continuous Improvement

* [ ] Automate manual procedures where possible
* [ ] Implement preventive measures
* [ ] Enhance monitoring coverage
* [ ] Improve communication processes
* [ ] Update training materials

## Contact Information

### Emergency Contacts

* **On-call Engineer:** +1-555-0123
* **DevOps Team:** [devops@learnille.com](mailto:devops@learnille.com)
* **Management:** [management@learnille.com](mailto:management@learnille.com)
* **External Support:** [aws-support@learnille.com](mailto:aws-support@learnille.com)

### Communication Channels

* **Slack:** #incidents, #devops
* **Email:** [alerts@learnille.com](mailto:alerts@learnille.com)
* **PagerDuty:** For critical alerts
* **Status Page:** [https://status.learnille.com](https://status.learnille.com)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.