> ## Documentation Index
> Fetch the complete documentation index at: https://ihub.learnille.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Rollback Instructions

> Migrated from the previous Learnille internal documentation hub

> The Kubernetes, ECS, PM2, and Docker Compose procedures below are legacy examples. For the current backend deployment, use Dokploy rollback and the release checks in [the production deployment guide](https://ihub.learnille.com/engineering/deployment/guide.md).

## Overview

This document provides procedures for rolling back deployments in the Learnille platform. Rollbacks may be necessary due to application failures, performance issues, or other critical problems.

## Rollback Types

### 1. Automated Rollback

* **Trigger**: Health check failures, error thresholds exceeded
* **Execution**: Immediate, no manual intervention required
* **Duration**: \< 5 minutes
* **Scope**: Application container rollback

### 2. Manual Rollback

* **Trigger**: Detected issues requiring human judgment
* **Execution**: Manual command execution
* **Duration**: 10-30 minutes
* **Scope**: Application, database, or infrastructure

### 3. Emergency Rollback

* **Trigger**: Critical system failure, security breach
* **Execution**: Immediate action by on-call engineer
* **Duration**: \< 15 minutes
* **Scope**: Complete environment restoration

## Pre-Rollback Checklist

### Assessment

* [ ] Identify the issue and impact
* [ ] Determine rollback scope (application, database, infrastructure)
* [ ] Check current system status and monitoring
* [ ] Notify stakeholders if needed
* [ ] Gather evidence (logs, metrics, user reports)

### Preparation

* [ ] Verify backup availability and integrity
* [ ] Check previous version stability
* [ ] Prepare rollback commands/scripts
* [ ] Ensure access to deployment tools
* [ ] Coordinate with team members if needed

## Automated Rollback Procedures

### Application Rollback (Kubernetes)

#### Immediate Rollback

```bash theme={null}
# Check current deployment status
kubectl get deployments -n learnille-prod

# Rollback to previous revision
kubectl rollout undo deployment/learnille-api -n learnille-prod

# Monitor rollback progress
kubectl rollout status deployment/learnille-api -n learnille-prod
```

#### Blue-Green Rollback

```bash theme={null}
# Check which environment is active
kubectl get service learnille-api -n learnille-prod -o yaml

# Switch traffic back to blue environment
kubectl patch service learnille-api -n learnille-prod -p '{"spec":{"selector":{"version":"blue"}}}'

# Verify traffic switch
kubectl get endpoints learnille-api -n learnille-prod
```

### Database Rollback

#### Migration Rollback

```bash theme={null}
# Check migration status
npm run migration:status -- --env production

# Rollback last migration
npm run migration:rollback -- --env production

# Verify database integrity
npm run migration:status -- --env production
```

#### Data Restore (if needed)

```bash theme={null}
# Restore from backup (if data corruption)
pg_restore -h production-db -U learnille -d learnille_prod backup_file.sql

# Verify data integrity
psql -h production-db -U learnille -d learnille_prod -c "SELECT COUNT(*) FROM users;"
```

## Manual Rollback Procedures

### 1. Application Rollback

#### Self-Hosted PM2 / Git Rollback

```bash theme={null}
# SSH into self-hosted production server
ssh admin@server.learnille.dev

# Navigate to application root
cd /opt/learnille

# Checkout previous stable Git commit / tag
git checkout v1.2.0

# Rebuild and reload PM2 process
cd server && npm install && npm run build
pm2 reload learnille-api --update-env
```

#### AWS ECS rollback (legacy)

```bash theme={null}
aws ecs update-service \
  --service learnille-api \
  --task-definition learnille-api:1.2.2 \
  --force-new-deployment

# Monitor deployment
aws ecs describe-services \
  --cluster learnille-prod \
  --services learnille-api \
  --query 'services[0].deployments'
```

#### Using Docker Compose

```bash theme={null}
# Stop current containers
docker-compose -f production.yml down

# Update image version in compose file
sed -i 's/learnille\/api:1.2.3/learnille\/api:1.2.2/' production.yml

# Start previous version
docker-compose -f production.yml up -d

# Verify containers
docker-compose -f production.yml ps
```

### 2. Infrastructure Rollback

#### Terraform Rollback

```bash theme={null}
# Check current state
terraform show

# Rollback to previous state
terraform plan -state=terraform.tfstate.backup
terraform apply -state=terraform.tfstate.backup

# Verify infrastructure
terraform show
```

#### CloudFormation Rollback

```bash theme={null}
# Check stack status
aws cloudformation describe-stacks --stack-name learnille-prod

# Rollback to previous version
aws cloudformation update-stack \
  --stack-name learnille-prod \
  --template-body file://template-v1.2.2.yaml \
  --parameters file://parameters.json

# Monitor stack update
aws cloudformation describe-stack-events --stack-name learnille-prod
```

### 3. Configuration Rollback

#### Environment Variables

```bash theme={null}
# Backup current environment
cp .env.production .env.production.backup

# Restore previous environment
cp .env.production.v1.2.2 .env.production

# Restart application
kubectl rollout restart deployment/learnille-api -n learnille-prod
```

#### Configuration Files

```bash theme={null}
# Backup current config
cp config/production.yaml config/production.yaml.backup

# Restore previous config
cp config/production-v1.2.2.yaml config/production.yaml

# Apply configuration
kubectl apply -f config/production.yaml
```

## Emergency Rollback Procedures

### Critical System Failure

1. **Immediate Assessment**
   ```bash theme={null}
   # Check system status
   curl -f https://api.learnille.com/health || echo "API down"

   # Check database connectivity
   psql -h production-db -U learnille -d learnille_prod -c "SELECT 1;"

   # Check infrastructure status
   aws ecs describe-services --cluster learnille-prod --services learnille-api
   ```

2. **Execute Emergency Rollback**
   ```bash theme={null}
   # Force rollback to last known good state
   kubectl set image deployment/learnille-api app=learnille/api:1.2.2 -n learnille-prod
   kubectl rollout status deployment/learnille-api -n learnille-prod

   # Switch to backup database if needed
   aws rds failover-db-cluster --db-cluster-identifier learnille-prod
   ```

3. **Traffic Management**
   ```bash theme={null}
   # Enable maintenance mode
   kubectl patch configmap learnille-config -p '{"data":{"maintenance":"true"}}'

   # Switch to backup load balancer
   aws elbv2 set-ip-address-type \
     --load-balancer-arn $BACKUP_LB_ARN \
     --ip-address-type ipv4
   ```

### Security Incident Response

1. **Isolate Affected Systems**
   ```bash theme={null}
   # Block suspicious traffic
   aws waf update-ip-set \
     --name suspicious-ips \
     --scope REGIONAL \
     --id $IP_SET_ID \
     --addresses $SUSPICIOUS_IP

   # Disable compromised service
   kubectl scale deployment learnille-api --replicas=0 -n learnille-prod
   ```

2. **Data Recovery**
   ```bash theme={null}
   # Restore from clean backup
   aws s3 cp s3://learnille-backups/clean-backup.sql /tmp/
   psql -h production-db -U learnille -d learnille_prod < /tmp/clean-backup.sql
   ```

3. **Security Validation**
   ```bash theme={null}
   # Run security scan
   docker run --rm -v $(pwd):/src securecodebox/scanner

   # Update security groups
   aws ec2 revoke-security-group-ingress --group-id $SG_ID --protocol tcp --port 80 --cidr 0.0.0.0/0
   ```

## Post-Rollback Procedures

### Validation Steps

* [ ] Application health checks passing
* [ ] Database connectivity verified
* [ ] User functionality tested
* [ ] Performance metrics normal
* [ ] Monitoring alerts cleared

### Documentation

```bash theme={null}
# Document rollback in incident log
echo "$(date): Rollback executed - $(whoami)" >> rollback-log.txt
echo "Reason: $ROLLBACK_REASON" >> rollback-log.txt
echo "Impact: $ROLLBACK_IMPACT" >> rollback-log.txt

# Update status page
curl -X POST https://status.learnille.com/api/incidents \
  -H "Authorization: Bearer $STATUS_API_KEY" \
  -d '{"status": "resolved", "message": "Issue resolved via rollback"}'
```

### Communication

* Notify internal team via Slack
* Update external status page
* Send customer communication if needed
* Schedule post-mortem meeting

### Investigation

* Analyze root cause of deployment failure
* Review monitoring data during incident
* Update deployment processes if needed
* Document lessons learned

## Monitoring During Rollback

### Key Metrics to Monitor

* Application response time
* Error rates by endpoint
* Database connection pool usage
* Infrastructure resource utilization
* User session success rate

### Alerting During Rollback

```yaml theme={null}
# Temporary rollback monitoring
groups:
  - name: rollback-monitoring
    rules:
      - alert: RollbackHealthCheck
        expr: up{job="learnille-api"} == 0
        for: 2m
        labels:
          severity: critical
          rollback: "true"
      - alert: RollbackErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
          rollback: "true"
```

## Rollback Testing

### Regular Testing

* Monthly rollback drills
* Automated rollback testing in staging
* Documentation review and updates
* Team training sessions

### Test Scenarios

* Application deployment failure
* Database migration failure
* Infrastructure configuration error
* Security vulnerability discovery
* Performance degradation

## Tools and Resources

### Rollback Scripts

```bash theme={null}
#!/bin/bash
# emergency-rollback.sh
set -e

echo "Starting emergency rollback..."

# Get last stable version
STABLE_VERSION=$(curl -s https://api.github.com/repos/learnille/learnille/releases/latest | jq -r .tag_name)

# Execute rollback
kubectl set image deployment/learnille-api app=learnille/api:$STABLE_VERSION
kubectl rollout status deployment/learnille-api

echo "Rollback completed successfully"
```

### Monitoring Dashboard

* Rollback status dashboard
* Real-time metrics during rollback
* Historical rollback data
* Success/failure analytics

## Contact Information

### Emergency Contacts

* **Primary On-call**: +1-555-0123 (DevOps Engineer)
* **Secondary On-call**: +1-555-0124 (Senior Engineer)
* **Management**: +1-555-0125 (Engineering Manager)

### Communication Channels

* **Slack**: #incidents
* **Email**: [incidents@learnille.com](mailto:incidents@learnille.com)
* **Status Page**: [https://status.learnille.com](https://status.learnille.com)
* **Runbook Repository**: [https://github.com/learnille/runbooks](https://github.com/learnille/runbooks)

## Continuous Improvement

### Metrics Tracking

* Rollback frequency and success rate
* Mean time to rollback
* Customer impact assessment
* Process efficiency improvements

### Process Updates

* Regular review of rollback procedures
* Incorporation of lessons learned
* Tool and automation improvements
* Team training and preparedness


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.