Rollback Instructions

The Kubernetes, ECS, PM2, and Docker Compose procedures below are legacy examples. For the current backend deployment, use Dokploy rollback and the release checks in the production deployment guide.

Overview

Rollback Types

1. Automated Rollback

• Trigger: Health check failures, error thresholds exceeded
• Execution: Immediate, no manual intervention required
• Duration: < 5 minutes
• Scope: Application container rollback

2. Manual Rollback

• Trigger: Detected issues requiring human judgment
• Execution: Manual command execution
• Duration: 10-30 minutes
• Scope: Application, database, or infrastructure

3. Emergency Rollback

• Trigger: Critical system failure, security breach
• Execution: Immediate action by on-call engineer
• Duration: < 15 minutes
• Scope: Complete environment restoration

Pre-Rollback Checklist

Assessment

• Identify the issue and impact
• Determine rollback scope (application, database, infrastructure)
• Check current system status and monitoring
• Notify stakeholders if needed
• Gather evidence (logs, metrics, user reports)

Preparation

• Verify backup availability and integrity
• Check previous version stability
• Prepare rollback commands/scripts
• Ensure access to deployment tools
• Coordinate with team members if needed

Automated Rollback Procedures

Application Rollback (Kubernetes)

Immediate Rollback

# Check current deployment status
kubectl get deployments -n learnille-prod

# Rollback to previous revision
kubectl rollout undo deployment/learnille-api -n learnille-prod

# Monitor rollback progress
kubectl rollout status deployment/learnille-api -n learnille-prod

Blue-Green Rollback

# Check which environment is active
kubectl get service learnille-api -n learnille-prod -o yaml

# Switch traffic back to blue environment
kubectl patch service learnille-api -n learnille-prod -p '{"spec":{"selector":{"version":"blue"}}}'

# Verify traffic switch
kubectl get endpoints learnille-api -n learnille-prod

Database Rollback

Migration Rollback

# Check migration status
npm run migration:status -- --env production

# Rollback last migration
npm run migration:rollback -- --env production

# Verify database integrity
npm run migration:status -- --env production

Data Restore (if needed)

# Restore from backup (if data corruption)
pg_restore -h production-db -U learnille -d learnille_prod backup_file.sql

# Verify data integrity
psql -h production-db -U learnille -d learnille_prod -c "SELECT COUNT(*) FROM users;"

Manual Rollback Procedures

1. Application Rollback

Self-Hosted PM2 / Git Rollback

# SSH into self-hosted production server
ssh admin@server.learnille.dev

# Navigate to application root
cd /opt/learnille

# Checkout previous stable Git commit / tag
git checkout v1.2.0

# Rebuild and reload PM2 process
cd server && npm install && npm run build
pm2 reload learnille-api --update-env

AWS ECS rollback (legacy)

aws ecs update-service \
  --service learnille-api \
  --task-definition learnille-api:1.2.2 \
  --force-new-deployment

# Monitor deployment
aws ecs describe-services \
  --cluster learnille-prod \
  --services learnille-api \
  --query 'services[0].deployments'

Using Docker Compose

# Stop current containers
docker-compose -f production.yml down

# Update image version in compose file
sed -i 's/learnille\/api:1.2.3/learnille\/api:1.2.2/' production.yml

# Start previous version
docker-compose -f production.yml up -d

# Verify containers
docker-compose -f production.yml ps

2. Infrastructure Rollback

Terraform Rollback

# Check current state
terraform show

# Rollback to previous state
terraform plan -state=terraform.tfstate.backup
terraform apply -state=terraform.tfstate.backup

# Verify infrastructure
terraform show

CloudFormation Rollback

# Check stack status
aws cloudformation describe-stacks --stack-name learnille-prod

# Rollback to previous version
aws cloudformation update-stack \
  --stack-name learnille-prod \
  --template-body file://template-v1.2.2.yaml \
  --parameters file://parameters.json

# Monitor stack update
aws cloudformation describe-stack-events --stack-name learnille-prod

3. Configuration Rollback

Environment Variables

# Backup current environment
cp .env.production .env.production.backup

# Restore previous environment
cp .env.production.v1.2.2 .env.production

# Restart application
kubectl rollout restart deployment/learnille-api -n learnille-prod

Configuration Files

# Backup current config
cp config/production.yaml config/production.yaml.backup

# Restore previous config
cp config/production-v1.2.2.yaml config/production.yaml

# Apply configuration
kubectl apply -f config/production.yaml

Emergency Rollback Procedures

Critical System Failure

1. Immediate Assessment
   # Check system status
   curl -f https://api.learnille.com/health || echo "API down"
   
   # Check database connectivity
   psql -h production-db -U learnille -d learnille_prod -c "SELECT 1;"
   
   # Check infrastructure status
   aws ecs describe-services --cluster learnille-prod --services learnille-api
   
2. Execute Emergency Rollback
   # Force rollback to last known good state
   kubectl set image deployment/learnille-api app=learnille/api:1.2.2 -n learnille-prod
   kubectl rollout status deployment/learnille-api -n learnille-prod
   
   # Switch to backup database if needed
   aws rds failover-db-cluster --db-cluster-identifier learnille-prod
   
3. Traffic Management
   # Enable maintenance mode
   kubectl patch configmap learnille-config -p '{"data":{"maintenance":"true"}}'
   
   # Switch to backup load balancer
   aws elbv2 set-ip-address-type \
     --load-balancer-arn $BACKUP_LB_ARN \
     --ip-address-type ipv4
   

Security Incident Response

1. Isolate Affected Systems
   # Block suspicious traffic
   aws waf update-ip-set \
     --name suspicious-ips \
     --scope REGIONAL \
     --id $IP_SET_ID \
     --addresses $SUSPICIOUS_IP
   
   # Disable compromised service
   kubectl scale deployment learnille-api --replicas=0 -n learnille-prod
   
2. Data Recovery
   # Restore from clean backup
   aws s3 cp s3://learnille-backups/clean-backup.sql /tmp/
   psql -h production-db -U learnille -d learnille_prod < /tmp/clean-backup.sql
   
3. Security Validation
   # Run security scan
   docker run --rm -v $(pwd):/src securecodebox/scanner
   
   # Update security groups
   aws ec2 revoke-security-group-ingress --group-id $SG_ID --protocol tcp --port 80 --cidr 0.0.0.0/0
   

Post-Rollback Procedures

Validation Steps

• Application health checks passing
• Database connectivity verified
• User functionality tested
• Performance metrics normal
• Monitoring alerts cleared

Documentation

# Document rollback in incident log
echo "$(date): Rollback executed - $(whoami)" >> rollback-log.txt
echo "Reason: $ROLLBACK_REASON" >> rollback-log.txt
echo "Impact: $ROLLBACK_IMPACT" >> rollback-log.txt

# Update status page
curl -X POST https://status.learnille.com/api/incidents \
  -H "Authorization: Bearer $STATUS_API_KEY" \
  -d '{"status": "resolved", "message": "Issue resolved via rollback"}'

Communication

• Notify internal team via Slack
• Update external status page
• Send customer communication if needed
• Schedule post-mortem meeting

Investigation

• Analyze root cause of deployment failure
• Review monitoring data during incident
• Update deployment processes if needed
• Document lessons learned

Monitoring During Rollback

Key Metrics to Monitor

• Application response time
• Error rates by endpoint
• Database connection pool usage
• Infrastructure resource utilization
• User session success rate

Alerting During Rollback

# Temporary rollback monitoring
groups:
  - name: rollback-monitoring
    rules:
      - alert: RollbackHealthCheck
        expr: up{job="learnille-api"} == 0
        for: 2m
        labels:
          severity: critical
          rollback: "true"
      - alert: RollbackErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
          rollback: "true"

Rollback Testing

Regular Testing

• Monthly rollback drills
• Automated rollback testing in staging
• Documentation review and updates
• Team training sessions

Test Scenarios

• Application deployment failure
• Database migration failure
• Infrastructure configuration error
• Security vulnerability discovery
• Performance degradation

Tools and Resources

Rollback Scripts

#!/bin/bash
# emergency-rollback.sh
set -e

echo "Starting emergency rollback..."

# Get last stable version
STABLE_VERSION=$(curl -s https://api.github.com/repos/learnille/learnille/releases/latest | jq -r .tag_name)

# Execute rollback
kubectl set image deployment/learnille-api app=learnille/api:$STABLE_VERSION
kubectl rollout status deployment/learnille-api

echo "Rollback completed successfully"

Monitoring Dashboard

• Rollback status dashboard
• Real-time metrics during rollback
• Historical rollback data
• Success/failure analytics

Contact Information

Emergency Contacts

• Primary On-call: +1-555-0123 (DevOps Engineer)
• Secondary On-call: +1-555-0124 (Senior Engineer)
• Management: +1-555-0125 (Engineering Manager)

Communication Channels

• Slack: #incidents
• Email: incidents@learnille.com
• Status Page: https://status.learnille.com
• Runbook Repository: https://github.com/learnille/runbooks

Continuous Improvement

Metrics Tracking

• Rollback frequency and success rate
• Mean time to rollback
• Customer impact assessment
• Process efficiency improvements

Process Updates

• Regular review of rollback procedures
• Incorporation of lessons learned
• Tool and automation improvements
• Team training and preparedness