Skip to main content

GridPane Git Release Pruning Issue - Diagnosis & Fix

🚨 Issue Summary​

Problem: GridPane servers can stop pruning old Git releases when a single release fails to delete properly. This causes a cascading failure where every subsequent deployment fails silently until the server runs out of disk space.

Root Cause: When one release directory has permission issues or corrupted files that prevent deletion, the automated pruning process breaks completely, but deployments continue to create new releases without cleaning up old ones.

Impact: Server disk space exhaustion, potential site downtime, deployment failures.

📊 Proactive Monitoring​

Early Warning Signs​

1. BetterStack Alerts​

Monitor for disk space alerts:

  • Look for "New incident" alerts about disk usage
  • Watch for resolution patterns (incident resolved/started cycles)

2. GridPane Slack Notifications​

GridPane sends some (not all) notifications to Slack. Monitor the #gridpane channel for:

  • Deployment failures
  • Server disk space warnings
  • Unusual deployment patterns
  • Example Notification:
Issue Detected! 
On Server: 144.202.28.119
Resolution: FILESYSTEM Capacity Warning > 85%!DetailsThe Filesystem on digitalchurch-atlanta-02 144.202.28.119 has surpassed 85% disk capacity. 
Really, it would be a great idea to scale your disk up now...

3. Manual Checks​

If you suspect issues, SSH into the server and check:

# Check disk usage
df -h

# Count releases in /var/www/
ls -la /var/www/ | grep release | wc -l

# Check for excessive releases (normal = 5-10 max)
ls -la /var/www/releases-*

🔧 Diagnosis Commands​

Identify Current/Core Release​

# Check which release is currently active (core). Never delete the currently active release (symlinked from `/var/www/core`).
ls -l /var/www/core

# Example output
lrwxrwxrwx 1 root root 32 Mar 10 17:49 /var/www/core -> /var/www/core-release-1773164964

# List all releases in order (Last one is usually active core)
ls -d core-release-*

🛠️ Fix Procedure​

Remove Problematic Release​

# Remove the oldest release first to see if that clears everything
rm -rf core-release-<old-release-number>

# If deletion fails, reset permissions on that release
sudo chattr -R -i /var/www/core-release-<old-release-number>

# Try removal again after permissions fix
rm -rf core-release-<old-release-number>

Step 3: Trigger Automatic Cleanup​

Once the problematic release is removed:

# Deploy the latest release (this will trigger automatic pruning)
# Use GridPane's deployment interface or git push to trigger

Step 4: Verify Fix​

# Check that old releases are now being cleaned up
ls -la /var/www/ | grep release

# Verify disk space recovered
df -h

# Confirm current release is still correct
readlink /var/www/current

📋 GridPane CLI Commands Reference​

Based on GridPane support emails (Ticket #627013):

Release Management​

# Navigate to releases directory
cd /var/www/

# List all releases (should normally be 6)
ls -lat | grep release

# Check current active release
readlink current

# Manual cleanup (if needed)
rm -rf releases-YYYYMMDD-HHMMSS

Disk Space Analysis​

# Check overall disk usage
df -h

# Check usage by directory
du -sh releases-* | sort -hr

# Find largest files in stuck release
du -sh releases-YYYYMMDD-HHMMSS/*

🚨 Server Affected: Atlanta 02​

Historical Details:

  • Date: March 11, 2026
  • Server: Atlanta 02 (WPMT server)
  • Issue: Single release with permission/attribute issues blocked all pruning
  • Resolution: Force deleted problematic release, redeployed latest, automatic pruning resumed

🔍 Monitoring Setup​

Automated Checks​

Create monitoring for:

  1. Release count - Alert if > 8 releases exist
  2. Disk space - Alert at 80% usage
  3. Deployment success rate - Alert on consecutive failures

Weekly Maintenance​

# Weekly check script
#!/bin/bash
echo "=== Release Count Check ==="
ls /var/www/ | grep release | wc -l

echo "=== Disk Usage Check ==="
df -h | grep "/dev"

echo "=== Current Release ==="
readlink /var/www/current

📞 Escalation Path​

  1. First Signs: Check release count and disk space
  2. Confirmed Issue: Identify problematic release directory
  3. Force Delete: Use chmod + rm -rf on stuck release
  4. Trigger Cleanup: Redeploy to resume automatic pruning
  5. If Still Stuck: Contact GridPane support with specific release directory name

💡 Prevention​

  • Monitor release counts - Normal should be 5-10 releases max
  • Watch BetterStack alerts for disk space warnings
  • Regular deployments help identify issues before disk fills
  • Document server maintenance windows to check release pruning

📝 Support Ticket Pattern​

When contacting GridPane:

  • Subject: "GIT branches not pruning on [SERVER] - Release stuck"
  • Include: Server name, problematic release directory, error messages
  • Mention: Files with permission/attribute issues preventing removal