ice/docs/deployment.md

Great Lakes Ice Report - Deployment Guide

This guide covers both automated and manual deployment options for the Great Lakes Ice Report application.

Prerequisites

• Server: Debian 12 (ARM64 or x86_64)
• Access: Root or sudo access
• Domain: DNS pointing to your server (optional)
• Ports: 80 and 443 open for web traffic

Automated Deployment (Recommended)

Quick Start

1. Run the deployment script on your server:

   # Default: Downloads config from S3
   curl -sSL https://ice-puremichigan-lol.s3.amazonaws.com/scripts/deploy.sh | bash
   
   # Alternative: Use local files only (no S3)
   curl -sSL https://ice-puremichigan-lol.s3.amazonaws.com/scripts/deploy.sh | S3_BUCKET_NAME=none bash
   

2. Follow the printed instructions from the script:

   # Clone repository
   git clone https://git.deco.sh/deco/ice.git /opt/icewatch
   
   # Copy config files (only if using S3_BUCKET_NAME=none)
   sudo cp /opt/icewatch/scripts/icewatch.service /etc/systemd/system/
   sudo cp /opt/icewatch/scripts/Caddyfile /etc/caddy/Caddyfile
   
   # Set up application
   cd /opt/icewatch
   npm install
   npm run build  # Compile TypeScript and build CSS
   cp .env.example .env
   nano .env  # Add your MapBox token and admin password
   
   # Configure domain (if needed)
   sudo nano /etc/caddy/Caddyfile
   
   # Database files are created automatically by the application
   
   # Set permissions
   sudo chown -R icewatch:icewatch /opt/icewatch
   sudo chmod 660 /opt/icewatch/.env
   
   # Start services
   sudo systemctl daemon-reload
   sudo systemctl enable icewatch caddy
   sudo systemctl start icewatch caddy
   

What the Automated Script Does

The deployment script automatically:

1. System Updates: Updates package repositories and system packages
2. Node.js Installation: Installs Node.js 20.x with build tools
3. Go Installation: Installs Go (required for building Caddy with plugins)
4. Custom Caddy Build: Builds Caddy with rate limiting plugin using xcaddy
5. Service Configuration: Creates systemd services for both the app and Caddy
6. Security Setup: Configures users, permissions, and security settings

Manual Deployment

1. System Preparation

# Update system
sudo apt update && sudo apt upgrade -y

# Install Node.js 20.x
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs build-essential

# Install Git (if not already installed)
sudo apt install -y git

# Verify Node.js version (should be 18+ for TypeScript)
node --version

2. Install Go (for Custom Caddy)

# Architecture detection
ARCH=$(uname -m)
case $ARCH in
    x86_64)
        GO_ARCH="amd64"
        ;;
    aarch64|arm64)
        GO_ARCH="arm64"
        ;;
    *)
        echo "Unsupported architecture: $ARCH"
        exit 1
        ;;
esac

# Download and install Go
GO_VERSION="1.21.5"
GO_TARBALL="go${GO_VERSION}.linux-${GO_ARCH}.tar.gz"
wget -q "https://go.dev/dl/${GO_TARBALL}"
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf "${GO_TARBALL}"

# Add to PATH
export PATH=$PATH:/usr/local/go/bin
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc

# Clean up
rm -f "${GO_TARBALL}"

3. Build Custom Caddy with Rate Limiting

# Install xcaddy
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
export PATH=$PATH:$(go env GOPATH)/bin

# Build Caddy with rate limiting plugin
xcaddy build --with github.com/mholt/caddy-ratelimit

# Install Caddy
sudo mv caddy /usr/local/bin/caddy
sudo chmod +x /usr/local/bin/caddy

4. Create Users and Directories

# Create Caddy user
sudo groupadd --system caddy
sudo useradd --system --gid caddy --create-home --home-dir /var/lib/caddy --shell /usr/sbin/nologin caddy

# Create directories
sudo mkdir -p /etc/caddy /var/log/caddy
sudo chown -R caddy:caddy /var/log/caddy

# Create app user
sudo groupadd --system icewatch
sudo useradd --system --gid icewatch --create-home --home-dir /opt/icewatch --shell /usr/sbin/nologin icewatch

5. Deploy Application

# Clone repository
sudo git clone https://git.deco.sh/deco/ice.git /opt/icewatch
cd /opt/icewatch

# Set temporary ownership for installation
sudo chown -R $USER:$USER /opt/icewatch

# Install dependencies and build
npm install  # This automatically builds CSS
npm run build  # Compile TypeScript and build CSS

# Database files are created automatically by the application

# Set final ownership
sudo chown -R icewatch:icewatch /opt/icewatch

6. Configure Environment

# Copy environment template
sudo cp .env.example .env

# Edit environment file
sudo nano .env

Required environment variables:

# MapBox API token (get free token at https://account.mapbox.com/access-tokens/)
MAPBOX_ACCESS_TOKEN=pk.your_mapbox_token_here

# Admin panel password
ADMIN_PASSWORD=your_secure_password

# Server port (default: 3000)
PORT=3000

# Node environment
NODE_ENV=production

7. Configure Systemd Services

Create app service:

sudo cp scripts/icewatch.service /etc/systemd/system/
# Ensure the service file has the correct ExecStart:
# ExecStart=/usr/bin/node dist/server.js

Create Caddy service:

sudo tee /etc/systemd/system/caddy.service > /dev/null <<EOF
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target
EOF

8. Configure Caddy

# Copy Caddyfile
sudo cp scripts/Caddyfile /etc/caddy/

# Edit domain name in Caddyfile
sudo nano /etc/caddy/Caddyfile
# Change 'ice.puremichigan.lol' to your domain

9. Start Services

# Reload systemd
sudo systemctl daemon-reload

# Enable and start services
sudo systemctl enable icewatch
sudo systemctl start icewatch

sudo systemctl enable caddy
sudo systemctl start caddy

Security Configuration

Firewall Setup

# Install UFW if not present
sudo apt install -y ufw

# Configure firewall
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable

SSL Certificates

Caddy automatically obtains SSL certificates from Let's Encrypt. No manual configuration required.

Rate Limiting

The application includes multiple layers of rate limiting:

1. Application Level: 10 requests per 15 minutes per IP for location submissions
2. Caddy Level:
   • 30 requests per minute for general API endpoints
   • 5 requests per minute for location submissions

Monitoring and Maintenance

Service Status

# Check application status
sudo systemctl status icewatch

# Check Caddy status
sudo systemctl status caddy

# View logs
sudo journalctl -u icewatch -f
sudo journalctl -u caddy -f

Log Files

• Application logs: sudo journalctl -u icewatch
• Caddy access logs: /var/log/caddy/icewatch.log
• System logs: /var/log/syslog

Database Maintenance

The application automatically:

• Cleans up expired location reports (older than 48 hours)
• Maintains profanity filter database
• Uses SQLite with automatic cleanup

Updates

# Navigate to app directory
cd /opt/icewatch

# Stop the service
sudo systemctl stop icewatch

# Pull latest changes
sudo -u icewatch git pull

# Install new dependencies and rebuild
sudo -u icewatch npm install
sudo -u icewatch npm run build

# Restart service
sudo systemctl restart icewatch

Troubleshooting

Common Issues

1. Service won't start:

   sudo journalctl -u icewatch --no-pager
   

2. SSL certificate issues:

   sudo journalctl -u caddy --no-pager
   

3. Permission issues:

   sudo chown -R icewatch:icewatch /opt/icewatch
   

4. TypeScript compilation issues:

   cd /opt/icewatch
   sudo -u icewatch npm run build
   

5. Port conflicts:

   sudo netstat -tlnp | grep :3000
   sudo netstat -tlnp | grep :80
   sudo netstat -tlnp | grep :443
   

Performance Tuning

For high-traffic deployments:

1. Increase Node.js memory limit:

   # Edit service file
   sudo nano /etc/systemd/system/icewatch.service
   # Add: Environment=NODE_OPTIONS="--max-old-space-size=4096"
   

2. Configure log rotation:

   sudo nano /etc/logrotate.d/icewatch
   

3. Monitor resource usage:

   htop
   sudo iotop
   

Support

For deployment issues:

• Check system logs: sudo journalctl -f
• Verify environment variables: sudo -u icewatch env
• Test application directly: cd /opt/icewatch && sudo -u icewatch node dist/server.js
• Review this documentation and configuration files
• Ensure TypeScript is compiled: npm run build

The application includes comprehensive logging and monitoring to help diagnose issues quickly.