ice/docs/deployment.md

# Great Lakes Ice Report - Deployment Guide

This guide covers both automated and manual deployment options for the Great Lakes Ice Report application.

## Prerequisites

- **Server**: Debian 12 (ARM64 or x86_64)
- **Access**: Root or sudo access
- **Domain**: DNS pointing to your server (optional)
- **Ports**: 80 and 443 open for web traffic

## Automated Deployment (Recommended)

### Quick Start

1. **Run the deployment script on your server:**
   ```bash
   # Default: Downloads config from S3
   curl -sSL https://ice-puremichigan-lol.s3.amazonaws.com/scripts/deploy.sh | bash
   
   # Alternative: Use local files only (no S3)
   curl -sSL https://ice-puremichigan-lol.s3.amazonaws.com/scripts/deploy.sh | S3_BUCKET_NAME=none bash
   ```

2. **Follow the printed instructions from the script:**
   ```bash
   # Clone repository
   git clone https://git.deco.sh/deco/ice.git /opt/icewatch
   
   # Copy config files (only if using S3_BUCKET_NAME=none)
   sudo cp /opt/icewatch/scripts/icewatch.service /etc/systemd/system/
   sudo cp /opt/icewatch/scripts/Caddyfile /etc/caddy/Caddyfile
   
   # Set up application
   cd /opt/icewatch
   npm install
   npm run build  # Compile TypeScript and build CSS
   cp .env.example .env
   nano .env  # Add your MapBox token and admin password
   
   # Configure domain (if needed)
   sudo nano /etc/caddy/Caddyfile
   
   # Database files are created automatically by the application
   
   # Set permissions
   sudo chown -R icewatch:icewatch /opt/icewatch
   sudo chmod 660 /opt/icewatch/.env
   
   # Start services
   sudo systemctl daemon-reload
   sudo systemctl enable icewatch caddy
   sudo systemctl start icewatch caddy
   ```

### What the Automated Script Does

The deployment script automatically:

1. **System Updates**: Updates package repositories and system packages
2. **Node.js Installation**: Installs Node.js 20.x with build tools
3. **Go Installation**: Installs Go (required for building Caddy with plugins)
4. **Custom Caddy Build**: Builds Caddy with rate limiting plugin using xcaddy
5. **Service Configuration**: Creates systemd services for both the app and Caddy
6. **Security Setup**: Configures users, permissions, and security settings

## Manual Deployment

### 1. System Preparation

```bash
# Update system
sudo apt update && sudo apt upgrade -y

# Install Node.js 20.x
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs build-essential

# Install Git (if not already installed)
sudo apt install -y git

# Verify Node.js version (should be 18+ for TypeScript)
node --version
```

### 2. Install Go (for Custom Caddy)

```bash
# Architecture detection
ARCH=$(uname -m)
case $ARCH in
    x86_64)
        GO_ARCH="amd64"
        ;;
    aarch64|arm64)
        GO_ARCH="arm64"
        ;;
    *)
        echo "Unsupported architecture: $ARCH"
        exit 1
        ;;
esac

# Download and install Go
GO_VERSION="1.21.5"
GO_TARBALL="go${GO_VERSION}.linux-${GO_ARCH}.tar.gz"
wget -q "https://go.dev/dl/${GO_TARBALL}"
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf "${GO_TARBALL}"

# Add to PATH
export PATH=$PATH:/usr/local/go/bin
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc

# Clean up
rm -f "${GO_TARBALL}"
```

### 3. Build Custom Caddy with Rate Limiting

```bash
# Install xcaddy
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
export PATH=$PATH:$(go env GOPATH)/bin

# Build Caddy with rate limiting plugin
xcaddy build --with github.com/mholt/caddy-ratelimit

# Install Caddy
sudo mv caddy /usr/local/bin/caddy
sudo chmod +x /usr/local/bin/caddy
```

### 4. Create Users and Directories

```bash
# Create Caddy user
sudo groupadd --system caddy
sudo useradd --system --gid caddy --create-home --home-dir /var/lib/caddy --shell /usr/sbin/nologin caddy

# Create directories
sudo mkdir -p /etc/caddy /var/log/caddy
sudo chown -R caddy:caddy /var/log/caddy

# Create app user
sudo groupadd --system icewatch
sudo useradd --system --gid icewatch --create-home --home-dir /opt/icewatch --shell /usr/sbin/nologin icewatch
```

### 5. Deploy Application

```bash
# Clone repository
sudo git clone https://git.deco.sh/deco/ice.git /opt/icewatch
cd /opt/icewatch

# Set temporary ownership for installation
sudo chown -R $USER:$USER /opt/icewatch

# Install dependencies and build
npm install  # This automatically builds CSS
npm run build  # Compile TypeScript and build CSS

# Database files are created automatically by the application

# Set final ownership
sudo chown -R icewatch:icewatch /opt/icewatch
```

### 6. Configure Environment

```bash
# Copy environment template
sudo cp .env.example .env

# Edit environment file
sudo nano .env
```

**Required environment variables:**
```bash
# MapBox API token (get free token at https://account.mapbox.com/access-tokens/)
MAPBOX_ACCESS_TOKEN=pk.your_mapbox_token_here

# Admin panel password
ADMIN_PASSWORD=your_secure_password

# Server port (default: 3000)
PORT=3000

# Node environment
NODE_ENV=production
```

### 7. Configure Systemd Services

**Create app service:**
```bash
sudo cp scripts/icewatch.service /etc/systemd/system/
# Ensure the service file has the correct ExecStart:
# ExecStart=/usr/bin/node dist/server.js
```

**Create Caddy service:**
```bash
sudo tee /etc/systemd/system/caddy.service > /dev/null <<EOF
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target
EOF
```

### 8. Configure Caddy

```bash
# Copy Caddyfile
sudo cp scripts/Caddyfile /etc/caddy/

# Edit domain name in Caddyfile
sudo nano /etc/caddy/Caddyfile
# Change 'ice.puremichigan.lol' to your domain
```

### 9. Start Services

```bash
# Reload systemd
sudo systemctl daemon-reload

# Enable and start services
sudo systemctl enable icewatch
sudo systemctl start icewatch

sudo systemctl enable caddy
sudo systemctl start caddy
```

## Security Configuration

### Firewall Setup

```bash
# Install UFW if not present
sudo apt install -y ufw

# Configure firewall
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable
```

### SSL Certificates

Caddy automatically obtains SSL certificates from Let's Encrypt. No manual configuration required.

### Rate Limiting

The application includes multiple layers of rate limiting:

1. **Application Level**: 10 requests per 15 minutes per IP for location submissions
2. **Caddy Level**: 
   - 30 requests per minute for general API endpoints
   - 5 requests per minute for location submissions

## Monitoring and Maintenance

### Service Status

```bash
# Check application status
sudo systemctl status icewatch

# Check Caddy status
sudo systemctl status caddy

# View logs
sudo journalctl -u icewatch -f
sudo journalctl -u caddy -f
```

### Log Files

- **Application logs**: `sudo journalctl -u icewatch`
- **Caddy access logs**: `/var/log/caddy/great-lakes-ice-report.log`
- **System logs**: `/var/log/syslog`

### Database Maintenance

The application automatically:
- Cleans up expired location reports (older than 48 hours)
- Maintains profanity filter database
- Uses SQLite with automatic cleanup

### Updates

```bash
# Navigate to app directory
cd /opt/icewatch

# Stop the service
sudo systemctl stop icewatch

# Pull latest changes
sudo -u icewatch git pull

# Install new dependencies and rebuild
sudo -u icewatch npm install
sudo -u icewatch npm run build

# Restart service
sudo systemctl restart icewatch
```

## Troubleshooting

### Common Issues

1. **Service won't start**:
   ```bash
   sudo journalctl -u icewatch --no-pager
   ```

2. **SSL certificate issues**:
   ```bash
   sudo journalctl -u caddy --no-pager
   ```

3. **Permission issues**:
   ```bash
   sudo chown -R icewatch:icewatch /opt/icewatch
   ```

4. **TypeScript compilation issues**:
   ```bash
   cd /opt/icewatch
   sudo -u icewatch npm run build
   ```

4. **Port conflicts**:
   ```bash
   sudo netstat -tlnp | grep :3000
   sudo netstat -tlnp | grep :80
   sudo netstat -tlnp | grep :443
   ```

### Performance Tuning

For high-traffic deployments:

1. **Increase Node.js memory limit**:
   ```bash
   # Edit service file
   sudo nano /etc/systemd/system/icewatch.service
   # Add: Environment=NODE_OPTIONS="--max-old-space-size=4096"
   ```

2. **Configure log rotation**:
   ```bash
   sudo nano /etc/logrotate.d/great-lakes-ice-report
   ```

3. **Monitor resource usage**:
   ```bash
   htop
   sudo iotop
   ```

## Support

For deployment issues:
- Check system logs: `sudo journalctl -f`
- Verify environment variables: `sudo -u icewatch env`
- Test application directly: `cd /opt/icewatch && sudo -u icewatch node dist/server.js`
- Review this documentation and configuration files
- Ensure TypeScript is compiled: `npm run build`

The application includes comprehensive logging and monitoring to help diagnose issues quickly.