dlx-ansible/docs/NPM-SSH-PROXY-FOR-JENKINS.md

# NPM SSH Proxy for Jenkins Agents

**Date**: 2026-02-09
**Purpose**: Use Nginx Proxy Manager to proxy SSH connections to Jenkins agents
**Benefit**: Centralized access control, logging, and SSL termination

## Architecture

### Before (Direct SSH)
```
External → Router:22 → Jenkins:22
```
**Issues**:
- Direct SSH exposure
- No centralized logging
- Single point of failure

### After (NPM Proxy)
```
External → NPM:2222 → Jenkins:22
Jenkins Agent Config: Connect to NPM:2222
```
**Benefits**:
- ✅ Centralized access through NPM
- ✅ NPM logging and monitoring
- ✅ Easier to manage multiple agents
- ✅ Can add rate limiting
- ✅ SSL/TLS for agent.jar downloads via web UI

## NPM Configuration

### Step 1: Create TCP Stream in NPM

**Via NPM Web UI** (http://192.168.200.71:81):

1. **Login to NPM**
   - URL: http://192.168.200.71:81
   - Default: admin@example.com / changeme

2. **Navigate to Streams**
   - Click **Streams** in the sidebar
   - Click **Add Stream**

3. **Configure Incoming Stream**
   - **Incoming Port**: `2222`
   - **Forwarding Host**: `192.168.200.91` (jenkins server)
   - **Forwarding Port**: `22`
   - **TCP Forwarding**: Enabled
   - **UDP Forwarding**: Disabled

4. **Enable SSL/TLS Forwarding** (Optional)
   - For encrypted SSH tunneling
   - **SSL Certificate**: Upload or use Let's Encrypt
   - **Force SSL**: Enabled

5. **Save**

### Step 2: Update Firewall on NPM Server

The NPM server needs to allow incoming connections on port 2222:

```bash
# Run from ansible control machine
ansible npm -m community.general.ufw -a "rule=allow port=2222 proto=tcp" -b

# Verify
ansible npm -m shell -a "ufw status | grep 2222" -b
```

### Step 3: Update Jenkins Agent Configuration

**In Jenkins Web UI** (http://192.168.200.91:8080):

1. **Navigate to Agent**
   - Go to: **Manage Jenkins** → **Manage Nodes and Clouds**
   - Click on the agent that uses SSH

2. **Update SSH Host**
   - **Host**: Change from `45.16.76.42` to `192.168.200.71` (NPM server)
   - **Port**: Change from `22` to `2222`
   - **Credentials**: Keep as `dlx-key`

3. **Advanced Settings**
   - **JVM Options**: Add if needed: `-Djava.awt.headless=true`
   - **Prefix Start Agent Command**: Leave empty
   - **Suffix Start Agent Command**: Leave empty

4. **Save and Launch Agent**

### Step 4: Update Router Port Forwarding (Optional)

If you want external access through the router:

**Old Rule**:
- External Port: `22`
- Internal IP: `192.168.200.91` (jenkins)
- Internal Port: `22`

**New Rule**:
- External Port: `2222` (or keep 22 if you prefer)
- Internal IP: `192.168.200.71` (NPM)
- Internal Port: `2222`

## Testing

### Test 1: SSH Through NPM from Local Network
```bash
# Test SSH connection through NPM proxy
ssh -p 2222 dlxadmin@192.168.200.71

# Should connect to jenkins server
hostname  # Should output: dlx-sonar
```

### Test 2: Jenkins Agent Connection
```bash
# From jenkins server, test as jenkins user
sudo -u jenkins ssh -p 2222 -i /var/lib/jenkins/.ssh/id_rsa dlxadmin@192.168.200.71 'hostname'

# Expected output: dlx-sonar
```

### Test 3: Launch Agent from Jenkins UI
1. Go to: http://192.168.200.91:8080/computer/
2. Find the agent
3. Click **Launch agent**
4. Check logs for successful connection

## NPM Stream Configuration File

NPM stores stream configurations in its database. For backup/reference:

```json
{
  "incoming_port": 2222,
  "forwarding_host": "192.168.200.91",
  "forwarding_port": 22,
  "tcp_forwarding": true,
  "udp_forwarding": false,
  "enabled": true
}
```

## Troubleshooting

### Issue: Cannot connect to NPM:2222

**Check NPM firewall**:
```bash
ansible npm -m shell -a "ufw status | grep 2222" -b
ansible npm -m shell -a "ss -tlnp | grep 2222" -b
```

**Check NPM stream is active**:
- Login to NPM UI
- Go to Streams
- Verify stream is enabled (green toggle)

### Issue: Connection timeout

**Check NPM can reach Jenkins**:
```bash
ansible npm -m shell -a "ping -c 2 192.168.200.91" -b
ansible npm -m shell -a "nc -zv 192.168.200.91 22" -b
```

**Check Jenkins SSH is running**:
```bash
ansible jenkins -m shell -a "systemctl status sshd" -b
```

### Issue: Authentication fails

**Verify SSH key**:
```bash
# Get Jenkins public key
ansible jenkins -m shell -a "cat /var/lib/jenkins/.ssh/id_rsa.pub" -b

# Check it's in authorized_keys
ansible jenkins -m shell -a "grep jenkins /home/dlxadmin/.ssh/authorized_keys" -b
```

### Issue: NPM stream not forwarding

**Check NPM logs**:
```bash
ansible npm -m shell -a "docker logs nginx-proxy-manager --tail 100" -b

# Look for stream-related errors
```

**Restart NPM**:
```bash
ansible npm -m shell -a "docker restart nginx-proxy-manager" -b
```

## Advanced: Multiple Jenkins Agents

For multiple remote agents, create separate streams:

| Agent | NPM Port | Forward To | Purpose |
|-------|----------|------------|---------|
| jenkins-local | 2222 | 192.168.200.91:22 | Local Jenkins agent |
| build-agent-1 | 2223 | 192.168.200.120:22 | Remote build agent |
| build-agent-2 | 2224 | 192.168.200.121:22 | Remote build agent |

## Security Considerations

### Recommended Firewall Rules

**NPM Server** (192.168.200.71):
```yaml
common_firewall_allowed_ports:
  - "22/tcp"    # SSH admin access
  - "80/tcp"    # HTTP
  - "443/tcp"   # HTTPS
  - "81/tcp"    # NPM Admin panel
  - "2222/tcp"  # Jenkins SSH proxy
  - "2223/tcp"  # Additional agents (if needed)
```

**Jenkins Server** (192.168.200.91):
```yaml
common_firewall_allowed_ports:
  - "22/tcp"    # SSH (restrict to NPM IP only)
  - "8080/tcp"  # Jenkins Web UI
  - "9000/tcp"  # SonarQube
```

### Restrict SSH Access to NPM Only

On Jenkins server, restrict SSH to only accept from NPM:

```bash
# Allow SSH only from NPM server
ansible jenkins -m community.general.ufw -a "rule=allow from=192.168.200.71 to=any port=22 proto=tcp" -b

# Deny SSH from all others (if not already default)
ansible jenkins -m community.general.ufw -a "rule=deny port=22 proto=tcp" -b
```

## Monitoring

### NPM Access Logs
```bash
# View NPM access logs
ansible npm -m shell -a "docker logs nginx-proxy-manager --tail 50 | grep stream" -b
```

### Connection Statistics
```bash
# Check active SSH connections through NPM
ansible npm -m shell -a "ss -tn | grep :2222" -b

# Check connections on Jenkins
ansible jenkins -m shell -a "ss -tn | grep :22 | grep ESTAB" -b
```

## Backup and Recovery

### Backup NPM Configuration
```bash
# Backup NPM database
ansible npm -m shell -a "docker exec nginx-proxy-manager sqlite3 /data/database.sqlite .dump > /tmp/npm-backup.sql" -b

# Download backup
ansible npm -m fetch -a "src=/tmp/npm-backup.sql dest=./backups/npm-backup-$(date +%Y%m%d).sql" -b
```

### Restore NPM Configuration
```bash
# Upload backup
ansible npm -m copy -a "src=./backups/npm-backup.sql dest=/tmp/npm-restore.sql" -b

# Restore database
ansible npm -m shell -a "docker exec nginx-proxy-manager sqlite3 /data/database.sqlite < /tmp/npm-restore.sql" -b

# Restart NPM
ansible npm -m shell -a "docker restart nginx-proxy-manager" -b
```

## Migration Checklist

- [ ] Create TCP stream in NPM (port 2222 → jenkins:22)
- [ ] Update NPM firewall to allow port 2222
- [ ] Test SSH connection through NPM proxy
- [ ] Update Jenkins agent SSH host to NPM IP
- [ ] Update Jenkins agent SSH port to 2222
- [ ] Test agent connection in Jenkins UI
- [ ] Update router port forwarding (if external access needed)
- [ ] Restrict Jenkins SSH to NPM IP only (optional but recommended)
- [ ] Document new configuration
- [ ] Update monitoring/alerting rules

## Related Files

- NPM host vars: `host_vars/npm.yml`
- Jenkins host vars: `host_vars/jenkins.yml`
- NPM firewall playbook: `playbooks/configure-npm-firewall.yml` (to be created)
- This documentation: `docs/NPM-SSH-PROXY-FOR-JENKINS.md`