dlx-ansible/docs/JENKINS-SSH-AGENT-TROUBLESHOOTING.md

# Jenkins SSH Agent Authentication Troubleshooting

**Date**: 2026-02-09
**Issue**: Jenkins cannot authenticate to remote build agent
**Error**: `Authentication failed` when connecting to remote SSH agent

## Problem Description

Jenkins is configured to connect to a remote build agent via SSH but authentication fails:

```
SSHLauncher{host='45.16.76.42', port=22, credentialsId='dlx-key', ...}
[SSH] Opening SSH connection to 45.16.76.42:22.
[SSH] Authentication failed.
```

## Root Cause

The SSH public key associated with Jenkins's 'dlx-key' credential is not present in the `~/.ssh/authorized_keys` file on the remote agent server (45.16.76.42).

## Quick Diagnosis

From jenkins server:
```bash
# Test network connectivity
ping -c 2 45.16.76.42

# Test SSH connectivity (should fail with "Permission denied (publickey)")
ssh dlxadmin@45.16.76.42
```

## Solution Options

### Option 1: Add Jenkins Key to Remote Agent (Quickest)

**Step 1** - Get Jenkins's public key from Web UI:
1. Open Jenkins: http://192.168.200.91:8080
2. Go to: **Manage Jenkins** → **Credentials** → **System** → **Global credentials (unrestricted)**
3. Click on the **'dlx-key'** credential
4. Look for the public key display (if available)
5. Copy the public key

**Step 2** - Add to remote agent:
```bash
# SSH to the remote agent
ssh dlxadmin@45.16.76.42

# Add the Jenkins public key
echo "ssh-rsa AAAA... jenkins@host" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

# Verify authorized_keys format
cat ~/.ssh/authorized_keys
```

**Step 3** - Test connection from Jenkins server:
```bash
# SSH to jenkins server
ssh dlxadmin@192.168.200.91

# Test connection as jenkins user
sudo -u jenkins ssh -o StrictHostKeyChecking=no dlxadmin@45.16.76.42 'echo "Success!"'
```

### Option 2: Create New SSH Key for Jenkins (Most Reliable)

**Step 1** - Run the Ansible playbook:
```bash
ansible-playbook playbooks/setup-jenkins-agent-ssh.yml -e "agent_host=45.16.76.42"
```

This will:
- Create SSH key pair for jenkins user at `/var/lib/jenkins/.ssh/id_rsa`
- Display the public key
- Create helper script to copy key to agent

**Step 2** - Copy key to agent (choose one method):

**Method A - Automatic** (if you have SSH access):
```bash
ssh dlxadmin@192.168.200.91
/tmp/copy-jenkins-key-to-agent.sh
```

**Method B - Manual**:
```bash
# Get public key from jenkins server
ssh dlxadmin@192.168.200.91 'sudo cat /var/lib/jenkins/.ssh/id_rsa.pub'

# Add to agent's authorized_keys
ssh dlxadmin@45.16.76.42
echo "<paste-public-key>" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```

**Step 3** - Update Jenkins credential:
1. Go to: http://192.168.200.91:8080/manage/credentials/
2. Click on **'dlx-key'** credential (or create new one)
3. Click **Update**
4. Under "Private Key":
   - Select **Enter directly**
   - Copy content from: `/var/lib/jenkins/.ssh/id_rsa` on jenkins server
5. Save

**Step 4** - Test Jenkins agent connection:
1. Go to: http://192.168.200.91:8080/computer/
2. Find the agent that uses 45.16.76.42
3. Click **Launch agent** or **Relaunch agent**
4. Check logs for successful connection

### Option 3: Use Existing dlxadmin Key

If dlxadmin user already has SSH access to the agent:

**Step 1** - Copy dlxadmin's key to jenkins user:
```bash
ssh dlxadmin@192.168.200.91

# Copy key to jenkins user
sudo cp ~/.ssh/id_ed25519 /var/lib/jenkins/.ssh/
sudo cp ~/.ssh/id_ed25519.pub /var/lib/jenkins/.ssh/
sudo chown jenkins:jenkins /var/lib/jenkins/.ssh/id_ed25519*
sudo chmod 600 /var/lib/jenkins/.ssh/id_ed25519
```

**Step 2** - Update Jenkins credential with this key

## Verification Steps

### 1. Test SSH Connection from Jenkins Server
```bash
# SSH to jenkins server
ssh dlxadmin@192.168.200.91

# Test as jenkins user
sudo -u jenkins ssh -o StrictHostKeyChecking=no dlxadmin@45.16.76.42 'hostname'
```

Expected output: The hostname of the remote agent

### 2. Check Agent in Jenkins
```bash
# Via Jenkins Web UI
http://192.168.200.91:8080/computer/

# Look for the agent, should show "Connected" or agent should successfully launch
```

### 3. Verify authorized_keys on Remote Agent
```bash
ssh dlxadmin@45.16.76.42
cat ~/.ssh/authorized_keys | grep jenkins
```

Expected: Should show one or more Jenkins public keys

## Common Issues

### Issue: "Host key verification failed"
**Solution**: Add host to jenkins user's known_hosts:
```bash
sudo -u jenkins ssh-keyscan -H 45.16.76.42 >> /var/lib/jenkins/.ssh/known_hosts
```

### Issue: "Permission denied" even with correct key
**Causes**:
1. Wrong username (check if it should be 'dlxadmin', 'jenkins', 'ubuntu', etc.)
2. Wrong permissions on authorized_keys:
   ```bash
   chmod 700 ~/.ssh
   chmod 600 ~/.ssh/authorized_keys
   ```
3. SELinux blocking (if applicable):
   ```bash
   restorecon -R ~/.ssh
   ```

### Issue: Jenkins shows "dlx-key" but can't edit/view
**Solution**: Credential is encrypted. Either:
- Replace with new credential
- Use Jenkins CLI to export (requires admin token)

## Alternative: Password Authentication

If SSH key auth continues to fail, temporarily enable password auth (NOT RECOMMENDED for production):

```bash
# On remote agent
sudo vim /etc/ssh/sshd_config
# Set: PasswordAuthentication yes
sudo systemctl restart sshd

# In Jenkins, update credential to use password instead of key
```

## Files and Locations

- **Jenkins Home**: `/var/lib/jenkins/`
- **Jenkins SSH Keys**: `/var/lib/jenkins/.ssh/`
- **Jenkins Credentials**: `/var/lib/jenkins/credentials.xml` (encrypted)
- **Remote Agent User**: `dlxadmin`
- **Remote Agent SSH Config**: `/home/dlxadmin/.ssh/authorized_keys`

## Related Commands

```bash
# View Jenkins credential store (encrypted)
sudo cat /var/lib/jenkins/credentials.xml

# Check jenkins user SSH directory
sudo ls -la /var/lib/jenkins/.ssh/

# Test SSH with verbose output
sudo -u jenkins ssh -vvv dlxadmin@45.16.76.42

# View SSH daemon logs on agent
journalctl -u ssh -f

# Check Jenkins logs
sudo tail -f /var/log/jenkins/jenkins.log
```

## Summary Checklist

- [ ] Network connectivity verified (ping works)
- [ ] SSH port 22 is reachable
- [ ] Jenkins user has SSH key pair
- [ ] Jenkins public key is in agent's authorized_keys
- [ ] Permissions correct (700 .ssh, 600 authorized_keys)
- [ ] Jenkins credential 'dlx-key' updated with correct private key
- [ ] Test connection: `sudo -u jenkins ssh dlxadmin@AGENT_IP 'hostname'`
- [ ] Agent launches successfully in Jenkins Web UI