diff --git a/docs/DOCKER-SERVER-SECURITY.md b/docs/DOCKER-SERVER-SECURITY.md new file mode 100644 index 0000000..12d0eab --- /dev/null +++ b/docs/DOCKER-SERVER-SECURITY.md @@ -0,0 +1,236 @@ +# Docker Server Security - Saved Configuration + +**Date**: 2026-02-09 +**Server**: docker (192.168.200.200) +**Status**: Security updates applied ✅, Firewall configuration ready for execution + +## What Was Completed + +### ✅ Security Updates Applied (2026-02-09) + +- **Packages upgraded**: 107 +- **Critical updates**: All applied +- **Status**: System up to date + +```bash +# Packages updated include: +- openssh-client, openssh-server (security) +- systemd, systemd-sysv (security) +- libssl3, openssl (critical security) +- python3, perl (security) +- linux-libc-dev (security) +- And 97 more packages +``` + +## Pending: Firewall Configuration + +### Current State + +- **Firewall**: ❌ Not configured (currently INACTIVE) +- **Risk**: All Docker services exposed to network +- **Open Ports**: + - 22 (SSH) + - 5000, 8000, 8001, 8080, 8081, 8082, 8443, 9000, 11434 (Docker services) + +### Recommended Configuration Options + +#### Option A: Internal Only (Most Secure - Recommended) + +**Use Case**: Docker services only accessed from internal network + +```bash +ansible-playbook playbooks/secure-docker-server-firewall.yml -e "firewall_mode=internal" +``` + +**Result**: +- ✅ SSH (22): Open to all +- ✅ Docker services: Only accessible from 192.168.200.0/24 +- ✅ External web access: Through NPM proxy +- 🔒 Direct external access to Docker ports: Blocked + +#### Option B: Selective External Access + +**Use Case**: Specific Docker services need external access + +```bash +# Example: Allow external access to ports 8080 and 9000 +ansible-playbook playbooks/secure-docker-server-firewall.yml \ + -e "firewall_mode=selective" \ + -e "external_ports=8080,9000" +``` + +**Result**: +- ✅ SSH (22): Open to all +- ✅ Specified ports (8080, 9000): Open to all +- 🔒 Other Docker services: Only internal network + +#### Option C: Custom Configuration + +**Use Case**: You need full control + +1. Test first: + ```bash + ansible-playbook playbooks/secure-docker-server-firewall.yml --check + ``` + +2. Edit the playbook: + ```bash + nano playbooks/secure-docker-server-firewall.yml + # Modify docker_service_ports variable + ``` + +3. Apply: + ```bash + ansible-playbook playbooks/secure-docker-server-firewall.yml + ``` + +## Docker Services Identification + +These ports were found running on the docker server: + +| Port | Service | Typical Use | Recommend | +|------|---------|-------------|-----------| +| 5000 | Docker Registry? | Container registry | Internal only | +| 8000 | Unknown | Web service | Internal only | +| 8001 | Unknown | Web service | Internal only | +| 8080 | Common web | Jenkins/Tomcat/Generic | Via NPM proxy | +| 8081 | Unknown | Web service | Internal only | +| 8082 | Unknown | Web service | Internal only | +| 8443 | HTTPS service | Web service (SSL) | Via NPM proxy | +| 9000 | Portainer/SonarQube | Container mgmt | Internal only | +| 11434 | Ollama? | AI service | Internal only | + +**Recommendation**: Use NPM (nginx) at 192.168.200.71 to proxy external web traffic to internal Docker services. + +## Pre-Execution Checklist + +Before running the firewall configuration: + +- [ ] **Identify required external access** + - Which services need to be accessed from outside? + - Can they be proxied through NPM instead? + +- [ ] **Verify NPM proxy setup** + - Is NPM configured to proxy to Docker services? + - Test internal access first + +- [ ] **Have backup access** + - Ensure you have console access if SSH locks you out + - Or run from the server locally + +- [ ] **Test in check mode first** + ```bash + ansible-playbook playbooks/secure-docker-server-firewall.yml --check + ``` + +- [ ] **Monitor impact** + - Check Docker containers still work + - Verify internal network access + - Test external access if configured + +## Execution Instructions + +### Step 1: Decide on firewall mode + +Ask yourself: +1. Do any Docker services need direct external access? (Usually NO) +2. Are you using NPM proxy for web services? (Recommended YES) +3. Is everything accessed from internal network only? (Ideal YES) + +### Step 2: Run the appropriate command + +**Most Common** (Internal only + NPM proxy): +```bash +ansible-playbook playbooks/secure-docker-server-firewall.yml +``` + +**If you need external access to specific ports**: +```bash +ansible-playbook playbooks/secure-docker-server-firewall.yml \ + -e "firewall_mode=selective" \ + -e "external_ports=8080,9000" +``` + +### Step 3: Verify everything works + +```bash +# Check firewall status +ansible docker -m shell -a "ufw status verbose" -b + +# Check Docker containers still running +ansible docker -m shell -a "docker ps" -b + +# Test SSH access +ssh dlxadmin@192.168.200.200 + +# Test internal network access (from another internal server) +curl http://192.168.200.200:8080 + +# Test services work through NPM proxy (if configured) +curl http://your-service.directlx.dev +``` + +### Step 4: Make adjustments if needed + +```bash +# View current rules +ansible docker -m shell -a "ufw status numbered" -b + +# Delete a rule +ansible docker -m shell -a "ufw delete " -b + +# Add a new rule +ansible docker -m shell -a "ufw allow from 192.168.200.0/24 to any port 8000" -b +``` + +## Rollback Plan + +If something goes wrong: + +```bash +# Disable firewall temporarily +ansible docker -m ufw -a "state=disabled" -b + +# Reset firewall completely +ansible docker -m ufw -a "state=reset" -b + +# Re-enable with just SSH +ansible docker -m ufw -a "rule=allow port=22 proto=tcp" -b +ansible docker -m ufw -a "state=enabled" -b +``` + +## Monitoring After Configuration + +```bash +# Check blocked connections +ansible docker -m shell -a "grep UFW /var/log/syslog | tail -20" -b + +# Monitor active connections +ansible docker -m shell -a "ss -tnp" -b + +# View firewall logs +ansible docker -m shell -a "journalctl -u ufw --since '10 minutes ago'" -b +``` + +## Next Steps + +1. **Review this document** carefully +2. **Identify which Docker services need external access** (if any) +3. **Choose firewall mode** (internal recommended) +4. **Test in check mode** first +5. **Execute the playbook** +6. **Verify services** still work +7. **Document any port exceptions** you added + +## Files + +- Playbook: `playbooks/secure-docker-server-firewall.yml` +- This guide: `docs/DOCKER-SERVER-SECURITY.md` +- Security audit: `docs/SECURITY-AUDIT-SUMMARY.md` + +--- + +**Status**: Ready for execution when you decide +**Priority**: High (server currently has no firewall) +**Risk**: Medium (breaking services if not configured correctly) +**Recommendation**: Execute during maintenance window with console access available diff --git a/docs/JENKINS-CONNECTIVITY-FIX.md b/docs/JENKINS-CONNECTIVITY-FIX.md new file mode 100644 index 0000000..f8e23c3 --- /dev/null +++ b/docs/JENKINS-CONNECTIVITY-FIX.md @@ -0,0 +1,126 @@ +# Jenkins Server Connectivity Fix + +**Date**: 2026-02-09 +**Server**: jenkins (192.168.200.91) +**Issue**: Ports blocked by firewall, SonarQube containers stopped + +## Problem Summary + +The jenkins server had two critical issues: + +1. **Firewall Blocking Ports**: UFW was configured with default settings, only allowing SSH (port 22) + - Jenkins running on port 8080 was blocked + - SonarQube on port 9000 was blocked + +2. **SonarQube Containers Stopped**: Both containers had been down for 5 months + - `sonarqube` container: Exited (137) + - `postgresql` container: Exited (0) + +## Root Cause + +The jenkins server lacked a `host_vars/jenkins.yml` file, causing it to inherit default firewall settings from the common role that only allowed SSH access. + +## Solution Applied + +### 1. Created Firewall Configuration + +Created `/source/dlx-src/dlx-ansible/host_vars/jenkins.yml`: + +```yaml +--- +# Jenkins server specific variables + +# Allow Jenkins and SonarQube ports through firewall +common_firewall_allowed_ports: + - "22/tcp" # SSH + - "8080/tcp" # Jenkins Web UI + - "9000/tcp" # SonarQube Web UI + - "5432/tcp" # PostgreSQL (SonarQube database) - optional +``` + +### 2. Applied Firewall Rules + +```bash +ansible jenkins -m community.general.ufw -a "rule=allow port=8080 proto=tcp" -b +ansible jenkins -m community.general.ufw -a "rule=allow port=9000 proto=tcp" -b +``` + +### 3. Restarted SonarQube Services + +```bash +ansible jenkins -m shell -a "docker start postgresql" -b +ansible jenkins -m shell -a "docker start sonarqube" -b +``` + +## Verification + +### Firewall Status +``` +Status: active + +To Action From +-- ------ ---- +22/tcp ALLOW IN Anywhere +8080/tcp ALLOW IN Anywhere +9000/tcp ALLOW IN Anywhere +``` + +### Running Containers +``` +CONTAINER ID IMAGE STATUS PORTS +97c85a325ed9 sonarqube:community Up 6 seconds 0.0.0.0:9000->9000/tcp +29fe0ededb3e postgres:15 Up 14 seconds 5432/tcp +``` + +### Listening Ports +``` +Port 8080: Jenkins (Java process) +Port 9000: SonarQube (Docker container) +Port 5432: PostgreSQL (internal Docker networking) +``` + +## Access URLs + +- **Jenkins**: http://192.168.200.91:8080 +- **SonarQube**: http://192.168.200.91:9000 + +## Future Maintenance + +### Check Container Status +```bash +ansible jenkins -m shell -a "docker ps -a" -b +``` + +### Restart SonarQube +```bash +ansible jenkins -m shell -a "docker restart postgresql sonarqube" -b +``` + +### View Logs +```bash +# SonarQube logs +ansible jenkins -m shell -a "docker logs sonarqube --tail 100" -b + +# PostgreSQL logs +ansible jenkins -m shell -a "docker logs postgresql --tail 100" -b +``` + +### Apply Firewall Configuration via Ansible +```bash +# Apply common role with updated host_vars +ansible-playbook playbooks/site.yml -l jenkins -t firewall +``` + +## Notes + +- PostgreSQL container only exposes port 5432 internally to Docker network (not 0.0.0.0), which is the correct configuration +- SonarQube takes 30-60 seconds to fully start up after container starts +- Jenkins is running as a system service (Java process), not in Docker +- Future updates to firewall rules should be made in `host_vars/jenkins.yml` and applied via the common role + +## Related Files + +- Host variables: `host_vars/jenkins.yml` +- Inventory: `inventory/hosts.yml` (jenkins @ 192.168.200.91) +- Common role: `roles/common/tasks/security.yml` +- Playbook (WIP): `playbooks/fix-jenkins-connectivity.yml` diff --git a/docs/JENKINS-NPM-PROXY-QUICK-REFERENCE.md b/docs/JENKINS-NPM-PROXY-QUICK-REFERENCE.md new file mode 100644 index 0000000..b45b2ff --- /dev/null +++ b/docs/JENKINS-NPM-PROXY-QUICK-REFERENCE.md @@ -0,0 +1,149 @@ +# Jenkins NPM Proxy - Quick Reference + +**Date**: 2026-02-09 +**Status**: ✅ Firewall configured, NPM stream setup required + +## Current Configuration + +### Infrastructure +- **NPM Server**: 192.168.200.71 (Nginx Proxy Manager) +- **Jenkins Server**: 192.168.200.91 (dlx-sonar) +- **Proxy Port**: 2222 (NPM → Jenkins:22) + +### What's Done +✅ Jenkins SSH key created: `/var/lib/jenkins/.ssh/id_rsa` +✅ Public key added to jenkins server: `~/.ssh/authorized_keys` +✅ NPM firewall configured: Port 2222 open +✅ Host vars updated: `host_vars/npm.yml` +✅ Documentation created + +### What's Remaining +⏳ NPM stream configuration (requires NPM Web UI) +⏳ Jenkins agent configuration update +⏳ Testing and verification + +## Quick Commands + +### Test SSH Through NPM +```bash +# After configuring NPM stream +ssh -p 2222 dlxadmin@192.168.200.71 +``` + +### Test as Jenkins User +```bash +ansible jenkins -m shell -a "sudo -u jenkins ssh -p 2222 -o StrictHostKeyChecking=no -i /var/lib/jenkins/.ssh/id_rsa dlxadmin@192.168.200.71 hostname" -b +``` + +### Check NPM Firewall +```bash +ansible npm -m shell -a "ufw status | grep 2222" -b +``` + +### View Jenkins SSH Key +```bash +# Public key +ansible jenkins -m shell -a "cat /var/lib/jenkins/.ssh/id_rsa.pub" -b + +# Private key (for Jenkins credential) +ansible jenkins -m shell -a "cat /var/lib/jenkins/.ssh/id_rsa" -b +``` + +## NPM Stream Configuration + +**Required Settings**: +- Incoming Port: `2222` +- Forwarding Host: `192.168.200.91` +- Forwarding Port: `22` +- TCP Forwarding: `Enabled` +- UDP Forwarding: `Disabled` + +**Access NPM UI**: +- URL: http://192.168.200.71:81 +- Default: admin@example.com / changeme +- Go to: **Streams** → **Add Stream** + +## Jenkins Agent Configuration + +**Update in Jenkins UI** (http://192.168.200.91:8080): +- Path: **Manage Jenkins** → **Manage Nodes and Clouds** → Select agent → **Configure** +- Change **Host**: `192.168.200.71` (NPM server) +- Change **Port**: `2222` +- Keep **Credentials**: `dlx-key` + +## Troubleshooting + +### Cannot connect to NPM:2222 +```bash +# Check firewall +ansible npm -m shell -a "ufw status | grep 2222" -b + +# Check if stream is configured +# Login to NPM UI and verify stream exists and is enabled +``` + +### Authentication fails +```bash +# Verify public key is authorized +ansible jenkins -m shell -a "grep jenkins /home/dlxadmin/.ssh/authorized_keys" -b +``` + +### Connection timeout +```bash +# Check NPM can reach Jenkins +ansible npm -m shell -a "nc -zv 192.168.200.91 22" -b +``` + +## Files + +- **Documentation**: `docs/NPM-SSH-PROXY-FOR-JENKINS.md` +- **Quick Reference**: `docs/JENKINS-NPM-PROXY-QUICK-REFERENCE.md` +- **Setup Instructions**: `/tmp/npm-stream-setup.txt` +- **NPM Host Vars**: `host_vars/npm.yml` +- **Jenkins Host Vars**: `host_vars/jenkins.yml` +- **Playbook**: `playbooks/configure-npm-ssh-proxy.yml` + +## Architecture Diagram + +``` +Before: + Jenkins Agent → Router:22 → Jenkins:22 + +After (with NPM proxy): + Jenkins Agent → NPM:2222 → Jenkins:22 + ↓ + Centralized logging + Access control + SSL/TLS support +``` + +## Benefits + +✅ **Security**: Centralized access point through NPM +✅ **Logging**: All SSH connections logged by NPM +✅ **Flexibility**: Easy to add more agents on different ports +✅ **SSL Support**: Can add SSL/TLS for encrypted tunneling +✅ **Monitoring**: NPM provides connection statistics + +## Next Steps After Setup + +1. ✅ Complete NPM stream configuration +2. ✅ Update Jenkins agent settings +3. ✅ Test connection +4. ⏳ Update router port forwarding (if external access needed) +5. ⏳ Restrict Jenkins SSH to NPM only (optional security hardening) +6. ⏳ Set up monitoring/alerts for connection failures + +## Advanced: Restrict SSH to NPM Only + +For additional security, restrict Jenkins SSH to only accept from NPM: + +```bash +# Allow SSH only from NPM +ansible jenkins -m community.general.ufw -a "rule=allow from=192.168.200.71 to=any port=22 proto=tcp" -b + +# Remove general SSH rule (if you want strict restriction) +# ansible jenkins -m community.general.ufw -a "rule=delete port=22 proto=tcp" -b +``` + +⚠️ **Warning**: Only do this after confirming NPM proxy works, or you might lock yourself out! diff --git a/docs/JENKINS-SSH-AGENT-TROUBLESHOOTING.md b/docs/JENKINS-SSH-AGENT-TROUBLESHOOTING.md new file mode 100644 index 0000000..543bf3a --- /dev/null +++ b/docs/JENKINS-SSH-AGENT-TROUBLESHOOTING.md @@ -0,0 +1,232 @@ +# Jenkins SSH Agent Authentication Troubleshooting + +**Date**: 2026-02-09 +**Issue**: Jenkins cannot authenticate to remote build agent +**Error**: `Authentication failed` when connecting to remote SSH agent + +## Problem Description + +Jenkins is configured to connect to a remote build agent via SSH but authentication fails: + +``` +SSHLauncher{host='45.16.76.42', port=22, credentialsId='dlx-key', ...} +[SSH] Opening SSH connection to 45.16.76.42:22. +[SSH] Authentication failed. +``` + +## Root Cause + +The SSH public key associated with Jenkins's 'dlx-key' credential is not present in the `~/.ssh/authorized_keys` file on the remote agent server (45.16.76.42). + +## Quick Diagnosis + +From jenkins server: +```bash +# Test network connectivity +ping -c 2 45.16.76.42 + +# Test SSH connectivity (should fail with "Permission denied (publickey)") +ssh dlxadmin@45.16.76.42 +``` + +## Solution Options + +### Option 1: Add Jenkins Key to Remote Agent (Quickest) + +**Step 1** - Get Jenkins's public key from Web UI: +1. Open Jenkins: http://192.168.200.91:8080 +2. Go to: **Manage Jenkins** → **Credentials** → **System** → **Global credentials (unrestricted)** +3. Click on the **'dlx-key'** credential +4. Look for the public key display (if available) +5. Copy the public key + +**Step 2** - Add to remote agent: +```bash +# SSH to the remote agent +ssh dlxadmin@45.16.76.42 + +# Add the Jenkins public key +echo "ssh-rsa AAAA... jenkins@host" >> ~/.ssh/authorized_keys +chmod 600 ~/.ssh/authorized_keys + +# Verify authorized_keys format +cat ~/.ssh/authorized_keys +``` + +**Step 3** - Test connection from Jenkins server: +```bash +# SSH to jenkins server +ssh dlxadmin@192.168.200.91 + +# Test connection as jenkins user +sudo -u jenkins ssh -o StrictHostKeyChecking=no dlxadmin@45.16.76.42 'echo "Success!"' +``` + +### Option 2: Create New SSH Key for Jenkins (Most Reliable) + +**Step 1** - Run the Ansible playbook: +```bash +ansible-playbook playbooks/setup-jenkins-agent-ssh.yml -e "agent_host=45.16.76.42" +``` + +This will: +- Create SSH key pair for jenkins user at `/var/lib/jenkins/.ssh/id_rsa` +- Display the public key +- Create helper script to copy key to agent + +**Step 2** - Copy key to agent (choose one method): + +**Method A - Automatic** (if you have SSH access): +```bash +ssh dlxadmin@192.168.200.91 +/tmp/copy-jenkins-key-to-agent.sh +``` + +**Method B - Manual**: +```bash +# Get public key from jenkins server +ssh dlxadmin@192.168.200.91 'sudo cat /var/lib/jenkins/.ssh/id_rsa.pub' + +# Add to agent's authorized_keys +ssh dlxadmin@45.16.76.42 +echo "" >> ~/.ssh/authorized_keys +chmod 600 ~/.ssh/authorized_keys +``` + +**Step 3** - Update Jenkins credential: +1. Go to: http://192.168.200.91:8080/manage/credentials/ +2. Click on **'dlx-key'** credential (or create new one) +3. Click **Update** +4. Under "Private Key": + - Select **Enter directly** + - Copy content from: `/var/lib/jenkins/.ssh/id_rsa` on jenkins server +5. Save + +**Step 4** - Test Jenkins agent connection: +1. Go to: http://192.168.200.91:8080/computer/ +2. Find the agent that uses 45.16.76.42 +3. Click **Launch agent** or **Relaunch agent** +4. Check logs for successful connection + +### Option 3: Use Existing dlxadmin Key + +If dlxadmin user already has SSH access to the agent: + +**Step 1** - Copy dlxadmin's key to jenkins user: +```bash +ssh dlxadmin@192.168.200.91 + +# Copy key to jenkins user +sudo cp ~/.ssh/id_ed25519 /var/lib/jenkins/.ssh/ +sudo cp ~/.ssh/id_ed25519.pub /var/lib/jenkins/.ssh/ +sudo chown jenkins:jenkins /var/lib/jenkins/.ssh/id_ed25519* +sudo chmod 600 /var/lib/jenkins/.ssh/id_ed25519 +``` + +**Step 2** - Update Jenkins credential with this key + +## Verification Steps + +### 1. Test SSH Connection from Jenkins Server +```bash +# SSH to jenkins server +ssh dlxadmin@192.168.200.91 + +# Test as jenkins user +sudo -u jenkins ssh -o StrictHostKeyChecking=no dlxadmin@45.16.76.42 'hostname' +``` + +Expected output: The hostname of the remote agent + +### 2. Check Agent in Jenkins +```bash +# Via Jenkins Web UI +http://192.168.200.91:8080/computer/ + +# Look for the agent, should show "Connected" or agent should successfully launch +``` + +### 3. Verify authorized_keys on Remote Agent +```bash +ssh dlxadmin@45.16.76.42 +cat ~/.ssh/authorized_keys | grep jenkins +``` + +Expected: Should show one or more Jenkins public keys + +## Common Issues + +### Issue: "Host key verification failed" +**Solution**: Add host to jenkins user's known_hosts: +```bash +sudo -u jenkins ssh-keyscan -H 45.16.76.42 >> /var/lib/jenkins/.ssh/known_hosts +``` + +### Issue: "Permission denied" even with correct key +**Causes**: +1. Wrong username (check if it should be 'dlxadmin', 'jenkins', 'ubuntu', etc.) +2. Wrong permissions on authorized_keys: + ```bash + chmod 700 ~/.ssh + chmod 600 ~/.ssh/authorized_keys + ``` +3. SELinux blocking (if applicable): + ```bash + restorecon -R ~/.ssh + ``` + +### Issue: Jenkins shows "dlx-key" but can't edit/view +**Solution**: Credential is encrypted. Either: +- Replace with new credential +- Use Jenkins CLI to export (requires admin token) + +## Alternative: Password Authentication + +If SSH key auth continues to fail, temporarily enable password auth (NOT RECOMMENDED for production): + +```bash +# On remote agent +sudo vim /etc/ssh/sshd_config +# Set: PasswordAuthentication yes +sudo systemctl restart sshd + +# In Jenkins, update credential to use password instead of key +``` + +## Files and Locations + +- **Jenkins Home**: `/var/lib/jenkins/` +- **Jenkins SSH Keys**: `/var/lib/jenkins/.ssh/` +- **Jenkins Credentials**: `/var/lib/jenkins/credentials.xml` (encrypted) +- **Remote Agent User**: `dlxadmin` +- **Remote Agent SSH Config**: `/home/dlxadmin/.ssh/authorized_keys` + +## Related Commands + +```bash +# View Jenkins credential store (encrypted) +sudo cat /var/lib/jenkins/credentials.xml + +# Check jenkins user SSH directory +sudo ls -la /var/lib/jenkins/.ssh/ + +# Test SSH with verbose output +sudo -u jenkins ssh -vvv dlxadmin@45.16.76.42 + +# View SSH daemon logs on agent +journalctl -u ssh -f + +# Check Jenkins logs +sudo tail -f /var/log/jenkins/jenkins.log +``` + +## Summary Checklist + +- [ ] Network connectivity verified (ping works) +- [ ] SSH port 22 is reachable +- [ ] Jenkins user has SSH key pair +- [ ] Jenkins public key is in agent's authorized_keys +- [ ] Permissions correct (700 .ssh, 600 authorized_keys) +- [ ] Jenkins credential 'dlx-key' updated with correct private key +- [ ] Test connection: `sudo -u jenkins ssh dlxadmin@AGENT_IP 'hostname'` +- [ ] Agent launches successfully in Jenkins Web UI diff --git a/docs/NPM-SSH-PROXY-FOR-JENKINS.md b/docs/NPM-SSH-PROXY-FOR-JENKINS.md new file mode 100644 index 0000000..6524939 --- /dev/null +++ b/docs/NPM-SSH-PROXY-FOR-JENKINS.md @@ -0,0 +1,300 @@ +# NPM SSH Proxy for Jenkins Agents + +**Date**: 2026-02-09 +**Purpose**: Use Nginx Proxy Manager to proxy SSH connections to Jenkins agents +**Benefit**: Centralized access control, logging, and SSL termination + +## Architecture + +### Before (Direct SSH) +``` +External → Router:22 → Jenkins:22 +``` +**Issues**: +- Direct SSH exposure +- No centralized logging +- Single point of failure + +### After (NPM Proxy) +``` +External → NPM:2222 → Jenkins:22 +Jenkins Agent Config: Connect to NPM:2222 +``` +**Benefits**: +- ✅ Centralized access through NPM +- ✅ NPM logging and monitoring +- ✅ Easier to manage multiple agents +- ✅ Can add rate limiting +- ✅ SSL/TLS for agent.jar downloads via web UI + +## NPM Configuration + +### Step 1: Create TCP Stream in NPM + +**Via NPM Web UI** (http://192.168.200.71:81): + +1. **Login to NPM** + - URL: http://192.168.200.71:81 + - Default: admin@example.com / changeme + +2. **Navigate to Streams** + - Click **Streams** in the sidebar + - Click **Add Stream** + +3. **Configure Incoming Stream** + - **Incoming Port**: `2222` + - **Forwarding Host**: `192.168.200.91` (jenkins server) + - **Forwarding Port**: `22` + - **TCP Forwarding**: Enabled + - **UDP Forwarding**: Disabled + +4. **Enable SSL/TLS Forwarding** (Optional) + - For encrypted SSH tunneling + - **SSL Certificate**: Upload or use Let's Encrypt + - **Force SSL**: Enabled + +5. **Save** + +### Step 2: Update Firewall on NPM Server + +The NPM server needs to allow incoming connections on port 2222: + +```bash +# Run from ansible control machine +ansible npm -m community.general.ufw -a "rule=allow port=2222 proto=tcp" -b + +# Verify +ansible npm -m shell -a "ufw status | grep 2222" -b +``` + +### Step 3: Update Jenkins Agent Configuration + +**In Jenkins Web UI** (http://192.168.200.91:8080): + +1. **Navigate to Agent** + - Go to: **Manage Jenkins** → **Manage Nodes and Clouds** + - Click on the agent that uses SSH + +2. **Update SSH Host** + - **Host**: Change from `45.16.76.42` to `192.168.200.71` (NPM server) + - **Port**: Change from `22` to `2222` + - **Credentials**: Keep as `dlx-key` + +3. **Advanced Settings** + - **JVM Options**: Add if needed: `-Djava.awt.headless=true` + - **Prefix Start Agent Command**: Leave empty + - **Suffix Start Agent Command**: Leave empty + +4. **Save and Launch Agent** + +### Step 4: Update Router Port Forwarding (Optional) + +If you want external access through the router: + +**Old Rule**: +- External Port: `22` +- Internal IP: `192.168.200.91` (jenkins) +- Internal Port: `22` + +**New Rule**: +- External Port: `2222` (or keep 22 if you prefer) +- Internal IP: `192.168.200.71` (NPM) +- Internal Port: `2222` + +## Testing + +### Test 1: SSH Through NPM from Local Network +```bash +# Test SSH connection through NPM proxy +ssh -p 2222 dlxadmin@192.168.200.71 + +# Should connect to jenkins server +hostname # Should output: dlx-sonar +``` + +### Test 2: Jenkins Agent Connection +```bash +# From jenkins server, test as jenkins user +sudo -u jenkins ssh -p 2222 -i /var/lib/jenkins/.ssh/id_rsa dlxadmin@192.168.200.71 'hostname' + +# Expected output: dlx-sonar +``` + +### Test 3: Launch Agent from Jenkins UI +1. Go to: http://192.168.200.91:8080/computer/ +2. Find the agent +3. Click **Launch agent** +4. Check logs for successful connection + +## NPM Stream Configuration File + +NPM stores stream configurations in its database. For backup/reference: + +```json +{ + "incoming_port": 2222, + "forwarding_host": "192.168.200.91", + "forwarding_port": 22, + "tcp_forwarding": true, + "udp_forwarding": false, + "enabled": true +} +``` + +## Troubleshooting + +### Issue: Cannot connect to NPM:2222 + +**Check NPM firewall**: +```bash +ansible npm -m shell -a "ufw status | grep 2222" -b +ansible npm -m shell -a "ss -tlnp | grep 2222" -b +``` + +**Check NPM stream is active**: +- Login to NPM UI +- Go to Streams +- Verify stream is enabled (green toggle) + +### Issue: Connection timeout + +**Check NPM can reach Jenkins**: +```bash +ansible npm -m shell -a "ping -c 2 192.168.200.91" -b +ansible npm -m shell -a "nc -zv 192.168.200.91 22" -b +``` + +**Check Jenkins SSH is running**: +```bash +ansible jenkins -m shell -a "systemctl status sshd" -b +``` + +### Issue: Authentication fails + +**Verify SSH key**: +```bash +# Get Jenkins public key +ansible jenkins -m shell -a "cat /var/lib/jenkins/.ssh/id_rsa.pub" -b + +# Check it's in authorized_keys +ansible jenkins -m shell -a "grep jenkins /home/dlxadmin/.ssh/authorized_keys" -b +``` + +### Issue: NPM stream not forwarding + +**Check NPM logs**: +```bash +ansible npm -m shell -a "docker logs nginx-proxy-manager --tail 100" -b + +# Look for stream-related errors +``` + +**Restart NPM**: +```bash +ansible npm -m shell -a "docker restart nginx-proxy-manager" -b +``` + +## Advanced: Multiple Jenkins Agents + +For multiple remote agents, create separate streams: + +| Agent | NPM Port | Forward To | Purpose | +|-------|----------|------------|---------| +| jenkins-local | 2222 | 192.168.200.91:22 | Local Jenkins agent | +| build-agent-1 | 2223 | 192.168.200.120:22 | Remote build agent | +| build-agent-2 | 2224 | 192.168.200.121:22 | Remote build agent | + +## Security Considerations + +### Recommended Firewall Rules + +**NPM Server** (192.168.200.71): +```yaml +common_firewall_allowed_ports: + - "22/tcp" # SSH admin access + - "80/tcp" # HTTP + - "443/tcp" # HTTPS + - "81/tcp" # NPM Admin panel + - "2222/tcp" # Jenkins SSH proxy + - "2223/tcp" # Additional agents (if needed) +``` + +**Jenkins Server** (192.168.200.91): +```yaml +common_firewall_allowed_ports: + - "22/tcp" # SSH (restrict to NPM IP only) + - "8080/tcp" # Jenkins Web UI + - "9000/tcp" # SonarQube +``` + +### Restrict SSH Access to NPM Only + +On Jenkins server, restrict SSH to only accept from NPM: + +```bash +# Allow SSH only from NPM server +ansible jenkins -m community.general.ufw -a "rule=allow from=192.168.200.71 to=any port=22 proto=tcp" -b + +# Deny SSH from all others (if not already default) +ansible jenkins -m community.general.ufw -a "rule=deny port=22 proto=tcp" -b +``` + +## Monitoring + +### NPM Access Logs +```bash +# View NPM access logs +ansible npm -m shell -a "docker logs nginx-proxy-manager --tail 50 | grep stream" -b +``` + +### Connection Statistics +```bash +# Check active SSH connections through NPM +ansible npm -m shell -a "ss -tn | grep :2222" -b + +# Check connections on Jenkins +ansible jenkins -m shell -a "ss -tn | grep :22 | grep ESTAB" -b +``` + +## Backup and Recovery + +### Backup NPM Configuration +```bash +# Backup NPM database +ansible npm -m shell -a "docker exec nginx-proxy-manager sqlite3 /data/database.sqlite .dump > /tmp/npm-backup.sql" -b + +# Download backup +ansible npm -m fetch -a "src=/tmp/npm-backup.sql dest=./backups/npm-backup-$(date +%Y%m%d).sql" -b +``` + +### Restore NPM Configuration +```bash +# Upload backup +ansible npm -m copy -a "src=./backups/npm-backup.sql dest=/tmp/npm-restore.sql" -b + +# Restore database +ansible npm -m shell -a "docker exec nginx-proxy-manager sqlite3 /data/database.sqlite < /tmp/npm-restore.sql" -b + +# Restart NPM +ansible npm -m shell -a "docker restart nginx-proxy-manager" -b +``` + +## Migration Checklist + +- [ ] Create TCP stream in NPM (port 2222 → jenkins:22) +- [ ] Update NPM firewall to allow port 2222 +- [ ] Test SSH connection through NPM proxy +- [ ] Update Jenkins agent SSH host to NPM IP +- [ ] Update Jenkins agent SSH port to 2222 +- [ ] Test agent connection in Jenkins UI +- [ ] Update router port forwarding (if external access needed) +- [ ] Restrict Jenkins SSH to NPM IP only (optional but recommended) +- [ ] Document new configuration +- [ ] Update monitoring/alerting rules + +## Related Files + +- NPM host vars: `host_vars/npm.yml` +- Jenkins host vars: `host_vars/jenkins.yml` +- NPM firewall playbook: `playbooks/configure-npm-firewall.yml` (to be created) +- This documentation: `docs/NPM-SSH-PROXY-FOR-JENKINS.md` diff --git a/docs/SECURITY-AUDIT-SUMMARY.md b/docs/SECURITY-AUDIT-SUMMARY.md new file mode 100644 index 0000000..5b211eb --- /dev/null +++ b/docs/SECURITY-AUDIT-SUMMARY.md @@ -0,0 +1,230 @@ +# Security Audit Summary + +**Date**: 2026-02-09 +**Servers Audited**: 16 +**Full Report**: `/tmp/security-audit-full-report.txt` + +## Executive Summary + +Security audit completed across all infrastructure servers. Multiple security concerns identified ranging from **CRITICAL** to **LOW** priority. + +## Critical Security Findings + +### 🔴 CRITICAL + +1. **Root Login Enabled via SSH** (`ansible-node`, `gitea`) + - **Risk**: Direct root access increases attack surface + - **Affected**: 2 servers + - **Recommendation**: Disable root login immediately + ```yaml + PermitRootLogin no + ``` + +2. **No Firewall on Multiple Servers** + - **Risk**: All ports exposed to network + - **Affected**: `ansible-node`, `gitea`, and others + - **Recommendation**: Enable UFW with strict rules + +3. **Password Authentication Enabled on Jenkins** + - **Risk**: We enabled this for temporary AWS access + - **Status**: Known configuration (for AWS Jenkins Master) + - **Recommendation**: Switch to key-based auth when possible + +### 🟠 HIGH + +4. **Automatic Updates Not Configured** + - **Risk**: Servers missing security patches + - **Affected**: `ansible-node`, `docker`, and most servers + - **Recommendation**: Enable unattended-upgrades + +5. **Security Updates Available** + - **Critical**: `docker` has **65 pending security updates** + - **Recommendation**: Apply immediately + ```bash + ansible docker -m apt -a "upgrade=dist update_cache=yes" -b + ``` + +6. **Multiple Services Exposed on Docker Server** + - **Risk**: Ports 5000, 8000-8082, 8443, 9000, 11434 publicly accessible + - **Firewall**: Currently disabled + - **Recommendation**: Enable firewall, restrict to internal network + +### 🟡 MEDIUM + +7. **Password-Based Users on Multiple Servers** + - **Users with passwords**: root, dlxadmin, directlx, jenkins + - **Risk**: Potential brute-force targets + - **Recommendation**: Enforce strong password policies + +8. **PermitRootLogin Enabled** + - **Affected**: Several Proxmox nodes + - **Risk**: Root SSH access possible + - **Recommendation**: Disable after confirming Proxmox compatibility + +## Server-Specific Findings + +### ansible-node (192.168.200.106) +- ✅ Password auth: Disabled +- ❌ Root login: **ENABLED** +- ❌ Firewall: **NOT CONFIGURED** +- ❌ Auto-updates: **NOT CONFIGURED** +- Services: nginx (80, 443), MySQL (3306), Webmin (12321) + +### docker (192.168.200.200) +- ✅ Root login: Disabled +- ❌ Firewall: **INACTIVE** +- ❌ Auto-updates: **NOT CONFIGURED** +- ⚠️ Security updates: **65 PENDING** +- Services: Many Docker containers on multiple ports + +### jenkins (192.168.200.91) +- ✅ Firewall: Active (ports 22, 8080, 9000, 2222) +- ⚠️ Password auth: **ENABLED** (intentional for AWS) +- ⚠️ Keyboard-interactive: **ENABLED** (intentional) +- Services: Jenkins (8080), SonarQube (9000) + +### npm (192.168.200.71) +- ✅ Firewall: Active (ports 22, 80, 443, 81, 2222) +- ✅ Password auth: Disabled +- Services: Nginx Proxy Manager, OpenResty + +### hiveops, smartjournal, odoo +- ⚠️ Firewall: **DISABLED** (intentional for Docker networking) +- ❌ Auto-updates: **NOT CONFIGURED** +- Multiple Docker services running + +### Proxmox Nodes (proxmox-00, 01, 02) +- ✅ Firewall: Active +- ⚠️ Root login: Enabled (may be required for Proxmox) +- Services: Proxmox web interface + +## Immediate Actions Required + +### Priority 1 (Critical - Do Now) + +1. **Disable Root SSH Login** + ```bash + ansible all -m lineinfile -a "path=/etc/ssh/sshd_config regexp='^PermitRootLogin' line='PermitRootLogin no'" -b + ansible all -m service -a "name=sshd state=restarted" -b + ``` + +2. **Apply Security Updates on Docker Server** + ```bash + ansible docker -m apt -a "upgrade=dist update_cache=yes" -b + ``` + +3. **Enable Firewall on Critical Servers** + ```bash + # For servers without firewall + ansible ansible-node,gitea -m apt -a "name=ufw state=present" -b + ansible ansible-node,gitea -m ufw -a "rule=allow port=22 proto=tcp" -b + ansible ansible-node,gitea -m ufw -a "state=enabled" -b + ``` + +### Priority 2 (High - This Week) + +4. **Enable Automatic Security Updates** + ```bash + ansible all -m apt -a "name=unattended-upgrades state=present" -b + ansible all -m copy -a "dest=/etc/apt/apt.conf.d/20auto-upgrades content='APT::Periodic::Update-Package-Lists \"1\";\nAPT::Periodic::Unattended-Upgrade \"1\";' mode=0644" -b + ``` + +5. **Configure Firewall for Docker Server** + ```bash + ansible docker -m ufw -a "rule=allow port={{ item }} proto=tcp" -b + # Add specific ports needed for services + ``` + +6. **Review and Secure Open Ports** + - Audit what services need external access + - Close unnecessary ports + - Use NPM proxy for web services + +### Priority 3 (Medium - This Month) + +7. **Implement Password Policy** + ```yaml + # In /etc/login.defs + PASS_MAX_DAYS 90 + PASS_MIN_DAYS 1 + PASS_MIN_LEN 12 + PASS_WARN_AGE 7 + ``` + +8. **Enable Fail2Ban** + ```bash + ansible all -m apt -a "name=fail2ban state=present" -b + ``` + +9. **Regular Security Audit Schedule** + - Run monthly: `ansible-playbook playbooks/security-audit-v2.yml` + - Review findings + - Track improvements + +## Positive Security Practices Found + +✅ **Jenkins Server**: Well-configured firewall with specific ports +✅ **NPM Server**: Good firewall configuration, SSL certificates managed +✅ **Most Servers**: Password SSH auth disabled (key-only) +✅ **Most Servers**: Root login restricted +✅ **Proxmox Nodes**: Firewalls active + +## Recommended Playbooks + +### security-hardening.yml (To Be Created) +```yaml +- Enable automatic security updates +- Disable root SSH login (except where needed) +- Configure UFW on all servers +- Install fail2ban +- Set password policies +- Remove world-writable files +``` + +### security-monitoring.yml (To Be Created) +```yaml +- Monitor failed login attempts +- Alert on unauthorized access +- Track open ports +- Monitor security updates +``` + +## Compliance Checklist + +- [ ] All servers have firewall enabled +- [ ] Root SSH login disabled (except Proxmox) +- [ ] Password authentication disabled (except where needed) +- [ ] Automatic updates enabled +- [ ] No pending critical security updates +- [ ] Strong password policies enforced +- [ ] Fail2Ban installed and configured +- [ ] Regular security audits scheduled +- [ ] SSH keys rotated (90 days) +- [ ] Unnecessary services disabled + +## Next Steps + +1. **Review this report** with stakeholders +2. **Execute Priority 1 actions** immediately +3. **Schedule Priority 2 actions** for this week +4. **Create remediation playbooks** for automation +5. **Establish monthly security audit** routine +6. **Document exceptions** (e.g., Jenkins password auth for AWS) + +## Resources + +- Full audit report: `/tmp/security-audit-full-report.txt` +- Individual reports: `/tmp/security-audit-*/report.txt` +- Audit playbook: `playbooks/security-audit-v2.yml` + +## Notes + +- Jenkins password auth is intentional for AWS Jenkins Master connection +- Firewall disabled on hiveops/smartjournal/odoo due to Docker networking requirements +- Proxmox root login may be required for management interface + +--- + +**Generated**: 2026-02-09 +**Auditor**: Ansible Security Audit v2 +**Next Audit**: 2026-03-09 (monthly) diff --git a/host_vars/jenkins.yml b/host_vars/jenkins.yml new file mode 100644 index 0000000..5202eb6 --- /dev/null +++ b/host_vars/jenkins.yml @@ -0,0 +1,9 @@ +--- +# Jenkins server specific variables + +# Allow Jenkins and SonarQube ports through firewall +common_firewall_allowed_ports: + - "22/tcp" # SSH + - "8080/tcp" # Jenkins Web UI + - "9000/tcp" # SonarQube Web UI + - "5432/tcp" # PostgreSQL (SonarQube database) - optional, only if external access needed diff --git a/host_vars/npm.yml b/host_vars/npm.yml index ebb8626..c2713a6 100644 --- a/host_vars/npm.yml +++ b/host_vars/npm.yml @@ -6,3 +6,11 @@ common_firewall_allowed_ports: - "80/tcp" # HTTP - "443/tcp" # HTTPS - "81/tcp" # NPM Admin panel + - "2222/tcp" # Jenkins SSH proxy (TCP stream) +# BEGIN ANSIBLE MANAGED BLOCK - Jenkins SSH Proxy +# Jenkins SSH proxy port (TCP stream forwarding) +# Stream configuration must be created in NPM UI: +# Incoming Port: 2222 +# Forwarding Host: 192.168.200.91 +# Forwarding Port: 22 +# END ANSIBLE MANAGED BLOCK - Jenkins SSH Proxy diff --git a/playbooks/configure-npm-ssh-proxy.yml b/playbooks/configure-npm-ssh-proxy.yml new file mode 100644 index 0000000..36d53ff --- /dev/null +++ b/playbooks/configure-npm-ssh-proxy.yml @@ -0,0 +1,116 @@ +--- +- name: Configure NPM firewall for Jenkins SSH proxy + hosts: npm + become: true + gather_facts: true + + vars: + jenkins_ssh_proxy_port: 2222 + + tasks: + - name: Display current NPM firewall status + ansible.builtin.shell: ufw status numbered + register: ufw_before + changed_when: false + + - name: Show current firewall rules + ansible.builtin.debug: + msg: "{{ ufw_before.stdout_lines }}" + + - name: Allow Jenkins SSH proxy port + community.general.ufw: + rule: allow + port: "{{ jenkins_ssh_proxy_port }}" + proto: tcp + comment: "Jenkins SSH proxy" + + - name: Display updated firewall status + ansible.builtin.shell: ufw status numbered + register: ufw_after + changed_when: false + + - name: Show updated firewall rules + ansible.builtin.debug: + msg: "{{ ufw_after.stdout_lines }}" + + - name: Update NPM host_vars file + ansible.builtin.blockinfile: + path: "{{ playbook_dir }}/../host_vars/npm.yml" + marker: "# {mark} ANSIBLE MANAGED BLOCK - Jenkins SSH Proxy" + block: | + # Jenkins SSH proxy port (TCP stream forwarding) + # Stream configuration must be created in NPM UI: + # Incoming Port: {{ jenkins_ssh_proxy_port }} + # Forwarding Host: 192.168.200.91 + # Forwarding Port: 22 + create: false + delegate_to: localhost + become: false + + - name: Check if NPM container is running + ansible.builtin.shell: docker ps --filter "name=nginx" --format "{{ '{{.Names}}' }}" + register: npm_containers + changed_when: false + + - name: Display NPM containers + ansible.builtin.debug: + msg: "{{ npm_containers.stdout_lines }}" + + - name: Instructions for NPM UI configuration + ansible.builtin.debug: + msg: + - "===== NPM Configuration Required =====" + - "" + - "Firewall configured successfully! Port {{ jenkins_ssh_proxy_port }} is now open." + - "" + - "Next steps - Configure NPM Stream:" + - "" + - "1. Login to NPM Web UI:" + - " URL: http://192.168.200.71:81" + - " Default: admin@example.com / changeme" + - "" + - "2. Create TCP Stream:" + - " - Click 'Streams' in sidebar" + - " - Click 'Add Stream'" + - " - Incoming Port: {{ jenkins_ssh_proxy_port }}" + - " - Forwarding Host: 192.168.200.91" + - " - Forwarding Port: 22" + - " - TCP Forwarding: Enabled" + - " - UDP Forwarding: Disabled" + - " - Click 'Save'" + - "" + - "3. Test the proxy:" + - " ssh -p {{ jenkins_ssh_proxy_port }} dlxadmin@192.168.200.71" + - " (Should connect to jenkins server)" + - "" + - "4. Update Jenkins agent configuration:" + - " - Go to: http://192.168.200.91:8080/computer/" + - " - Click on the agent" + - " - Click 'Configure'" + - " - Change Host: 192.168.200.71" + - " - Change Port: {{ jenkins_ssh_proxy_port }}" + - " - Save and launch agent" + - "" + - "Documentation: docs/NPM-SSH-PROXY-FOR-JENKINS.md" + +- name: Test Jenkins SSH connectivity through NPM (manual verification) + hosts: localhost + gather_facts: false + + tasks: + - name: Test instructions + ansible.builtin.debug: + msg: + - "" + - "===== Testing Checklist =====" + - "" + - "After configuring NPM stream, run these tests:" + - "" + - "Test 1 - SSH through NPM:" + - " ssh -p 2222 dlxadmin@192.168.200.71" + - "" + - "Test 2 - Jenkins user SSH:" + - " ansible jenkins -m shell -a 'sudo -u jenkins ssh -p 2222 -o StrictHostKeyChecking=no -i /var/lib/jenkins/.ssh/id_rsa dlxadmin@192.168.200.71 hostname' -b" + - "" + - "Test 3 - Launch agent in Jenkins UI:" + - " http://192.168.200.91:8080/computer/" diff --git a/playbooks/fix-jenkins-connectivity.yml b/playbooks/fix-jenkins-connectivity.yml new file mode 100644 index 0000000..945d6bd --- /dev/null +++ b/playbooks/fix-jenkins-connectivity.yml @@ -0,0 +1,106 @@ +--- +- name: Fix Jenkins and SonarQube connectivity issues + hosts: jenkins + become: true + gather_facts: true + + tasks: + - name: Display current firewall status + ansible.builtin.shell: ufw status verbose + register: ufw_before + changed_when: false + + - name: Show current firewall rules + ansible.builtin.debug: + msg: "{{ ufw_before.stdout_lines }}" + + - name: Apply common role to configure firewall + ansible.builtin.include_role: + name: common + tasks_from: security.yml + + - name: Display updated firewall status + ansible.builtin.shell: ufw status verbose + register: ufw_after + changed_when: false + + - name: Show updated firewall rules + ansible.builtin.debug: + msg: "{{ ufw_after.stdout_lines }}" + + - name: Check if SonarQube containers exist + ansible.builtin.shell: docker ps -a --filter "name=sonarqube" --format "{{.Names}}" + register: sonarqube_containers + changed_when: false + + - name: Start PostgreSQL container for SonarQube + community.docker.docker_container: + name: postgresql + state: started + when: "'postgresql' in sonarqube_containers.stdout" + register: postgres_start + + - name: Wait for PostgreSQL to be ready + ansible.builtin.pause: + seconds: 10 + when: postgres_start.changed + + - name: Start SonarQube container + community.docker.docker_container: + name: sonarqube + state: started + when: "'sonarqube' in sonarqube_containers.stdout" + + - name: Wait for services to start + ansible.builtin.pause: + seconds: 30 + when: postgres_start.changed + + - name: Check Jenkins service status + ansible.builtin.shell: ps aux | grep -i jenkins | grep -v grep + register: jenkins_status + changed_when: false + failed_when: false + + - name: Display Jenkins status + ansible.builtin.debug: + msg: "Jenkins process: {{ 'RUNNING' if jenkins_status.rc == 0 else 'NOT FOUND' }}" + + - name: Check listening ports + ansible.builtin.shell: ss -tlnp | grep -E ':(8080|9000|5432)' + register: listening_ports + changed_when: false + failed_when: false + + - name: Display listening ports + ansible.builtin.debug: + msg: "{{ listening_ports.stdout_lines }}" + + - name: Test Jenkins connectivity from localhost + ansible.builtin.uri: + url: "http://localhost:8080" + status_code: [200, 403] + timeout: 10 + register: jenkins_test + failed_when: false + + - name: Display Jenkins connectivity test result + ansible.builtin.debug: + msg: "Jenkins HTTP status: {{ jenkins_test.status | default('FAILED') }}" + + - name: Summary + ansible.builtin.debug: + msg: + - "===== Fix Summary =====" + - "Firewall: Updated to allow ports 22, 8080, 9000, 5432" + - "Jenkins: {{ 'Running on port 8080' if jenkins_status.rc == 0 else 'NOT RUNNING' }}" + - "SonarQube: {{ 'Started' if postgres_start.changed else 'Already running or not found' }}" + - "" + - "Access URLs:" + - " Jenkins: http://192.168.200.91:8080" + - " SonarQube: http://192.168.200.91:9000" + - "" + - "Next steps:" + - " 1. Test access from your browser" + - " 2. Check SonarQube logs: docker logs sonarqube" + - " 3. Verify PostgreSQL: docker logs postgresql" diff --git a/playbooks/secure-docker-server-firewall.yml b/playbooks/secure-docker-server-firewall.yml new file mode 100644 index 0000000..8d7d7e0 --- /dev/null +++ b/playbooks/secure-docker-server-firewall.yml @@ -0,0 +1,146 @@ +--- +# Docker Server Firewall Configuration +# Status: READY FOR EXECUTION +# Created: 2026-02-09 +# +# IMPORTANT: Review and customize the firewall_allowed_ports variable +# based on which Docker services need external access +# +# Usage: +# Option A - Internal Only (Most Secure): +# ansible-playbook playbooks/secure-docker-server-firewall.yml -e "firewall_mode=internal" +# +# Option B - Selective Access: +# ansible-playbook playbooks/secure-docker-server-firewall.yml -e "firewall_mode=selective" -e "external_ports=8080,9000" +# +# Option C - Review Current State: +# ansible-playbook playbooks/secure-docker-server-firewall.yml --check + +- name: Configure Firewall on Docker Server + hosts: docker + become: true + gather_facts: true + + vars: + # Default mode: internal (most secure) + firewall_mode: "{{ firewall_mode | default('internal') }}" + + # Ports that are always allowed + essential_ports: + - "22/tcp" # SSH + + # Docker service ports (customize based on your needs) + docker_service_ports: + - "5000/tcp" # Docker service + - "8000/tcp" # Docker service + - "8001/tcp" # Docker service + - "8080/tcp" # Docker service + - "8081/tcp" # Docker service + - "8082/tcp" # Docker service + - "8443/tcp" # Docker service (HTTPS) + - "9000/tcp" # Docker service (Portainer/SonarQube?) + - "11434/tcp" # Docker service (Ollama?) + + # Internal network subnet + internal_subnet: "192.168.200.0/24" + + tasks: + - name: Display current configuration mode + ansible.builtin.debug: + msg: | + ╔════════════════════════════════════════════════════════════════╗ + ║ Docker Server Firewall Configuration ║ + ╚════════════════════════════════════════════════════════════════╝ + + Mode: {{ firewall_mode }} + Essential Ports: {{ essential_ports }} + Docker Ports: {{ docker_service_ports | length }} services + Internal Subnet: {{ internal_subnet }} + + - name: Install UFW if not present + ansible.builtin.apt: + name: ufw + state: present + update_cache: yes + + - name: Reset UFW to default (if requested) + community.general.ufw: + state: reset + when: reset_firewall | default(false) | bool + + - name: Set UFW default policies + community.general.ufw: + direction: "{{ item.direction }}" + policy: "{{ item.policy }}" + loop: + - { direction: 'incoming', policy: 'deny' } + - { direction: 'outgoing', policy: 'allow' } + + - name: Allow SSH (essential) + community.general.ufw: + rule: allow + port: "{{ item.split('/')[0] }}" + proto: "{{ item.split('/')[1] }}" + comment: "Essential - SSH access" + loop: "{{ essential_ports }}" + + - name: Allow Docker services from internal network only + community.general.ufw: + rule: allow + port: "{{ item.split('/')[0] }}" + proto: "{{ item.split('/')[1] }}" + from_ip: "{{ internal_subnet }}" + comment: "Docker service - internal only" + loop: "{{ docker_service_ports }}" + when: firewall_mode == 'internal' + + - name: Allow specific Docker services externally (selective mode) + community.general.ufw: + rule: allow + port: "{{ item.split('/')[0] }}" + proto: "{{ item.split('/')[1] }}" + comment: "Docker service - external access" + loop: "{{ external_ports.split(',') }}" + when: + - firewall_mode == 'selective' + - external_ports is defined + + - name: Enable UFW + community.general.ufw: + state: enabled + + - name: Display firewall status + ansible.builtin.shell: ufw status verbose + register: ufw_status + changed_when: false + + - name: Show configured firewall rules + ansible.builtin.debug: + msg: "{{ ufw_status.stdout_lines }}" + + - name: Display open ports + ansible.builtin.shell: ss -tlnp | grep LISTEN + register: open_ports + changed_when: false + + - name: Summary + ansible.builtin.debug: + msg: | + ╔════════════════════════════════════════════════════════════════╗ + ║ Firewall Configuration Complete ║ + ╚════════════════════════════════════════════════════════════════╝ + + Mode: {{ firewall_mode }} + Status: UFW Enabled + + {{ ufw_status.stdout }} + + Next Steps: + 1. Test SSH access: ssh dlxadmin@192.168.200.200 + 2. Test Docker services from internal network + 3. If external access needed, run with firewall_mode=selective + 4. Monitor: sudo ufw status numbered + + To modify rules later: + sudo ufw allow from 192.168.200.0/24 to any port + sudo ufw delete diff --git a/playbooks/security-audit-v2.yml b/playbooks/security-audit-v2.yml new file mode 100644 index 0000000..3ab259d --- /dev/null +++ b/playbooks/security-audit-v2.yml @@ -0,0 +1,149 @@ +--- +- name: Security Audit - Generate Reports + hosts: all:!localhost + become: true + gather_facts: true + + tasks: + - name: Create audit directory + ansible.builtin.file: + path: "/tmp/security-audit-{{ inventory_hostname }}" + state: directory + mode: '0755' + delegate_to: localhost + become: false + + - name: Collect SSH configuration + ansible.builtin.shell: | + sshd -T 2>/dev/null | grep -E '(permit|password|pubkey|port|authentication)' || echo "Unable to check SSH config" + register: ssh_check + changed_when: false + failed_when: false + + - name: Collect firewall status + ansible.builtin.shell: | + if command -v ufw >/dev/null 2>&1; then + ufw status numbered 2>/dev/null || echo "UFW not active" + else + echo "No firewall detected" + fi + register: firewall_check + changed_when: false + + - name: Collect open ports + ansible.builtin.shell: ss -tlnp | grep LISTEN + register: ports_check + changed_when: false + + - name: Collect sudo users + ansible.builtin.shell: getent group sudo 2>/dev/null || getent group wheel 2>/dev/null || echo "No sudo group" + register: sudo_check + changed_when: false + + - name: Collect password authentication users + ansible.builtin.shell: | + awk -F: '($2 != "!" && $2 != "*" && $2 != "") {print $1}' /etc/shadow 2>/dev/null | head -20 || echo "Unable to check" + register: pass_users_check + changed_when: false + failed_when: false + + - name: Collect recent failed logins + ansible.builtin.shell: | + journalctl -u sshd --no-pager -n 50 2>/dev/null | grep -i "failed\|authentication failure" | tail -10 || echo "No recent failures or unable to check" + register: failed_logins_check + changed_when: false + failed_when: false + + - name: Check automatic updates + ansible.builtin.shell: | + if [ -f /etc/apt/apt.conf.d/20auto-upgrades ]; then + echo "Automatic updates: ENABLED" + cat /etc/apt/apt.conf.d/20auto-upgrades + else + echo "Automatic updates: NOT CONFIGURED" + fi + register: auto_updates_check + changed_when: false + + - name: Check for available security updates + ansible.builtin.shell: | + apt-get update -qq 2>&1 | head -5 + apt list --upgradable 2>/dev/null | grep -i security | wc -l || echo "0" + register: security_updates_check + changed_when: false + failed_when: false + + - name: Generate security report + ansible.builtin.copy: + content: | + ╔════════════════════════════════════════════════════════════════╗ + ║ Security Audit Report: {{ inventory_hostname }} + ║ IP: {{ ansible_host }} + ║ Date: {{ ansible_date_time.iso8601 }} + ╚════════════════════════════════════════════════════════════════╝ + + === SYSTEM INFORMATION === + OS: {{ ansible_distribution }} {{ ansible_distribution_version }} + Kernel: {{ ansible_kernel }} + Architecture: {{ ansible_architecture }} + + === SSH CONFIGURATION === + {{ ssh_check.stdout }} + + === FIREWALL STATUS === + {{ firewall_check.stdout }} + + === OPEN NETWORK PORTS === + {{ ports_check.stdout }} + + === SUDO USERS === + {{ sudo_check.stdout }} + + === USERS WITH PASSWORD AUTH === + {{ pass_users_check.stdout }} + + === RECENT FAILED LOGIN ATTEMPTS === + {{ failed_logins_check.stdout }} + + === AUTOMATIC UPDATES === + {{ auto_updates_check.stdout }} + + === AVAILABLE SECURITY UPDATES === + Security updates available: {{ security_updates_check.stdout_lines[-1] | default('Unknown') }} + + dest: "/tmp/security-audit-{{ inventory_hostname }}/report.txt" + mode: '0644' + delegate_to: localhost + become: false + +- name: Generate Summary Report + hosts: localhost + gather_facts: false + + tasks: + - name: Find all audit reports + ansible.builtin.find: + paths: /tmp + patterns: "security-audit-*/report.txt" + recurse: true + register: audit_reports + + - name: Display report locations + ansible.builtin.debug: + msg: | + ╔════════════════════════════════════════════════════════════════╗ + ║ Security Audit Complete ║ + ╚════════════════════════════════════════════════════════════════╝ + + Reports generated for {{ audit_reports.files | length }} servers + + View individual reports: + {% for file in audit_reports.files %} + - {{ file.path }} + {% endfor %} + + View all reports: + cat /tmp/security-audit-*/report.txt + + Create consolidated report: + cat /tmp/security-audit-*/report.txt > /tmp/security-audit-full-report.txt diff --git a/playbooks/security-audit.yml b/playbooks/security-audit.yml new file mode 100644 index 0000000..51e4b3f --- /dev/null +++ b/playbooks/security-audit.yml @@ -0,0 +1,193 @@ +--- +- name: Comprehensive Security Audit + hosts: all + become: true + gather_facts: true + + tasks: + - name: Gather security information + block: + - name: Check SSH configuration + ansible.builtin.shell: | + echo "=== SSH Configuration ===" + sshd -T | grep -E '(permitrootlogin|passwordauthentication|pubkeyauthentication|permitemptypasswords|port)' + register: ssh_config + changed_when: false + + - name: Check for users with empty passwords + ansible.builtin.shell: | + echo "=== Users with Empty Passwords ===" + awk -F: '($2 == "" || $2 == "!") {print $1}' /etc/shadow 2>/dev/null | head -20 || echo "Unable to check (requires root)" + register: empty_passwords + changed_when: false + failed_when: false + + - name: Check sudo users + ansible.builtin.shell: | + echo "=== Sudo Users ===" + getent group sudo 2>/dev/null || getent group wheel 2>/dev/null || echo "No sudo group found" + register: sudo_users + changed_when: false + + - name: Check firewall status + ansible.builtin.shell: | + echo "=== Firewall Status ===" + if command -v ufw >/dev/null 2>&1; then + ufw status verbose 2>/dev/null || echo "UFW not enabled" + elif command -v firewall-cmd >/dev/null 2>&1; then + firewall-cmd --list-all + else + echo "No firewall detected" + fi + register: firewall_status + changed_when: false + + - name: Check open ports + ansible.builtin.shell: | + echo "=== Open Network Ports ===" + ss -tlnp | grep LISTEN | head -30 + register: open_ports + changed_when: false + + - name: Check failed login attempts + ansible.builtin.shell: | + echo "=== Recent Failed Login Attempts ===" + grep "Failed password" /var/log/auth.log 2>/dev/null | tail -10 || \ + journalctl -u sshd --no-pager -n 20 | grep -i "failed\|authentication failure" || \ + echo "No recent failed attempts or unable to check logs" + register: failed_logins + changed_when: false + failed_when: false + + - name: Check for automatic updates + ansible.builtin.shell: | + echo "=== Automatic Updates Status ===" + if [ -f /etc/apt/apt.conf.d/20auto-upgrades ]; then + cat /etc/apt/apt.conf.d/20auto-upgrades + elif [ -f /etc/dnf/automatic.conf ]; then + grep -E "^apply_updates" /etc/dnf/automatic.conf + else + echo "Automatic updates not configured" + fi + register: auto_updates + changed_when: false + failed_when: false + + - name: Check system updates available + ansible.builtin.shell: | + echo "=== Available Security Updates ===" + if command -v apt-get >/dev/null 2>&1; then + apt-get update -qq 2>/dev/null && apt-get -s upgrade | grep -i security || echo "No security updates or unable to check" + elif command -v yum >/dev/null 2>&1; then + yum check-update --security 2>/dev/null | tail -20 || echo "No security updates or unable to check" + fi + register: security_updates + changed_when: false + failed_when: false + + - name: Check Docker security (if installed) + ansible.builtin.shell: | + echo "=== Docker Security ===" + if command -v docker >/dev/null 2>&1; then + echo "Docker version:" + docker --version + echo "" + echo "Running containers:" + docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' | head -20 + echo "" + echo "Docker daemon config:" + if [ -f /etc/docker/daemon.json ]; then + cat /etc/docker/daemon.json + else + echo "No daemon.json found (using defaults)" + fi + else + echo "Docker not installed" + fi + register: docker_security + changed_when: false + failed_when: false + + - name: Check for world-writable files in critical directories + ansible.builtin.shell: | + echo "=== World-Writable Files (Sample) ===" + find /etc /usr/bin /usr/sbin -type f -perm -002 2>/dev/null | head -10 || echo "No world-writable files found or unable to check" + register: world_writable + changed_when: false + failed_when: false + + - name: Check password policies + ansible.builtin.shell: | + echo "=== Password Policy ===" + if [ -f /etc/login.defs ]; then + grep -E "^PASS_MAX_DAYS|^PASS_MIN_DAYS|^PASS_MIN_LEN|^PASS_WARN_AGE" /etc/login.defs + else + echo "Password policy file not found" + fi + register: password_policy + changed_when: false + failed_when: false + + always: + - name: Display security audit results + ansible.builtin.debug: + msg: | + + ╔════════════════════════════════════════════════════════════════╗ + ║ Security Audit Report: {{ inventory_hostname }} + ╚════════════════════════════════════════════════════════════════╝ + + {{ ssh_config.stdout }} + + {{ empty_passwords.stdout }} + + {{ sudo_users.stdout }} + + {{ firewall_status.stdout }} + + {{ open_ports.stdout }} + + {{ failed_logins.stdout }} + + {{ auto_updates.stdout }} + + {{ security_updates.stdout }} + + {{ docker_security.stdout }} + + {{ world_writable.stdout }} + + {{ password_policy.stdout }} + +- name: Generate Security Summary + hosts: localhost + gather_facts: false + + tasks: + - name: Create security report summary + ansible.builtin.debug: + msg: | + + ╔════════════════════════════════════════════════════════════════╗ + ║ Security Audit Complete ║ + ╚════════════════════════════════════════════════════════════════╝ + + Review the output above for each server. + + Key Security Checks Performed: + ✓ SSH configuration and hardening + ✓ User account security + ✓ Firewall configuration + ✓ Open network ports + ✓ Failed login attempts + ✓ Automatic updates + ✓ Available security patches + ✓ Docker security (if applicable) + ✓ File permissions + ✓ Password policies + + Next Steps: + 1. Review findings for each server + 2. Address any critical issues found + 3. Implement security recommendations + 4. Run audit regularly to track improvements diff --git a/playbooks/setup-jenkins-agent-ssh.yml b/playbooks/setup-jenkins-agent-ssh.yml new file mode 100644 index 0000000..394c578 --- /dev/null +++ b/playbooks/setup-jenkins-agent-ssh.yml @@ -0,0 +1,104 @@ +--- +# Setup SSH key for Jenkins to connect to remote agents +# Usage: ansible-playbook playbooks/setup-jenkins-agent-ssh.yml -e "agent_host=45.16.76.42" + +- name: Setup Jenkins SSH key for remote agent + hosts: jenkins + become: true + gather_facts: true + + vars: + jenkins_user: jenkins + jenkins_home: /var/lib/jenkins + agent_host: "{{ agent_host | default('') }}" + agent_user: "{{ agent_user | default('dlxadmin') }}" + + tasks: + - name: Validate agent_host is provided + ansible.builtin.fail: + msg: "Please provide agent_host: -e 'agent_host=45.16.76.42'" + when: agent_host == '' + + - name: Create .ssh directory for jenkins user + ansible.builtin.file: + path: "{{ jenkins_home }}/.ssh" + state: directory + owner: "{{ jenkins_user }}" + group: "{{ jenkins_user }}" + mode: '0700' + + - name: Check if jenkins SSH key exists + ansible.builtin.stat: + path: "{{ jenkins_home }}/.ssh/id_rsa" + register: jenkins_key + + - name: Generate SSH key for jenkins user + ansible.builtin.command: + cmd: ssh-keygen -t rsa -b 4096 -f {{ jenkins_home }}/.ssh/id_rsa -N '' -C 'jenkins@{{ ansible_hostname }}' + become_user: "{{ jenkins_user }}" + when: not jenkins_key.stat.exists + + - name: Set correct permissions on SSH key + ansible.builtin.file: + path: "{{ jenkins_home }}/.ssh/{{ item }}" + owner: "{{ jenkins_user }}" + group: "{{ jenkins_user }}" + mode: "{{ '0600' if item == 'id_rsa' else '0644' }}" + loop: + - id_rsa + - id_rsa.pub + + - name: Read jenkins public key + ansible.builtin.slurp: + path: "{{ jenkins_home }}/.ssh/id_rsa.pub" + register: jenkins_pubkey + + - name: Display jenkins public key + ansible.builtin.debug: + msg: + - "===== Jenkins Public Key =====" + - "{{ jenkins_pubkey.content | b64decode | trim }}" + - "" + - "Next steps:" + - "1. Copy the public key above" + - "2. Add it to {{ agent_user }}@{{ agent_host }}:~/.ssh/authorized_keys" + - "3. Test: ssh -i {{ jenkins_home }}/.ssh/id_rsa {{ agent_user }}@{{ agent_host }}" + - "4. Update Jenkins credential 'dlx-key' with this private key" + + - name: Create helper script to copy key to agent + ansible.builtin.copy: + dest: /tmp/copy-jenkins-key-to-agent.sh + mode: '0755' + content: | + #!/bin/bash + # Copy Jenkins public key to remote agent + AGENT_HOST="{{ agent_host }}" + AGENT_USER="{{ agent_user }}" + JENKINS_PUBKEY="{{ jenkins_pubkey.content | b64decode | trim }}" + + echo "Copying Jenkins public key to ${AGENT_USER}@${AGENT_HOST}..." + ssh ${AGENT_USER}@${AGENT_HOST} "mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${JENKINS_PUBKEY}' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys" + + echo "Testing connection..." + sudo -u jenkins ssh -o StrictHostKeyChecking=no -i {{ jenkins_home }}/.ssh/id_rsa ${AGENT_USER}@${AGENT_HOST} 'echo "Connection successful!"' + + - name: Instructions + ansible.builtin.debug: + msg: + - "" + - "===== Manual Steps Required =====" + - "" + - "OPTION A - Copy key automatically (if you have SSH access to agent):" + - " 1. SSH to jenkins server: ssh dlxadmin@192.168.200.91" + - " 2. Run: /tmp/copy-jenkins-key-to-agent.sh" + - "" + - "OPTION B - Copy key manually:" + - " 1. SSH to agent: ssh {{ agent_user }}@{{ agent_host }}" + - " 2. Edit: ~/.ssh/authorized_keys" + - " 3. Add: {{ jenkins_pubkey.content | b64decode | trim }}" + - "" + - "Then update Jenkins:" + - " 1. Go to: http://192.168.200.91:8080/manage/credentials/" + - " 2. Find credential 'dlx-key'" + - " 3. Update → Replace with private key from: {{ jenkins_home }}/.ssh/id_rsa" + - " 4. Or create new credential with this key"