8.5 KiB

Raw Blame History

Plan: hiveops-qds-monitor Microservice

Context

QDS (Quality Data Systems) sends automated ATM alert emails from atmalerts@qualitydatasystems.com via MoniManager to hiveops@branchcos.com. These are currently landing in a Gmail label called QDS but are never acted on programmatically. The goal is a new HiveOps microservice that monitors this inbox and creates (or resolves) incidents in hiveops-incident automatically.

Email Format (Observed)

Two event types, identified by subject prefix:

[External] Event Occurred → create incident
[External] Canceled → resolve existing incident

Body contains a structured key-value table:

Terminal #      763330
Fault Contents  Cash Dispenser Bill Cassette PCU3 Abnormal
Fault Device    Cash Dispenser
Error Code      (optional)
Date and Time   02/23/2026 06:56:58
Ticket No       287886
Remaining Total USD90,000.00  (optional)
Priority        High
Severity        Critical
Problem Desc    ...

New Service: `hiveops-qds-monitor`

Location: /source/hiveops-src/hiveops-qds-monitor/ (new repo, Java Spring Boot)

Tech: Spring Boot 3.4.x · Gmail API (OAuth2) · RestTemplate · H2 embedded DB · Spring Scheduler

Architecture

Gmail (QDS label)
        │ poll every 2 min (unread only)
        ▼
GmailPollerService
        │ parse body
        ▼
EmailParserService  ──→  QdsEmailEvent { terminalId, ticketNo, faultDevice,
        │                                faultContents, severity, eventType }
        │
  Event Occurred?                Canceled?
        │                              │
        ▼                              ▼
IncidentCreationFlow         IncidentResolutionFlow
  1. AuthService.getJwt()      1. TicketMappingRepo.findByTicketNo()
  2. AtmLookupService          2. PUT /api/incidents/{id} → RESOLVED
     GET /api/atms/search      3. TicketMappingRepo.delete()
  3. QdsIncidentMapper
     (fault → type, severity)
  4. POST /api/incidents
  5. TicketMappingRepo.save(ticketNo → incidentId)
        │                              │
        └────────── mark email as read ─┘

Project Structure

hiveops-qds-monitor/
├── pom.xml
├── Dockerfile
└── src/main/
    ├── java/com/hiveops/qdsmonitor/
    │   ├── HiveOpsQdsMonitorApplication.java
    │   ├── config/
    │   │   └── AppConfig.java            # RestTemplate, Gmail client beans
    │   ├── scheduler/
    │   │   └── GmailPollerService.java   # @Scheduled, calls Gmail API
    │   ├── service/
    │   │   ├── EmailParserService.java   # Parse raw email body → QdsEmailEvent
    │   │   ├── AtmLookupService.java     # GET /api/atms/search
    │   │   ├── AuthService.java          # Login + token cache + auto-refresh
    │   │   └── IncidentApiService.java   # POST /api/incidents, PUT .../status
    │   ├── mapper/
    │   │   └── QdsIncidentMapper.java    # Fault → IncidentType + Severity
    │   ├── dto/
    │   │   ├── QdsEmailEvent.java
    │   │   ├── CreateIncidentRequest.java
    │   │   └── UpdateIncidentRequest.java
    │   └── repository/
    │       └── TicketMappingRepository.java  # H2: ticketNo ↔ hiveops incidentId
    └── resources/
        └── application.properties

Key Implementation Details

1. Gmail Polling (`GmailPollerService`)

Use google-api-services-gmail Java client library
OAuth2 credentials (client_id, client_secret, refresh_token) from env vars
Query: label:QDS is:unread — fetch only unread messages
After processing: mark email as read (UNREAD label removed)
Run on @Scheduled(fixedDelayString = "${qds.poll.interval-ms:120000}")

2. Email Parsing (`EmailParserService`)

Detect event type from subject: contains "Event Occurred" vs "Canceled"
Parse body as key-value pairs using regex: ^(\w[\w\s]+?)\s{2,}(.+)$
Extract: Terminal #, Fault Contents, Fault Device, Error Code, Date and Time, Ticket No, Priority, Severity

3. Fault → IncidentType Mapping (`QdsIncidentMapper`)

Fault Device / Contents	IncidentType
Card Reader	CARD_READER_FAIL
Cash Dispenser + "cassette"	CASSETTE_LOW / CASSETTE_EMPTY
Cash Dispenser (other)	DISPENSER_JAM
Network	NETWORK_ERROR
Power	POWER_FAILURE
Item Processing Module	HARDWARE_ERROR
Out of service / Service	SOFTWARE_ERROR
Default	HARDWARE_ERROR

QDS Severity → HiveOps Severity:

Critical → CRITICAL
High (priority) → HIGH
Normal → MEDIUM
Low → LOW

4. Auth Service (`AuthService`)

POST http://hiveops-auth:8082/auth/api/login with service account credentials
Cache JWT token + expiry in memory
Auto-refresh using refresh token before expiry (check on each API call)
Env vars: SERVICE_ACCOUNT_EMAIL, SERVICE_ACCOUNT_PASSWORD

5. ATM Lookup (`AtmLookupService`)

GET http://hiveops-incident:8081/api/atms/search?query={terminalId}
Return first match's numeric id (Long)
If no match found: log WARN and skip incident creation (don't create orphan incidents)

6. Incident API (`IncidentApiService`)

Create: POST http://hiveops-incident:8081/api/incidents
- Description format: [QDS] Terminal: {terminalId} | Ticket: #{ticketNo} | {faultContents} | {dateTime}
Resolve: PUT http://hiveops-incident:8081/api/incidents/{id} with { "status": "RESOLVED" }

7. Ticket Mapping (H2 Embedded DB)

Table: ticket_mapping(qds_ticket_no VARCHAR PK, hiveops_incident_id BIGINT, created_at)
Spring Data JPA with H2 — no external DB needed
Persist across restarts (H2 file mode: ./data/qds-monitor)

Incident Description Format

[QDS] Terminal: 763330 | Ticket: #287886 | Cash Dispenser Bill Cassette PCU3 Abnormal | 02/23/2026 06:56:58

Files to Create

File	Action
`/source/hiveops-src/hiveops-qds-monitor/pom.xml`	New — Spring Boot 3.4, Gmail API, H2
`/source/hiveops-src/hiveops-qds-monitor/Dockerfile`	New — multi-stage Java build
`/source/hiveops-src/hiveops-qds-monitor/src/...`	New — all Java source files above
`/source/hiveops-src/hiveops-qds-monitor/src/main/resources/application.properties`	New

Files to Modify

File	Change
`hiveops/instances/services/docker-compose.yml`	Add `hiveops-qds-monitor` service block
`hiveops/instances/services/.env`	Add Gmail OAuth + service account env vars
`hiveops/.env`	Add `QDS_MONITOR_VERSION=latest`

Docker Service Definition (to add to docker-compose.yml)

hiveops-qds-monitor:
  image: ${REGISTRY_URL}/hiveops-qds-monitor:${QDS_MONITOR_VERSION:-latest}
  restart: unless-stopped
  environment:
    - GMAIL_CLIENT_ID=${GMAIL_CLIENT_ID}
    - GMAIL_CLIENT_SECRET=${GMAIL_CLIENT_SECRET}
    - GMAIL_REFRESH_TOKEN=${GMAIL_REFRESH_TOKEN}
    - GMAIL_LABEL_NAME=QDS
    - INCIDENT_API_URL=http://hiveops-incident:8081
    - AUTH_API_URL=http://hiveops-auth:8082
    - SERVICE_ACCOUNT_EMAIL=${QDS_SERVICE_ACCOUNT_EMAIL}
    - SERVICE_ACCOUNT_PASSWORD=${QDS_SERVICE_ACCOUNT_PASSWORD}
    - POLL_INTERVAL_MS=120000
  volumes:
    - ./data/qds-monitor:/app/data
  depends_on:
    - hiveops-incident
    - hiveops-auth
  networks:
    - hiveops-network
  logging:
    driver: "json-file"
    options:
      max-size: "10m"
      max-file: "3"
  healthcheck:
    test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/8080 && echo -e 'GET /actuator/health HTTP/1.0\r\n\r\n' >&3 && cat <&3 | grep -q 'UP'"]
    interval: 30s
    timeout: 10s
    retries: 3

Required Gmail OAuth Setup (one-time, before deployment)

Before deploying, a one-time OAuth2 flow must be run to generate a refresh token for hiveops@branchcos.com:

Create a Google Cloud project with Gmail API enabled
Create OAuth2 credentials (Desktop app type)
Run OAuth consent flow to get refresh_token
Store client_id, client_secret, refresh_token in .env

Verification

Build the jar locally: mvn clean package -DskipTests
Run locally with env vars set, verify it connects to Gmail and logs emails found
Deploy to docker-compose, check logs: docker compose logs -f hiveops-qds-monitor
Send a test email to the QDS label and confirm incident created in hiveops-incident UI
Mark the incident as resolved by sending a "Canceled" test email and confirm status change
Check H2 ticket mapping is persisted across service restart

8.5 KiB Raw Blame History