Supai AI – Smoke-Test Review & Action Items

Overview of the Session and Its Goals

• Informal working session focused on smoke-test/UAT results for the “Supai” AI-assistant that:
 – Auto-routes ServiceNow Service-Catalog tickets (requests only; incidents not yet in scope).
 – Reaches out to requesters through Microsoft Teams when required information is missing.
• Attendees include Justine (primary SME for IMS queue), Dana, James (developer), Phil, David and QA representatives (Mayra, etc.).
• Objective: review anomalies discovered after the most recent deployment, decide what to fix before the next build, and refine the manual smoke-test procedure so QA can continue while devs correct logic.

Key Problems Observed in Latest Deployment

• Auto-Routing Failure
 – Field “Can this ticket be routed?” sometimes records Yes but business rule still refuses to route.
 – James suspects an inverted boolean check (e.g. a “not” where it should be positive logic) in the business rule; will correct.
• Teams/Missing-Info Pings Mis-Firing
 – Supai prompts users for missing data even when ticket shows “Has required fields – True”.
 – Example: Password-reset for Redshift listed all needed values, AI still asked for client, cloud-account, instance, etc.
 – Hypothesis: Routing guide doesn’t differentiate between “Redshift build/modify” vs “Redshift password reset”.
• Confidence-Score Oddities
 – Very low routing confidence even on blatantly obvious keywords (e.g. “Globalscape”).
 – Suspected same root cause as routing bug or training data misalignment.
• Supai Chat Message Lacks Context
 – When it IMs a user it does NOT quote the RITM number or short description, making it hard for requester or analyst to correlate.

ServiceNow Object Hierarchy Refresher

• Requested Item (RITM)
 – User-facing container; cannot be assigned to a group.
• Service Catalog Task (SC Task)
 – Child of RITM; actual work object Supai routes.
 – Tracker/spreadsheet must record the SC Task number, not the RITM.
• Each smoke-test request therefore yields:
 – 1 RITM (parent) + ≥1 SC Task(s) (children).
 – Supai’s business rules and assignment groups act only on SC Tasks.

Business Rules & Desired AI Logic (Current vs. Expected)

• Rule 1 – Missing-Info Interrogation
 – If \text{required_fields} = \text{False} then Supai sends a Teams message listing missing/unclear fields.
 – Rule should run independently of routing; Supai may route immediately after pinging.
• Rule 2 – Auto-Routing
 – If \text{can_be_routed} = \text{True} then assign to correct group regardless of Rule 1 outcome.
 – Bug: current implementation blocks routing when any field marked missing.
• Three-Strike Rule (Human Process, not coded)
 – If user fails to answer after three prompts a human dispatcher manually cancels/returns ticket.
 – AI should NOT automatically close tickets on the third strike yet; leave status “Open – Waiting on user”.

Smoke-Test / UAT Procedure Walk-Through

• Where to Launch:
 1. Open Dev Service Portal → IMS Services Catalog → Request form.
 2. Copy template rows from the shared “Smoke Test Tickets” sheet (type, body, priority, etc.).
• Important Form Fields:
 – Short Description: often auto-generated except for generic “I need something else” item.
 – Description: large free-text; new form will inject all answers here so AI can parse.
• After Submit:
 1. Locate the SC Task in tech back-end (Mayra needs access granting).
 2. Record into spreadsheet:
  • SC Task ID, computed category, predicted assignment group, routing confidence, solution doc + confidence, etc.
  • Boolean flags: Auto-routed? Supai reached out? Fields-required value? etc.
 3. Visually inspect comments:
  – Discrepancy #1 – Supai IM’d but “Has required fields = True”.
  – Discrepancy #2 – “Can be routed = True” but no routing occurred.
 4. (Optional deep-dive) Use Ticket-Routing Guide to cross-check keyword hits vs. confidence-score anomalies.
• Edge-Cases to Add:
 – High-priority requests.
 – Various password-reset scenarios (Redshift, Globalscape, etc.).
 – Tickets intentionally missing each distinct required field to validate prompts.

Documentation & Knowledge Sources

• Ticket-Routing Guide
 – Live editable copy in SharePoint (link in chat). Periodically copied to S3 → Dev → Prod.
 – Action: add explicit entry for “Redshift – Password Reset” to stop false prompts.
• SOP Library
 – Manually exported from Confluence into S3 “input_solution” bucket; used for resolution suggestions.
• Server Naming Conventions & Client List
 – Exist as separate docs; plan to upload so AI can recognize hostnames & client acronyms.

Backlog / Action Items Generated

  1. Bug-Fix: Reverse the misplaced boolean in auto-routing business rule (James).
  2. Investigate & correct routing-confidence calculations.
  3. Update Routing Guide:
     • Add “Redshift password reset”.
     • Add list of clients and allow hostnames as alternative to cloud-account field.
  4. Supai Message Enhancements:
     • Start message with RITM number & short description.
     • Change tone to: “Your ticket MAY be missing or must CLARIFY the following information…”
     • Rename “Missing Fields” section to “Fields Requiring Clarification”.
  5. AI-Training Enhancements:
     • Teach model to parse client names, acronyms and hostname formats (feed server-naming-convention doc).
     • Automate periodic migration of updated routing guide & SOPs from SharePoint to S3.
  6. QA Spreadsheet Tweaks:
     • Remove legacy columns “Ticket has required fields?” and “Can ticket be routed?” or note that they are informational only (logic handled elsewhere).
  7. Meeting Scheduling / Logistics:
     • Team to reconvene Monday after Friday holiday for deeper dive on flagged cases.
     • Justine will continue demo sessions; Mayra to gain SC-Task access; QA to keep populating smoke-test sheet.

Anticipated Future Work & Dependencies

• New Request Form (Justine): will push richer structured data into description, aiding AI accuracy. Target: end-of-month production.
• Global-Side Rollout: lacking a “power user” like Justine; will require extra knowledge-harvesting sessions.
• Once routing & prompting stabilize → one more deployment → full UAT cycle.

Practical / Ethical / User-Experience Considerations

• Clear, non-accusatory wording prevents user frustration when AI incorrectly flags missing data.
• Providing ticket identifiers in Teams chat supports accessibility & auditability.
• Ensuring AI auto-routes even after prompting maintains SLA by preventing tickets from idling.
• Need to respect out-of-office scenarios before triggering three-strike closures (currently manual).

Connections to Earlier Work / Principles

• Continues long-standing goal: reduce manual dispatcher triage time.
• Builds on prior smoke-tests that created “ticket has fields / route?” comment logic (now outdated).
• Follows DevOps principle of rapid iterative deployment + UAT feedback loop.
• Ethos: “Lead a horse to water; can’t make it drink” – AI ensures the requester is asked, but team still auto-routes to keep workflow moving.

Next Concrete Steps Before Monday Check-In

• James: patch boolean logic & inspect confidence-score code.
• QA: create high-priority & varied reset tickets, record discrepancies.
• Justine: update routing guide entries, draft new Teams prompt copy, begin testing updated request form.
• Dana/Phil: enter backlog items into tracker (message re-phrasing, hostname knowledge, automation tasks).