1/71
Looks like no tags are added yet.
Purpose of the app
A web app that takes a plain description of a security control or environment and gives a starting-point SOC 2 style assessment: interview questions, evidence to request, likely weaknesses, control mappings, a risk score, and a maturity rating. With a POA&M for remediation.
Reason for building the app
Writing first drafts by hand at ROKS was slow, I wanted to see how far AI could take the starting point while a human still owns the judgment.
Intended users
A GRC analyst or consultant who needs a fast first draft before the real assessment work.
Technology stack
React front end
Vercel serverless functions on the back
The Claude API for the AI calls.
Three-call workflow
The user picks a framework and environment, then describes the control.
Call 1 assesses,
Call 2 (the judge) checks each control mapping
Call 3 (Plain Talk) rewrites the findings for a business reader.
Each call is its own endpoint.
Models used in the app
Claude Haiku 4.5 at temperature 0 for the assessment and the business-language rewrite; Claude Opus 5 at medium effort as the judge.
Why use two models
The judge needs stronger reasoning. In testing, the smaller model kept misjudging which criteria cover MFA, so the bigger model was assigned the checking job.
Why temperature is set to 0
So the same input gives the same output. Consistency matters in audit work.
Role of the judge
For each mapped control, it asks whether the control really fits what was described.
It returns relevant or not, plus a rationale.
If the judge call fails, the app keeps the original mappings.
How made-up controls are prevented
Sixty-one verified SOC 2 summaries of the AICPA Trust Services Criteria are used.
The server, not the model, decides whether a control is labeled verified or AI-generated.
Strict inclusion rule
A control appears only if the input text directly describes something that control governs.
No 'commonly associated' controls are added
A missing control is a reason to leave it out, not to add it.
Flag, never delete
When the judge disagrees with a mapping, the app flags it for human review. It never removes the mapping automatically, so a human decides.
Verified versus AI-generated controls
Verified means the control ID has an AICPA definition in the reference set.
AI-generated means the model produced the control from its own knowledge, so the user should double-check it.
Plain-talk toggle that removes jargon for executives
A toggle that rewrites the findings in business language with no jargon or control IDs so an executive can read them.
It is its own AI call, and it only runs when you click it, which is lazy loading. The result is cached, so switching back and forth is instant.
Risk score scale
A number from 1 to 10 with a short justification, where 10 is the highest risk. It is a starting suggestion, not a final score.
Maturity rating levels
One of five levels: Initial, Developing, Defined, Managed, or Optimizing, plus a one or two sentence justification. The model picks it from the description, and a human assessor confirms it.
Number of supported frameworks
15 including SOC 2 Type I and Type II, ISO 27001, NIST 800-53, and PCI DSS. Control IDs follow each framework's own numbering.
How the app is secured
Prompts and model choices live on the server, not in the browser.
The API key stays server-side
Output is forced to JSON and validated
Inputs are capped at 5,000 characters
A prompt-injection screen blocks phrases like 'ignore previous instructions'
And there is an origin allowlist with field validation.
Defense against prompt injection
Two layers: a pattern screen on input, and server-side prompts so the user never touches the instructions. Output is also validated against a fixed shape.
Where data is stored
In the browser's local storage, holding the history and the POA&M list. Nothing is saved on a server.
How the app is tested
27 automated tests with a mocked API, run through GitHub Actions on each push, plus a build-time secret scan so no key ships in the build.
Limits of the app
It is a starting point, not an audit. It only knows what the description tells it, and it can still be wrong, which is why there is a judge and a human review.
Planned improvements
Adding more verified control references beyond SOC 2, and adding an export so a report can leave the tool cleanly.
Use on a client
It was used to get a starting point. It was never connected to client systems, and the report itself was written by hand.
Connection to this role
The same idea as fullCircle's Assist: AI speeds up the first draft, and the consultant owns the conclusion.
How is the maturity rating determined?
There is no formula. The AI reads the control description and picks one of five levels: Initial, Developing, Defined, Managed, or Optimizing. Initial means ad hoc and Optimizing means continuously improving. It also writes a one or two sentence reason.
Does the judge set the maturity rating?
No. The first call, the assessment on Haiku, sets it. The judge only checks the control mappings.
Is code calculating the maturity rating?
No. The prompt limits the answer to the five levels and the server checks the format. The rating is the model's reading of the description.
How is the risk score determined?
The model gives a number from 1 to 10 based on the gaps in the description, with a 2 to 3 sentence reason. The app reads 1 to 3 as Low, 4 to 6 as Medium, and 7 to 10 as High.
Will the maturity rating change between runs?
The assessment runs at temperature 0, so it is steady but not guaranteed. A vaguer description gets a lower maturity, so the input matters.
Does Plain Talk change the maturity rating?
No. Plain Talk only rewrites the explanations in business language. The level and score stay the same.
How could maturity ratings be made more consistent?
Write a clear definition of each level into the prompt and test it on known examples. Right now the prompt names the levels but does not define them.
Who has the final say on the rating?
A human. The rating is a starting point for a GRC professional to confirm, never a finding on its own.
Six outputs of the Risk Whisperer assessment
6 to 8 interview questions, 6 to 8 evidence items, 4 to 6 weaknesses with severity and a fix, 1 to 6 control mappings, a risk score, and a maturity rating.
Frameworks and environments supported
15 frameworks and 9 environments: AWS, AWS GovCloud, Azure, Azure Gov, GCP, Oracle OCI, multi-cloud, hybrid, and on-prem.
POA&M tracker
Under each weakness, the user sets an owner, a target date, and a status (Open, In Progress, Closed, or Deferred). It saves automatically in the browser.
Other features of the app
History that survives closing the tab, PDF export, copy buttons, a regenerate button, 4 example presets, and an installable PWA.
Three endpoints and why they are separate
/api/assess, /api/judge, and /api/plain. Each makes one AI call, so each stays under the time limit and one failure does not take down the others.
Token and time limits
Assess and Plain Talk get 6,000 tokens, and the judge gets 4,000. Vercel allows 60 seconds, and the server stops at 55 with a clear timeout message.
Architecture in one sentence
The browser sends only data, the server checks it, builds the prompt, adds the secret key, calls Claude, and returns parsed JSON.
Where the API key lives
In a Vercel environment variable on the server, without the REACT_APP prefix, so it never reaches the browser.
Security hook
Right way, right door, right shape: POST with JSON, an allowed origin, and only expected fields with allowed values and size limits.
Is the origin check authentication?
No. The origin check only blocks other websites, which is why every field is still validated.
Why extra fields are rejected
So no one can send their own prompt or pick a different model. Extra fields get a 400 error.
Input size limits
5,000 characters of input and 6 control mappings.
Why Plain Talk skips the injection screen
The AI's own findings use words like 'exposed' or 'leak' that would trip the screen, so shape and size checks protect that call instead.
Meaning of the API status codes
400 invalid data, 403 wrong origin, 405 wrong method, 415 wrong content type, 502 bad AI response, 504 timeout.
Handling bad AI output
The parser tolerates stray text around the JSON and says plainly if the answer was cut off at the token limit.
What the judge checks
For SOC 2, every mapped control. For other frameworks, only controls with a verified SOC 2 summary. If there is nothing to judge, the call is skipped to save money.
If the judge call fails
The assessment still shows with its original explanations.
Who decides whether a control is verified
The server, never the model. Only controls with a verified summary can have their explanation rewritten.
Verification labels
'Control name verified, explanation AI-generated' or 'not yet verified against source'. A doubtful control also gets a Weak fit flag.
Is the control text the official AICPA text?
No. It is verified summaries of the AICPA 2017 Trust Services Criteria, all 61 of them.
What overrides AI-written control names
A hardcoded table of verified SOC 2 names overrides AI-written control names. Code beats AI where accuracy matters.
Control that MFA maps to
CC6.1, logical access. CC6.2 covers registering and removing users, and MFA is not evidence of that process.
Result of temperature 0 testing
Four IAM runs returned the same controls (CC6.1, CC6.2, CC6.3, and CC5.2). A later run varied at the edges and the judge flagged the doubtful ones. Consistent, not guaranteed.
Why the judge moved to Opus
Haiku misjudged MFA twice, flagging CC6.1 and passing CC6.2. A stronger model on the checking job fixed it, and the main call stays on Haiku for speed and cost.
Why not ask Haiku to remove its own weak mappings
Haiku writes in one pass and does not remove what it wrote. A separate judge fixes this through architecture, not a longer prompt.
Judge story: flag, never delete
Problem: the judge deleted correct SOC 2 mappings on overly literal readings.
Cause: it was allowed to remove controls.
Fix: flag, never delete, with a stronger judge grounded in the criteria.
Test: a live re-run kept the correct mappings and flagged the doubtful ones.
Debugging story: timeouts and JSON parsing
The first live test said the AI took too long, so the limit was raised to 60 seconds with a cutoff at 55. The next failure was unparseable JSON, so the parser was made to tolerate stray text.
Mistake story: syntax fix rewrite
A syntax fix replaced a 670-line app with a minimal rewrite that dropped features. Git restored the last good commit and the work was re-applied. Every diff is now reviewed before merging.
How is the app tested?
27 automated tests run against a mocked API. They cover method, origin and content checks, rejection of browser-supplied prompts and models, field and size validation, each endpoint's model settings, error mapping, and em dash removal. A regression test checks SOC 2 control ID to category pairings.
How was the security refactor verified?
With automated tests, a production build check for prompt text, a byte-for-byte comparison of requests across 1,620 framework, environment, and control family combinations, and live tests.
What is the build-time secret scan?
After every build, a script fails the build if the output contains a key-like string such as sk-ant. Env files are git-ignored.
Continuous integration (CI)
GitHub Actions runs the tests on every push and pull request, and Vercel builds a preview for every pull request.
Where does the data go?
Input goes to Anthropic's API for the AI calls, so I tell users to use sanitized descriptions, never real client secret
Current gaps: login and rate limiting
No login and no rate limiting exist yet. The origin check is not authentication, so fixed models and token caps are what keep cost down.
Next security step
Rate limits and optional sign-in.
Frameworks with verified control summaries
Only SOC 2. NIST, ISO, PCI, and the other frameworks rely on the model's own knowledge, so their controls are labeled AI-generated.
Planned roadmap features
Verified control tables for NIST 800-53 and ISO 27001, a fixed test set to measure mapping accuracy, POA&M export to CSV, and linking POA&M items to collector evidence so a finding closes only when evidence proves it.
Project history: original name
Started in June 2026 as AuditTrail, then renamed Risk Whisperer.
Development timeline
Security and Plain Talk first, then POA&M, then SOC 2 accuracy with the judge, server-side prompts in late September, and verified summaries in October.