Rapid Fire - Risk Whip

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/71

encourage image

There's no tags or description

Looks like no tags are added yet.

72 Terms

1
New cards

Purpose of the app

A web app that takes a plain description of a security control or environment and gives a starting-point SOC 2 style assessment: interview questions, evidence to request, likely weaknesses, control mappings, a risk score, and a maturity rating. With a POA&M for remediation.

2
New cards

Reason for building the app

Writing first drafts by hand at ROKS was slow, I wanted to see how far AI could take the starting point while a human still owns the judgment.

3
New cards

Intended users

A GRC analyst or consultant who needs a fast first draft before the real assessment work.

4
New cards

Technology stack

  • React front end

  • Vercel serverless functions on the back

  • The Claude API for the AI calls.


5
New cards

Three-call workflow

The user picks a framework and environment, then describes the control.

  • Call 1 assesses,

  • Call 2 (the judge) checks each control mapping

  • Call 3 (Plain Talk) rewrites the findings for a business reader.

Each call is its own endpoint.


6
New cards

Models used in the app

Claude Haiku 4.5 at temperature 0 for the assessment and the business-language rewrite; Claude Opus 5 at medium effort as the judge.

7
New cards

Why use two models

The judge needs stronger reasoning. In testing, the smaller model kept misjudging which criteria cover MFA, so the bigger model was assigned the checking job.

8
New cards

Why temperature is set to 0

So the same input gives the same output. Consistency matters in audit work.

9
New cards

Role of the judge

  • For each mapped control, it asks whether the control really fits what was described.

  • It returns relevant or not, plus a rationale.

  • If the judge call fails, the app keeps the original mappings.


10
New cards

How made-up controls are prevented

  • Sixty-one verified SOC 2 summaries of the AICPA Trust Services Criteria are used.

  • The server, not the model, decides whether a control is labeled verified or AI-generated.


11
New cards

Strict inclusion rule

  • A control appears only if the input text directly describes something that control governs.

  • No 'commonly associated' controls are added

  • A missing control is a reason to leave it out, not to add it.


12
New cards

Flag, never delete

When the judge disagrees with a mapping, the app flags it for human review. It never removes the mapping automatically, so a human decides.

13
New cards

Verified versus AI-generated controls

  • Verified means the control ID has an AICPA definition in the reference set.

  • AI-generated means the model produced the control from its own knowledge, so the user should double-check it.


14
New cards

Plain-talk toggle that removes jargon for executives

A toggle that rewrites the findings in business language with no jargon or control IDs so an executive can read them.

It is its own AI call, and it only runs when you click it, which is lazy loading. The result is cached, so switching back and forth is instant.

15
New cards

Risk score scale

A number from 1 to 10 with a short justification, where 10 is the highest risk. It is a starting suggestion, not a final score.

16
New cards

Maturity rating levels

One of five levels: Initial, Developing, Defined, Managed, or Optimizing, plus a one or two sentence justification. The model picks it from the description, and a human assessor confirms it.

17
New cards

Number of supported frameworks

15 including SOC 2 Type I and Type II, ISO 27001, NIST 800-53, and PCI DSS. Control IDs follow each framework's own numbering.

18
New cards

How the app is secured

  • Prompts and model choices live on the server, not in the browser.

  • The API key stays server-side

  • Output is forced to JSON and validated

  • Inputs are capped at 5,000 characters

  • A prompt-injection screen blocks phrases like 'ignore previous instructions'

  • And there is an origin allowlist with field validation.


19
New cards

Defense against prompt injection

Two layers: a pattern screen on input, and server-side prompts so the user never touches the instructions. Output is also validated against a fixed shape.

20
New cards

Where data is stored

In the browser's local storage, holding the history and the POA&M list. Nothing is saved on a server.

21
New cards

How the app is tested

27 automated tests with a mocked API, run through GitHub Actions on each push, plus a build-time secret scan so no key ships in the build.

22
New cards

Limits of the app

It is a starting point, not an audit. It only knows what the description tells it, and it can still be wrong, which is why there is a judge and a human review.

23
New cards

Planned improvements

Adding more verified control references beyond SOC 2, and adding an export so a report can leave the tool cleanly.

24
New cards

Use on a client

It was used to get a starting point. It was never connected to client systems, and the report itself was written by hand.

25
New cards

Connection to this role

The same idea as fullCircle's Assist: AI speeds up the first draft, and the consultant owns the conclusion.

26
New cards

How is the maturity rating determined?

There is no formula. The AI reads the control description and picks one of five levels: Initial, Developing, Defined, Managed, or Optimizing. Initial means ad hoc and Optimizing means continuously improving. It also writes a one or two sentence reason.

27
New cards

Does the judge set the maturity rating?

No. The first call, the assessment on Haiku, sets it. The judge only checks the control mappings.

28
New cards

Is code calculating the maturity rating?

No. The prompt limits the answer to the five levels and the server checks the format. The rating is the model's reading of the description.

29
New cards

How is the risk score determined?

The model gives a number from 1 to 10 based on the gaps in the description, with a 2 to 3 sentence reason. The app reads 1 to 3 as Low, 4 to 6 as Medium, and 7 to 10 as High.

30
New cards

Will the maturity rating change between runs?

The assessment runs at temperature 0, so it is steady but not guaranteed. A vaguer description gets a lower maturity, so the input matters.

31
New cards

Does Plain Talk change the maturity rating?

No. Plain Talk only rewrites the explanations in business language. The level and score stay the same.

32
New cards

How could maturity ratings be made more consistent?

Write a clear definition of each level into the prompt and test it on known examples. Right now the prompt names the levels but does not define them.

33
New cards

Who has the final say on the rating?

A human. The rating is a starting point for a GRC professional to confirm, never a finding on its own.

34
New cards

Six outputs of the Risk Whisperer assessment

6 to 8 interview questions, 6 to 8 evidence items, 4 to 6 weaknesses with severity and a fix, 1 to 6 control mappings, a risk score, and a maturity rating.

35
New cards

Frameworks and environments supported

15 frameworks and 9 environments: AWS, AWS GovCloud, Azure, Azure Gov, GCP, Oracle OCI, multi-cloud, hybrid, and on-prem.

36
New cards

POA&M tracker

Under each weakness, the user sets an owner, a target date, and a status (Open, In Progress, Closed, or Deferred). It saves automatically in the browser.

37
New cards

Other features of the app

History that survives closing the tab, PDF export, copy buttons, a regenerate button, 4 example presets, and an installable PWA.

38
New cards

Three endpoints and why they are separate

/api/assess, /api/judge, and /api/plain. Each makes one AI call, so each stays under the time limit and one failure does not take down the others.

39
New cards

Token and time limits

Assess and Plain Talk get 6,000 tokens, and the judge gets 4,000. Vercel allows 60 seconds, and the server stops at 55 with a clear timeout message.

40
New cards

Architecture in one sentence

The browser sends only data, the server checks it, builds the prompt, adds the secret key, calls Claude, and returns parsed JSON.

41
New cards

Where the API key lives

In a Vercel environment variable on the server, without the REACT_APP prefix, so it never reaches the browser.

42
New cards

Security hook

Right way, right door, right shape: POST with JSON, an allowed origin, and only expected fields with allowed values and size limits.

43
New cards

Is the origin check authentication?

No. The origin check only blocks other websites, which is why every field is still validated.

44
New cards

Why extra fields are rejected

So no one can send their own prompt or pick a different model. Extra fields get a 400 error.

45
New cards

Input size limits

5,000 characters of input and 6 control mappings.

46
New cards

Why Plain Talk skips the injection screen

The AI's own findings use words like 'exposed' or 'leak' that would trip the screen, so shape and size checks protect that call instead.

47
New cards

Meaning of the API status codes

400 invalid data, 403 wrong origin, 405 wrong method, 415 wrong content type, 502 bad AI response, 504 timeout.

48
New cards

Handling bad AI output

The parser tolerates stray text around the JSON and says plainly if the answer was cut off at the token limit.

49
New cards

What the judge checks

For SOC 2, every mapped control. For other frameworks, only controls with a verified SOC 2 summary. If there is nothing to judge, the call is skipped to save money.

50
New cards

If the judge call fails

The assessment still shows with its original explanations.

51
New cards

Who decides whether a control is verified

The server, never the model. Only controls with a verified summary can have their explanation rewritten.

52
New cards

Verification labels

'Control name verified, explanation AI-generated' or 'not yet verified against source'. A doubtful control also gets a Weak fit flag.

53
New cards

Is the control text the official AICPA text?

No. It is verified summaries of the AICPA 2017 Trust Services Criteria, all 61 of them.

54
New cards

What overrides AI-written control names

A hardcoded table of verified SOC 2 names overrides AI-written control names. Code beats AI where accuracy matters.

55
New cards

Control that MFA maps to

CC6.1, logical access. CC6.2 covers registering and removing users, and MFA is not evidence of that process.

56
New cards

Result of temperature 0 testing

Four IAM runs returned the same controls (CC6.1, CC6.2, CC6.3, and CC5.2). A later run varied at the edges and the judge flagged the doubtful ones. Consistent, not guaranteed.

57
New cards

Why the judge moved to Opus

Haiku misjudged MFA twice, flagging CC6.1 and passing CC6.2. A stronger model on the checking job fixed it, and the main call stays on Haiku for speed and cost.

58
New cards

Why not ask Haiku to remove its own weak mappings

Haiku writes in one pass and does not remove what it wrote. A separate judge fixes this through architecture, not a longer prompt.

59
New cards

Judge story: flag, never delete

  • Problem: the judge deleted correct SOC 2 mappings on overly literal readings.

  • Cause: it was allowed to remove controls.

  • Fix: flag, never delete, with a stronger judge grounded in the criteria.

  • Test: a live re-run kept the correct mappings and flagged the doubtful ones.


60
New cards

Debugging story: timeouts and JSON parsing

The first live test said the AI took too long, so the limit was raised to 60 seconds with a cutoff at 55. The next failure was unparseable JSON, so the parser was made to tolerate stray text.

61
New cards

Mistake story: syntax fix rewrite

A syntax fix replaced a 670-line app with a minimal rewrite that dropped features. Git restored the last good commit and the work was re-applied. Every diff is now reviewed before merging.

62
New cards

How is the app tested?

27 automated tests run against a mocked API. They cover method, origin and content checks, rejection of browser-supplied prompts and models, field and size validation, each endpoint's model settings, error mapping, and em dash removal. A regression test checks SOC 2 control ID to category pairings.

63
New cards

How was the security refactor verified?

With automated tests, a production build check for prompt text, a byte-for-byte comparison of requests across 1,620 framework, environment, and control family combinations, and live tests.

64
New cards

What is the build-time secret scan?

After every build, a script fails the build if the output contains a key-like string such as sk-ant. Env files are git-ignored.

65
New cards

Continuous integration (CI)

GitHub Actions runs the tests on every push and pull request, and Vercel builds a preview for every pull request.

66
New cards

Where does the data go?

Input goes to Anthropic's API for the AI calls, so I tell users to use sanitized descriptions, never real client secret

67
New cards

Current gaps: login and rate limiting

No login and no rate limiting exist yet. The origin check is not authentication, so fixed models and token caps are what keep cost down.

68
New cards

Next security step

Rate limits and optional sign-in.

69
New cards

Frameworks with verified control summaries

Only SOC 2. NIST, ISO, PCI, and the other frameworks rely on the model's own knowledge, so their controls are labeled AI-generated.

70
New cards

Planned roadmap features

Verified control tables for NIST 800-53 and ISO 27001, a fixed test set to measure mapping accuracy, POA&M export to CSV, and linking POA&M items to collector evidence so a finding closes only when evidence proves it.

71
New cards

Project history: original name

Started in June 2026 as AuditTrail, then renamed Risk Whisperer.

72
New cards

Development timeline

Security and Plain Talk first, then POA&M, then SOC 2 accuracy with the judge, server-side prompts in late September, and verified summaries in October.