Work flashcards

What did I build

I built a Playwright + TypeScript automation framework for AVEVA Universal HMI. It controls a browser to perform operator-like actions, then verifies whether the HMI/runtime responded correctly.

What is Chromium?

Chromium is the browser that my current Playwright automation controls. It is closely related to Google Chrome, but it is a separate browser project.

What is Playwright?

Playwright is the browser-automation library my TypeScript framework uses. It tells Chromium to navigate, click, type, wait, take screenshots, and inspect the web application.

What is the difference between Chromium and Playwright?

Chromium is the browser that actually renders Universal HMI. Playwright is the automation software that controls that browser.

What is a Page?

A Page is Playwright's object representing a browser tab/page. My framework uses the Page object to interact with the Universal HMI loaded in that tab—for example navigating, clicking, locating elements, typing, and taking screenshots.

What is the difference between Browser, BrowserContext, and Page?

1. Browser = Chromium itself

2. BrowserContext = an isolated browser session with its own cookies/login/storage

3. Page = an individual tab inside that context

What is the DOM? And why is it important to this project?

The DOM is the browser's structured representation of the elements on a webpage. Playwright can use that structure to locate things such as buttons, textboxes, and navigation items.

Normal DOM elements can often be targeted directly with Playwright locators, but many AVEVA graphic objects are rendered inside a canvas-like surface and are not individually exposed as convenient DOM elements.

Why does the canvas problem lead to normalized coordinates and pixel verification?

Because canvas graphics are not always individually available in the DOM, Playwright may need to interact with them by position and verify them visually. Normalized coordinates provide reusable canvas-relative targeting, while pixel analysis lets the framework measure what changed on the rendered screen.

What is a bounding box?

A bounding box describes where something currently exists on the screen and how large it currently is. It typically includes the dimensions (width and height) and position (x and y coordinates) of the element. But to find the buttons or specific graphics inside of that bounding box, I would need normalized coordinates, because with a new bounding box comes new coordinates for the validation that we are targeting.

What is a normalized coordinate?

A normalized coordinate is describing the target as a percentage/fraction of the canvas.

How does playwright verify a visual animation has the expected behavior.

Playwright verifies an animation like Fill through a process where it will take a screenshot, decode the pixel data, extract the ROI, measure the region filled, compared to what expected, and then pass or fail

How does Decoding the PNG work?

In a PNG, there is the position (x,y), color with values 0-255, and Alpha. To measure it programmatically, we need to decode it into pixel data — arrays of numbers. Then we can write code that counts pixels, finds color boundaries, and calculates ratios. You can't do that with an image file; you need the raw pixel values.

Why does PNG work in this automation framework?

Decoding the PNG works because in terms of normalized coordinates, we are able to get the ROI and extract just the pixels in the screenshot that we care about. When measuring the fill, it sees how far the “filled” pixels extend to. Counting the consecutive pixels will give a ratio, which can be compared to the expected.

What is the point of the discovery layer?

Discovery attempts to establish target-specific config fields automatically when there is deterministic evidence. Fields it cannot establish are explicitly marked MANUAL_REQUIRED instead of being guessed.

Q: How does the framework discover Fill direction?

This method is only valid when multiple trustworthy states can actually be produced/observed. It is not currently proven for Graphic_FillStyle

What are the two jobs of g1FlowPidLifecycleConfig.ts?

Product bindings and test intent/safety policy.

1. For Product bindings, things such as mode, setpoint, operation, equioment, process response bindings are things that will be recorded with references

2. For Test Intent, things like what mode testing, what values are safe to test, failure mode, stop cases, how long wait.

What does automatic discovery try to eliminate?

Manual product bindings like controls, references, geometry, equipment, and mappings.
1. Would have to manually discover and update src/profiles/g1FlowPidLifecycleConfig.ts . It would have to located PID toggle, measure its position as normalized coordinateds, type those numbers into config, and then same for other buttons. It had to put down safe test values. It had to put in timeout values. It had to put in equipment references. Would have to say when a pump is active and where is the ROI coordinates.

What is the target/future discovery architecture for G1?

Generates a product model that a scenario planner turns into lifecycle config.

What does structural discovery answer?

What controls, signals, graphics, and equipment exist?

What does semantic role resolution answer?

Which discovered objects are Start, Stop, Setpoint, PV, equipment, etc.

What does scenario matching do?

Determines whether the discovered product fits a known reusable lifecycle like PID.

What does discovery do with PID modes?

Finds the selector and available modes.

Who decides which PID mode to test?

Test policy—or a policy can automatically test all discovered modes.

What does discovery do with Start and Stop?

Finds their controls, actions, geometry, and associated runtime state.

What happens to equipment.members after discovery?

In the intended discovery architecture, equipment members would be enumerated automatically. Today pidLifecycle.ts is reusable across configured members, but discovery of those members is separate work. Discovery automatically enumerates the equipment group

Why is G1 equipment handling already reusable?

The scenario detects whichever configured pump activates instead of hardcoding P01/P02/P03.

• In src/scenarios/pidLifecycle.ts. pidLifecycle.ts should consume resolved/configured equipment,

What is flowMapping trying to represent?

Semantic roles such as Setpoint, Process Value, status, and related signals.

Does discovery replace pidLifecycle.ts?

No. It generates the bindings/config that pidLifecycle.ts already consumes.

What if discovery finds a new animation with no verifier?

Mark it unsupported until a reusable verifier is added.

What is the future G1 flow?

Discovery → product model → scenario planner → generated config → pidLifecycle.ts.

What is an ROI

ROI is a bounded region of interest containing only the pixel the verifier needs to measure

Why verify semantic state and pixels separately?

A correct runtime value does not guarantee correct rendering, and correct looking pixels do not prove the underlying value changed

What does EaeLiveDataObserver do?

Independently observes live EAE runtime values/ReferenceStrings.

Q: What does CanvasInputMutation do?

A: Performs operator-like canvas input, commits the value, verifies readback, and supports restoration.

Q: What does quantitativeFillVerifier do?

A: Measures rendered Fill quantitatively from pixels and determines whether it matches the expected state.

Q: What is a verifier?

A: Reusable logic that judges whether an observed result is correct.

Q: Why are normalized coordinates still useful if discovery becomes automatic?

A: Discovery can derive normalized coordinates dynamically; the execution layer still needs them to interact reliably with the current canvas geometry.

Q: What does boundingBox() give us versus normalized coordinates?

A: Bounding box gives the current canvas position/size; normalized coordinates locate a target relative to that canvas.

Q: What is a ReferenceString?

A: A runtime identity used to observe a specific EAE value through live data

Q: What was the main architectural pivot in Phase 2?

From testing one product lifecycle to building reusable verification for supported HMI behaviors

Why did controlled graphics matter?

They gave deterministic inputs and deterministic expected outputs for testing animations and interactions.

What is a controlled graphic?

A Studio-authored fixture designed to isolate a known behavior. It may support controlled mutation if a safe mutation path is proven; otherwise it can still support observation/verification.

What is the basic controlled-graphic chain?

Custom property → behavior/animation → runtime rendering → generic verifier.

What is the key difference between Fill and Blink verification?

Fill is spatial measurement; Blink is temporal/state-change measurement.

What does Blink prove?

The rendered visual state changes over time in response to its controlling state.

What does Indicator Light prove?

A discrete visual state can switch between on/off states based on its source.

Q: What is the main difference between controlled testing and integration testing?

• A: Controlled testing isolates one known behavior; integration testing checks whether framework concepts still work in a more complex ViewApp.

Q: What was ViewApp_006 mainly used to prove?

• A: Integrated Alarm/Trend baselines and selected cross-layout/framework behavior in a multi-pane application.

What did the Alarm baseline prove?

Structural presence, readiness, and stable rendering of the Alarm component.

What did the Alarm/Trend baseline NOT prove?

• Alarm generation, acknowledgment, clearing, filtering, or full operational workflows.

What does “baseline test” mean?

A bounded proof of essential structure/readiness/stability, not a comprehensive functional test.

Q: What is the Phase 2 → Phase 3 progression?

• A: Phase 2 proved generic behavior contracts; Phase 3 tested integration, component baselines, and reuse across different contexts.

Q: What is the discovery connection to Phase 3?

• A: If generic execution already works across multiple compatible targets, discovery can later automate finding and configuring those targets.

Q: Why is EaeLiveDataObserver reusable?

• A: Its observation logic is generic; different products supply different runtime references.

Q: Does EaeLiveDataObserver know what a property means?

• A: No. Product/config/discovery supplies the semantic role; the observer just reads the runtime value.

Q: Why preserve evidence from accepted runs?

• A: To support reproducibility, regression investigation, and defensible status claims.

42. Q: When can you call something a framework defect?

A: Only when evidence isolates the reusable framework logic as the cause—not simply because an integrated test failed.

Q: What are the five framework layers?

A: Config = what/where; Helper = how; Scenario = sequence; Verifier = judge; Spec/Test = wire + assert.

Q: What is the difference between a helper and a scenario?

A: A helper performs one reusable technical operation; a scenario coordinates multiple operations into a workflow.

Q: What is configuration in the old architecture?

A: Engineer-authored target/product information consumed by generic framework code.

Q: What changes about configuration under discovery?

A: The equivalent config still exists, but the framework generates much of it instead of the engineer manually authoring it.

Q: What is semantic state?

A: The actual underlying runtime state/value, separate from how it is visually rendered.

Q: What is visual state?

A: What the operator sees rendered in the HMI.

Q: What is the G1 Flow PID lifecycle?

A: Baseline → setpoint → Start → active state → equipment activation → process response → Stop → rundown → restoration.

Q: What is a semantic settlement gate?

A: After a UI action, wait until the corresponding runtime value reaches the expected state before continuing.

Q: Can this framework work on any product today?

A: Not automatically. It supports products that fit implemented scenario/verifier contracts and currently may require product-specific config.

Q: What is the biggest current limitation?

The execution/verifier framework is mature, but not every target-specific authored fact can be discovered automatically from EAE or Universal HMI. The current architecture therefore combines deterministic discovery with explicit manual fallback.

What is the new hybrid discovery architecture?

Caller context → discovery providers → partial generated contract → manual overrides for unresolved fields → resolved config → existing deterministic runtime framework.

Why is it called hybrid?

Because some fields can be established automatically, while other required target-specific facts may need explicit manual input when no deterministic provider can establish them.

What is FieldResult?

A generic wrapper representing the state of one discovered/configured field, including its status, optional value, provenance, evidence, and reason.

What is fieldContract.ts?

The new generic field-level contract layer. It defines how individual fields report whether they are discovered, unresolved, not applicable, or blocked.

What are the field statuses?

DISCOVERED, MANUAL_REQUIRED, NOT_APPLICABLE, and BLOCKED.

What is field provenance?

Metadata describing where a field value came from, such as caller input, deployment metadata, runtime discovery, framework mapping, or manual override.

Why is provenance important?

It prevents inferred/manual values from being mistaken for authoritative or automatically discovered facts.

What is FRAMEWORK_MAPPING provenance?

A deterministic value derived from framework knowledge rather than target discovery. Example: once behavior=Fill is legitimately resolved, the framework can map it to the Fill verifier.

What is MANUAL_OVERRIDE provenance?

The engineer explicitly supplied the value because automatic providers could not resolve it.

What does manualOverrideProvider.ts do?

It merges explicit engineer input into only the unresolved parts of a partial contract instead of requiring the engineer to rewrite the whole config.

What does resolveWithManualOverrides() do?

It applies validated manual overrides to unresolved field results while preserving provenance.

Why shouldn't a manual override silently replace a discovered value?

Because that would destroy provenance and could hide a disagreement between discovery evidence and human configuration.

What is FillOnboardingContract?

The field-level representation of everything needed or potentially relevant to onboard a Fill behavior before producing the old executable FillConfig.

What does buildFillOnboardingContract() do?

Builds the current partial Fill onboarding contract from caller context and whatever accepted providers can legitimately establish.

What does fillExecutableConfig.ts do?

Converts a sufficiently resolved Fill onboarding contract into the existing FillConfig consumed by the deterministic Fill verifier.

What was wrong with the old fillConfigGenerator fallback pattern?

It could silently fill missing discovery values using a predefined historical Fill config instead of explicitly reporting them as unresolved/manual.

Why did we investigate EAE authored metadata?

Ideally, EAE would provide behavior, source binding, range, direction, geometry, and input metadata so the framework could generate config directly from authored truth.

What did the EAE metadata investigation conclude?

We found useful topology/navigation metadata, but no supported/self-describing machine-readable interface on this environment exposing the complete authored graphic behavior contract.

Did EAE itself lack the information?

No. Studio obviously knows it. The issue is that we did not identify a supported programmatic interface that exposes those facts to our framework.

What did Linux runtime/network investigation find?

Universal HMI exposed navigation/topology data and runtime plumbing, but no self-describing Fill source/binding/range/direction/geometry contract.

FillConfig

Existing executable configuration consumed by the quantitative Fill verification path. The new discovery layer should eventually generate/resolve this instead of an engineer manually writing it.

quantitativeFillVerifier

Existing generic pixel-based Fill verification logic. Do not replace it with discovery logic.

measureFillExtent()

Core quantitative operation that measures Fill extent from decoded image data using the Fill configuration.

BrowserNativeScreenshotDecoder

The accepted screenshot-decoding implementation used to convert PNG evidence into pixel data for quantitative verification.

CanvasInputMutation

Generic operator-style canvas mutation helper used when a valid editable target and mutation topology are known.

SignalR / eaeHub

Real-time runtime messaging mechanism Universal HMI uses for some server/browser communication and live data.