Comprehensive Study Notes on AI Integration, Governance, and Future Infrastructure

Global Competitiveness and Internal Growth at CDL

Recent developments at CDL highlight the company's position at the forefront of technological innovation, particularly in the use of Model Control Planes (MCPs). Executives Matt Bright and others noted that while the rapid pace of change can be overwhelming, the organization is leading the industry. During a professional visit to Germany, CDL representatives shared insights on their utilization of Dynatrace as an MCP with a major German bank. This bank, despite being a highly regulated entity seeking similar agility, struggled to replicate CDL's speed, underscoring the company's competitive edge. The rate of change within the internal harness has been significant, with 89 merge requests recorded over the last six months alone. This equates to an average of one technical change every other day. Notably, approximately 20 to 30 of these changes were driven by community ideas rather than top-down architecture, demonstrating a highly collaborative environment. This level of engagement extends to communication platforms; the MS Teams chat for these updates became so popular that it broke Microsoft’s limit for chat participants, making it the second most popular channel in the company, surpassed only by the zone menu.

The Philosophy of Adaptation: Finna.ai and Intercom

The organizational strategy centers on the principle of "adapt or die," a sentiment echoed by Des Trainer, a key figure from Finna.ai and Intercom. Finna.ai was originally an internal AI research and development spinoff of Intercom, a prominent call center application provider. The company's recent rebranding and acquisition by Salesforce for millions of dollars within just three years illustrates the power of rapid adaptation. CDL is applying these same principles to its internal operations, ensuring the harness and team workflows evolve to remain relevant. The goal is to move away from "Wild West" style changes where the only feedback loop was manual chat responses, towards a more rigorous and deterministic systematic approach.

Engineering Evaluations and Regression Testing Frameworks

To manage high rates of change without sacrificing quality, the team has implemented an evaluations test for regression checking, modeled after the AI testing blueprint. This process involves the Artificial Intelligence Development Life Cycle (AIDLC), which consists of three phases: Inception, Construction, and a third phase associated with operations. Initial attempts to use an "LLM as a Judge" method—where one Large Language Model (LLM) reviews the responses of another against a ground truth—proved inefficient. This method was costly, required expensive models, and took approximately $2 \times 10^0$ to $3 \times 10^0$ hours to run. Furthermore, the judge model displayed a tendency to fabricate answers or "mark its own homework" if the primary model was too slow. Consequently, the team transitioned to a deterministic "Golden Example" evaluation method, which compares outputs against a static ground truth, ensuring more reliable regression packs.

Hardware Capabilities and the Economics of Local Inference

CDL is exploring local inference to manage the rising costs and token usage associated with AI. Industry data from Google indicates that engineers are using four times as many tokens as they were six months ago, a trend mirrored at CDL. Local inference involves running models directly on employee laptops via tools like OLAMA, rather than relying on AWS-hosted models. This transition is dependent on two factors: hardware and model efficiency. New M5 processors feature integrated neural processors capable of reaching the target speed of $50 \text{ tokens/s}$. Speeds below this threshold result in perceptible lag for the user. For context, Gemini 3.6 Flash operates at roughly $260 \text{ tokens/s}$, while Opus 5 runs at approximately $50 \text{ tokens/s}$ to $60 \text{ tokens/s}$.

Model Parameters, Memory, and Quantization Metrics

The technical requirements for running models locally involve managing parameter counts and memory. Historically, a dense model required roughly $1 \text{ GB}$ of RAM per $10^9$ parameters, as every request activated all parameters. Modern Mixture of Experts (MoE) or sparse models optimize this by only highlighting specific experts for each character inferred, reducing memory requirements by $60\%$ to $80\%$. For example, a $3.5 \times 10^{10}$ parameter Quen model might only require $5 \text{ GB}$ of memory. Additionally, quantization acts as a form of memory compression; however, higher compression ratios lead to a loss of accuracy. If a model with $2.1 \times 10^{11}$ parameters is quantized down to a 1-bit representation, the loss would be so high the model would become unusable, despite fitting into $128 \text{ GB}$ of RAM. CDL aims to balance these factors to achieve Sonnet-level results locally.

The Evolution of Harness Version 2: Agentic Workflows

The next iteration of the CDL harness, Version 2 (v2), represents a shift from monolithic workflows to modular, agentic structures. This version utilizes sub-agents and specialized "skills," allowing for over 20 independent capabilities to be triggered and parallelized. This shift redefines the "human in the loop" model. In current workflows, humans are involved from beginning to end. In v2, humans provide the initial intent, the agent generates the output, and the human verifies the final result. While the generation process is automated, human accountability remains a constant. Lessons learned from managing nine major MCP vendors, including the contrast between Orca’s complex 50-tool interface and GitLab’s streamlined Orbit MCP, are being applied to this new architecture to avoid past inefficiencies.

Security Governance and the GRC/AIG Framework

The Governance, Risk, Compliance, and Security (GRC) group, in collaboration with the Artificial Intelligence Group (AIG), is establishing new frameworks to ensure CDL is fit for the future. As the company engages with entities like the Post Office and international banks, security becomes the paramount concern. Key governance measures include Approval Gates to prevent "agent sprawl" (creating agents without clear utility) and Control Reviews to categorize agents by risk. A simple document summarization agent carries low risk, whereas an agent connected to production APIs carries a significantly higher risk profile. Ownership and recertification are also critical; agents must be treated as living software that requires ongoing oversight even after going live.

Cybersecurity Threats, Prompt Injection, and Blast Radius

Cybersecurity at CDL is shifting focus from traditional vulnerabilities to AI-specific threats like prompt injection and excessive autonomy. Prompt injection occurs when malicious text within a document or code influences an LLM to perform unauthorized actions. To counter this, CDL is developing threat modeling techniques to understand the "blast radius" of an agent. For instance, an HR onboarding agent with excessive permissions could be targeted via a CV containing hidden malicious text, potentially exposing the entire organization. Therefore, agents should be viewed as digital identities—analogous to human employees—requiring strict access controls. The company is automating the enforcement of its 21 current policies (comprising approximately 2,000 items) by integrating them into the context store, ensuring that agents automatically check for requirements like encryption during the development process.

Questions & Discussion

Job Security and AI Substitution: A listener expressed concern regarding job security and the potential for AI to render human roles obsolete. Rob and others clarified that while AI supercharges output—aiming for a 3x increase in productivity—the company has limitless roadmaps and aspirations for global growth that require more human effort, not less. They emphasized that humans remain the accountable party for all AI-generated work, and roles are evolving rather than disappearing. Comparisons were made to the transition to cloud computing, where roles changed from hardware maintenance to higher-level management.

Output Metrics vs. Cost: Regarding the reported $187 \%$ output increase, it was asked if costs rose proportionally. The response indicated that while AI costs (such as Kiro and tokens) are significant, they only represent a single-digit percentage increase relative to the massive gains in throughput. The team is continuously monitoring "token economics" and exploring local LLMs to further optimize these costs.

Sustainability and Environment: A question was raised about the environmental impact of AI energy consumption. The speaker noted that by using hyperscalers like AWS, CDL benefits from advanced liquid cooling and scaling efficiencies that exceed what the company could achieve on-site. They cautioned against using large models like Opus for minor tasks (e.g., writing an email) and encouraged using smaller, more efficient models to reduce the "carbon footprint" of their daily usage, jokingly referencing the energy waste of generating "funny cat videos."

Training and Skill Gaps: Some staff felt left behind or lacked access to training. The leadership group noted that introductory sessions and copilot training have been available for nearly a year, with high completion rates (around $92\%$). They encouraged anyone needing more support to reach out to their Team Development Lead (TDL) or SEM and to utilize internal sites and "agentatons" for hands-on experience.

Management and the Pod Structure: A manager felt they were losing control over their team due to the new "pod" structure. Executives explained that the pod concept was designed to prevent "brain drain" by facilitating knowledge sharing between senior engineers and apprentices while AI handles the bulk of code writing. They acknowledged that leadership roles are shifting to focus more on people management and high-level technical strategy.

Data Protection and International Models: Regarding the use of Chinese LLMs or other external labs, Mansha explained that all models are accessed through AWS Bedrock, which provides a secure sandbox and ensures that data privacy remains under CDL's control. Standard contracts are reviewed to prevent prompts from being used for malicious monitoring or model training by third parties.