Amazon and Paramount+ Measurement, Incrementality, and Testing Framework

Experimentation Framework and Measurement Pillars

• Amazon Incrementality Measurement (AIM):

  • Generates a randomized treatment and control group via ghost bidding mechanisms.
  • Provides an unpolluted baseline read on whether paid media directly causes behavioral changes.
  • Serves as the core foundational pillar for evaluating subscriber retention.

• Amazon Marketing Cloud (AMC):

  • Grants direct access to integrate event-level and user-level operational signals.
  • Enables integration of AIM treatment and control cohorts into AMC environments to compute critical downstream performance metrics, such as streaming hours and platform consumption signals.

• Causal Modeling Frameworks:

  • Employed in scenarios where randomized controlled trials (RCTs) cannot be implemented, allowing historical spend evaluation without live experimentation.
  • Primary mechanism planned for paid acquisition analysis (e.g., Ultimate Fighting Championship [UFC] media assets).
  • Campaign Lift: Constructs a synthetic matched control cohort composed of behaviorally similar users exposed to the campaign, generating a modeled counterfactual.
  • Graphical Modeling: Establishes a directed network of cause-and-effect dependencies to identify baseline activity in the counterfactual absence of an advertisement.

• Deterministic Audience Split:

  • Resolves measurement gaps stemming from global accommodation constraints and multi-cell test designs.
  • Employs AMC to contrast downstream outcomes between cohorts eligible for advertising exposure and true holdout groups.
  • Requires that target users have received Paramount+ advertising impressions within the preceding $13$-month window due to historical data ingestion parameters.

Quarter-over-Quarter Testing Evolution and Retention Metrics

• Quarter-over-Quarter Testing Progression:

  • Quarter 1: Deployed $6$ concurrent parallel tests to determine initial operational benchmarks.
  • Quarter 2: Consolidated testing architecture into $1$ unified test leveraging Q1 baseline findings.
  • Quarter 3: Executed a single point-in-time multi-cell test centered on an at-risk subscriber cohort, directly comparing two distinct ad formats.
  • Quarter 4: Transitioned to a recurring monthly testing paradigm utilizing three-cell test structures.
  • 2027 Objective: Standardize on a singular, quarter-over-quarter methodology to ensure longitudinal comparability and systematic performance tracking.

• Key Retention Benchmarks and Performance Trajectory:

  • Q1 Baseline Benchmarks (2026 Baseline):
  • Cost per incremental renewal: $\$148$.
  • Cost per incremental streaming hour: $\$2.47$.
  • Streaming TV Plus registered the lowest cost per incremental renewal across all evaluated ad products.
  • Q2 Performance Shifts:
  • Established an efficiency improvement target of $10\%$.
  • Attained a $24\%$ reduction in cost per incremental renewal.
  • Future Iteration Targets:
  • Continued target trajectory calls for an ongoing $10\%$ quarter-over-quarter cost efficiency gain.

• Extended Tracking Windows and Calculation Methodology:

  • $90$-Day Post-Test Window Analysis:
  • Cost per incremental renewal exhibits significant downward trends at $90$ days post-test relative to day-of-completion reads, driven by widening behavioral divergence between treatment and control groups over time.
  • Renewal Accounting Logic:
  • Renewals represent ongoing subscription continuity (active subscribers who refrain from churning out of Paramount+) rather than churn-and-return transactions.
  • Renewal metrics track aggregate renewal volume rather than unique subscriber counts. If a single user renews $3$ times within a $90$-day window, this is registered as $3$ renewals.
  • Mathematical illustration: If $100$ individuals in a control group generate $100$ total renewals, while $100$ individuals in a treatment group generate $300$ total renewals, the campaign yields $200$ incremental renewals.
  • Cohort Distinctions:
  • Metrics showing $\$0.88$ and $\$0.84$ apply specifically to the clean onboarding subscriber cohort isolated in Q1 and tracked into Q2, distinct from full experiment window calculations.

Multi-Cell Testing, Ad Products, and Extended Window Analysis

• Q3 Multi-Cell Experiment Structure and Findings:

  • Segmented the addressable ad-eligible universe into balanced partitions:
  • $50\%$ of the universe was served exclusively Streaming TV Plus.
  • $50\%$ of the universe was served exclusively Fire TV Plus.
  • Evaluated against a mixed control group.
  • Renewal rates within a $15$-day experiment window:
  • Control cohort baseline renewal rate: $18\%$.
  • Fire TV Plus cohort renewal rate: $21\%$.
  • Streaming TV Plus cohort renewal rate: $18\%$, reflecting a $4\times$ impact trajectory among exposed populations.

• Ad Product and Format Characteristics:

  • Streaming TV Plus: Consistently surfaces as the superior format for driving down incremental renewal costs and elevating streaming hours, particularly effective at reaching audiences outside native hardware environments.
  • Format Comparisons:
  • Future multi-cell roadmaps include direct comparative tests between Streaming TV Plus and Prime Video Premiere (PVP).
  • Creative formats differ in cost-efficiency: static display executions feature lower Cost Per Mille ($\text{CPM}$) rates and generally capture lower-funnel efficiencies, whereas video formats drive expanded lift over longer attribution windows.
  • Cross-environment performance variations exist across feature rotators, native Fire TV UI, the Prime Video App, Demand-Side Platform (DSP) off-Amazon inventory, and PVP.

Acquisition Incrementality and Media Mix Strategy

• Acquisition Efficiency Opportunities:

  • Media Mix Modeling (MMM) demonstrates a $15\%$ cost efficiency advantage per incremental acquisition through Amazon channels compared to broader paid media channels.
  • Display inventory and Amazon DSP video inventory account for only $10\%$ of Paramount's aggregate media allocation, indicating a significant under-investment relative to their efficiency potential.
  • DSP programmatic buying incorporates third-party ($3\text{P}$) off-Amazon inventory alongside Amazon-owned inventory; current implementations utilize always-on DSP regular and plus inventory, with active testing on single-tether optimization architectures.
  • Existing MMM datasets reflect Prime Video Channels (PVC) spend exclusively, omitting Direct-to-Consumer (D2C) spend executed via self-service Amazon DSP.

• Acquisition Framework Pillars for 2027:

  • Media Mix Reallocation: Scale spend across DSP video, display, and Streaming TV to achieve a target $10\%$ acquisition efficiency lift.
  • Causal Modeling Integration: Quantify the extended-window, non-immediate impacts of video media without running disruptive operational holdouts.
  • First-Party Signal Enhancement: Ingest complete first-party ($1\text{P}$) data feeds to execute absolute audience suppression of existing subscribers, guaranteeing that $100\%$ of acquisition ad spend targets net-new consumers.

• Performance Metrics for Acquisition and Win Backs:

  • Q3 Win Back test Cost Per Acquisition ($\text{CPA}$): approximately $\$324$.
  • Q3 Sponsored Tiles test $\text{CPA}$: approximately $\$180$.
  • Pure acquisition test datasets from late September are scheduled for ingestion and analysis in subsequent operational cycles.

Share of Voice (SOV) and Guaranteed Media Measurement

• Methodological Limitations of Randomized Controlled Trials on Guaranteed Media:

  • Standard RCT experimentation cannot be deployed on Share of Voice (SOV) or guaranteed programmatic sponsorships due to an inability to control delivery variables (e.g., impression frequency caps, absolute impression counts, delivery pacing).
  • Even when isolating designated market areas (DMAs) or testing split allocations (e.g., $25\%$ SOV versus $75\%$ SOV), RCT holdouts cannot isolate exposure variables reliably.

• Alternative SOV Testing Frameworks:

  • Causal Modeling on High-Impact Media: Deployed on guaranteed media placements, specifically utilized for the September 19 UFC live sports broadcast, removing the requirement for synthetic holdouts or delivery restrictions.
  • Geo-Targeted Testing: DMA-level execution scheduled for October 31, preparing structural measurement for high-tier live sports inventory.
  • Tier-Based Incrementality Measurement: Paramount evaluating third-party incrementality testing vendors (e.g., Haus, Neustar) utilizing tier-based geographical grouping, requiring partner whitelisting and compliance checks within Amazon's approved partner network.

First-Party Data Integration and Identity Resolution

• Structural Audience Disconnect Between Platforms:

  • Prime Video Channels (PVC) accounts for roughly $30\%$ (one-third) of Paramount's aggregate subscriber volume.
  • Paramount Direct-to-Consumer (D2C) platforms account for the remaining $70\%$.
  • Operational deficiency: Acquisition, retention, and win-back behavioral signals generated across the $70\%$ D2C subscriber base are not systematically fed back into Amazon to optimize media served across the $30\%$ PVC subscriber base.

• Data Ingestion Channels and Signal Loss:

  • Paramount generates first-party signals via Adobe Conversions API (CAPI) and tracks mobile application installs via the Kochava Mobile Measurement Partner (MMP) integration for mobile inventory.
  • These signals currently operate disconnected from Amazon optimization loops, creating severe attribution blind spots.
  • Ingesting off-Amazon D2C events through AMC via HighTouch provides full bi-directional visibility.

• Identity Spine and Match Rate Comparisons:

  • Current suppression relies strictly on LiveRamp audience uploads executed on a recurring $7$-day cadence, transmitting active subscriber lists without automated suppression delete tags.
  • LiveRamp forces tradeoffs between match fidelity and audience scale.
  • Amazon provides direct, deterministic match resolution leveraging verified user identity markers (hashed emails, verified household postal addresses, telephone numbers) linked across hardware ecosystems (Fire TV, Roku, Samsung, LG).
  • Amazon Match ID allows ingestion of third-party identifiers (e.g., LiveRamp RampIDs) directly into the clean room to execute deterministic joins against internal graphs.

• Strategic Entity Consolidation and Multi-Tenant Privacy:

  • In corporate merger environments (e.g., Paramount and Warner Bros. Discovery [WBD] / HBO), Amazon's clean-room architecture enables cross-service audience modeling.
  • Amazon maintains isolated Prime Video Channel audience datasets for both Paramount+ and HBO; subject to legal data permissions and whitelisting, AMC instances can coordinate cross-brand suppression or activation without violating regulatory boundaries (e.g., Video Privacy Protection Act [VPPA], Children's Online Privacy Protection Act [COPPA]).

Industry Case Studies: Footwear, Telecom, and Sports Betting

• D2C Footwear Enterprise Case Study:

  • Integration Structure: Integrated off-Amazon D2C transactional events directly with Amazon viewing and shopping signals inside AMC.
  • Performance Metrics:
  • Achieved a deterministic match rate of $94\%$.
  • $62\%$ of all recorded purchasers on Amazon.com were validated as strictly incremental consumers.
  • Outcome: Scaled total gross sales across both native D2C and Amazon.com storefronts without cannibalizing existing customer traffic.

• Telecommunications and Cable Provider Case Study:

  • Integration Structure: Deployed first-party D2C converter feeds directly through Conversions API (CAPI) to execute high-fidelity audience suppression.
  • Outcome: Uncovered extensive populations of previously untracked active subscribers, drastically minimized wasted ad spend across streaming footprints, and improved overall media incrementality by multiple percentage points.

• Sports Betting Enterprise Case Study:

  • Integration Structure: Weekly ingestion of first-party gaming events into AMC, including conversions, first-time deposits (FTDs), account activations, profit boost redemptions, and promotional plays.
  • Performance Metrics:
  • Achieved deterministic match rates of $80\%$ (with historical address-matched cohorts reaching between $80\%$ and $95\%$).
  • Recorded a $36\%$ efficiency improvement in CPA for first-time deposits over time.
  • Produced over $\$30{,}000{,}000$ in aggregate media cost savings within a single fiscal year.
  • Operational Impact: Direct integration eliminated the necessity of intermediary third-party ad tech vendors, extinguishing external platform licensing fees.

Budget Allocation, LTV Modeling, and Product Roadmaps

• Media Mix Modeling (MMM) vs. Test-and-Learn Applications:

  • MMM dictates macro-level operational budget decisions, channel-specific reductions, and media efficiency hurdles (Return on Ad Spend [ROAS]).
  • Granular over-the-top (OTT) streaming inventory is not yet fully programmatic-operationalized via automated MMM models.
  • Incrementality experimentation serves as an ongoing calibration pulse-check to directionalize and reconcile monthly MMM readouts.

• Customer Lifetime Value (LTV) Optimization Dynamics:

  • Programmatic bidding requires synthetic conversion value indexes based on regional LTV divergence to prevent misallocated media budgets.
  • Example: Acquisition costs in Mexico may stand at $\$4$ CPA compared to $\$12$ CPA in the United Kingdom, yet UK customer cohorts yield a $6\times$ higher LTV, justifying aggressive spend allocation toward higher-CPA markets.
  • Current platform constraints limit fluid cross-region programmatic capital reallocation, highlighting the need for custom conversion values and dynamic frequency/bid modifiers.

• Amazon Measurement Product Roadmap:

  • Developing integrated Multi-Touch Attribution (MTA) architectures that programmatically ingest randomized controlled trial (RCT) incrementality weights into downstream bidding engines.
  • Transitioning from isolated, post-hoc incrementality studies toward always-on quarterly studies featuring nested multi-cell tests.

Questions and Stakeholder Discussions

• Deterministic Split Application to Acquisition Testing:

  • Question: Can the deterministic audience split method be applied directly to acquisition tests planned for Q4?
  • Answer: Affirmative. However, addressable cohorts must be constrained to individuals exposed to Paramount+ advertising within the prior $13$ months due to the operational lookback limits of the AMC data spine.

• Industry Availability of AMC and AIM Joint Measurement:

  • Question: Is the unified integration of AIM ghost bidding data within the AMC clean room unique to Paramount, or is it commercially available to other marketplace partners?
  • Answer: Other enterprise partners currently leverage this joint capability. The offering was developed in response to widespread advertiser demands to replace inefficient, paid Public Service Announcements (PSAs) previously required to assess ghost-bidding lift.

• Simultaneous Execution of AIM and Deterministic Splits:

  • Question: Can a single user receive both AIM testing and deterministic audience splitting simultaneously to measure retention and acquisition concurrently?
  • Answer: Negative. Individual users cannot be allocated to both methodologies at the same time. The testing frameworks must be bifurcated across non-overlapping, distinct audience segments.

• Format Superiority: Video vs. Display in Retention Contexts:

  • Question: Do video formats structurally deliver higher incrementality relative to display or lower-funnel performance formats within retention workflows?
  • Answer: Incrementality performance remains channel- and placement-specific. Lower-funnel static creative typically registers significantly lower CPMs and superior cost efficiency per incremental renewal. Video drives long-tail awareness and retention value, but broader testing is underway to assess whether retention media justifies continued investment compared to reallocating capital into broad-reach acquisition.

• Evaluation of Third-Party Incrementality Measurement Partners:

  • Question: Which outside partners are being assessed to handle tier-based SOV testing, and what onboarding protocols are required?
  • Answer: Haus and Neustar are under evaluation. Third-party testing partners must be formally registered and approved within Amazon's measurement partner directory prior to operational deployment to ensure security and privacy compliance.

• Privacy Compliance for Direct Streaming Data Integrations:

  • Question: Given legal and industry privacy statutes like VPPA and COPPA, are leading streaming publishers currently executing direct first-party data uploads into AMC?
  • Answer: Multiple enterprise streaming clients (both major entities and long-tail publishers) currently execute direct, multi-field first-party data integrations into AMC under compliant privacy frameworks.