C40 CIA MODELS

Def’n of Model

Practical representation of relationships among entities using Financial, Economic, Mathematical, Statistical (FEMS) concepts

• Composed of → SIR

  • Model Specification

  • Model Implementation

  • Model Runs

Model Implementation

Systems developed to perform calculation for a model specification

Model Risk

Risk that user will draw inappropriate conclusions due to limitations of model

• Measured by Severity & Likelihood of model failure

• Model is a simplification of reality → risk always exists

Model Run

Inputs & outputs of a model implementation

Model Specification

Description of components of a model & their interactions

Model vs Calculation + Exs

Main Distinction: model requires more documentation

• How it’s chosen & used

Ex of Model

• Using GLMs to segment business

• LDFs

Ex of Calculation

• Adding a column of numbers

• Calculating least squares regresion line

4 Criteria to assess Severity of Model Failure

FINF

• Financial significance of results

• Importance of model

• Freq of use

• Non-financial impact

4 Criteria to assess Likelihood of Model Failure

ACED

• Adequacy of testing

• Complexity of model

• Expertise of user

• Documentation

Why should you reuse models if possible?

Saves time & money

Validate Model’s Data Input → Considerations

Sufficiency

• Data meets model specification?

• If model used repeatedly, are data always in consistent format?

Reliability

• Reconcile to other sources

• Compare to prior periods

• How are missing data handled?

Validate Model Assumptions → Considerations

• Regular peer review

• Using intended assumptions for each run?

• Unchanged unless meant to be changed?

Validate Model Results → Considerations

• Outputs consistent with inputs?

• Are results as expected? → Direction & Magnitude

• Materiality of errors

• Attribution analysis → can chg in results from prev period be explained?

Do actuaries have more control over Severity or Likelihood of model failure?

Likelihood

• Choose more reliable model

• Test more thoroughly

When to reassess model risk rating?

• When model fails

• Regular cycle (ex: every 5 years)

• When model use changes

Types of Models to Validate

• New (or substantially changed) Model

• Existing model used in new way

• Model approved by others

• Model outside actuary’s expertise

Validate a New (or substantially changed) Model

SILD

Review Specifications

• Data, methods, assumptions

Validate Implementation

• Test & compare w/ other tested models to verify calculations

Deal w/ Limitations

• Understand range of uses model was designed & tested for

Keep Documentation

• Why it was chosen & any limitations

Validate an Existing Model used in a new way

• Review limitations in new application that were not relevant before

• Consider if risk rating has changed

Validate a Model approved by others

Review & approve validation report

Validate a Model outside actuary’s expertise

Determine appropriate level of reliance on others

• Are they an expert in their field?

• Extent model has been reviewed by experts

• Risk rating of model

Make reasonable attempt at understanding model’s:

• Basic workings of model

• Testing & validations completed

Ex: credit scoring model

Sensitivity Testing → Purpose

• Validate model

• Understand relationship between inputs & outputs

Sensitivity Testing Assumption → Considerations

• Test assumptions outside expected range

• Consider interplay between related assumptions

• Cases w/ non-linear relationship between inputs/outputs

Stochastic Models → What to review

• Results from a carefully chosen sample of deterministic scenarios

• Distr of output results for reasonability

• Scenarios that lie near a boundary

Model Risk Rating Schemes

Unidimensional

• Give scores on several risk factors

• Sum to get total

Two-Dimensional

• Assess severity & likelihood of model failure separately

• Balance the two to get risk rating