Inside the CAS’s New Monograph on Exam 8
recently had the pleasure of reading the new CAS Monograph, “From GLMs to Comprehensive Insurance Pricing: Techniques and Challenges,” by Alan Chalk, David Deacon, Montserrat Guillen, and Max Martinelli. The monograph will be part of the Exam 8 course outline starting in October of this year.
The monograph is structured around three case studies used to explore major practical challenges in modeling once you get beyond the basic generalized linear model (GLM) fit. Within the monograph, there is robust discussion of data validation splits, baseline modeling, penalized regression, black-box sub models, and more. The monograph does an excellent job of calling out the real-life issues one runs into when running pricing models and the compromises one must make.
I recommend the monograph to anyone currently performing insurance pricing or anyone with an interest in why pricing GLMs is actually pretty tricky, even if you think GLMs are easy. I recently chatted with the main technical author, Alan Chalk, about the development of the monograph. I also confess, as the lead of a pricing data science team, I couldn’t help showing my cards a little bit as I poked at some of the more slippery topics in the monograph.
Andrew Somers: What motivated you to write this monograph?
Alan Chalk: Giving something back to the profession is always a motivator. But beyond that, writing something like this together with phenomenal co-authors is fun and a great learning experience.
AS: The introduction includes a disclaimer that “the potential negative consequences of mindlessly creating rating plans without human input and review remain as large now as they ever have been.” That theme recurs throughout. Is this a deliberate counter-message to where they see the industry headed with automation?
AC: Not really. It’s another fact of life — I have made mistakes in my career in relying on the output of statistical methods without properly running rates past underwriters or other experts. And we just want to encourage others not to go down the same route. I once gave a presentation where the whole content was on mistakes I had made. In software engineering tests, we talk about edge cases. The same goes for the output from statistical or machine learning models. We need to know how edge cases arise and then how to protect ourselves from them.
AS: The case study data in the monograph does not always behave well, which is very true to life. I found the honest discussion about this topic unusual for a monograph but refreshing. In particular, you couldn’t get GLMMs [generalized linear mixed models] to converge reliably. Is that a software maturity issue, a data sparsity issue, or something more fundamental?
AC: This is a life issue. Time is short, whether getting a result out at work or writing a monograph. Once we have a method which works, we always need to think: how much more benefit am I likely to get by doing this a slightly better way. Most of the methods for HCCVs (high-cardinality categorical variables) discussed in the monograph are simple and mechanical and, when used carefully in a penalized regression framework, provide reliable outcomes (subject to edge cases). GLMMs are not simple. Whilst they should be an improvement, if we can’t get them to converge and provide consistent results, it’s probably time to move on.
AS: Another time data did not behave well was with PCA [principal component analysis]. You noted that although it didn’t provide a great solution, it was an interesting way to check the quality of data preparation. Is using it as a standard check reasonable or was it just sort of circumstantially useful in this example?
AC: I think that the more tools an actuary has in their toolbox, the better. And then use them whenever each is right. I remember one time a team I was working in was doing a manual audit trying to work out which combination of factors led to a systems problem. I suggested a decision tree, and the answer fell out almost immediately. PCA is not complicated and so it does not take long to learn, so it’s a no-brainer. Actuaries should be aware of it and methods like it — and then use them as fits the situation.
AS: You picked pseudo-R² calculated on held-out folds and stuck with it throughout. I know there’s a trade-off of picking one metric for simplicity versus showing too many metrics. In reality we tend to pick a couple of metrics to evaluate our models in a few different ways. How did you think about that tension, and what would you say to someone who asks why not also report Gini, double lift, residual studies, or quantile loss-ratio plots in the case studies?
AC: In one role I held in an insurance company in a somewhat advisory role, on the first day that I went in, I was asked exactly this question. They typically looked at around 10 performance metrics, and they wondered if they could cut it down to fewer.
In this case, I think it is a fair challenge. We could have had an extra chapter on the different metrics that are potentially useful — and maybe that is yet something for a future edition. Whatever the metric used, the message of the monograph holds: During fitting, decisions should be made using cross-validation, and a final test set should be held out.
Given this is done, one of my favorite single-number metrics is to imagine there are only two companies in the market: a company defined by your existing rate structure and a company defined by the new rates. Rebase both sets of rates so that in total they sum to the incurred claims in the test set. And then play either winner takes all (cheapest wins the quote) or use some elasticity estimate. Each company should end up with roughly the same amount of business — but what is the difference in loss ratio?
AS: Chapter 10 raises the scenario where cross-validation will silently fail because a small, poorly priced segment grows after launch through adverse selection. How should pricing teams operationalize the “monitor business mix post rate change” guidance without it becoming a vague aspiration?
AC: This is probably the most important point in the whole monograph. So long as we have decent post-implementation monitoring, even if we do make mistakes (I think most people make mistakes), we can fix them. Short-term and longer-term back testing are crucial. I guess it does depend on the marketplace. To what extent do you have quote and conversion data? Is it a live daily feed or delayed? At least we can probably know what business has been written soon after it has been written. A detailed monitoring of mix of business should be possible in most situations.
AS: Finally, getting back out of the weeds a little bit, what is the main takeaway you hope readers gain?
AC: Well, hopefully they pass Exam 8! As far as what’s in the monograph: Remember that however clever the technique, there are always going to be edge cases that can lead to anti-selection. As far as what’s not in the monograph — a realization that penalized regression is just the beginning of what data science and AI have to offer actuaries. Large language models (LLMs) are game changers in all aspects of what we do. I would not be surprised if everything we know today is out of date by next year. It’s our job as professionals to learn and adapt.