Scorecards form the backbone of decision making for many financial institutions. They are used in the account management of key decision areas like collections and authorisations, for example. They can tell us whether to accept or decline a customer for a particular credit-based product or tell us the percentage of a customer’s outstanding balance that will be recovered over a certain period of time.
In this blog post, we’ll be covering what scorecard monitoring is, its importance and the consequences of not carrying out the exercise regularly.
A change in internal or external factors can affect a scorecard
A change in internal or external factors can affect a scorecard’s performance over time and it is therefore important to monitor it for any significant changes and adjust it if necessary. By monitoring scorecards regularly, any factors that could have an effect on this can be identified. A changing economic environment, or any significant changes to a company’s operating environment – such as new products or a change to marketing strategies – could influence a scorecard’s performance. It is important that your scorecard reflects these changes.
Read more on Credit Risk Scorecards.
The importance of monitoring the model
Once a scorecard has been developed and implemented it is essential that the model be monitored regularly. The key assumption in scorecard development is that the past is representative of the future. If today’s population is different to the population on which the model was developed, there is a good chance that the performance of the model has also changed.
Behind the score is a probability of a customer behaving in a certain way. By regularly monitoring the models that are being used, the effectiveness of the scorecard can be tested. Monitoring is the easiest and most fundamental way of confirming whether or not the model is still predictive of the current population. If the model is performing below levels that a business considers acceptable, then the model should be redeveloped or adjusted to align with the current population.
The two components of Scorecard Monitoring
Scorecard monitoring reports typically consist of two components: Front-end reports and Back-end reports.
Front-end refers to the filtering function of the scorecard “at the front end” – i.e. guiding the accept / decline decisions.
Back-end refers to how the scorecard performs once the applicant has been on-boarded and is now a customer.
In the case of an application scorecard, the front-end reports could allow one to see if a change in marketing strategies has caused a shift in the behaviour of males and females or a certain age group (assuming of course gender and age are variables in the application scorecard) or, in the case of a behaviour scorecard, if there has been a shift in the payment patterns of the population.
One will be able to see the effect of this shift in the back-end reports by comparing the current gini to the development gini and reverse rank order can be observed. If the behaviour of customers has shifted over time, resulting in reverse rank-ordering, for example, this will be observed in the back-end reports and a scorecard re-alignment or re-development would need to be carried out.
Scorecard monitoring reports can also be used to check the trend in the accept rate (for Application Scorecards) over time and to identify if there have been any significant shifts in the number of applicants that have been accepted or declined, or those that have been accepted but have chosen to decline the product (Not Taken Up’s or NTU’s).
Front-end reports are typically made up of the following:
- Population Stability Report
- Characteristic Stability Report
The Population Stability Report assesses the current score distribution of accounts and compares them to the development or baseline sample.
This assessment will allow any shifts in the current population to be viewed and is usually the first point of reference in identifying whether or not a significant shift has occurred with regards to the current portfolio.
This analysis segments accounts into equally proportioned scorebands and compares the proportion of accounts that fall within each scoreband for both the development and more recent sample. The minimum number of accounts required for this report is 300 accounts.
The Characteristic Stability Report is similar to the Population Stability Report but looks at the population at a characteristic level over a period of time.
The analysis is useful in that, where there is a significant shift in the distribution at a population score level (as identified with the Population Stability Report) a more detailed characteristic level score analysis will allow one to be able to see where and why this change is possibly occurring and adjust the scorecard accordingly. Each characteristic is analysed at an attribute level and allows one to see where there has been a shift between the development data sample and the recent population.
Back-end reports are used to analyse the effectiveness of the scorecard and how predictive the model currently is in relation to when the scorecard was implemented. This report is usually comprised of:
- Analysis of the Gini co-efficient
- A score-to-odds analysis
The Gini co-efficient measures the strength of a scorecard and indicates how effective the model is in separating the bad customers from the good customers. It is sometimes useful to look at the Gini co-efficient over time to see the trend in the predictive power of the scorecard and to analyse if there have been any significant changes over a certain period of time.
The score-to-odds relationship informs us of the linear relationship between the scores and the log of the odds for the scorecard, i.e. the relationship between the number of good accounts and the number of bad accounts at a particular score. The current relationship is compared to the standard odds relationship that was used at development. If the observed odds deviate materially from the standard and the scorecard no longer ranks the probabilities, then a scorecard alignment will be required to restore the score-to-odds relationship.
The consequences of not monitoring and updating scorecards
In the case of application scorecards, the scorecard-driven decisions are sometimes overridden. This could be due to additional information, such as fraudulent warnings. One can also assess the override rate and determine whether the overriding of scorecard-driven decisions is changing considerably over time, allowing companies to assess whether this is having an impact on their business.
Declining a good customer (high-side override) has less impact on a portfolio than the potential loss from an accepted bad customer (low-side override). Too many high-side overrides means lost potential revenue, and if these overrides are above an acceptable level, then this needs to be monitored or investigated. However, overrides should generally be kept minimal, as this is why we have a scorecard in place – to filter out the good accounts from the bad ones.
Designing a scorecard with little available data
In some cases, scorecards are ‘expertly designed’ – where not much data is available. Once sufficient data becomes available, a monitoring report can be generated in order to see whether or not the population is performing as is expected. If not, a re-alignment of the scorecard will have to be carried out. Otherwise, in extreme cases where the scorecard is not performing as expected, the scorecard could be redeveloped using available data.
In Summary
It is important to track the performance of a scorecard so as to ensure that the model that was developed is still predictive of the current population. This will, in turn, result in business stakeholders being able to make more informed decisions and plan accordingly. Regular scorecard monitoring ensures the model’s stability and robustness, and if it is seen that the scorecard no longer performs as expected, an alignment of the scores may be required to restore the score-to-odds relationship.
We offer a range of credit analytics products designed and developed to minimise risk and maximise profitability along the customer lifecycle. Please visit our website for an overview of these Credit Risk products.
