Checklists have saved thousands of lives. How can we improve them?

Not everything that counts can be counted, and not everything that can be counted counts.

- William Bruce Cameron


We saw the power of checklists in their ability to harness the mundane, save lives and improve investment outcomes (read here) but we think they can be improved.

Scorecards - which are like a combination of checklists and screening tools - can dramatically increase the value of our analysis and leverage our investment process.

If checklists are so widely acclaimed, why change?

We think that checklists are brilliant at ensuring highly trained, experienced professionals take the time to consider all the components of a decision. They serve as a reminder to perform the same steps (even the mundane ones) every time. Confirm the accounting treatment of earnings. Look at off balance sheet financing. Check the maturity of the corporate bonds. However beyond this, checklists are not great at recording the outcome of these assessments in any structured or quantifiable form.


Scorecards enable both a checklist of steps to perform (for example: assess pricing power of the firm) but also to capture both the quantitative outcome (low, medium, high, 1-10, 4 stars) and the descriptive qualitative analysis behind the score (strong brand supported by industry leading R&D investment).


Using custom scorecards, investors can determine any number of fundamental characteristics they want to score (company competitive advantage, management quality, growth prospects, pricing power, M&A track record, ESG) and empower their analysts to capture their qualitative insights in a structured way.


Investors adopting a scorecard approach can build up a database of comparable metrics that can be used just like a universe screening algorithm. However, unlike generic screening, we now have information that is proprietary to our investment team, supported by analytical conclusions and integrated with our overall investment process.


Scorecards in Calibre RMS can be set up by the team to be consistently populated by analysts across the entire investment universe. These scorecards are attached as a step within an investment process stage and have threshold scores set that determine whether a company passes or fails that scorecard.

Scorecards can hold a rich level of information that is akin to an ultimate hybrid of a checklist, a quantitative screen and a focused research note.


Calibre RMS Features

Setup scorecard

Scorecards are much more powerful than checklists within Calibre RMS. You can see multiple examples of scorecards in the process library below, where companies can be scored using star ratings, numerical scores or from custom categories selected from a list. These selections are amongst many of the scorecard options available.

Examples of scorecards that can be fully configured in the system.

In the example below, we are using the Morningstar Economic Moat attributes as a way of measuring how likely a company is to keep competitors at bay for an extended period. The details behind this idea can be found here:

Scorecard attributes.

We can see we are in scorecard configuration mode as the header is purple. Our Economic Moat scorecard is set to be reviewed every year. Completed company scorecards that are more than one year old are considered stale and will not pass this step. We have averaged the scores for each of our scorecard items into an overall summary score for this scorecard. We have set up the five economic moat attributes that Morningstar uses so analyst assessments can be guided by this framework.


This scorecard has been configured with a “pass / fail” threshold of three stars. Companies in our investment universe must get an average score of three stars out of a possible five in order to have passed our Economic Moat test.


We can also see the ‘+’ icon at the bottom of this configuration screen. This allows us to add new items or attributes to our scorecard.


Complete Scorecard


As with checklists, once a team member has set up a scorecard and added it to an investment process, it is available to be used for any company that is part of the investable universe.


We can see below for this company (CSL Limited) the scorecard has been set up but is “Unrated” as it has yet to be completed.


A unrated scorecard.


Once the analyst starts to edit the scorecard, the header turns green and each of the Economic Moat attributes that apply to CSL can be scored and commented upon. Scorecards can be set up so that comments are either optional - analysts can quickly apply scores with no rationale - or mandatory where analysts must provide the reason behind their score.

Note the summary score is the average of the scores given for each attribute and is calculated by the system. The analyst cannot override this summary score.


Filling out the CSL Scorecard.


The completed and saved scorecard is now stored in the system. Although CSL has a poor score for the Network Effects attribute, it scores very highly in the other four attributes. The company has an overall Economic Moat score of four out of five - which means it passes this process step.


This scorecard is a simple example, with only a few lines of analysis text for each attribute. The system can hold thousands of words of analysis for each scorecard metric, should analysts want to provide high levels of detail behind each one.

A completed scorecard.


View scorecards for all our companies


Of course investors care not just about the results of a specific company, but how the universe of companies compare. We can look at all investable companies in our universe through the lens of our Economic Moat scorecards, which will have been completed by different members of our team. Regardless of industry, the level of analyst experience or their personal bias, the scorecard process provides a way to compare qualitative analysis in a consistent way across all investments.


In the example below, both CSL and Amcor pass our economic moat scorecard process.  This scorecard is sitting within the Industry Analysis stage of our investment process. Moreover, we can see the source of their Economic Moat differs.

Both Amcor and CSL pass our economic moat scorecard process.

We can also see that AGL and BKL fail to score an Economic Moat above the three star threshold because of poor scores across a range of our attributes.

AGL and BKL both fail our economic moat scorecard process.

In a forthcoming Calibre Insights, we will look at how to overlay business rules (such as supporting evidence) onto scorecard metrics, how to undertake detailed screening based on the integration of scorecards and other quantitative data and resolve the major drawbacks of using Excel for scorecard metrics.


What to continue the conversation?

Contact us to discuss how we can help your team.