How is performance inference done?

                                                        Return to FAMQ

There are a number of methods for performance inference:

• Testing – One of the most reliable is to “test” into the unknown market by accepting applications that would be rejected or not normally included in the “known” sample and weighted up to represent the full TTD sample.
  • Advantages
    • Control the number and type of accounts that are accepted.
    • Management of these accounts is the same as they would be if the company actually expanded into this part of the market.
    • Surprises – The results of testing may show that this group is not as risky as expected and may represent a niche market in which the company can excel.
  • Disadvantages
    • Cost, since accounts are being accepted that are riskier than the price justifies. For example, a credit company might accept 10% of the normally rejected accounts at the same price structure as the normally accepted accounts and incur higher losses than the price justifies.
    • Time – It takes a year or two to get a good read on the actual performance in some situations.
• Surrogate Accounts – Try to look at the unknown accounts and find a record of how those accounts may have behaved at a competitor over the same time frame or look at a generic data set and try to match up the unknown accounts based on other variables such as FICO, demographics, income, …
  • Advantages
    • No uncertain costs – The account performance is with another company so there is no cost other than the fixed cost of buying the information
    • Time – Results are immediately available
  • Disadvantages
    • The account performance was not under the same management conditions so the performance inference may be off.
    • Matching logic may be inexact and difficult to confirm.
• Analytical Inference – This is a technique using expert knowledge of the domain in which the known model is developed and then “engineering” the model based on the known population to account for any model anomalies or inconsistencies in the inferred population. Analytical Performance Inference is a methodology that helps protect against the “truncation” phenomenon.
  • Advantages
    • No additional data needed
    • Results are immediately available.
    • Current account management strategies are accounted for in the analysis
  • Disadvantages
    • Accuracy is dependent on the quality of the analyst and the tools available.
    • It is a lengthy process requiring multiple iterations with the developer and an experienced reviewer. Even then the results are not guaranteed.

At the end of the performance inference stage, there are a number of items that should be reviewed, depending upon the inference technique being used. The primary goal here is to ensure that the inference results indicate that any “truncation” or other problems with the original data have been ameliorated. The results will include:

• Specification
• Validation
• Known vs. Inferred Bad rates
• Accept/Reject reports
• Low-side over-rides
• Score segment coverage
• Unmatched rate
• Bias in match
• Sample adjustment process using inference
• Special analysis depending on Inference Methodology used.
  
                                                          Return to FAMQ