Why Analytical Modeling Will Never Be The Same – Part 1

What’s to become of analytical models in our real-time, machine learning age filled with new and complex data sources?

I tackled these questions by interviewing expert colleagues across the globe. This post is Part 1 of a three-part series that explores these challenges – and provides solutions.  The series is also being published in my newsletter.

Machine learning and other dynamic technologies are transforming the analytic domain, changing the way businesses act and ideally enhancing the way they perform. In the world of consumer lending, machine learning has the potential to be a significant disrupter to the traditional methodology and infrastructure of model building.

Let’s demonstrate how this disruption can happen by using an example of a market segment where the code has not yet been cracked…

Just consider the following findings in the most recent FDIC study*. According to the FDIC, over 30% of U.S. households qualify as “unbanked” or “underbanked.” By definition, these households make use of “Alternative Financial Services” or AFS’s – “check-cashing outlets, money transmitters, car title lenders, payday loan stores, pawnshops, and rent-to-own stores are all considered AFS providers.” *

The implications of these findings are significant:

  • Opportunities to expand the banking universe by better understanding AFS segments.
  • Opportunities to improve existing banking models as many existing banking customers are using Alternative Financial Services.

Now, machine learning has the potential to include these AFS segments onto mainstream financial services. First, we would need to identify and integrate the data sources that house AFS households. Using new technologies to integrate new data with existing data should open up opportunities for a deeper understanding of these segments.

Given the significant size and implication of the population identified in the FDIC study, these segments would later be included in the analytical models being deployed.

In summary, all players in this ecosystem must review traditional methods and data sources with the business goals of:

  • Expanding the eligible universe for inclusion in traditional (FDIC backed) financial institutions.
  • Improving business performance on existing target universe within traditional financial institutions.

I believe new modeling techniques that employ machine learning can aid both consumers and financial institutions by developing more accurate models through capturing and using these new sources of data.

To get to this point, I reached out to colleagues from across the globe to gain varied perspectives on the expansive data and analytic methodologies impacting predictive modeling today. These experts span regulatory bodies, credit bureaus and financial institutions. Together, we have nearly two centuries of combined experience in credit policy and compliance, consumer behavior and advanced analytics, as well as consumer lending and payments in both developed and emerging markets.

What emerged from these discussions was a clear understanding that the financial industry seriously needs to integrate business acumen into sophisticated analytics. Utilizing sophisticated analytic techniques, such as machine learning, requires rethinking the basics.

To ground the discussions we began with the five steps by which FICO** defines the basic methodology of building models:


Integrating new modeling techniques employing machine learning into this methodology requires that we answer the following questions:

  • How can a discipline that requires consistency and stability of model performance rely on the inherent flux of machine learning algorithms?
  • How is it possible to demonstrate stability of output with constantly changing inputs?
  • How will the regulatory environment evolve to embrace analytic applications of dynamic technological capabilities?

Through these discussions, three key themes emerged. Interestingly, all were associated with the first two steps in the model building methodology which are not often discussed:

  • We do not spend the appropriate amount of time on defining the business problem.
  • Is there a need for a development sample in the world of machine learning and adaptive predictive models?
  • We must create a new management system to support and integrate new capabilities with existing tools and processes.

This newsletter and the following series will explore steps 1 & 2 through a lens of the evolving disciplines required to advance predictive modeling in consumer lending.

Challenge: We do not spend the appropriate amount of time on defining the business problem.

The consensus from industry experts is that defining the business problem is the most impactful component of the modeling process and one that is dependent on deep business understanding with sophisticated analytic domain expertise. Analysts who don’t understand business objectives and goals will not produce the relevant models necessary to achieve those goals.

Clearly, in order to predict future outcomes, there must exist a clear understanding of the specific objective one is trying to achieve.

For example, say your broad business problem is revenue growth in a credit card portfolio. You are developing a predictive model that will generate more revenue. The business leaders and analytic community must determine if this revenue will be a result of account balance growth OR new sales generated by this account – as the identifiers in these scenarios differ.

Once this first determination is made, business leaders and the analytic community must establish the mix of balance growth or sales from acquiring new customers and optimizing existing customers. The behaviors and expected outcomes associated with new customers and existing customers differ.

The underlying reason for the balance growth or new sales also plays a role in identifying the business problem a predictive model will address.  Understanding customer attitudes and characteristics is required to define the business objective.

  • Does the customer pay their balance in full or pay the minimum payment?
  • Does the customer pay their balance on time?
  • Is the customer generating new sales as a result of a loyalty program?  A rewards product?  A marketing campaign?
  • What are the spending patterns for customers?
  • How ‘sticky” is the program in generating future sales?
  • How do these customers contribute to revenue growth for the portfolio?
  • Is the predictive model being developed for a specific strategy, such as balance consolidation – or a specific behavior, such as customer attrition?

Analyzing these scenarios (which of course influence expected outcomes) and their associated customer behaviors helps focus the business problem, define the objective function for the model, and begin the modeling process.

Quite a lot of analytical work and segment understanding needs to take place before beginning the modeling process. The objective function for the model is based on the targeted specific behaviors that resolve the business problem – in this example, behaviors that identify and generate increased revenue. Data, variable selection and modeling techniques will all differ based on the objective function you establish.

Connect Business Leaders with Analytics Experts.

It is critical to connect the business’s objective function, model development, model performance, and P&L performance. To do this requires the combination of business P&L leaders and business analysts to define the business problem and objective, define key success metrics, and become accountable for measuring performance. Further, a management process must be in place to review progress regularly.

In the example above, if the business leaders determined that a specific predictive model is needed for their balance consolidation strategy, the outcomes of the model are measured for account balance growth over a period of time, based on the customer segment and specific balance consolidation offer. The incremental balances associated with customers as a result of the predictive model are directly integrated into the program’s P&L and portfolio performance.

Looking Forward: How do we blend nontraditional data sources, new data sources and new data elements  for example, that missing data from Alternative Financial Services  and unify the process with the importance of having a business objective up front?

Adding nontraditional data sources into the evolving model development discipline has the potential to:

  • Expand the eligible universe for inclusion in traditional (FDIC backed) financial institutions.
  • Improve business performance on existing target universe within traditional financial institutions.

With a view towards expanding the eligible universe and improving business performance, Part 2 of this newsletter series will explore the human element of model building and the current need for a development sample in the world of machine learning and adaptive predictive models.


“2011 FDIC National Survey of Unbanked and Underbanked Households,” FDIC, updated December 2012, fdic.gov.

** “Understanding Predictive Analytics,” FICO, Fair Isaac Corporation, fico.com.