To be competitive in the modern banking market, community bankers must identify and validate appropriate business use cases for Generative Artificial Intelligence Large Language Models (GenAI LLM) [Footnote 1] . Possible use cases include writing meeting minutes, drafting internal-policy documents, writing quarterly CECL credit loss allowance reports, drafting FDIC Call Report submissions, writing consumer credit underwriting policy, etc. If the bank’s assets exceed the FDIC $10B threshold for model validation, the bank’s GenAI use cases must undergo an SR11-7-style model validation. In this note we demonstrate the impact the business use case has on the model risk tier assignment of the GenAI model.
The first step in a model validation is to estimate the model risk tier of the model to be validated. Martingale Solution Group (MSG) creates model risk tiering systems customized and calibrated to an institution’s model inventory. For this case study example, MSG built a risk tier scorecard with the following seven Model Risk Factors:
- Materiality / Exposure
- Use Case / Regulatory Exposure
- Model Complexity
- Data Quality & Lineage
- Governance & Controls
- Model Validation & Monitoring
- GenAI / LLM-Specific
Each model risk factor has 1-10 components which are linearly combined to obtain an overall model risk score. Lower scores correspond to lower model risk tier. Models with higher risk tiers are prioritized for deep-dive model validation.
This case study scores two business use cases for the GenAI LLM model at a community bank that is just above the FDIC’s $10B model validation threshold:
a. Internal Use Case – Writing strictly internal general business support documents that do not impact the firm’s books and records. Examples include meeting invitations, meeting minutes, project plans, etc.
___________________
Footnote 1 – A GenAI LLM is a machine‑learning model that is trained on very large corpora of text (and sometimes code or multimodal data) to learn statistical patterns of language and produce fluent, contextually relevant natural‑language outputs. Community banks license GenAI software products for internal business workflows.
b. Regulatory and Policy Use Case – Writing FDIC Call Report submissions, writing policy documents (e.g., consumer credit underwriting, employee handbook, BSA/AML policy, etc.)
Each business use case undergoes model risk tiering. Since the GenAI LLM model is identical in both use cases, most of the model risk factor component scores are identical for each of the two business use cases. The Regulatory and Policy Use case obtained the maximum Use Case / Regulatory Exposure model risk factor score due to its employment for quarterly FDIC regulatory submissions. In addition, since no policy fine-tuning data is available to customize the LLM for this use case, the model earns a high Data Quality & Lineage model risk factor score. In addition, the use case received an elevated Model Validation & Monitoring score because systematic policy back testing is not possible. The GenAI/LLM Specific score for the Regulatory use case is elevated because:
- Data leakage risk – the GenAI LLM license does not guarantee the bank’s internal data remains secure
- Red-team testing – the bank in this case study does not have the resources to perform adequate adversarial attack cybersecurity testing
- The bank does not have adequate monitoring of the LLM’s behavior
As a result, the overall Model Risk Tier for the two model use cases are as follows:
The take-away for this case study is that community banks must proceed on the GenAI LLM model deployment journey with caution. Validation of GenAI models on each individual business use case reduces model risk and adds value to the enterprise. Martingale Solution Group is your go-to GenAI LLM validation provider. Creation of a model inventory with complete Model Risk Tiers as well as full validations with benchmarking are available. Contact Martingale Solution Group for a consultation.
