Impact Measurement Framework
Time Periods
Initial Measurement
This first stage should focus on defining indicators for sandbox applicants in line with the objectives of the sandbox framework. It should include not only business metrics but regulatory and market outcomes, including those that test assumptions and provide policy insights. These initial metrics defined by the policy makers will help assess an applicant’s feasibility for the sandbox and how the applicant will contribute to and test sandbox goals. The metrics can be monitored by sandbox staff throughout the testing stage and evaluated during the exit stage, with a focus on potential wider policy implications. Authorities should avoid turning the assessment and monitoring of the applicant into a “check-the-box” exercise and should instead ensure the applicant has a clear link to enhancing sandbox policy goals.
Ongoing Monitoring
Throughout the sandbox process, regulators should implement a series of ongoing assessments to measure the progress of the sandbox framework and the firms within it. Ongoing assessments often measure, for instance,
- continued suitability and relevance of each sandbox firm and cohort against sandbox metrics.
- direct and indirect institutional changes and benefits that can be attributed to the sandbox.
- operational efficiency of the sandbox process, both for regulators and for firms that move through the sandbox process.
Such assessments should measure the progress and outcomes of the sandbox on an on-going basis and support policy makers in remaining agile, understanding policy implications, and adjusting their sandbox and legal or regulatory framework as needed.
Periodic and Final Evaluations
Periodic and final evaluations should be conducted at the end of a sandbox process or after a defined duration. This is a point-in-time evaluation and should be positioned to help determine the impact of a sandbox on broader financial sector and national goals, such as building institutional capacity, enabling firms to come to market, growing the broader fintech ecosystem, or contributing to national financial inclusion progress. Such assessments often require broader data collection efforts and, in some cases, econometric modeling.
Impact Measurement Framework
Country-Level Outcomes
- Country-level outcome indicators should focus on how well the sandbox contributes to broader financial sector outcomes. This may include, for instance, national financial inclusion goals, economic measures such as the ability to attract foreign talent and improve growth, or broader digital development.
- Note that these are difficult to measure or attribute to the workings of a sandbox and hence should relate to goals clearly defined for the sandbox at its outset.
- Some examples of specific outcomes include nationwide rise in financial inclusion levels, increased ease of doing business, or increased numbers of products or services targeting the unbanked.
- Outcomes can be measured through global data indicators, as in the World Bank Findex, or in-country measurements, such as national financial inclusion surveys assessing innovation uptake and/or feedback surveys from users of services offered by firms participating in sandbox tests.
Impact Measurement Framework
Regulatory Outcomes
- This level should include regulatory outcomes from innovations operating within the sandbox or from knowledge and intelligence gathered that impacts regulation, supervision, or policy. While direct regulatory change.
- The metrics should test regulatory assumptions. For this, a clear understanding of the policy questions that each applicant raises is critical.
- Key considerations to developing policy-led indicators include assessing risk, micro and macro shocks, behavioral reactions, sector-wide interactions, and contagion.
Sample Policy Indicators:
- Introducing new regulation or amendments to existing regulation to support digitization. This could include laws and acts, regulations, directives, circulars, guidelines, or explanatory notes. Contributing to policy projects through analytical insights.
- Adapting the supervisory process in response to market developments.
Supervision Indicators:
- Implementing proportionate regulation to lower the barrier to responsible innovation that might have been faced with unduly burdensome regulatory requirements.
- Including certain types of innovations within the regulatory perimeter, as a direct consequence of assessment of risk posed.
- Reviewing and addressing consumer protection issues that may arise with new innovations.
- Interacting fairly and transparently with new entrants.
- Improving inter- and intra-regulator coordination.
- Assessing imbalances in level playing fields (i.e., monitoring of market distortion, anticompetitive behaviors, etc.).
- Adjusting or thoroughly evaluating current frameworks based on sandbox tests.
Impact Measurement Framework
Firm & Cohort-Level Outcomes
Both qualitative and quantitative indicators should be considered when looking at the firm-level outcomes. Aside and beyond business metrics collected through firms, such as number of consumers, value of transactions, and so on, the sandbox framework should also develop metrics to assess the sandbox’s impact on the market and market players.
Metrics are nuanced and defined on a case-by-case basis, but some examples include:
- Case studies demonstrating the response of incumbents to new entrants to the market.
- The nature of the support provided by the sandbox that may have contributed to greater regulatory certainty for firms.
- Evidence of how the sandbox has helped firms establish themselves in the market.
- Number of new innovations, products, or services that have entered the market.
- Increased volumes or values of particular services, like e-lending, wealth management through robo- advisory services, online accounts, etc.
- Increased number of financial service providers competing in the market and hence bringing in more competition and consumer-centric products.
- Increased access to regulatory expertise to get innovative ideas to market.
- Ratio of firms adapting to existing regulatory frameworks, compared to those requiring new or greatly modified regulation.
- Firm reports around increased regulatory certainty, guidance, and engagement from authorities.
Impact Measurement Framework
Operational Outcomes
At the operational or institutional level, indicators should assess the ongoing appropriateness of the sandbox internally, analyze the resources and capacities used during implementation, and evaluate if the sandbox is contributing to overarching institutional goals.
For instance, a regulator may hypothesize that a sandbox will strengthen an institution’s capacity to regulate fintech. To test this hypothesis, a regulator may want to consider indicators that herald particular institutional changes such as the following:
- Widespread buy-in and catalytic change across the institution.
- Increased regulatory capacity in certain areas in response to market trends.
- Improved knowledge of the fintech sector and gaps identified in business areas’ fintech knowledge and know-how.
- Signaling and market perception of regulator’s openness to enabling fintech.
- The establishment of new, related units or increased departmental capacity of the sandbox.
- New staff hired with backgrounds more aligned to regulating or supervising particular elements of fintech, per sandbox recommendations.
- An understanding of information flows from sandbox trails, and how the information is used and acted upon.
- Cost-benefit analysis of capacity and resources used by the sandbox compared to policy outcomes.