Vom now allows you to run a proposed policy change against all past decisions with a couple of clicks. That is, to run a backtest. Here's how it works:
Anytime you are editing a version you can click on the backtest button on the toolbar:
Backtests are a kind of batch execution. That is, a large amount of decisions that run together in a few minutes. While the more general batch execution feature takes a CSV with the decisions to run, backtest use up to 20k fairly sampled decisions from the policies execution history.
When you click the backtest button on a policy you will see the batch execution dialog configured to run a backtest:
Just select the period and the count of past decisions to use, and after about 5 minutes you will get a CSV with all the input data used for each original decision, their historical result and the result from running the backtest version. This allows you to easily do any data analysis you want in Excel or another tool. For instance how will approval rate change with the proposed change.
We know making large changes to decision policies can be risky, so Vom with this new feature Vom now offers a full suite of testing functionality to make you comfortable before flipping the switch. We already had policy tests (AKA unit tests) and a strong A/B test feature, so backtest completes the set.
Comments