For around 40 years, we’ve all heard the business idiom roughly similar to “only that which gets measured, improves.” The exact source and wording are sometimes debated, but the main point is that you must measure performance in order to determine where you can make improvements. Various metrics and Key Performance Indicators (KPIs) are used by different companies and S&OP team leads to gain insights into forecasting/planning accuracy as well as measure performance of major players. Different companies prioritize different KPIs and metrics based on what makes the most sense to them, how comfortable and familiar their S&OP leads are with the potential calculations involved, and how the companies prioritize various factors influencing the relative value of any particular metric. In general, we can say that nearly all metrics and KPIs can potentially shed light on at least some aspect of the S&OP process and can have value—if their limitations are understood. However, most statisticians and planning experts would agree that some metrics are objectively more useful than others.
Some common S&OP metrics include MAPE (Mean Absolute Percent Error), MAD (Mean Absolute Deviation, also known as MAE or Mean Absolute Error), Bias/Mean Forecast Error (MFE), Forecast Accuracy (FA), Tracking Signals (TS), inventory turns, customer service level (CSL), capacity utilization, demand variability, MSE (Mean Square Error), RMSE (Root Mean Square Error), and RSPE (Running Sum Percent Error), among others. Again, all of these metrics (and many more) can be useful within their sphere when viewed in the proper perspective and with appropriate caveats, but two or three—RSPE (combined with Tracking Signals) and the variants of MAPE—are probably more valuable than others overall in measuring and contributing to S&OP and particularly demand planning success. First let’s go over some of the more common S&OP metrics and KPIs, and briefly discuss their relative value.
There are dozens of potential metrics and KPIs relating to demand planning and the entire S&OP process, with a great variability in their complexity. Some are easily understood via basic math, and some require significant understanding of statistics and more advanced mathematical equations. As noted above, each company will need to determine how valuable or useful each KPI/metric is to them and their situation. Here are some of the more common ones.
This is one of the most basic metrics and is calculated by simply subtracting the actual sales from the forecasted sales. If you forecasted 100 sales but sold only 80 over a selected period, the Forecast Error would be 20. You can also express this in a dollar amount for sales teams, as in a forecasted total sales dollar figure of $8 million annually, minus the actual sales dollar figure of $6.5 million, for a Forecast Error of $1.5 million. Obviously this metric is limited in nuance but it does convey the overall, bottom-line figure quite effectively.
MAD is a fun acronym and the metric is used by many demand planners. Mean Absolute Deviation (or Mean Absolute Error) is a somewhat basic metric that helps evaluate forecast accuracy by averaging the magnitudes of the forecast errors, helping to uncover any variation in forecasts over a specified time period. The simplified version of the formula to determine MAD is:
MAD = Sum of observed absolute forecast errors over multiple periods / Number of periods
MAD is expressed as a number of units rather than a percentage. This means that one of the limitations of using MAD is that the result is not scaled relative to the amount of demand. For example, if you arrive at a MAD of 10 units, this is a fantastic KPI if your demand is 500 units, but a pretty dismal KPI if your demand is 20 units. Since MAD does not account for the scale of the product being forecasted, a product with very high demand may have a similar MAD to a product with low demand, even though the impact to the business of any errors would be very different. Again, intelligent understanding of each metric is needed.
Tracking Signals are another fairly simple metric that helps expose any significant bias in forecasting. This metric can be examined for a single month, but analyzing the tracking signal over a longer period of time can help evaluate the accuracy of the forecasting model.
There are several ways of calculating TS, varying from fairly simple to quite complicated. The most basic way to elaborate it is:
TS = accumulated forecast errors / Mean Absolute Deviation
Another source explains it thus:
TS = (actual sales for one month – forecast sales for that month) / absolute value of (actual sales for one month – forecast sales for that month)
When performed for one month, as in the second example above, the result is either a 1 or -1. A TS of 1 (in this case) indicates that the forecast was lower than actual demand. On the other hand, a TS of -1 would mean that actual demand was lower than what was forecast.
Clearly, tracking signals become more accurate and relevant the greater number of time periods are included. A minimum of 7 weeks is recommended to develop a better idea of whether there is any consistent bias. If you tracked 7 weeks, for example, the highest possible result would be 7 and the lowest would be -7. Planners should aim for a TS of zero or close to it, indicating that there is no significant bias, at least using this metric.
Another fairly simple concept, the Bias metric reveals any trend for any demand/sales forecast errors to be consistently higher or lower than what actually occurs. Running Sum Percent Error (RSPE) is a total deviation between the actual data and the forecast, expressed as a percentage of the total sum of the forecast. Bias is calculated by taking the sum of forecast errors over more than one period and dividing it by the sum of the forecast lags for that same period. For consistency, a simplified version of the equation follows:
Bias = Sum of observed forecast errors over multiple periods / sum of forecast lags
Obviously, the greater number of time periods examined, the more accurately the RSPE figure will reflect any bias. By tracking bias regularly, planning and S&OP leads can uncover any trends for forecasts to come in under or over actual figures. If a team is consistently under-forecasting, the company may be losing sales due to inadequate supply. On the other hand, if a team is consistently over-forecasting, the company may lose resources maintaining excess inventory. Ideally, you want to see minimal forecast errors, and they should distribute both slightly above and slightly below the actual sales figures. This shows that there is effectively no bias. This could also be a KPI for evaluating forecast team leads, since producing unbiased, accurate planning data and reports is to be commended.
When we evaluate RSPE we also have to take into consideration the Tracking Signal (TS), which, as elaborated above, is the running sum of forecast errors divided by the average absolute forecast error.
These 2 metrics together (RSPE and Tracking Signal) give us our accurate “bias.” RSPE tells us on average how much above or below actual sales the forecast is (expressed as a percentage), and TS then clues us in on how statistically significant our bias is.
For example, if you have an RSPE of 10% over the last 7 months, that means that on average over the last 7 months your forecast has been 10% less than the actual sales. If your TS were 4 for that same scenario, that means that the forecast was less than sales 4 periods out of the last 7, meaning that there’s a statistically significant pattern of forecasting low. Or, in other words, you are biased to under-forecast or over-sell your forecast (depending on how you want to view it). If, on the other hand, your tracking signal were 2, that would mean you have some periods above and some below, meaning there isn’t a significant pattern of bias to one side or the other. You can calculate bias over any historical time range but we often advise 7 periods, as this provides enough data to identify patterns, but not so many that data points get smoothed out or lost in the noise.
This is a broader supply-chain metric that shows how effective your company is at fulfilling orders and meeting customers’ expectations. Only orders fulfilled completely and on time are considered. A simplified version of the CSL calculation is:
CSL = (number of fulfilled orders / total number of orders) x 100
Obviously, you are aiming for a score of 100, which would mean that every single order was fulfilled completely and on time. This may happen in an ideal world, but rarely happens for businesses selling significant volume. Still, maintaining high CSL numbers should be a priority for every business that sells anything.
Forecast Accuracy (FA) is a high-level demand planning metric that helps you see how accurate your forecasts for sales and demand are. Obviously, the more accurate your forecast, the less you waste on unnecessary overstock or stockout situations, and the higher your profits.
The simplified equation for determining FA is:
FA = 1 – [absolute value of (actual sales for time period – forecast sales for same time period) / actual sales for time period]
So what’s considered a “good” FA percentage? If you have no historical sales/demand data, an FA of 75%-80% would be considered very good. After all, without any real data, the demand planner would essentially be making an educated guess. If you have products that have been on the market for months or years, you should strive for an FA of 90% or better.
Mean Square Error is a metric of forecast accuracy that averages the squares of the forecast errors, which removes all the negative terms (since any negative number times itself results in a positive number). This is a double-edged sword, in that it gives more weight to the larger errors in the data (which seems logically desirable), but it also means the resulting error rate may overemphasize the magnitude of errors. The simplified formula is:
MSE = Sum of the squares of forecast errors over multiple time periods / total number of time periods examined
The lower the number, the more accurate the forecast, taking the above caveats into account.
In simple terms, the root mean squared error is simply the square root of the MSE (above), or the square root of the average squared error. RMSE can be helpful in evaluating how severe forecasting errors are. Although RMSE, like MSE, is not scaled according to the actual demand, this deficiency is mitigated somewhat (compared to MSE) since RMSE utilizes the square root of the average error. The simplified formula for finding RMSE is:
RMSE = Square root [squared (forecast sales for a time period – actual sales for the same time period)]
MAPE is probably the most widely used metric in S&OP/demand planning, and is a statistical measure of a forecast’s accuracy. The calculation results in the expression of the degree of error as a percentage, so it’s still relevant regardless of scale. It also makes it easy to explain to various teams across the organization who may not be up on the other metrics relating to demand planning. The simplified version of the formula is:
MAPE = Sum of (forecast error for a time period / actual sales for that period) / total number of forecast errors x 100
It has become de rigueur over the past few years to eschew MAPE or at least point out its limitations when dealing with highly volatile demand. However, for many planners it’s still highly relevant today, particularly when supplemented by its improved or fortified cousins, Symmetrical Mean Absolute Percentage Error (SMAPE) and Weighted Mean Absolute Percentage Error (WMAPE). More on these in the section below.
We mentioned some of the shortcomings of the various metrics above, but let’s talk a little more in-depth about MAPE (and variations).
However, MAPE is still very useful in cases where sales data is plentiful and where there are no occurrences of zero forecasting errors, particularly when planners are aware of the limitations of using MAPE without considering its caveats. Furthermore, when bolstered by SMAPE and WMAPE, a more holistic picture of forecasting performance emerges.
Symmetrical Mean Absolute Percentage Error (SMAPE) is useful in compensating for the shortcomings of MAPE in those cases where data is limited. The advantage of using this metric to express forecasting errors is that it has both a lower (0%) and an upper (200%) limit in order to overcome some of the situation-dependent disadvantages of MAPE. Weighted Mean Absolute Percentage Error is another improvement that can in some cases result in more accurate KPIs since it gives more emphasis to actual sales/demand volume and observations.
Mean Squared Error (MSE) is another metric that can help flesh out MAPE numbers in certain instances. MSE helps illuminate the impact of significant discrepancies between forecasts and actual demand. The simplified version of this calculation is:
MSE = (1 / number of observations) x sum of (actual – predicted)2
Since this metric squares the errors (forecasted demand minus actual demand) and then calculates the average, MSE emphasizes the impact of larger errors. It can also help uncover potential issues in smaller, but consistent forecasting errors. A theoretical example would be a situation where demand for a particular product is consistently underestimated or under-forecasted by a small margin. A standard MAPE metric calculation might not flag this as a significant issue. However, MSE would likely reveal the cumulative impact of these small errors, highlighting the need for improvement.
RSPE combined with Tracking Signals can be very useful because, as noted above, this gives you the directional bias and also the magnitude of the bias. However, we must make an important point here. Rather than adhering to one or two metrics, the main goal should be simply to improve the accuracy regardless of which metric you chose. Use whatever metric/KPI makes the most sense for your situation to get your baseline, then get to work trying to improve on it month to month. For example, it’s not that a 10% MAPE has much meaning by itself, but if it was 13% last month, then it tells you that you are doing the right things and moving in the right direction. Similarly, if one product is 10% and another product of similar volume is showing 15% MAPE, then you know which one needs more attention, and this helps you dedicate your time and resources more intelligently.
If you were to simply calculate every available KPI/metric for S&OP and demand planning, you’d have endless numbers to look at, but unless you had a solid understanding of the limitations and advantages of each type of metric, your perspective and understanding of your company’s performance would be somewhat incomplete, and potentially skewed. Often, planners are faced with either an inadequate amount of usable data, or information overload to the point of paralysis. Today’s AI-assisted planning solutions help prioritize metrics that actually matter, producing data and reports that are easy to understand, easy to share with team leads, and easy to act on.
The ORI Excellence Platform is the only software and AI tool purpose-built to help you achieve top-tier sales and operations planning and execution in one place. Reduce risk, manage uncertainty, and increase resilience with ORI.
Get a free demo and find out how ORI’s 30 years of experience and best-in-class technology can help you.