Before a manufacturer adds AI to demand planning, it should find out which of its planners' changes to the system forecast make it worse and stop those, starting with the ones that raise it. A forecast raised by hand flows into the production plan and material orders once stock on hand and safety stock are netted off, so an upward change that proves wrong can end up as inventory the company pays to carry.
An analysis published in the International Journal of Forecasting in December 2024 pooled around 147,000 forecasts from six earlier studies. Planners' changes typically led to improvements in bias and accuracy for only just over half of stock keeping units, with variation across datasets; bias is a steady lean toward forecasting too high or too low. Changes that raised the forecast were confirmed as more likely to worsen performance, while changes that lowered it usually helped, most of all when they were large. The evidence that planners made effective use of information the software lacked was weak, and they appeared to respond to irrelevant cues instead.
A 2021 case study by two of the same authors, in the same journal, covered fifteen years of forecasting at one pharmaceutical company. Most of its forecasts came from managers overriding the system's statistical forecasts in sales and operations planning, and the study found that those overrides took considerable management effort and often reduced forecast accuracy.
Planners go on adjusting once the forecast comes from AI, according to a 2026 study in the Journal of Operations Management, summarized by the University of Arkansas's Walton College. Using two lab experiments and about 575,000 observations from a multinational retailer that moved from a traditional forecasting model to an AI one, the researchers found managers making larger adjustments when AI performs poorly than when a traditional model made the same error, and making smaller ones as the algorithm improved.
The check is what demand planners call forecast value added analysis: measuring what hand changes do to the accuracy and bias of the system forecast, the statistical forecast the planning software produces before anyone changes it. For each item and month it needs the system forecast as it stood when production and material orders were set, the planner's final number, and what customers ordered, which includes orders for items that ran out. Where the software overwrote old system forecasts, start with the months it kept and save a copy each month from now on. Ask the demand planning lead for four numbers from the last twelve months: the share of forecasts changed by hand, and the units of forecast error, the gap between forecast and orders, that hand changes added or removed against the system forecast, for all changes, upward changes and downward changes.
At a manufacturer that makes to stock, read the upward line first. A kind of change that added error made the forecast worse, and it stops: the system forecast stands, and a planner who expects more demand brings the evidence behind that view, such as a retailer's confirmed promotion or a new customer contract, to the monthly planning meeting. Rerun the four numbers every month.
The 2024 analysis also found that a debiasing procedure applied to adjusted forecasts proved effective at improving forecast performance, meaning a correction for that lean, and a later AI project can be tested against that correction on the same records. Joined to the production plans and material orders of those months, the records also give the CFO an estimate of the working capital tied up in stock built on upward changes that added error.
