This article previously appeared in the Society for Workforce Planning Professionals online journal, On Target. They appear here with the gracious permission of SWPP and Vicky Herrell.

As you get older, you notice things.  For me, one of those things is almost biblical: “new things are old things.” Eighteen years ago(!), I wrote an article for On Target called “Does Accuracy Matter?”.  In it, we discussed issues and observations about our WFM industry and ways to improve the accuracy of our contact center models.  What surprises me is that— even with all the improvements in computing, algorithms, and AI— that old article is still relevant.

We’ve seen many WFM products come and go, but it seems that every ‘new’ solution eventually reveals the same old assumptions underneath.

We’ve ignored the basics

When I was in college, my studies turned toward simulation, optimization, and forecasting modeling. In the first week of every modeling course, each professor would discuss the modeling golden rule: prove that any mathematical representation of an operation or process is close to the physical reality; that your model is accurate. Each of my classes would start by discussing the importance of model validation.

But for all the talk about innovation in workforce management, that’s one topic we’ve collectively tiptoed around for decades. Not forecasting accuracy— we talk about that plenty­­— but model accuracy, the accuracy of the core staffing math itself. The accuracy of the assumptions. The accuracy of the engine underneath all the dashboards and workflows we rely on.

And here’s the uncomfortable truth: as an industry, we’ve ignored it.

Why?

WFM grew up focusing on scheduling efficiency, not science or statistics. The earliest tools were built to assign people to shifts, not to validate the mathematical models that determine how many people we need in the first place. Forecasting and capacity planning were bolted on much later, and the underlying math never really caught up with the complexity of our flexible operations.

We also learned to work around the problem. Real contact centers are messy, noisy systems. Requirements fluctuate. Customer patience varies. Routing rules can change on a whim. WFM vendors avoided accuracy claims because they didn’t try to measure it, and planners compensated inaccuracies with fudges, gut-feel, and a healthy amount of operational heroics.

And let’s be honest: validation is hard. Everyone of your contact streams really is different, and to have an accurate staffing model, you need to build unique models for each of them. Validation requires simulation, data science, and transparency — and an unvalidated, generic, and inaccurate Erlang model is so much easier to implement. So, accuracy became something we assumed rather than something we tested. Erlang formulas have been treated as gospel (when they shouldn’t be). Vendors assumed their models were “close enough” but they never actually examined them. Customers assumed the vendors had validated their models and no one asked the uncomfortable question.

What is model validation?

Validation is a simple process.  We ask our models to answer a modest question: “Model, if you had perfect information, could you predict our service correctly?”

We look back in time, look at the actual as-happened staffing, volumes, and handle times, week-over-week. It makes sense that if I knew these numbers each week, my models would predict service and abandons and occupancy correctly, right?

We plug those historic weekly numbers into our staffing models and predict what the model thinks the service levels would be.  And we compare them to what actually happened. The question: do our models reflect reality?

If our models predict the actuals closely, we know we have a model that reflects the complexities of our operation. 

If our models’ outputs are not close to the actuals, it means the basis for whatever system or process or spreadsheet we are using is deeply flawed.  If we are using a non-validated model for scheduling or real time analytics, the number of people it is telling us it needs is a guess. If we are using it for capacity planning, it is likely inflating our budgets and introducing slop into our operation.  Our model is off and gives us wrong predictions.

Also, building a weekly staffing model isn’t as simple as adding up our arrival distribution and applying an Erlang model to estimate staffing requirements.  Customer patience, handle time distributions, agent schedules, schedule non-adherence, and shrink all contribute to a capacity planning model’s accuracy. This complexity is exactly why these models require data and validation, they are not easy to get right out of the box.

Below, I am including two pretty good validations of a contact center’s service level and abandon predictions.  Remember, this does not show the accuracy of a forecast, it shows the accuracy of the staffing calculation and the service level prediction.  It says that if I knew the weekly volume, the handle time, and the staff available, I could very accurately tell what the service levels and abandonment would be.  But it also means that any what-if analyses I do will be rock solid.

What do we get when we validate? Bad decision avoidance.

It seems like I’ve been off on a tangent, talking about validation, when I promised to discuss whether accuracy even matters.

TL;DR: Accuracy matters. A lot. (OK, funny that the TL;DR thingy is in the middle of this, instead of at the beginning).

When we validate we get confidence that our models will help us precisely analyze our operation, and everyone should be confident that our analyses are solid. But so, what?

Our models—especially our capacity/strategic planning models—are used to answer very important, very big and costly questions.  Things like:

• What should be our budget?

• Should we close a center or a line of business?

• What happens if we enact a hiring freeze?

• Should we offer a cross-sell program?

• Should we hire 100 agents? When?

• What happens if we combine agent groups?

• Should we offer chat?

• Should we use AI agents to reduce our staffing costs? What sort of operational impact will AI agents have?

• Should we outsource? How much of the business should we outsource?

We all know that Erlang equations overstaff, right?  So, let’s say we were doing some analysis on questions like those above.  If we assume a fairly typical Erlang bias, say anywhere from 5 to 15% overstaffing, is it possible that our model’s analysis to close a center or to outsource more might lead us to make the wrong decision? Ask yourself, “If I overstate my staffing requirements, and put 8% unneeded additional cost into my plan, will we be more or less likely to outsource my center?”  These are big, many-million-dollar strategic decisions.

Stuff you can do with a validated model

If your models are accurate on all dimensions, you can do sensitivity analytics that make clear the trade-offs of competing business objectives.  Below is a simple sensitivity analysis graph.  In it, we ask, what happens as we hire more agents to our service levels and abandons?  The graph makes the results of the decision to hire or not clearer.

Because our models are validated, we can be confident that any trade-offs we make with them are solid.

It may not always feel like it, but we are pretty important

Of all the decisions our organizations make, few are as important as those informed by our strategic analyses. When we make changes in our workforce management software or our contact center routing, the decisions are tactical and if we make a mistake, correctible.

Not so, with our capacity planning what-ifs.  One big decision, poorly decided, can lead to months of operational pain.  I was recently chatting with a dear friend who plans for a very large contact center network.  Her latest concern— and it is significant— is that her company’s push toward AI agents will be made much too fast to recover if it doesn’t work out.  Her company will buy the tech and make staffing reductions immediately. If the experiment is a bust, or even if customer response to AI agents is less enthusiastic than expected, the operation could take months, maybe even a year to recover. Call centers can enter that “death spiral” with one poor choice. Capacity planning decisions are strategic, and mistakes are expensive and long-lasting.

Because of this, we planners are important. I bet if you asked the CFO or COO, they’d say so too.

Finally, some rules of thumb

I asked one of my AI-bot buddies to look at our industry, and all the WFM vendors, and see if they discuss accuracy and validation.  When the search came up blank, I asked the AI why he thought it was so. He said, “Its because we don’t hold them accountable.  Most other industries that use modeling require their providers to demonstrate model accuracy.” 

Here are some red flags, when evaluating WFM and analytic tools that use staffing models:

• If the system doesn’t require history to estimate staffing or service, it’s not accurate.

• If you have an option to enter distributions (say volume distribution), rather than using history directly, it’s not accurate.

• If there is no validation graph, they are using the same model for tech support that they are using for ticket sales.

• Does it only calculate staffing requirements?  Simulation works the other way and only simulates the service expected. Also, only showing requirements is a sneaky way of showing something that can’t be easily proved wrong.  You need service predictions.

• Does it show you expected service levels, ASA, concurrency, and abandons?  If not, it is most likely using some form of an Erlang C calculation.

• The obvious one: if they tout the Erlang C calculation on their website or technical documents, it won’t be accurate.

• Erlang A equations, like all other models, require validation as part of the model building process.  Ask for one.

We now have the computing power, the data, and the AI reasoning layers to build models that behave like real contact centers, not idealized, theoretical ones. We can validate. We can simulate. We can know.

Let’s hold our vendors accountable, ask for a model validation on your specific data.

If we don’t validate the models, as has been the habit in our industry, then our staffing plans and what-ifs may look precise, but they’re not grounded in anything real. They aren’t real numbers.

Ric and Chris are both founders of Real Numbers, a contact center capacity planning and modeling company.  Ric can be reached at ric@realnumbers.com  or (410) 562-1217, Chris at chris@realnumbers.com.  Please know that we are *very* interested in learning about your business problems and challenges (and what you think of these articles). Want to improve that capacity plan? You can find Ric’s calendar and can schedule time with him at realnumbers.com.  Follow Ric on LinkedIn!  (www.linkedin.com/in/ric-kosiba/)