Why Most AI Impact Assessments Miss the Real Socio-Economic Story

Disclaimer: Client scenarios described in this article are illustrative composites drawn from common patterns seen across socio-economic and business analysis work, not references to any specific identifiable client or engagement.

Over the past couple of years, our team has fielded a version of the same request more times than I can count: an organization — sometimes a government agency, sometimes a development institution, sometimes a private employer bracing for change — wants to understand what AI adoption is actually going to do to their workforce and the community around it. Everyone wants the same headline number. How many jobs will be lost. How many will be created. Net positive or net negative.

I understand the appeal of that number. It’s clean, it’s quotable, and it fits in a press release. But after being part of enough of these studies, I’ve become genuinely skeptical of how most AI impact assessments are structured, because the net-jobs number is usually the least useful thing they produce, and it tends to crowd out the analysis that would actually help an organization or a community prepare for what’s coming.

The net number hides more than it reveals

Here’s the problem with leading a socio-economic study with a single net employment figure: it implicitly treats a labor market as one undifferentiated pool, where a job lost in one place and a job created somewhere else cancel out cleanly. In practice, that’s almost never how displacement and job creation actually distribute.

I worked on a study, alongside our research team, looking at automation exposure across a mid-sized regional economy. The topline projection was mildly positive — more roles created over a five-year horizon than eliminated. On paper, that reads as good news. But when we broke the data down by sector, geography, and skill level, the picture split apart. The jobs being displaced were concentrated in specific towns, held disproportionately by workers over 45 without recent formal retraining, in roles that had been stable for decades. The jobs being created were concentrated in different metro areas entirely, requiring skill sets that had almost no overlap with what the displaced workers actually had. The net number was positive. The lived reality for the specific population being displaced was not, and no policy response designed around the topline figure would have addressed what was actually happening on the ground.

This is the core methodological failure I keep seeing: aggregation at the wrong level of analysis. A national or even regional net-employment number is the socio-economic equivalent of reporting average income without mentioning the distribution around it. It’s technically accurate and almost useless for designing an actual response.

Timing matters more than most studies admit

A second pattern I’ve noticed is that most AI impact assessments treat displacement and job creation as though they happen on the same timeline, when in reality they almost never do. Job losses from automation tend to be relatively fast and concentrated — a call center consolidates its staffing once a new system handles routine inquiries, and that happens over a matter of months, not years. Job creation, on the other hand, tends to be slower, more diffuse, and dependent on entirely separate conditions being met — capital investment, training infrastructure, employer willingness to hire and train rather than simply automate further.

When a study models both as if they occur on the same curve, it systematically understates the transition period where displaced workers exist without an adequate pipeline of new roles to move into. I think this matters enormously for how socio-economic research actually gets used, because the policy question an organization needs answered isn’t “will this balance out eventually.” It’s “what happens to the people in the gap, and how long is that gap.” A study that smooths over the timing mismatch gives decision-makers false comfort at exactly the point where they need the opposite — a clear, honest picture of how long the transition window actually is and what it will require to bridge it.

Who actually gets displaced is rarely who gets studied

There’s a selection bias I’ve seen recur across multiple engagements, where the roles that get the most analytical attention in an AI impact study are the roles that are easiest to model — usually white-collar, well-documented positions with clear task inventories, like customer service, basic data entry, or first-tier analysis work. Meanwhile, workers whose jobs sit at the edges of formal employment — gig workers, informal sector labor, contractors moving between short-term engagements — tend to get much thinner treatment, not because they’re less affected, but because they’re harder to count and their employment data is messier.

I think this is a genuine methodological gap, not just an inconvenience. In several regions we’ve studied, informal and gig-adjacent work makes up a substantial share of actual economic activity, and it’s often more exposed to automation pressure than formal employment, because it tends to concentrate in exactly the kind of routine, transactional tasks that current AI tools handle well — scheduling, basic customer interaction, simple logistics coordination. A socio-economic study that under-samples this population isn’t just incomplete. It’s likely to systematically underestimate both the scale and the concentration of near-term impact, because the workers most exposed are the ones least visible in the standard data sources most studies default to.

The skills-gap framing oversimplifies a harder problem

Almost every AI impact study I’ve reviewed eventually arrives at a recommendation involving “reskilling” or “upskilling,” and I don’t disagree that training matters. But I think the framing usually oversimplifies what’s actually a much harder structural problem, and treating it as primarily a training gap can lead organizations toward solutions that don’t match the real barrier.

In the regional study I mentioned earlier, we did a deeper dive into why displaced workers weren’t moving into the newly created roles, even where training programs existed and were reasonably well funded. The barrier, in most cases, wasn’t a lack of available training. It was a combination of geography — the new roles were concentrated in a different metro area than where displaced workers lived — and risk tolerance, since many displaced workers, particularly those with families and mortgages, were unwilling to relocate or take on months of unpaid training with uncertain outcomes, especially mid-career. A training-focused recommendation, however well designed, doesn’t address either of those barriers. What actually moved the needle in that case was a combination of relocation support, employer-sponsored paid training rather than self-funded, and partnerships with local employers to guarantee interview pipelines for program graduates. None of that shows up if your analysis stops at “there’s a skills gap, so build a training program.”

What a more honest impact study actually looks at

Given all of this, the framework our team has moved toward for these engagements looks different from the standard net-employment model. A few things we now build in as a matter of course:

• Disaggregated exposure mapping, broken down by geography, sector, age cohort, and formal versus informal employment status, rather than a single aggregate exposure score. This is the piece that actually tells you where the pain will concentrate, which is the information a policymaker or employer needs to design a targeted response rather than a generic one.
• Separate timelines for displacement and replacement, modeled independently rather than netted against each other, so decision-makers can see the actual transition window they’re planning for, not a smoothed-over average that hides it.
• Explicit inclusion of informal and gig-adjacent labor, even when the available data is imperfect, because excluding it entirely tends to be a bigger distortion than including it with clearly stated uncertainty.
• Barrier analysis alongside skills analysis — looking at geography, financial risk tolerance, and access to paid (versus self-funded) training as distinct obstacles from raw skill gaps, since conflating them leads to recommendations that don’t address the actual reason people aren’t moving into new roles.

None of this is more analytically complex than a standard net-employment projection. It’s more granular, and it requires being honest that the useful answer is messier and less quotable than a single headline figure. But in our experience, it’s the difference between a study that gets cited once and a study that actually shapes what an organization or a government does next — which, at the end of the day, is the entire point of doing socio-economic research in the first place.

Why this matters right now

AI-driven labor market disruption is, without question, one of the most searched and most commissioned socio-economic research topics of the moment, and that demand isn’t slowing down. But a rushed or oversimplified impact study can do real harm — it can lead an organization to under-invest in the transition support that displaced workers actually need, or to over-invest in generic training programs that don’t address the real barriers people face. Our team’s approach has been shaped by watching those mistakes play out in real engagements, and by the specific, sometimes uncomfortable data that emerges once you stop averaging the picture and start actually looking at who’s affected, where, and on what timeline. That’s the level of rigor a topic this consequential deserves.