AI Changed Engineering. Hiring Hasn’t Caught Up.

Reflections | Business Evolution

References: Code is Cheap, Context is King

We’re still testing for skills that matter less and less on the job.

In a previous piece (see References at the top), I wrote about how AI is shifting the role of engineers away from pure implementation and toward problem framing, context, and decision-making.

That shift is already happening in day-to-day work.

But there’s a lag in another part of the system that matters just as much: how we hire.

I’ve been running a number of engineering interviews recently, and one thing keeps standing out.

The way we evaluate engineers hasn’t really changed—but the job itself has.

We still ask candidates to solve algorithm problems on the spot, write code from scratch without help, and recall syntax under pressure. These exercises are familiar, easy to standardise, and relatively simple to score.

They’re also increasingly disconnected from how the work actually happens.

Most interview loops are designed to test individual performance in a constrained environment:

• no external tools

• limited time

• clean, well-defined problems

In that setting, the signal you get is fairly narrow. You learn how quickly someone can recall patterns, how comfortable they are with syntax, and how they perform under pressure.

But that’s not what most engineering work looks like.

In practice, the job is messier. Problems are often unclear at the start. Requirements evolve. Systems have history and constraints. And almost no one is writing code in isolation without access to documentation, teammates, or tools.

The gap between the interview environment and the actual work has always existed.

What’s changed—especially with AI in the loop—is how wide that gap has become.

As I argued in the previous note (see References), AI hasn’t made engineering trivial—it has changed where the difficulty sits.

Writing code is becoming easier. Knowing what to build, why, and how it fits is not.

Given a well-defined problem, an AI can generate a large portion of the implementation quickly. That doesn’t remove the need for skill—it raises the importance of judgment.

The hard part is increasingly:

• understanding what needs to be built

• asking the right questions early

• guiding the solution in the right direction

• validating that the output actually solves the problem

Good engineers don’t just “use AI.” They work with it deliberately. They know how to guide it, when to trust it, and when to question it.

That combination—context, judgment, and effective tool usage—is what drives real productivity.

It’s also not something most interviews are designed to measure.

There’s a deeper mismatch underneath all of this.

In interviews, we often remove tools to “see how candidates think.” We ask them to work from memory, to avoid external help, and to solve problems on their own.

But once they’re hired, we expect the opposite.

We expect them to:

• use documentation

• collaborate with others

• leverage AI tools

• move quickly by using everything available to them

So we end up in a strange position.

We filter candidates based on how well they perform without tools, and then hire them into jobs where success depends on how well they use them.

There’s an obvious pushback here: fundamentals are still important.

That’s true.

Understanding data structures, systems, and core concepts still matters. But memorising solutions or recalling syntax under pressure is a weak proxy for that understanding.

You can be excellent at solving interview-style problems and still struggle with real-world systems. The reverse is also true.

If the goal is to predict how someone will perform on the job, we need to be careful about what signals we rely on.

Take something as straightforward as a login system.

If you ask an AI to build one, it will give you a working result—something with authentication, validation, and a basic user flow. On the surface, it looks complete.

But the moment you step outside that narrow definition, the gaps start to show. Should it support enterprise SSO? Are there compliance or audit requirements? How does it behave under failure conditions, or integrate with the rest of the product?

None of those questions are about code. They’re about context.

The AI didn’t get it wrong—it was never given the full picture to begin with.

If the work has changed, the evaluation should follow.

That doesn’t mean lowering the bar. It means testing the right things.

For example:

Give candidates problems that are slightly ambiguous and see how they clarify them

Let them use documentation or AI tools, and evaluate how they work with them

Ask them to modify or debug an existing piece of code instead of starting from scratch

Spend time understanding their decisions, not just their final answer

These approaches are harder to standardise, but they’re much closer to the actual job.

Engineering has been moving in this direction for a while, and AI has accelerated it.

The role is shifting away from pure implementation and toward problem framing, decision-making, and effective use of tools.

Hiring, for the most part, hasn’t caught up.

And until it does, we’ll keep selecting for strengths that matter less, while overlooking the ones that matter more.