[Testing] When an expert meets an LLM, or what the "hybrid neuro-symbolic" paradigm is and how it will save your project from AI slop.

The truth is that programmers are lazy. And when it comes to tests, everyone wants to have them, but hardly anyone wants to write them. In today’s AI era, we often hand that job, just like writing the actual code, to coding agents. Unfortunately, agents tend to repeat in their tests the same mistakes they made in the code. Or they test only the simple, obvious cases.

But AI is not the only way to create tests. The first ideas for generating tests automatically appeared as early as the 1970s. In 1976, Webb Miller’s research laid an excellent theoretical foundation for test generation. Years later, building on that research, Mark Harman and Bryan Jones coined the term SBST, Search-Based Software Testing, in 2001.

In this article I want to show that combining SBST with AI-generated tests gives remarkable results: tests that not only catch hard-to-predict bugs, but are also readable and easy to maintain. I will also try to show how to bring this hybrid into your workflow.

What is SBST?

TL;DR SBST works a bit like natural selection. It generates a batch of tests and checks which of them exercise the most code. It simply throws away the weak or repetitive tests, and slightly modifies the best ones to create a new, improved version. It repeats this cycle until it has squeezed the maximum number of bugs out of the code.

Search-Based Software Testing rests on a rather bold assumption: that the code, as it is, works correctly. It also ignores entirely the business meaning of the code under test. It works as a simple loop.

1️⃣ Initialization: We take existing test cases or generate completely random starting scenarios.

2️⃣ Mutation and modification: The algorithm changes the selected tests at the level of code and data. It does this in 3 ways:

• Data (parameter) mutation: swapping values for extreme ones (e.g. 0, MAX_INT, null, unusual dates).
• Step-sequence mutation: changing the structure of a test (adding, removing or reordering steps).
• Crossover: splitting two tests and joining them into one (e.g. the polished setup from the first + the assertions from the second).

3️⃣ Evaluation (fitness function): We run the modified tests and score them. The algorithm checks not only whether a test covered a new part of the code, but also how close it came to satisfying hard conditions (e.g. entering a particular if).

4️⃣ Selection and succession: Based on the scores, the algorithm decides which tests survive. To find the best ones and avoid getting stuck in a dead end, it uses several strategies:

• Elitism: a guarantee that the very best test passes to the next iteration unchanged.
• Tournament selection: drawing a small group of tests and picking the best of them (it gives weaker tests a chance and keeps the population diverse).
• Discarding duplicates: tests that give the same result are removed, so the algorithm does not get stuck generating the same thing over and over.

Problems with SBST

As I wrote above, SBST works on the assumption that the code under test works correctly. That caveat touches one of the most important problems in testing theory: the test oracle problem.

There is one exception to that assumption. SBST can detect bugs without assuming the code is correct if the program fails critically (e.g. throws an unhandled NullPointerException or overflows a buffer). To the algorithm, a crash is a universal signal of a bug, whatever the business logic.

That is why, in the chart below, which shows how effective SBST is using the popular tool EvoSuite as an example, the rate of detecting real bugs is high.

It also does fairly well on path and branch coverage. But that is a rather deceptive metric: SBST is indeed good at finding possible scenarios when simple numbers or text variables are involved. But when a test needs to guess an exact Polish national ID number (PESEL) or a URL ending in .png, the algorithm may need a great many iterations to find the right values. Sometimes there are too many of those iterations to run them in real conditions.

On top of that, as the chart shows, a test generated this way is hard for a person to read. This is a direct consequence of the test’s weak semantics. For the person reading the code, it is a chaotic collection of tests and assertions where they have to work out the meaning of each test every time.

LLM to the rescue

The answer to these problems is to use an LLM in some steps of the SBST loop. More precisely, when the algorithm detects that further SBST loops are not giving enough improvement, AI is used to mutate the current pool of tests. Knowing the business context better, the LLM improves the test cases more than the existing SBST algorithms do. On top of that, AI is better at describing those tests, so they naturally become much more readable for people.

That is exactly the approach taken by the group of researchers who developed CodaMosa. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri and Siddhartha Sen, on behalf of Microsoft, combined OpenAI Codex with the Pynguin tool, which uses an SBST algorithm called MOSA. And the results they achieved in 2023 turned out to be stunning. Hybrid test generation proved satisfactory on almost every front.

This approach has become the flagship example of the so-called neuro-symbolic paradigm in software engineering. It combines the “neuro” (the semantic intuition and understanding of the code’s intent that language models bring) with the “symbolic” (the mathematical, impartial and algorithmic rigor of SBST systems). This sidesteps the biggest weaknesses of both technologies: the LLM protects SBST from semantic blindness, and SBST protects the code from the hallucinations and compilation errors that AI generates.

The results published at the ICSE conference in 2023 confirmed the large advantage of this approach. The researchers tested CodaMosa on 486 demanding Python modules. The hybrid achieved statistically significantly higher code coverage than SBST alone on 173 modules, and higher than the Codex model alone on 279 modules. That translates into an average increase in code coverage of 9% to 10%.

What next? How can we use this in our daily work?

Moving from traditional testing, or from uncritical AI test generation, to a reliable hybrid workflow (a Neuro-Symbolic Developer Workflow) does not require writing your own research tools from scratch. We can bring these principles into our teams with well-designed CI/CD and configuration of AI agents (such as Cursor or Claude Code).

Here are four concrete steps for turning the findings of the empirical research into everyday engineering practice:

1. Replace raw prompts with the “intent extraction” technique. Make the agent analyze the business requirement before it writes tests, e.g. by reading it from a GitHub issue or a Jira ticket. Let it try to describe the business requirement in its own words.
2. Set up an automatic repair loop with feedback (Feedback Loop). Do not let developers fix compilation errors in AI-generated tests by hand. Your development workflow or agent script should automatically run a local build, capture the errors from the console (stdout/stderr) and send them back to the LLM with a request for a fix
3. Introduce an iron rule of testing with tautological validation (“Fails Without, Passes With”); it is a bit like test-driven development. First, the agent writes a test based on the business requirement, and checks that the test fails; otherwise it writes it again (the “fails without” phase). Only then does the agent implement the production code, until the test it wrote starts passing (the “passes with” phase).
4. Use AI to break through coverage plateaus in SBST. Instead of asking LLMs to generate whole test suites from scratch (which gives low coverage in large projects), integrate them with SBST tools following the same methodology as CodaMosa. Let a traditional SBST tool do fast, cheap and systematic exploration of the code. When the tool hits a coverage barrier, run a script that asks the LLM to generate only the missing, semantically correct input data (e.g. a correctly formatted JSON object). Inject that data as semantic seeds back into the SBST genetic algorithm