AI-generated test scripts can save time, especially when they draft the first version of a login flow, a checkout path, or a form submission test. The problem starts when the script looks correct, runs without obvious errors, and still asserts the wrong thing. That is where hallucinated assertions show up: checks that appear precise but are disconnected from actual product behavior, unstable UI state, or the real contract you meant to verify.

This guide is about how to debug hallucinated assertions in AI-generated test scripts without falling into the trap of trusting the output just because it is syntactically valid. The goal is not to reject AI-generated tests outright. It is to treat them like untrusted code that needs review, instrumentation, and evidence before it becomes part of your regression suite.

A test that passes for the wrong reason is often worse than a failing test, because it teaches the team the wrong lesson.

What a hallucinated assertion actually is

A hallucinated assertion is any assertion that claims to validate a behavior, but in practice does not verify the intended user outcome or system contract. In AI-generated tests, this usually happens in one of four ways:

  1. The assertion checks a UI string that is unrelated to the behavior under test.
  2. The assertion targets a locator that matches the wrong element, but still exists on the page.
  3. The assertion validates a state that is briefly true during rendering, not the final state.
  4. The assertion is too generic, so it passes even when the product is broken.

For example, an AI might generate a Playwright assertion like this:

typescript

await expect(page.getByText('Success')).toBeVisible();

That looks reasonable until you realize the page has multiple “Success” labels, including a hidden toast template, a help tooltip, or a banner shown on a different workflow. The test may pass even if the actual order submission failed.

The underlying issue is not just bad code generation. It is a mismatch between the model’s pattern completion and the product’s actual verification needs. To debug these failures, you need to inspect the assertion at three levels, the selector, the state being checked, and the business meaning of the check.

Why AI-generated assertions fail in believable ways

AI-generated tests often fail subtly because the model is good at writing plausible automation patterns, but it does not truly know your application state or data model. It guesses from context. That creates several common failure modes.

1. Locator optimism

The model picks a locator that is readable and likely to exist, such as text, role, label, or CSS class. But readable does not mean unique or stable. A locator like getByText('Submit') may match multiple buttons, off-screen clones, or a translated string that changes with locale.

2. Assertion drift

The script may be generated against one screen, while the app has since changed. The test still “works” because it asserts on a nearby piece of text, not the real outcome. This is common when a generated test is copied into a suite without human review.

3. Phantom success conditions

AI-generated tests may rely on visible success messages, but those messages can be shared by multiple flows. A generic toast or notification can create a false positive if it appears for the wrong action.

4. Waits that hide bugs

Some generated tests use broad waits, sleeps, or generic network idle checks. This can mask race conditions. The assertion passes because the app eventually reaches a predictable state, not because the user journey is correct.

5. Overfitting to the current DOM

AI can infer a selector from the live DOM and generate a check that works only for the current implementation. If the same UI element changes structure, the assertion fails, or worse, keeps passing while pointing at the wrong node.

First rule, debug the assertion before you debug the app

When a generated test misbehaves, teams often start by assuming the product is broken. That is the wrong default. First ask whether the assertion itself is trustworthy.

A good debugging sequence is:

  1. Confirm that the selector identifies the intended element.
  2. Confirm that the element’s state is the one you want to validate.
  3. Confirm that the assertion reflects a user-visible or contract-level outcome.
  4. Confirm that the assertion fails when the behavior is intentionally broken.

That last step is the most important. If you cannot make the assertion fail on demand, it is not a useful test.

A practical workflow for debugging hallucinated assertions

Step 1: Reduce the test to the smallest meaningful path

Strip the test down to the minimum interaction needed to expose the assertion. If the issue is a checkout confirmation, remove unrelated navigation, setup, and cleanup. This helps you see whether the failure is in the assertion or earlier in the flow.

For example, in Playwright:

import { test, expect } from '@playwright/test';

test('submits the form', async ({ page }) => {
  await page.goto('/contact');
  await page.getByLabel('Email').fill('qa@example.com');
  await page.getByRole('button', { name: 'Send' }).click();
  await expect(page.getByText('Thanks')).toBeVisible();
});

If this test passes, do not stop there. Ask whether “Thanks” is specific enough, whether it appears on other pages, and whether it is tied to the exact submission you care about.

Step 2: Inspect the locator in the browser, not just in the script

Open the page, run the same locator in devtools or your automation runner, and verify what it actually matches. Many hallucinated assertions come from a locator that is syntactically valid but semantically wrong.

Check for:

  • multiple matches
  • hidden matches
  • duplicated components in modals, drawers, or responsive layouts
  • localization changes
  • components rendered by portals

If your locator is too broad, tighten it using accessibility role, name, hierarchy, or a test id with a stable contract.

Step 3: Verify the asserted state is observable and stable

A frequent mistake is asserting immediately after an action that triggers asynchronous rendering. The test may pass or fail depending on timing. If the generated assertion uses a transient state, replace it with a stable condition.

For example, instead of checking for an element to appear immediately after a click, assert on the resulting URL, API response, or a persisted entity count if that is the real outcome.

typescript

await expect(page).toHaveURL(/\/orders\/\d+$/);
await expect(page.getByRole('heading', { name: 'Order Details' })).toBeVisible();

This is better than asserting on a success toast alone, because it verifies a durable state transition.

Step 4: Create a negative test to prove the assertion can fail

If you can, intentionally break the condition the assertion is supposed to validate. For example, change the selector to a different element, mock the API to return an error, or insert a wrong value. The assertion should fail.

If it still passes, the assertion is too weak.

A useful test must be falsifiable. If it cannot fail when the behavior changes, it is not protecting anything.

Patterns that often produce false assertions

Generic success text

AI models love phrases like “Success,” “Saved,” or “Completed” because they are common UI patterns. The problem is that these words often appear in multiple contexts.

Prefer assertions that tie the outcome to the object under test. For example, verify the order number, the submitted email, or the new record count, not just a success label.

CSS class assertions

Generated scripts sometimes assert on class names because they are easy to query. This is fragile and often misleading, especially in CSS module, Tailwind, or component library setups where classes are implementation details.

Avoid checks like:

typescript

await expect(page.locator('.alert-success')).toBeVisible();

Unless .alert-success is an explicit test contract, this is just implementation coupling.

Partial text matching

Text matching is a frequent source of accidental passes. A test that looks for "Paid" might match "Unpaid", "Paid out", or "Payment failed" in a surrounding context. Use the narrowest useful scope.

Checkpoints based on loading states

Some AI-generated tests assert that loading spinners disappear or network requests complete, but they do not verify the final user-facing effect. That can hide backend failures if the UI clears the spinner regardless of response content.

How to validate locators before trusting assertions

Locator validation is one of the fastest ways to catch hallucinated assertions. Treat each locator as a hypothesis about the DOM, not as a truth.

Use accessibility-first locators when possible

Roles, accessible names, labels, and text relationships are usually more stable than CSS selectors. In Playwright, this often means using getByRole, getByLabel, or getByText with caution.

typescript

await page.getByRole('button', { name: 'Create invoice' }).click();
await expect(page.getByRole('heading', { name: 'Invoice created' })).toBeVisible();

This is stronger than a structural selector because it captures how a user would identify the element.

Check uniqueness explicitly

If your framework allows it, confirm that the locator resolves to one element only. Multiple matches are a red flag for hallucinated assertions.

typescript

const saveButtons = page.getByRole('button', { name: 'Save' });
await expect(saveButtons).toHaveCount(1);

Verify the locator is visible and interactable

A hidden element can still satisfy a text assertion. You want to know whether the actual control is visible to the user.

Tie locators to a business event

If a test is about order completion, use selectors that correspond to the order summary, order number, or persisted confirmation, not just a shared toast.

Distinguishing a false assertion from a flaky check

Hallucinated assertions and flaky checks are related, but they are not the same problem.

  • A false assertion is wrong in principle, because it validates the wrong thing.
  • A flaky check is right in intent, but unstable in execution.

For example, checking for a confirmation message after an API call may be a valid assertion, but if the UI animation delays rendering, it becomes flaky. Meanwhile, checking for the text "Saved" anywhere on the page might be stable, but still false if it does not refer to the right action.

When debugging, decide which one you have:

  • If the assertion passes even when the feature is intentionally broken, it is false.
  • If the assertion sometimes fails on healthy builds, it is flaky.

The fix for each is different. False assertions need better semantics. Flaky checks need better synchronization, better locator strategy, or stronger state control.

Replace weak UI assertions with contract-level checks

If the point of the test is business correctness, the best assertion may not be a UI assertion at all. AI-generated tests often default to visible text checks because they are easy to generate, but you may get better reliability from API or storage validation.

For example, after submitting a form, you could verify the backend record directly:

typescript

const response = await page.request.get('/api/orders/123');
expect(response.ok()).toBeTruthy();
const order = await response.json();
expect(order.status).toBe('confirmed');

This is especially useful when the UI is highly dynamic and the real contract is an API response or database record. Be careful, though, because overusing backend assertions can miss UI integration issues. The right balance is usually one or two contract checks for each critical user path, plus UI coverage for rendering and accessibility.

Build a failure-oriented review checklist

When reviewing AI-generated test scripts, use a checklist that assumes the assertion is guilty until proven otherwise.

Assertion review questions

  • What exact behavior does this assertion verify?
  • Could it pass if the wrong element is rendered?
  • Could it pass if the underlying action failed?
  • Is the asserted text unique to this workflow?
  • Does the assertion depend on timing or animation?
  • Would a user consider this a meaningful success condition?
  • What happens if the page contains duplicate labels or translated text?

Locator review questions

  • Does the locator target a stable public contract, such as accessibility role or test id?
  • Can the same locator match multiple elements?
  • Is the element hidden, disabled, or off-screen when the assertion runs?
  • Is the locator sensitive to layout changes?

Data review questions

  • Is the test using realistic data, or can any value satisfy the assertion?
  • Does the test reuse fixed emails, IDs, or names that might collide?
  • Is the data isolated between runs?

A debugging pattern for Playwright, Selenium, and Cypress teams

The framework matters less than the debugging discipline, but each tool has slightly different strengths.

Playwright

Playwright gives you strong locator APIs, traces, and web-first assertions. Use them to confirm locator behavior and timing.

typescript

await expect(page.getByRole('alert')).toContainText('Invoice saved');

If this passes unexpectedly, inspect whether there are multiple alerts or whether the message is reused elsewhere in the app.

Selenium

Selenium-based suites often need more deliberate waits and more explicit locator inspection. Avoid ad hoc sleeps. Use explicit waits for concrete conditions, then verify the resulting state.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10) wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ‘[role=”alert”]’))) assert ‘Invoice saved’ in driver.page_source

Even here, page_source is only a temporary debugging aid. For production assertions, target the specific element text, not the whole DOM.

Cypress

Cypress command chaining can hide timing assumptions if you are not careful. Use targeted assertions and inspect the yielded subject.

javascript cy.contains(‘[role=”alert”]’, ‘Invoice saved’).should(‘be.visible’);

If this is too broad, narrow the scope further by selecting the container that corresponds to the specific workflow.

Use logging and trace data to expose the mismatch

Most hallucinated assertions become obvious when you inspect the actual state transitions. Use trace files, screenshots, console logs, network logs, and DOM snapshots to answer a single question, what exactly did the test observe when it passed?

Look for:

  • the wrong toast or banner
  • stale data from a previous run
  • cached API responses
  • hidden duplicate elements
  • wrong route after navigation
  • mismatched test fixture data

If your automation tool supports traces, compare a passing run and a failing run side by side. A hallucinated assertion often reveals itself as “the page changed, but not in the way the test author thought.”

Fixing the assertion, not just the symptom

Once you find the issue, choose the fix based on the type of mismatch.

If the locator is too broad

Narrow it with role, label, parent context, or a stable test id.

If the message is too generic

Assert on a unique identifier, record ID, URL fragment, API response, or row content.

If the state is transient

Wait for the stable endpoint of the workflow, not the transient loading indicator.

If the assertion is not user-relevant

Replace it with an outcome that a real user or API consumer would recognize.

If the test is checking implementation details

Move the assertion closer to the product contract, or split it into one test for UI behavior and one for backend correctness.

A simple rule for choosing better assertions

A good assertion usually satisfies at least two of these three conditions:

  1. It is specific to the action performed.
  2. It is stable across minor UI changes.
  3. It reflects a meaningful user or system outcome.

If your assertion only satisfies one, it is probably too weak or too brittle.

CI makes hallucinated assertions more expensive

In local debugging, a false assertion can look like a harmless green check. In CI, it becomes a silent risk. In a pipeline, especially one tied to continuous integration, a passing but meaningless test can let regressions ship for weeks.

That is why AI-generated tests need stricter review than manually written tests. A developer who wrote the assertion usually has some mental model of its intent. A generated test may only have surface validity. Before merging it into CI, confirm that the assertion fails when the feature is broken and passes only when the intended state is present.

For teams building a robust test automation stack, this also means maintaining a higher bar for acceptance than “the script runs.” The better question is whether the script provides signal.

Practical debugging checklist

Use this checklist whenever an AI-generated test seems suspiciously green:

  • Run the test with tracing or verbose logging.
  • Confirm the locator matches exactly one intended element.
  • Check whether the assertion depends on shared text or shared UI components.
  • Verify the action produces a durable state change.
  • Replace transient UI-only checks with a more stable outcome when appropriate.
  • Break the product on purpose and ensure the assertion fails.
  • Review whether the test is validating business value or implementation detail.

If you want a shorthand: locate, observe, falsify, tighten.

When to keep the generated test and when to rewrite it

Keep the test if:

  • the locator is stable and unique
  • the assertion maps clearly to the user outcome
  • the test fails reliably when the behavior breaks
  • the assertion is not overspecified

Rewrite the test if:

  • the assertion uses generic success text
  • the locator matches multiple elements
  • the test passes despite a broken workflow
  • the AI has inferred a brittle DOM structure
  • the test is really checking implementation details, not behavior

Sometimes the fastest fix is not to patch the generated test, but to replace it with a simpler, stronger assertion. Fewer assertions with clearer meaning often beat a long generated script full of incidental checks.

Final takeaway

Debugging hallucinated assertions in AI-generated test scripts is mostly about discipline, not tooling. The model can help you draft the outline, but it cannot know whether your assertion reflects the real contract unless you verify it. Treat every generated assertion as provisional, validate the locator, test the failure mode, and prefer stable outcomes over attractive but generic text checks.

If you do that consistently, AI-generated tests become useful accelerators instead of quiet sources of false confidence.