MINISTRY OF TESTING

Self-explaining
test automation

Fast root cause diagnosis for PR-blocking E2E failures

Alexander Chumakin / QA Lead @ JetBrains

FAILED ASSERTION

+ SERVICE DIFF

+ LOGS / TRACE

=> WHY?

WHY I CARE ABOUT THIS LOOP

QA Lead @ JetBrains

I work around the boundary between product teams, E2E automation, CI/CD, and release confidence.

tests
pipelines
process
shared context

THE OLD COMMUNICATION PROBLEM

Two tracks, one merge gate

DEVELOPERS
feature PR review
QA AUTOMATION
test case fixture report
?

same product,
different context

PROCESS IS STILL HARD IN AGILE TEAMS

The expensive loop

CI fails
Dev asks QA
QA digs artifacts
cause explained

handoff latency

TOO MUCH DATA, NOT ENOUGH DIAGNOSIS

What a red check says

FAILED

But the developer still has to ask:

what changed?

JUnit XMLtrace.zipvideo screenshotservice.logdiffHTML report

REPORTING ALREADY DOES A LOT

The tools are not the problem

evidence
is already
there

AllureReportPortalscreenshots videofailure contexttracecustom dashboards

THE MISSING STEP

Diagnosis is correlation

assertion service log trace/video code diff
synthesis layer

WHY

A diagnosis is a small story backed by evidence.

NOT MAGIC, JUST SYNTHESIS

Where AI helps

logstest source AI diffassertion

Root cause  /  Evidence  /  Suggested fix

HOW WE THINK ABOUT IT AT JETBRAINS

AI adoption is process adoption

code reviewautomationsmart reminders JetBrains teams process feedbackbug fixing

not only code gen

FROM GATEKEEPER TO DEVELOPER ALLY

Shorten the feedback loop

BEFORE

red check
ask QA
manual digging

AFTER

red check
AI diagnosis in PR
developer fixes

PUBLIC DEMO SHAPE

Reusable workflow architecture

service repo
checkout UI
reusable E2E
workflow
artifacts
+ diff
OpenAI
analysis
PR summary
comment
Focused service diff

PR-READY, NOT REPORT-READY

What should it explain?

Root cause
Evidence
Suggested fix

Useful only if tied to collected artifacts and the suspicious change.

AI diagnosis artifact

KEEP THE TEST AS SOURCE OF TRUTH

Guardrails

tests
still
decide

AI summarizes evidence No flaky claims without logs Fallback if API is unavailable

A SMALL BUG WITH A CLEAR SIGNAL

Demo story

clean main
accepts public email
bug branch
requires @company.test

E2E catches it

Checkout UI failure

BEFORE WE OPEN GITHUB

Three takeaways

Context

collect evidence around the failure

Ownership

write for the developer who changed the code

Flow

publish diagnosis where the decision happens

LIVE DEMO

Let's break
the checkout

PR → E2E → artifacts → AI → comment