Skip to main content Skip to page footer

Automated Tests

Why we rely on Playwright for the AI Suite

Automated testing is a central part of our development strategy – especially for a complex application like the AI Suite, which connects TYPO3, AI APIs, and various system components. We face diverse challenges: different TYPO3 versions, complex backend interfaces, cross-system processes, and costly AI interfaces.

In this article, we will show you how we have built a sustainable testing strategy, which tools we have tried, why we ultimately chose Playwright – and how we are currently automating over 300 stable, repeatable test cases with it.

Table of Contents

What are the challenges?

Anyone who develops TYPO3 projects with AI features professionally knows the typical hurdles:

  • Testing in the TYPO3 backend: complex UI, partly nested content, and iframes
  • Cross-system tests: interaction with servers, APIs, and other frameworks
  • Different TYPO3 versions: test coverage for v11, v12, and v13
  • Costs due to API requests: especially when testing AI services, every call is cost-relevant

These aspects significantly complicate automated testing – both technically and organizationally.

Where to start?

At the beginning, we searched for a structured, practical solution. We wanted to develop a testing strategy that works in the long term. For this, we analyzed various frameworks and clearly defined our requirements:

  • Scalable for multiple TYPO3 versions
  • Decoupling of frontend and server logic
  • Good maintainability and stability
  • Reduction of unnecessary costs in AI tests

Which frameworks did we try?

During the evaluation phase, we tested four tools:

TYPO3 Testing Framework

Well integrated, but limited to TYPO3-specific scenarios.

More about the TYPO3 Testing Framework

Codeception

Good with Selenium, but with complex dependencies.

More about Codeception

Cypress.io

Basically offers good approaches for our challenges – but runs tests within its own iframe, which causes problems in the TYPO3 backend.

More about Cypress.io

Playwright

Provides out-of-the-box solutions for many of our challenges.

More about Playwright

After several trials, the choice fell on Playwright – particularly because of its flexibility, speed, multi-browser compatibility, and the ability to test our systems independently of the framework.

The first attempts ... and the pains

Our first attempts quickly "hit the nail on the head," but also offered immediate optimization potential:

Tests against live data

Including real AI requests, which were expensive and unreliable

Random generation

The random generation led to unstable results

Identical tests for different TYPO3 versions

For each TYPO3 version, nearly identical tests ran with high maintenance effort

The consequence: unstable tests

Additionally, little reliability as well as high costs and maintenance effort

New/better approaches ... and optimizations

Through targeted changes, we were able to make tests significantly more stable and efficient:

Separation of TYPO3 frontend and server/API tests

Fewer dependencies, clearer responsibilities

Unified test database

Same data basis for client, server, and shop

Standardization of test cases

Version-specific logic is outsourced

Result: More stable, repeatable tests

Additionally, less maintenance effort and lower costs

Conclusion

Automated end-to-end testing with Playwright is not just a "nice to have" for us, but essential. As the functionality of the AI Suite grows, so does the complexity – and thus the demands on our quality assurance.
Playwright gives us the assurance that everything works as it should – even in future TYPO3 versions, in different browsers, and with various user workflows.

The combination of modern AI and a solid testing strategy makes the AI Suite what it should be: Reliable. Future-proof. Strong in everyday use.

Demo Repository

Our demo repository exemplifies how we use Playwright in practice – including visual regression tests, test structure, browser setup, and a simple use case. Ideal for understanding our approach or starting your own tests on a similar basis.

Get started now