Plugin life support with E2E tests

Those who have plugins in the WordPress Plugin Directory and rely on manual testing face a challenge in avoiding the plugins being marked as "out of date".

When this occurs, a notice is displayed:

This plugin hasn't been tested with the latest 3 major releases of WordPress. It may no longer be maintained or supported and may have compatibility issues when used with more recent versions of WordPress.

This is often interpreted as a sign that the plugin has been "abandoned."

A lot has been said about "abandoned projects" in the open-source world, including the WordPress space. Various ideas have been proposed to address this issue, from promoting plugin adoption to creating maintenance programs.

Are we in a better situation than we were ten years ago?

One thing is for sure: there's no way around testing. The only way to know something is still working as expected, as advertised, is by somehow checking it.

Manual testing over time is draining, unsustainable, and definitely not scalable. And I would go as far as to say the reason for "abandonment" in some cases.

The best chance: E2E tests

Among the many types of automated tests, end-to-end (E2E) can be deployed the easiest. It is the least invasive and does not require any upfront code changes.

Since most of the plugins were released, E2E testing has become easier and more accessible. And we are close to some reliable no-code, "AI" assisted E2E tools and solutions.

But until then, the most future-proof solution is to use Playwright. This is the tool that WordPress core plans to migrate to, and it is already being used by Gutenberg.

Maintaining the tests for others

While updating a small plugin, I added some E2E tests. However, they were not merged because the maintainer did not want to handle the complexity of the tool.

This decision was completely fair, considering the size and usage of the plugin, among other factors.

Nevertheless, the E2E tests themselves are not the problem; it's the maintainability of the tool. That's why I decided not to discard the tests and took on the responsibility of maintaining the Playwright setup myself.

My plan is to run the tests regularly and notify the developer if anything breaks. Likewise, maybe give them a ping that everything is smooth. Finger crossed.

Who knows, I might add tests for other plugins over time. There are a few plugins that I would like to see around in the next few years.

If nothing else, when the time eventually comes, these tests can provide a solid foundation for forking the project, refactoring it, and adding new features.