E2E tests for small WordPress community plugins

There are more than 100,000 plugins avaliable on the WordPress.org Plugins Directory.

A significant percentage of them are likely (quasi-)abandoned. They might be incompatible with the latest version of WordPress, or perhaps they are still functioning. However, without anyone testing them, we just don't know.

Rediscovering an old plugin

The same was true for the Comment Saver plugin, which was first released in 2008. It received a few updates in 2009, but over time, it became one those plugins.

14 years later, in 2023, I stumbled upon this plugin and got in touch with Will, its original author. I offered my help to bring it up to date, and we made it happen.

However, nearly two years have passed since then. While the plugin remains functional, the "Tested up to" metadata hasn't been updated. As a result, its page now shows the dreaded message:

This plugin hasn't been tested with the latest 3 major releases of WordPress. It may no longer be maintained or supported and may have compatibility issues when used with more recent versions of WordPress.

Will has been inactive in the WordPress space for quite some time, so it's no surprise that manually testing the plugin's compatibility multiple times a year and updating the "Tested up to" metadata isn't high on his priority list.

The role of the community

Currently, only those with commit access to the repository, typically the plugin author, can indicate that their plugin is compatible with the most recent WordPress version.

But what happens if those individuals are no longer available? Without automated tests, who ensures that the plugin still functions as intended?

In general, should it always be the plugin author's sole responsibility to test compatibility?

How many of us voluntarily and purposefully test the compatibility of plugins with newer WordPress versions and inform the authors of our findings?

Of course, unintentionally, anyone using a plugin is testing its compatibility with newer WordPress versions when they update their installation. However, they typically only share their findings on the plugin's support forum if something goes wrong.

It would be interesting to explore ways for the larger community to contribute to determining and signaling a plugin's compatibility with recent WordPress versions.

Reducing the burden with E2E tests

Another way we could help reduce the burden of maintaining a plugin is by writing and maintaining automated tests as an alternative to manual testing.

When I updated Comment Saver, I also wrote E2E tests to cover its functionality. However, the test code, along with all the configuration and environment setup, ended up being significantly larger than the plugin itself.

In the end, Will and I agreed that it didn't make sense for him to take on the added complexity and maintenance of the tests. So, we decided not to merge that part.

However, it's not entirely uncommon to keep E2E tests separate from the main chunk of the code. Similarly, it's also not unusual for someone other than the developer to write these tests.

Since I have already written E2E tests for Comment Saver, as well as for another plugin, I plan to maintain and periodically run these myself.

If the tests ever fail when these small, mostly unknown plugins become incompatible, someone will know.