On Wednesday, March 8th, 2023, we will release Matador Jobs 3.8.7 to all our users. This release is a performance-focused update and we are excited to get it in the hands of our users. Let’s dig in.
Performance Improvements
The bulk of our effort that went into this release was to improve performance when communicating with with Bullhorn and maintaining connection reliability. The change notes are short, and don’t really capture the significant effort we put into reviewing every piece of our interface with Bullhorn.
We found some of our users’ issues with performance is the result of Matador’s growth in size and scope impacting the total number of times we needed to communicate with Bullhorn. In 2018, Matador “talked” to Bullhorn once per 30 minutes, now it can “talk” to Bullhorn every few minutes, especially when a logged in administrator is doing work in Matador settings or the WordPress admin. One of the key causes of unexpected disconnections, we found, were many “conversations” happening between Bullhorn and users’ sites at once causing “race condition” bugs. We identified ways to prevent these from occurring in several instances, including moving some on-demand connections into the sync routine and forcing other connections to be prioritized over others. We will continue monitoring this part of our code for further improvements.
While doing the work around “race condition” bugs, we found that one of the worst offenders was a developer/debugging tool, included on all installs of Matador Jobs, that would incorrectly trigger some Bullhorn connections even when not in use. More connections mean an increased likelihood of these “race condition” bugs. We had previously intended to move the tool away from core into a separate “Developer Tools” extension, and we expedited that plan while also fixing the bug should you install the Developer Tools extension.
We also found several issues with how some caching and load balancing systems, which are employed by web hosts to speed up your sites and ensure uptime, were handling WordPress transients. Transients are pieces of information that is temporary in nature but necessary to be persisted between instances of WordPress and PHP while valid. Matador used WordPress transients to sustain aspects of Bullhorn connections. We found some load balancers and caching systems were, as expected, syncing transient data between instances, while others, and especially since WordPress 6.1 which modified performance at a core level including adjusting transients management, were not. To protect our connection transient data, we opted to move away from using WordPress transients for anything connection-related in place of an alternate system we have greater control over and can more reliably expect caching and load balancing systems to handle on a uniform basis.
The key performance improvements, which have massively improved connectivity consistency on the sites we’ve tested them on, are summarized as:
- Prevented “race condition” bugs when multiple concurrent Bullhorn connections are running at once. These will now be considerably more uncommon.
- Removed a developer debugging feature that could interrupt a regular Bullhorn communication if a logged in user was accessing WP Admin during a specific window of time during an active Bullhorn communication.
- Removed any reliance on the WordPress Transients API for handling Bullhorn connection variables, falling back to explicit use of the options database. This is to prevent inconsistent handling of transients by load balancers and persistent object caching systems we’ve encountered since WordPress 6.1 and Matador 3.8.0.
Modifications Admin Notice Emails; No Emails Outside of Production
Admin notice emails were the next on our list to review. They weren’t the cause of connection or performance issues, but they were failing at their job of helping mitigate them. We identified a few key issues with Admin emails:
- Admin Emails all shared a single “timeout.” So an admin email warning about a “Private Candidate” would block an admin email about a disconnection if the former was queued to be sent before the latter.
- Admin Emails would send on all sites, including any set up for testing or development. This led to confusion about when an email required action.
The result was some users had become numb to them. So, we made the following changes:
- Each Admin Notice email (Disconnection, private candidate entitlements, consent object entitlements, cron error) have a separate timeout.
- Admin Notice emails will no longer send for sites that are Development, Test, or Staging environments. It is recommended you leverage the
WP_ENVIRONMENT_TYPE constant, but some common URL schemes from popular hosts automatically trigger the Dev/Test/Staging check and prevents emails on those installs.
The result should be much fewer admin emails and admin emails only when they concern issues about your Production, working web site.
Sidestepped Linux Bug Causing Invalid Resume Errors
Some of our users began noticing some applications would process but get no resume file. They’d check their Matador copy of the application and see a resume file, which they could download, and which would open. Yet Matador was not submitting the file to Bullhorn.
We investigated and found that Matador, through its use of WordPress’s file type validation functions, which used PHP’s file type validation classes, which used the operating systems file system tools, was indeed declaring the files invalid.
Why? Well, it took a while to figure out. And it is a heck of a story, but let us first explain how Matador handles files:
- When a file is input into the application, it is validated in that moment against several checks. Maximum file size and file type are a few. This is to help prevent dangerous files being uploaded to your site.
- Later, when a resume file is submitted to Bullhorn, as well as when any files are saved into Bullhorn, before the file is sent to Bullhorn it is validated again, just in case. This is to prevent your site sending dangerous files to Bullhorn which could get your account revoked or damage your data.
- Even though we check the files going in, we check the files again going out in case your site is compromised and the files are modified by the hack.
The issue, we found, is wild! Here is what we learned:
- For some time and/or still today, when a user creates/created a resume in Google Docs and exports the file as a Microsoft Office Open Office Document (docx), Google Docs would save the file with a doubled MIME-type string.
- A MIME-type is important for systems to validate the file and determine it is safe. The doubled MIME-type string is not valid.
- That said, most of the software that reads these files are able to figure out the bug and validate the file. Even if the bug is now fixed at Google Docs, any docs created with this doubled MIME-type will persist. That said, it appears Unix/Linux’s
libmagic module has not added special handling for this.
- The PHP dev team, which updated the programming language that WordPress and Matador uses, all the way back 2019 considered the issue and decided not to address it as it was being caused “upstream”;
libmagic needs to fix it.
- Since WordPress uses PHP’s filesystem tools that rely on
libmagic, and since Matador uses WordPress’s filesystem tools, the issue, which is still unfixed three and a half years later, impacts Matador users.
Now that we understand the issue, we feel like we can’t wait for it to be fixed “upstream”, so we added special code that would test for the doubled MIME-type string and trick WordPress into accepting the file otherwise. Exact patch notes, in the Changelog, are therefore:
- Fixed an issue causing .docx files generated by Google Docs to fail validation prior to processing to external systems. This actually wasn’t “fixed,” but rather avoided, as the cause of the issue is in
libmagic, the Unix/Linux file handler that PHP relies on. The fix runs an after-validation check on files that fail validation to detect the bugged Google Docs generated MS Office Open Office formatted files.
More Improvements for Salary Range
As we continue to see how you all use Salary Range in the wild, we have additional improvements to the feature.
- Sites that do not have the PHP
intl module will now fail gracefully and return an unformatted string instead of an error. The PHP intl module is recommended by WordPress and required for Matador to format a number like 105000 into something localized, like $105,000. Some users who were running PHP builds without intl module were getting errors during job sync.
- Salary Range would fall back to the value of the
salary field when the high and low were the same non-zero value. Some users, we learned, have begun to use the default salary field for internal uses only and thus we modified the behavior to use the value of the high salary field when high and low are the same.
Other Updates
There are several other smaller and less notable bug fixes and changes. Here are some worth mentioning:
- Added compatibility with the Akismet anti-spam plugin, which was adding junk data to Matador Application submissions.
- Added infrastructure for the rapid development of Developer Tools. Extension developers can explore these in
/src/DeveloperTools/.
- Registered Taxonomies will now display on sidebar menu. This may cause some admin areas to feel cluttered but 3rd party templating tools including Elementor would hide taxonomies that were not in menus.
Update Now!
Matador Jobs 3.8.7 is released for automatic* update to all subscribers as of Wednesday, March 8, 2023. If your subscription has expired, renew it on your account page. If you find any issues, please send a support request.
* Please Note: if your current installed version is 3.8.4 or earlier, you need to manually install this update.
