Thinking: Our Blog

Ruminations on the WordPress Taxonomy System

Thought up on January 3, 2012 ! What do you think?

One of our most recent projects was to create a WordPress theme that offered versions in six different languages (two of which were UK vs. US English dialects.) The main feature of this theme was to be easily able to switch between languages, translating not only static screen elements, but also dynamic elements that come from the database. Oomph developed a theme that offers all the functionality necessary for these requirements, but in the development process, a flaw in the WordPress taxonomy system was revealed. The succinct definition of this problem is this:

Identical terms that appear in two different taxonomies are still the same term.

Yet, They are treated by the WordPress interface as if they are not.

The Problem

For a simple example, suppose we had two taxonomies, fruits and colors. In the fruits taxonomy, we had the following terms:

Apple
Orange
Banana

And in colors, the following:

Red
Orange
Yellow

Suppose we wanted to change all of the colors to use lower-case instead:

red
orange
yellow

We now see that the change to the term “Orange” is also reflected in the fruits taxonomy:

Apple
orange
Banana

This is obviously not what we want. Conceptually, Orange the fruit and Orange the color are distinct entities, but to WordPress, they are the same thing.

The Project

The problem above was fully fleshed out while developing one of our most recent and most challenging projects. This project required us to be able to translate everything seen on the screen for an end-user into 5 distinct languages: English, French, Italian, German and Dutch; as well as into 2 variants of English, UK and US, for a total of 6 different translations for each string seen by the user. Besides regular translateable strings that were coded into the theme using the WordPress I8n API, we also had to manage translations for certain taxonomies, such as category.

Since these strings would be coming from the database, employing the WordPress I18n API for translating these strings via .PO/.MO files would not have been the correct way of approaching this problem, as then we would have been passing variable values into the __() suite of functions. This poses a problem since the scripts we use to generate the .POT files to be translated would not know which string is actually in need of translation.

Another down-side of having used PO files for translating these strings would be that there would be no separation of language category taxonomies, which means that per-language category terms would have to mirror each of the other ones at all times, and there would be no opportunity for using a foreign-language term that does not appear in either the base (untranslated) taxonomy.

In order to achieve translated taxonomies, we first created a taxonomy, “languages” that would store each of the languages that were used within our system, with the terms:

The -language suffix on these term slugs was a necessity because of slug collisions with another taxonomy.

   en-uk-language - English (UK)
   en-us-language - English (US)
   fr-fr-language - French
   nl-nl-language - Dutch
   de-de-language - German
   it-it-language - Italian

The terms from this taxonomy were in turn used to define a taxonomy for each one of the taxonomies we needed to translate:

   category_en-uk-language
   category_fr-fr-language
   ...

And so forth. For each term used in one of these translated taxonomies, the helpful “description” field is used to link it back to its untranslated base:

As an aside, the hijacking of the “description” field as a substitute for proper Taxonomy metadata is a time-honored WordPress hack that is used by many themes. The alternative would be to store taxonomy metadata in the options table, but this method does not scale as well, since it bloats the options table and would require two database reads to pull meta data out for every term. Another alternative for dealing with this issue would be to use a custom table, but this sort of thing is simply not allowed in the WordPress.com/Wordpress.com VIP environment, adds extra complexity, and would still require extra database reads to pull the taxonomy meta data.

In (US) English, we might have a category structure akin to the following:

Job Market
-- Pay
Advice
-- Insight
-- Resumés

Which in French, for example, would look like:

Marche de l'emploi
-- Salaires
Conseils
-- Aperçu
-- CV's

But also might have slight differences for UK English vs. our base theme language of US English:

Job Market
-- Salaries
Advice
-- Insight
-- CV's

Looking at our category structure above, we see that both UK English and US English are using “Job Market” as a category, but UK English and French both use the term “CV’s” instead of the US English “Resumés.”

Suppose we wanted to change the UK English category to read “Job market” instead, with the second word in lower-case. We would make this change using the edit terms screen in the WordPress back-end. Now we look at the US English version of “category”:

Job market
-- Salaries
Advice
-- Insight
-- Resumés

Whoah! Back the truck up! That change also got applied to the “Job Market” term in the US English category. What happened here?! It seems that even though the “Job Market” category appears twice in separate category taxonomies, changes to one will affect the other. We’d encounter the same problem if we tried to change French “CV’s” into “CVs”, without an apostrophe: the change will also propagate to the UK English category name. This is definitely not what we want.

Problem Details

The complication here arises from the WordPress taxonomy data schema. There are two tables in the WordPress schema that are responsible for storing taxonomy data:

wp_terms

and

wp_term_taxonomy

A third table, wp_term_relationships links taxonomy terms with the posts to which they are attached, but I don’t necessarily consider it part of the taxonomy data storage system.

The fields of interest are:

wp_term_taxonomy.taxonomy
wp_term_taxonomy.parent
wp_term_taxonomy.description
wp_terms.name
wp_terms.slug

  and finally, the relating field: 

wp_term_taxonomy.term_id = wp_terms.term_id

This is a “normalized” schema, in the sense that objects that are ostensibly the same appear only once: a distinct term will only ever appear once in the wp_terms table, keyed by its slug.

Here is where we get to the core of the above-stated problem:

Identical terms that appear in two different taxonomies are still the same term.

Yet, They are treated by the WordPress interface as if they are not.

This is because WordPress is doing everything it can to keep the terms/term_taxonomy tables normalized. If we have a term, “Job Market”, and it appears in both UK English Category and US English Category, then any changes to one will necessarily be made to the other, since they are sharing the same entry in wp_terms.

Why is this a problem? For me, it boils down to the following issues:

1. Terms in two different taxonomies are very rarely conceptually the same thing, even if they have the same text.
2. The WordPress administrative interface implies that a term in two or more taxonomies are NOT the same thing (they appear in different locations, after all.)
3. From a database engineering standpoint, it’s an extra, unnecessary table in the schema.

Why is it designed this way? As with many quirks in today’s software, it boils down to “legacy code.” WordPress grew organically over time, and certain decisions were made at the time in order to add features, that seemed reasonable at the time, given the problem being solved at the time, that did not fully predict future implications. WordPress was originally devised as blogging software, but is more and more being used for general CMS purposes. As a general-purpose CMS, complex taxonomy schemas such as the one we used for this recent project would be required for storing data into distinct buckets, which might have complex and overlapping domains. But suppose we could fix this? What could we do to address this issue, and maybe even simplify code and user experience in the process?

wp_taxonomy

The wp_taxonomy table is my proposed solution to this problem. It would consolidate all of the information in wp_terms with wp_term_taxonomy:

taxonomy_id
taxonomy
name
slug
description
parent
count
term_group

Notice that here there’s only two more fields in this table than in wp_term_taxonomy, one of which is the oh-so-mysterious-and-little-used term_group from wp_terms. Since linking to a separate terms table is no longer required, we can completely drop term_id, and then add the two relevant fields from the terms table, name and slug. Uniqueness on this table would then be enforced on the pair (taxonomy,slug). A schema like this would result in one less database table, one less field, and a lot more flexibility, since each taxonomy would now create a unique bucket of terms.

How much work would it take to adapt the API to use this proposed schema? It’s hard to say. A brief run-through of the functions that touch the database in wp-include/taxonomy.php shows that at least the following functions would need to be changed:

get_object_in_term()
WP_Tax_Query::get_sql()
WP_Tax_Query::transform_query()
get_term()
get_term_by()
get_terms()
term_exists()
wp_delete_object_term_relationships()  (Likely obviated entirely by proposed schema)
wp_delete_term()
wp_get_object_terms()
wp_insert_term()
wp_set_object_terms()
wp_unique_term_slug()
wp_update_term()
wp_update_term_count_now()
clean_term_cache()
_pad_term_counts()
_update_post_term_count()

This is most definitely not a complete list of all the functions that would require updating in order to work with this proposed schema. There are also all of the functions that hook into the various term-related actions and filters, as well as all of the back-end wp-admin code that would need modification. The above represents only the minimal set of core taxonomy API functions that would need updating, but I think it’s a good start.

If only it were so easy!

In an ideal world, where WordPress wasn’t already the beast that it has grown to be, with millions of sites depending on it, including many of the big players, (14.7% percent of the top million sites!) such a change might be a reasonable undertaking. But WordPress is a giant system with a huge community (some of which I’m sure will have something to say about my proposal!) and a monstrous ecosystem. While my proposed changes would (I believe) generally result in simpler code and a more predictable taxonomy system, it’s simply not feasible to make such a drastic change to core without a completely separate code fork.

As millions upon millions of sites are built upon WordPress these days, making dramatic changes to the database schema is a daunting task that would likely break thousands of sites, even if the API managed to stay the same. This is because there is still plenty of theme code out there that reaches directly into the two extant taxonomy data tables, and this would be the code that would most definitely be broken by any change. Perhaps a forked version of WordPress would be needed as a proof-of-concept, but even then, we’re looking at a development cycle that would span years to get such a major initiative off the ground.