Menu

WordPress REST API Team

Skip to content

Ryan McCue 06:16 on May 12, 2014
Tags: meta, metadata   

Handling Post Meta

One of the longest standing feature requests for the API is to add support for modifying post meta. While at first it may seem like a reasonably simple request, it becomes quite the rabbit hole when you begin digging into the issues surrounding it. Here’s a quick summary of the issues surrounding meta, and how we’re looking at solving it.

The Issues

The biggest issue with post meta is the difference between the data model in WordPress, and the most common usage of meta. In general, the main meta usage is as a key-value store; that is, a one-to-one mapping between a meta key and a meta value. However, the data model in WordPress allows multiple values per key. Handling these in a coherent way is challenging, as we want to handle the most common case easily while also making the multiple value model possible.

The other main issue with meta is serialized data. As anyone who’s used post meta in WP knows, WP will store any type of data (except resources or closures) by serializing the data under the hood. This isn’t usually an issue, as the process is transparent for most uses of post meta. However, this presents a problem when accessing the data via the API, as we cannot expose this data transparently.

JSON has no distinction between associative arrays and objects, as associative arrays only exist in PHP. In addition, JSON cannot pass objects and their types; that is, the representation of stdClass and a custom MyAwesomeObject are the same in JSON. Exposing this data using the default JSON encoding semantics would cause data loss. On the flip side, while exposing the raw serialized string would not cause data loss, this would expose protected and private properties on objects, as well as the internal implementation details including the class. This could expose critical internal data.

In addition to these issues, combining the two can cause further problems. With a naive approach of mapping key-to-value in a JSON object and allowing multiple keys and serialization, we could have a result like:

{
	"my_key": [
		"value1",
		"value2"
	],
}

However, it’s impossible to tell whether this is a key with multiple values, or a key with a single value of a PHP array.

If we treat this as a key with multiple values, updating the values could prove problematic. How do we distinguish between adding elements, updating existing elements, and removing elements? Simply leaving elements out does not necessarily mean we want to remove them, as we may just want to reduce the amount of data being sent over the wire.

Proposals

With these issues considered, there’s a few resolutions we need to implement.

The first resolution is to not handle serialized data at all. That includes displaying it and allowing modification. For all intents and purposes, serialized data will be treated as protected meta. We cannot avoid this, due to the object data loss issue.

We’ve now come up with two proposals on how to handle updates.

First Proposal

The first proposal by Rachel Baker and Taylor Lovett uses the following format for reading the data:

{
	"post_meta": {
		"my_key": [
			"value1"
		],
		"my_other_key": [
			"value1",
			"value2"
		]
	}
}

For updating meta, you can then pass in data thusly:

{
	"post_meta": [
		{
			"key": "my_key",
			"value": "newvalue1",
			"action": "update"
		}
	]
}

For adding meta:

{
	"post_meta": [
		{
			"key": "my_new_key",
			"value": "newvalue1",
			"action": "add"
		}
	]
}

And for deleting:

{
	"post_meta": [
		{
			"key": "my_key",
			"action": "delete"
		}
	]
}

This approach has the advantage of keeping data access fairly simple, and means the most common case of key-value storage is simply post_meta.my_key[0], while multiple values can iterate post_meta.my_key.

However, it has the disadvantage that the input format does not match the output format. This means that you cannot send the post data straight back to the server without causing an error. In addition, it mixes actions into the data itself that is sent to the server. Multiple values are also not handled by this, however this could be corrected by including a previous_value when updating.

Second Proposal

The second proposal by myself builds on Rachel and Taylor’s work, but changes the format slightly. Data looks like the following when reading a post:

{
	"post_meta": [
		{
			"ID": 33,
			"key": "my_key",
			"value": "value1"
		},
		{
			"ID": 34,
			"key": "my_other_key",
			"value": "value1"
		},
		{
			"ID": 35,
			"key": "my_other_key",
			"value": "value2"
		}
	]
}

For updating meta:

{
	"post_meta": [
		{
			"ID": 33,
			"key": "my_key",
			"value": "newvalue1"
		}
	]
}

For adding meta:

{
	"post_meta": [
		{
			"key": "my_new_key",
			"value": "newvalue1"
		}
	]
}

And for deleting meta:

{
	"post_meta": [
		{
			"ID": 33,
			"key": null,
			"value": null
		}
	]
}

This approach has the advantage that the input format matches the output format. Submitting the post data back to itself will have no effect, as the value will already match. In addition, the action to take is implied from the data, rather than specifying it in the data; updating a value requires just updating the value, adding a value requires adding a value without an ID, and deleting a value requires renaming the key to null (that is, specifying an empty value). This format uses the meta ID from the database as the primary key, allowing manipulating multiple values easily.

One of the disadvantages of this approach is that reading data becomes more complicated. Accessing all data for a key now involves filtering the meta values on the client side, rather than a simple lookup. While this is reasonably easy to achieve, it’s not as obvious as a straight access (and also has worse performance characteristics). The updating format is less obvious, as it’s implied from the data format rather than being spelled out explicitly.

Other Approaches

One approach that isn’t considered above is using the first approach’s data for the post itself, and exposing meta in the second form via another endpoint (e.g. /posts/[id]/meta). While this would enable both simple and complicated uses nicely, it also introduces significant fragmentation and duplication. This means developers would need to learn and support two separate methods of achieving the same result, and also work out internally which to use. In practice, clients would end up simply supporting a single approach for consistency. This approach would also violate the Decisions, Not Options mantra of WordPress.

Decisions

We need to make a decision on how we handle meta data. Personally, I’m biased towards the solution I wrote, but it’s not the perfect solution, and we’ll never have a truly perfect situation. Both approaches are a compromise, and we need to decide on which compromise we want to choose.

I’d love to hear thoughts on which approach people would prefer, and anything we may have missed during consideration.

#meta, #metadata

  1. theorboman 06:39 on May 12, 2014

    The second proposal gets my vote. Seems a bit cleaner & more CRUD-like.

  2. Bronson Quick 07:02 on May 12, 2014

    I’m a fan of the second approach as well. From past experiences I’ve found its been frustrating doing dev for APIs where the input and output formats don’t match.

  3. slaFFik 07:50 on May 12, 2014

    I think the 2nd approach is better. It’s easier to understand and work with (imo).

  4. matthewhainesyoung 08:40 on May 12, 2014

    I think it is wrong to assume that the primary use of meta is key => single_value, it makes more sense to assume that meta is key => array( values ) – the second method does not make iterating over the values for a single key easy.
    I would like to see something like this – based on approach 1.
    To get all post meta: /posts/[id]/meta

    [
    	{
    		key: 'my_key',
    		value: [ 'value1' ]
    	},
    	{
    		key: 'my_key_2',
    		value: [ 'value1', 'value2' ]
    	},
    ]
    

    You would not use this endpoint to update a single meta value. Instead you would use the endpoint /posts/[id]/meta/[key]. A POST request would then replace all existing meta values with the passed values.

    {
    	key: 'my_key_2',
    	value: [ 'value1', 'value2' ]
    },
    

    You could also get a single meta value and handle deleting using the same endpoint. Perhaps we could also do something here to handle adding a single value.

    • Ryan McCue 11:43 on May 12, 2014

      Note: It’s reasonably simple to filter down in JS with Underscore:

      _.filter( post.get('meta'), { 'key': "my_key" } )
      

      Ditto in most languages; the only issue is that it’s an O(N) operation rather than an O(1), so it’s more complex.

    • Daniel Bachhuber 00:33 on May 14, 2014

      /posts/[id]/meta/[id]

      Why not use id explicitly?

      • Ryan McCue 02:10 on May 14, 2014

        Why not use id explicitly?

        With the first proposal, we never expose the meta ID anywhere; everything is key-based.

  5. Max Cutler 14:30 on May 12, 2014

    The second proposal is almost identical to how XML-RPC handles post meta, except that XML-RPC requires all the meta pairs to be supplied in updates and then it deletes any entries that are missing. I think it’s slightly easier for clients to just delete a key locally and not send it rather than null out values and then filter those null values out of their own APIs/UIs, but I like the incremental/individual update that your approach allows.

    In practice, I have rarely encountered a WP deployment that uses enough custom fields for the O(n) vs O(1) performance impact would be noticeable, even on mobile phones. If a particular API client knows it will be doing lots of lookups by key name, the client can trivially build it’s own dictionary/hashtable with a single pass over the array.

    • Ryan McCue 00:14 on May 13, 2014

      The second proposal is almost identical to how XML-RPC handles post meta, except that XML-RPC requires all the meta pairs to be supplied in updates and then it deletes any entries that are missing. I think it’s slightly easier for clients to just delete a key locally and not send it rather than null out values and then filter those null values out of their own APIs/UIs, but I like the incremental/individual update that your approach allows.

      The main issue I have with this is that you then can’t send partial updates. At the moment, you can send requests like:

      {
          "title": "My New Title"
      }
      

      This doesn’t delete all the other fields. IMO, it’s strange to require that post meta deletes missing fields with this behaviour. Maybe this behaviour is wrong, but I find it works quite well in that it allows you to only send deltas rather than the full data each time.

  6. K.Adam 14:52 on May 12, 2014

    If multiple post_meta entries with the same key get different IDs inside WP today, then I’m in favor of option 2: I agree that grouping and filtering meta values within the API client application is trivial from an implementation standpoint and negligible in most cases from a performance standpoint.

    However, I do agree that “null”ing out the values to delete feels unintuitive. In my own work (primarily working on JavaScript client-side applications), I’m most used to just popping or otherwise removing a value from a list, and having the backend take care of the deletion. I’m fine going with “null” if that’s a more common pattern, just didn’t feel as familiar as what Max described above.

    Also, naive question: Would there be any situations where a post_meta key would be coming from the server with a “null” value?

    • Ryan McCue 00:11 on May 13, 2014

      Also, naive question: Would there be any situations where a post_meta key would be coming from the server with a “null” value?

      *sigh*, it looks like the meta_key is marked as nullable in the database, so that may be a non-option.

  7. Daniel Bachhuber 00:35 on May 14, 2014

    Sorry if this is a dumb question, but why aren’t we using HTTP verbs to specify the operation?

    • Ryan McCue 03:18 on May 14, 2014

      Originally, I was thinking that we wouldn’t need to make meta first-class objects with their own routes and such. See the Other Approaches section above.

      I’m now thinking it’s probably a better way to handle it. If we handle it a different way, we’re going to have to compromise on the data structure (such as including "action": "delete"). It’s also definitely more RESTful.

      • mikeschinkel 04:57 on May 14, 2014

        @ryan – In the comment above are you saying that including "action": "delete" in the JSON is RESTful, or am I misunderstanding? Or are you saying we would not include "action": "delete" in the JSON in order to be RESTful? Color me confused by wording.

        • Ryan McCue 20:46 on May 15, 2014

          Sorry; meant that including the action within the data is not RESTful.

          • mikeschinkel 21:16 on May 15, 2014

            Gotcha. Thanks.

  8. mikeschinkel 04:56 on May 14, 2014

    My vote is #2 more than #1 because #1 I don’t think RESTful and thus it looses the benefits that come from RESTfulness such as HTTP caching.

    That said, I would recommend a #3 which is a variation of #2. I’ve posted it to a Gist for easier viewing.

    P.S. Sorry I haven’t been involved lately. I’ve been working on the #metadata FaaP and that’s taken all my free time.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Post navigation

← Weekly meeting in 10 hours;…
Also, before I forget about… →

About this o2

This o2 is the organisational hub for the WP REST API team, currently developing the JSON REST API for WordPress core.

You can get the current beta version from WordPress.org, or get a development version from GitHub.

Comments are open for all to discuss. If you'd like to start a new post, suggest a new topic.

You can also submit completely anonymous feedback.

Team Members

Tags

136 205 3434DskcckDS455Mcnskfwepwoemcmvweoivmo4453 a core-restapi meeting meetingnotes meta metadata new25682452WWW568

Meta

  • Register
  • Log in
  • Entries feed
  • Comments feed
  • WordPress.com
Create a free website or blog at WordPress.com.
s
search
c
compose new post
r
reply
e
edit
t
go to top
j
go to the next post or comment
k
go to the previous post or comment
o
toggle comment visibility
esc
cancel edit post or comment
  • Follow Following
    • WordPress REST API Team
    • Join 250 other followers
    • Already have a WordPress.com account? Log in now.
    • WordPress REST API Team
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Copy shortlink
    • Report this content
    • View post in Reader
    • Manage subscriptions
    • Collapse this bar
%d bloggers like this: