Now that we’ve had time for the dust to settle, we’d like to publicly debrief on the information disclosure security vulnerability from last Thursday, April 9th.
What the problem was:
- Any user, whether authenticated or unauthenticated, could access any post revision. Post revisions should only be accessible to authenticated users who have permission to edit the post.
- Thursday morning, 6:39 am PT, a WP-API user submitted a public GH issue with the information disclosure security vulnerability.
- Daniel saw the issue, saw it for what it was, and had a bit of an “oh shit moment”. His first step was to attempt to reach out to the WordPress.org security team for guidance.
- Around 7:45 am PT, Nacin responded and created a private Slack channel to coordinate the response. He suggested releasing a fix as quick as we could, and being loud about the release.
- Daniel created reasonably private release branches on his own fork to share with testing. Joe and Rachel helped out with testing. We set 10:30 am PT as the announcement time.
- As we got closer to announcement, we added a variety of WordPress community members to the private Slack channel to help verify the fix.
- In this process, Joe found a third, less critical but still severe bug to fix. This delayed the release by an hour. We had already shipped v1.1.2, v1.0.1, v0.9.1 and v0.8.1 to WordPress.org, which necessitated creating new versions. v1.2.1 wasn’t affected by the third bug.
- Fixed versions of WP-API and announcement post were live by 1:39 pm PT, exactly 7 hours after the Github issue was opened. Auto-updates started rolling shortly after that.
What went well:
- Daniel: We shipped the fix out in a timely manner.
- Daniel: Private Slack room to gather applicable parties. As the issue progressed, we added representatives from some hosts, etc.
- Daniel: One person dedicated to committing fixes, and another dedicated to testing.
- Daniel: Multiple people that could’ve taken the lead on getting the fixes shipped.
- Daniel: Auto-updates greatly increase confidence of getting the fix deployed to live sites.
- Daniel: Representative from hosts were attentive once we had them identified.
- Joe: Prioritised other commitments, Daniel especially – gave up a whole day to fix the issue. Not possible for everyone to do at the drop of a hat, but as a team, we reacted well.
- Joe: Communication throughout was very good – the private group on slack provided a good way to converse. Also, external comms with numerous stakeholders was very good.
- Joe: Reviewing / testing – lots of people were fast to react and test fixes, I also put in a bunch of time to verify, this made the push live much smoother when it finally happened.
What can be done better next time
- Daniel: Bug shouldn’t have been publicly disclosed. It was only a matter of timing that Daniel saw and acted on it promptly. Could’ve been much worse.
- Daniel: Difficulty getting ahold of the security team. There was an hour delay before we had Nacin involved and a security backchannel going. Next time need to email email@example.com
- Daniel: Would be nice to have some advance list of sites, people, etc. to reach out in advance of publicly disclosing.
- Joe: Having a more structured approach / run book document on what we should have done. For example, who can commit to the SVN repo, list of people we should reach out to.
- Joe: Perhaps a definition of how many version back we decide to patch – perhaps case by case depending on the severity of the issue though.
- Rachel: Explicitly mention everyone should update, even those who have checked it out via version control. The language of the announcement post could’ve communicated this better.
- Rachel: Identify a dedicated “press” contact for everyone who wants to write about it.
Feel free to ask questions, add comments, etc. in the comments.