If you’re in any way involved with OpenPype and Ynput community, you probably noticed that on Friday 3.2.2023 around 19:20 CET, the main OpenPype repository has been disabled with this note.
Access to this repository has been disabled by GitHub Staff as a result of a private information removal request. You may contact GitHub Support for more information or to appeal this decision. Read more about GitHub’s Private Information Removal Policy here: GitHub’s Private Information Removal Policy.
This is when we at Ynput have found out about it as well. Strange, we thought, but surely it’s either a mistake that will soon be resolved or we’ll at least find out from support what’s up, to be able to react. So we contacted GitHub support right away. It was Friday late evening which didn’t fill us with confidence in hearing back anytime soon and at this moment we started being more worried because it turned out that all of the forks have been disabled as well. At that moment, things turned a lot more serious.
To our big surprise, GitHub never replied to any of our appeals, and we sent many from multiple people (most admins and owners of our organization). We asked, we pleaded, we were polite, and after 6 days of being cut off from the core of our company livelihood also rude… The community and owners of the forks also sent plenty of requests for appeal.
Considering we got nowhere with GitHub support at all, you might wonder how did we get the repo and all of the forks back? Well, by working with the community, going through all of our GitHub notifications again and again, and most importantly, because we were incredibly lucky that the originator of the removal request who is part of our community was as confused as us about what happened. GitHub reacted at least to the support ticket that he originally opened to ask them for help fixing a fairly common mistake that can happen to absolutely anyone. Accidentally committing sensitive information to a file.
Here’s the sequence of events as we pieced it together. Sherlock style
Wednesday 1.2.
9:16 - A PR has been opened to ynput/OpenPype repository by an external contributor
9:23 - The author noticed that he’s accidentally committed sensitive information as part of the PR, so he closed the PR. We didn’t even notice it existed. Unfortunately, the author wasn’t able to remove the file himself, so he asked GH support for assistance. To be specific, he asked for help removing a single file from his fork.
18:47- GitHub proceeded to send a notification to the owner of our repository, one of Ynput administrators, that there was a request filed to remove private content which was part of a particular PR with a link to an exact commit. A commit in a source branch in an external fork that we have no control over whatsoever. We were given 48 hours to resolve the issue, unfortunately, this e-mail has been sent to only one of our administrators/organization owners who missed the e-mail. Even though we have multiple (maybe too many) owners to prevent exactly this situation, redundancy and multiple points of contact are clearly lost on GitHub support. People go on holidays, they have tough days, family situations, or just plain miss an e-mail from GitHub in swamp of other GitHub e-mails. And it really went to only one person, we checked, many times… yes… in spam too.
Friday 3.2.
~19:00 Because we haven’t responded to the warning, the repository and all of its forks (over 70 of them) have been disabled. Why the full repository and not only the incriminating PR? Why all of the forks, even though they never even touched the incriminating file?
The original requester got this message from GH
Hi, Access to the following reported content has now been disabled: URL to the commit in the pr URL to the incriminating file Please note that, at the owner’s request, we may re-enable the repository to allow them to remove any sensitive data at some point in the future. Let us know if you have any questions or concerns at this time.
I’ve sent GH email appeal #1
Saturday 4.2.
We found the e-mail from GitHub about the takedown and start digging through local copies of the repo to find the commit and PR to figure out what exactly happened, we haven’t found anything because we had zero access to the repo, its history or settings and local data, including our internal ticketing that syncs with GH found nothing at all.
email appeal to GH #2 with more information on the matter
Sunday 5.2.
Nothing interesting apart from us still trying to untangle the events and me not sleeping for a second night in a row. Thank you GitHub for a lovely weekend with the family in the mountains.
Monday 6.2.
Things are moving. Thanks to @BigRoy who has masochistically frequent e-mail notifications regarding PRs into the repository, we were able to trace down the title and the author of the incriminating PR and get in touch first thing in the morning. Thomas (the author) was as confused as we were. Based on the GH support message he received on Friday (that you can see above), he thought GH only removed the PR and the single file.
We agreed that he will contact GH as well about the situation. This is the reply he got:
Unfortunately, we are unable to fulfill this request. At your request, we made an attempt to reach out the repository’s owner to have them remove the files themselves, but did not receive a response. At that time, we took action to disable the entire repository. If you would like to retract your PIR request, we can re-enable the repository and you can reach out to the repository’s owner to have files you mentioned to be removed.
Email appeal to GH #3
Tuesday 7.2.
~ 12:00 The author has canceled the PIR (Personal Information Removal) request
~ 16:00 The repository has been restored… without the forks, including the fork where the offending file was coming from. So now the file is visible in the closed PR history (that only GH support staff can delete), and the author can’t remove it either because he can’t access his own for that has been disabled.
email appeal to GH #4
Wednesday 8.2.
The requester has asked GH support to restore all the forks.
Thursday 9.2.
22:40 - Forks have been re-enabled… but not the only fork that actually had the file that needs to be removed.
A few facts to sum up:
- The main repo was out of action for 4 days.
- Forks were out for 6 days!
- Having multiple admins and owners in the repo is useless, GitHub will clearly just randomly pick one when action is required.
- You can seemingly take down any repo network really easily by filing a PIR request. If the owner misses the single notification email they are out of luck.
- Communication from GitHub about the scorched earth approach to a simple request is totally non-existent.
- If the person who filed the PIR in the first place wasn’t incredibly helpful, GitHub would have effectively killed a fairly large FOSS project network until someone in the support team takes pity on the owners. (it took 60 days for github support to answer our requests)
- As a repository owner, you have absolutely no way of removing files from the history once a PR has been created. Even if it’s closed, its content stays visible and only GitHub staff can remove it… that’s if they ever reply to you, of course.
The really scary part of this story is that even though we’ve just been through it, there seems to be very little we can learn from it that could prevent such a situation at a later date, apart from being less unlucky about who and when is notified about any issues.
Ynput is a company with open-source at its core, built around the community and as such, we rely heavily on the infrastructure provided by Github. There are other options out there for hosting a large open-source project; however, Github is the only one that not only provides the infrastructure but more importantly visibility and community tools to manage it. We are considering steps to strengthen our day-to-day operations with better repository and issues mirroring, but realistically we appear to be at the mercy of GitHub’s bizarre support approach if anything similar happens again.
I want to personally thank everyone for reaching out, sending appeals to GitHub, and being extremely supportive for the whole week while the repos were down. Special thank you goes to Thomas who accidentally caused the whole incided
To have a proper ending to this little saga, two months after the incident, we received an official reply from Github.
Apr 10, 2023, 6:28 PM UTC Hi, Sorry for the delay. We are reaching out to confirm your repository was reinstated as we identified the flag was made in error. Please rest assured that we don’t take these types of action lightly. We’re taking this opportunity to improve our processes to make sure this doesn’t impact other users going forward. Our sincere apologies for the mistake and the inconvenience it caused. Regards, GitHub Trust & Safety
