GitHub header
All Systems Operational
Git Operations ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
API Requests ? Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Sep 19, 2024

No incidents reported today.

Sep 18, 2024

No incidents reported.

Sep 17, 2024

No incidents reported.

Sep 16, 2024
Incident with Pages and Actions
Resolved - On September 16, 2024, between 21:11 UTC and 22:20 UTC, Actions and Pages services were degraded. Customers who deploy Pages from a source branch experienced delayed runs. Approximately 1,100 runs were delayed long enough to get marked as abandoned. The runs that weren't abandoned completed successfully after we recovered from the incident. Actions jobs experienced average delays of 23 minutes, with some jobs experiencing delays as high as 45 minutes. During the course of the incident, 17% of runs were delayed by more than 5 minutes. At peak, as many as 80% of runs experienced delays exceeding 5 minutes. The root cause was a misconfiguration in the service that manages runner connections, which caused CPU throttling and led to a performance degradation in that service.

We mitigated the incident by diverting runner connections away from the misconfigured nodes. We are working to improve our internal monitoring and alerting to reduce our time to detection and mitigation of issues like this one in the future.
Sep 16, 22:08 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 16, 21:55 UTC
Update - The team is investigating issues with some Actions jobs being queued for a long time and a percentage of jobs failing. A mitigation has been applied and jobs are starting to recover.
Sep 16, 21:53 UTC
Update - Pages is operating normally.
Sep 16, 21:52 UTC
Update - Actions is experiencing degraded availability. We are continuing to investigate.
Sep 16, 21:37 UTC
Investigating - We are investigating reports of degraded performance for Actions and Pages
Sep 16, 21:31 UTC
Disruption with Git SSH
Resolved - This incident has been resolved.
Sep 16, 14:28 UTC
Update - We are no longer seeing dropped Git SSH connections and believe we have mitigated the incident. We are continuing to monitor and investigate to prevent reoccurrence.
Sep 16, 14:27 UTC
Update - We have taken suspected hosts out of rotation and have not seen any impact in the last 20 minutes. We are continuing to monitor to ensure the problem is resolved and are investigating the cause.
Sep 16, 14:11 UTC
Update - We are seeing up to 2% of Git SSH connections failing.

We have taken suspected problematic hosts out of rotation and are monitoring for recovery and continuing to investigate.
Sep 16, 13:38 UTC
Update - We are investigating failed connections for Git SSH. Customers may be experiencing failed SSH connections both in CI and interactively. Retrying the connection may be successful. Git HTTP connections appear to be unaffected.
Sep 16, 13:30 UTC
Investigating - We are currently investigating this issue.
Sep 16, 13:29 UTC
Sep 15, 2024

No incidents reported.

Sep 14, 2024
Incident with Pull Requests
Resolved - This incident has been resolved.
Sep 14, 22:43 UTC
Update - Pull Requests is operating normally.
Sep 14, 22:43 UTC
Update - we believe we have mitigated and are confirming recovery.
Sep 14, 22:41 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests
Sep 14, 22:10 UTC
Sep 13, 2024
Processing delays to some Issues, Pull Requests and Webhooks
Resolved - On Sep 13, 2024, between 05:03 UTC and 07:13 UTC, the Webhooks and Actions services were degraded resulting in some customers experiencing delayed processing of Webhooks and Actions Runs. 0.5% of Webhook deliveries were delayed more than 2 minutes during the incident. 15% of Actions Runs started between 05:03 and 05:24 UTC saw run start delays or failures. At 05:24 UTC, we implemented a mitigation to shift traffic to healthy infrastructure and new Actions Runs resumed normal operations. During the rest of the incident window, Actions runs started before 05:24 UTC continued to see delays publishing logs or job results. No Actions runs or Webhook deliveries were lost, only delayed.

We mitigated the incident by immediately shifting traffic to a healthy cluster while investigating. The incident was caused by an erroneous configuration change on our eventing platform. A permanent fix was deployed at 06:22 UTC after which services began to recover and burn down their backed up queues, with full recovery by 07:13 UTC.

We are working to reduce our time to detection and develop test automation to prevent issues like this one in the future.
Sep 13, 07:13 UTC
Update - We are seeing improvements in telemetry and are monitoring the delivery of delayed Webhooks and Actions job statuses.
Sep 13, 06:49 UTC
Update - We've applied a mitigation to fix the issues being experienced in some cases with delays to webhook deliveries, and the delayed reporting of the outcome of some running Actions jobs. We are monitoring for full recovery.
Sep 13, 06:23 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 13, 05:59 UTC
Investigating - We are investigating reports of degraded performance for Issues, Pull Requests and Webhooks
Sep 13, 05:42 UTC
Sep 12, 2024

No incidents reported.

Sep 11, 2024

No incidents reported.

Sep 10, 2024

No incidents reported.

Sep 9, 2024

No incidents reported.

Sep 8, 2024

No incidents reported.

Sep 7, 2024

No incidents reported.

Sep 6, 2024

No incidents reported.

Sep 5, 2024
Disruption with some GitHub services
Resolved - Between August 27, 2024, 15:52 UTC and September 5, 2024, 17:26 UTC the GitHub Connect service was degraded. This specifically impacted GHES customers who were enabling GitHub Connect for the first time on a GHES instance. Previously enabled GitHub Connect GHES instances were not impacted by this issue.

Customers experiencing this issue would have received a 404 response during GitHub Connect enablement and subsequent messages about a failure to connect. This was due to a recent change in configuration to GitHub Connect which has since been rolled back.

Subsequent enablement failures on re-attempts were caused by data corruption which has been remediated. Customers should now be able to enable GitHub Connect successfully.

To reduce our time to detection and mitigation of such issues in the future, we are working to improve observability of GitHub Connect failures. We are also making efforts to prevent future misconfiguration of GitHub Connect.
Sep 5, 17:24 UTC
Update - We continue to work on making GitHub Connect setup available for customers that experienced errors since Sept 2. We will share another update as we make progress.
Sep 5, 16:49 UTC
Update - We've reverted a related change and customers can now setup GitHub Connect again. However, customers who attempted to setup GitHub Connect after September 2nd that saw a 404 will see continued failures until we complete an additional repair step. We will provide additional updates as this work progresses.
Sep 5, 16:15 UTC
Update - Enterprise administrators are seeing failures in the form of 404s when attempting to setup GitHub Connect for the first time. Existing Connect setups are not impacted. We are working on a fix for this issue.
Sep 5, 15:56 UTC
Investigating - We are currently investigating this issue.
Sep 5, 15:55 UTC
← Incident History Powered by Atlassian Statuspage