Recap Time Squad's infra - Squad Wiki temporarily unavailable due to issues on one of Miraheze's servers – Incident details

Squad Wiki temporarily unavailable due to issues on one of Miraheze's servers

Resolved
Major outage
Started almost 2 years agoLasted about 1 month

Affected

Wiki (hosted on Miraheze)

Major outage from 7:47 AM to 2:54 PM

Updates
  • Resolved
    Resolved

    We just resolved the issue, literally days ago! Sorry for the late update, but the wiki is now officially kicking in, albeit with some backend issues, missing pages and stuff.

  • Monitoring
    Update

    We're technically back, although an issue with our custom domain has been fixed behind the scenes. Because we recreated the wiki without any backups (while the drives where the database cluster is the wiki's database is under Owen's custody in the meanwhile), this could take longer to make things back into its state before the unexpected drive issues happened.

    Please keep tuning in to the linked GitLab issue for updates regarding the situation, but for now, we downgraded the severity of this incident to `Degraded performance` per this update.

  • Monitoring
    Monitoring

    Sorry for this late update, but we're being notified days ago by Miraheze team that our wiki is affected by recent `cloud14` issues, particularly the disk issues on `db141`. We're currently monitoring the situation behind the scenes via this GitLab issue.

  • Identified
    Identified

    The sysadmins at Miraheze are working on bringing cloud14 back online, albeit without the databases. Once cloud14 is back, MediaWiki speeds should improve, and mail should work once again. There is no ETA for when wikis will return, however. (update from <https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283340>)