Recap Time Squad's infra - Notice history

Experiencing partially degraded performance

Landing Website - Operational

100.0% uptime
2022-11 · 100.0%-12 · 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

Vaultwarden - Operational

100.0% uptime
2022-11 · 100.0%-12 · 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

Wiki (hosted on Miraheze) - Operational

53.52% uptime
2022-11 · 77.75%-12 · 0.00%2023-01 · 85.09%

Previous page

Next page

2022-11
2022-12
2023-01

Email - Operational

100.0% uptime
2022-11 · 100.0%-12 · 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

SSO - Operational

100.0% uptime
2022-11 · 100.0%-12 · 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

Third Party: GitLab Pages - Operational

Third Party: GitLab -> Website - Operational

Third Party: Github -> Actions - Operational

Third Party: Docker -> Docker Hub Web - Operational

100.0% uptime

Third Party: Discord -> API - Operational

Third Party: Cloudflare API - Operational

Third Party: GitLab -> API - Operational

Third Party: GitLab -> Container Registry - Operational

Third Party: Docker -> Docker Authentication - Operational

Third Party: Docker -> Docker Hub Registry - Operational

100.0% uptime

Third Party: Cloudflare DNS Root Servers - Operational

Third Party: Cloudflare -> Cloudflare Sites and Services -> DNS Updates - Degraded performance

Third Party: Cloudflare -> Cloudflare Sites and Services -> Cloudflare Dashboard - Operational

Third Party: Cloudflare Workers - Degraded performance

Third Party: Cloudflare Tunnel - Degraded performance

100.0% uptime

Third Party: Vercel -> CI/CD -> Git Integrations - Operational

Third Party: Vercel Dashboard - Operational

Third Party: Vercel Certificate Issuance - Operational

Third Party: Vercel API - Operational

Third Party: Vercel DNS - Operational

Third Party: Netlify -> Collaborative Deploy Previews - Operational

Third Party: Netlify API - Operational

Third Party: Netlify DNS - Operational

100.0% uptime

Third Party: Let's Encrypt -> acme-v02.api.letsencrypt.org (Production) - Operational

Third Party: Let's Encrypt -> oak.ct.letsencrypt.org (Production) - Operational

Third Party: Let's Encrypt -> ocsp.root-x1.letsencrypt.org - Operational

Third Party: Let's Encrypt -> {e1,e2,r3,r4}.o.lencr.org & ocsp.int-x{3..4}.letsencrypt.org - Operational

100.0% uptime

Third Party: Discord -> Media Proxy - Operational

Third Party: Discord -> Voice -> US Central - Operational

Third Party: Discord -> Voice -> Singapore - Operational

Third Party: Railway -> Dashboard - Operational

Third Party: Railway -> Deployment Layer - Operational

Third Party: Railway -> Routing Layer - Operational

Third Party: Railway -> Builder - Operational

Notice history

View current status

2023-01

JAN28
Planned email migration to biz.mail.ru
CompletedMaintenance1 hour 30 minutes
  • Completed2023-01-28 at 4:00 PMUTC

    Maintenance has completed successfully

  • In progress2023-01-28 at 2:30 PMUTC

    Maintenance is now in progress

  • Planned2023-01-28 at 2:30 PMUTC

    We are planning for a scheduled maintenance, which might affect deliverability and reachability of @recaptime.eu.org email addresses due to DNS propagation by the weekend before end of the month.

    For squad members with existing @recaptime.eu.org email address on Yandex.Mail, please coordinate with Andrei Jiroh for migrating the emails to the new provider. We're done migrating mail for our legacy domain (recaptime.tk) there earlier this week (2023-01-04).

2022-12

DEC18
Miraheze server maintenance might cause Squad Wiki to go read-only or offline
CompletedMaintenance
  • Planned2022-12-18 at 9:00 PMUTC

    Miraheze will be doing server maintenance beginning at 21:00 UTC on Sunday, 18 December 2022 lasting until 03:00 UTC on Monday, 19 December 2022. During this time, some wikis may be intermittently unavailable or go into read-only.

  • Completed2022-12-18 at 9:00 PMUTC

    Maintenance has completed successfully, but a few hiccups were happened. Please see https://meta.miraheze.org/wiki/Tech:SRE_noticeboard for further updates.

2022-11

Squad Wiki temporarily unavailable due to issues on one of Miraheze's servers
ResolvedMajor outage1015 hours 6 minutes
  • ResolvedUTC

    We just resolved the issue, literally days ago! Sorry for the late update, but the wiki is now officially kicking in, albeit with some backend issues, missing pages and stuff.

  • UpdateUTC

    We're technically back, although an issue with our custom domain has been fixed behind the scenes. Because we recreated the wiki without any backups (while the drives where the database cluster is the wiki's database is under Owen's custody in the meanwhile), this could take longer to make things back into its state before the unexpected drive issues happened.

    Please keep tuning in to the linked GitLab issue for updates regarding the situation, but for now, we downgraded the severity of this incident to `Degraded performance` per this update.

  • MonitoringUTC

    Sorry for this late update, but we're being notified days ago by Miraheze team that our wiki is affected by recent `cloud14` issues, particularly the disk issues on `db141`. We're currently monitoring the situation behind the scenes via this GitLab issue.

  • IdentifiedUTC

    The sysadmins at Miraheze are working on bringing cloud14 back online, albeit without the databases. Once cloud14 is back, MediaWiki speeds should improve, and mail should work once again. There is no ETA for when wikis will return, however. (update from <https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283340>)

NOV20
Scheduled database migration for Vaultwarden
CompletedMaintenance88 hours 24 minutes
  • Completed2022-11-24 at 7:39 AMUTC

    Maintenance has completed successfully days ago. Sorry for this late update, but we're now kicking tires behind the scenes.

  • Update2022-11-21 at 10:42 AMUTC

    Sorry for the long downtime, we're still fixing some stuff behind the scenes. We'll be back soon.

  • Update2022-11-20 at 5:12 PMUTC

    Maintenance is still in progress, but we're done migrating to a new database hosting provider. You'll be notified one things are ready.

  • In progress2022-11-20 at 3:15 PMUTC

    Maintenance is now in progress, and the server will be offline shortly.

  • Update2022-11-20 at 3:15 PMUTC

    Maintenace has been rescheduled to tomorrow at 11:30 PM, Philippine Standard Time.

  • Planned2022-11-20 at 3:15 PMUTC

    We'll conduct a database migration for our Vaultwarden instance on 2022-11-20 at 15:30 (UTC time) which you might experience up to 1-2 hours depending on the activity. This maintenance is needed as we migrate the database into a separate provider for availability reasons (although we're not going full HA yet) and to do an upgrade deployment afterwards.

    For more details, please see https://mau.dev/RecapTime/infra/issue-tracker/issues/1 for up-to-date information.

Previous

2022-11 to 2023-01

Next