Recap Time Squad's infra - Notice history

All systems operational

Landing Website - Operational

100% - uptime
2022-11 · 100.0%-12· 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

Vaultwarden - Operational

100% - uptime
2022-11 · 100.0%-12· 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

Wiki (hosted on Miraheze) - Operational

54% - uptime
2022-11 · 77.75%-12· 0.00%2023-01 · 85.09%

Previous page

Next page

2022-11
2022-12
2023-01

Email - Operational

100% - uptime
2022-11 · 100.0%-12· 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

SSO - Operational

100% - uptime
2022-11 · 100.0%-12· 100.0%2023-01 · 100.0%

Previous page

Next page

2022-11
2022-12
2023-01

Third Party: GitLab Pages - Operational

Third Party: GitLab -> Website - Operational

Third Party: Github -> Actions - Operational

Third Party: Docker -> Docker Hub Web - Operational

100% - uptime

Third Party: Discord -> API - Operational

Third Party: Cloudflare API - Operational

Third Party: GitLab -> API - Operational

Third Party: GitLab -> Container Registry - Operational

Third Party: Docker -> Docker Authentication - Operational

Third Party: Docker -> Docker Hub Registry - Operational

100% - uptime

Third Party: Cloudflare DNS Root Servers - Operational

Third Party: Cloudflare -> Cloudflare Sites and Services -> DNS Updates - Operational

Third Party: Cloudflare -> Cloudflare Sites and Services -> Cloudflare Dashboard - Operational

Third Party: Cloudflare Workers - Operational

Third Party: Cloudflare Tunnel - Operational

100% - uptime

Third Party: Vercel -> CI/CD -> Git Integrations - Operational

Third Party: Vercel Dashboard - Operational

Third Party: Vercel API - Operational

Third Party: Vercel Certificate Issuance - Operational

Third Party: Vercel DNS - Operational

Third Party: Netlify -> Collaborative Deploy Previews - Operational

Third Party: Netlify API - Operational

Third Party: Netlify DNS - Operational

100% - uptime

Third Party: Let's Encrypt -> acme-v02.api.letsencrypt.org (Production) - Operational

Third Party: Let's Encrypt -> oak.ct.letsencrypt.org (Production) - Operational

Third Party: Let's Encrypt -> ocsp.root-x1.letsencrypt.org - Major outage

Third Party: Let's Encrypt -> {e1,e2,r3,r4}.o.lencr.org & ocsp.int-x{3..4}.letsencrypt.org - Operational

100% - uptime

Third Party: Discord -> Media Proxy - Operational

Third Party: Discord -> Voice -> US Central - Operational

Third Party: Discord -> Voice -> Singapore - Operational

Third Party: Railway -> Dashboard - Operational

Third Party: Railway -> Deployment Layer - Degraded performance

Third Party: Railway -> Routing Layer - Operational

Third Party: Railway -> Builder - Operational

Notice history

2023-01

Planned email migration to biz.mail.ru
  • Completed
    2023-01-28 at 4:00 PM
    Completed
    2023-01-28 at 4:00 PM

    Maintenance has completed successfully

  • In progress
    2023-01-28 at 2:30 PM
    In progress
    2023-01-28 at 2:30 PM

    Maintenance is now in progress

  • Planned
    2023-01-28 at 2:30 PM
    Planned
    2023-01-28 at 2:30 PM

    We are planning for a scheduled maintenance, which might affect deliverability and reachability of @recaptime.eu.org email addresses due to DNS propagation by the weekend before end of the month.

    For squad members with existing @recaptime.eu.org email address on Yandex.Mail, please coordinate with Andrei Jiroh for migrating the emails to the new provider. We're done migrating mail for our legacy domain (recaptime.tk) there earlier this week (2023-01-04).

2022-12

Miraheze server maintenance might cause Squad Wiki to go read-only or offline
  • Planned
    2022-12-18 at 9:00 PM
    Planned
    2022-12-18 at 9:00 PM

    Miraheze will be doing server maintenance beginning at 21:00 UTC on Sunday, 18 December 2022 lasting until 03:00 UTC on Monday, 19 December 2022. During this time, some wikis may be intermittently unavailable or go into read-only.

  • Completed
    2022-12-18 at 9:00 PM
    Completed
    2022-12-18 at 9:00 PM

    Maintenance has completed successfully, but a few hiccups were happened. Please see https://meta.miraheze.org/wiki/Tech:SRE_noticeboard for further updates.

2022-11

Squad Wiki temporarily unavailable due to issues on one of Miraheze's servers
  • Resolved
    Resolved

    We just resolved the issue, literally days ago! Sorry for the late update, but the wiki is now officially kicking in, albeit with some backend issues, missing pages and stuff.

  • Update
    Update

    We're technically back, although an issue with our custom domain has been fixed behind the scenes. Because we recreated the wiki without any backups (while the drives where the database cluster is the wiki's database is under Owen's custody in the meanwhile), this could take longer to make things back into its state before the unexpected drive issues happened.

    Please keep tuning in to the linked GitLab issue for updates regarding the situation, but for now, we downgraded the severity of this incident to `Degraded performance` per this update.

  • Monitoring
    Monitoring

    Sorry for this late update, but we're being notified days ago by Miraheze team that our wiki is affected by recent `cloud14` issues, particularly the disk issues on `db141`. We're currently monitoring the situation behind the scenes via this GitLab issue.

  • Identified
    Identified

    The sysadmins at Miraheze are working on bringing cloud14 back online, albeit without the databases. Once cloud14 is back, MediaWiki speeds should improve, and mail should work once again. There is no ETA for when wikis will return, however. (update from <https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283340>)

Scheduled database migration for Vaultwarden
  • Completed
    2022-11-24 at 7:39 AM
    Completed
    2022-11-24 at 7:39 AM

    Maintenance has completed successfully days ago. Sorry for this late update, but we're now kicking tires behind the scenes.

  • Update
    2022-11-21 at 10:42 AM
    Update
    2022-11-21 at 10:42 AM

    Sorry for the long downtime, we're still fixing some stuff behind the scenes. We'll be back soon.

  • Update
    2022-11-20 at 5:12 PM
    Update
    2022-11-20 at 5:12 PM

    Maintenance is still in progress, but we're done migrating to a new database hosting provider. You'll be notified one things are ready.

  • In progress
    2022-11-20 at 3:15 PM
    In progress
    2022-11-20 at 3:15 PM

    Maintenance is now in progress, and the server will be offline shortly.

  • Update
    2022-11-20 at 3:15 PM
    Update
    2022-11-20 at 3:15 PM

    Maintenace has been rescheduled to tomorrow at 11:30 PM, Philippine Standard Time.

  • Planned
    2022-11-20 at 3:15 PM
    Planned
    2022-11-20 at 3:15 PM

    We'll conduct a database migration for our Vaultwarden instance on 2022-11-20 at 15:30 (UTC time) which you might experience up to 1-2 hours depending on the activity. This maintenance is needed as we migrate the database into a separate provider for availability reasons (although we're not going full HA yet) and to do an upgrade deployment afterwards.

    For more details, please see https://mau.dev/RecapTime/infra/issue-tracker/issues/1 for up-to-date information.

2022-11 to 2023-01

Next