Recap Time Squad's infra - Notice history

Third Party: Cloudflare Tunnel experiencing partial outage

Landing Website - Operational

100% - uptime
2022-09 · 100.0%-10· 100.0%-11· 100.0%

Previous page

Next page

2022-09
2022-10
2022-11

Vaultwarden - Operational

100% - uptime
2022-09 · 100.0%-10· 100.0%-11· 100.0%

Previous page

Next page

2022-09
2022-10
2022-11

Wiki (hosted on Miraheze) - Operational

93% - uptime
2022-09 · 100.0%-10· 100.0%-11· 77.75%

Previous page

Next page

2022-09
2022-10
2022-11

Email - Operational

100% - uptime
2022-09 · 100.0%-10· 100.0%-11· 100.0%

Previous page

Next page

2022-09
2022-10
2022-11

SSO - Operational

100% - uptime
2022-09 · 100.0%-10· 100.0%-11· 100.0%

Previous page

Next page

2022-09
2022-10
2022-11

Third Party: GitLab Pages - Operational

Third Party: GitLab -> Website - Operational

Third Party: Github -> Actions - Operational

Third Party: Docker -> Docker Hub Web - Operational

100% - uptime

Third Party: Discord -> API - Operational

Third Party: Cloudflare API - Operational

Third Party: GitLab -> API - Operational

Third Party: GitLab -> Container Registry - Operational

Third Party: Docker -> Docker Authentication - Operational

Third Party: Docker -> Docker Hub Registry - Operational

100% - uptime

Third Party: Cloudflare DNS Root Servers - Operational

Third Party: Cloudflare -> Cloudflare Sites and Services -> DNS Updates - Operational

Third Party: Cloudflare -> Cloudflare Sites and Services -> Cloudflare Dashboard - Operational

Third Party: Cloudflare Workers - Operational

Third Party: Cloudflare Tunnel - Partial outage

100% - uptime

Third Party: Vercel -> CI/CD -> Git Integrations - Operational

Third Party: Vercel Dashboard - Operational

Third Party: Vercel API - Degraded performance

Third Party: Vercel Certificate Issuance - Operational

Third Party: Vercel DNS - Operational

Third Party: Netlify -> Collaborative Deploy Previews - Operational

Third Party: Netlify API - Operational

Third Party: Netlify DNS - Operational

100% - uptime

Third Party: Let's Encrypt -> acme-v02.api.letsencrypt.org (Production) - Operational

Third Party: Let's Encrypt -> oak.ct.letsencrypt.org (Production) - Operational

Third Party: Let's Encrypt -> ocsp.root-x1.letsencrypt.org - Operational

Third Party: Let's Encrypt -> {e1,e2,r3,r4}.o.lencr.org & ocsp.int-x{3..4}.letsencrypt.org - Operational

100% - uptime

Third Party: Discord -> Media Proxy - Operational

Third Party: Discord -> Voice -> US Central - Operational

Third Party: Discord -> Voice -> Singapore - Operational

Third Party: Railway -> Dashboard - Operational

Third Party: Railway -> Deployment Layer - Operational

Third Party: Railway -> Routing Layer - Operational

Third Party: Railway -> Builder - Operational

Notice history

2022-11

Squad Wiki temporarily unavailable due to issues on one of Miraheze's servers
  • Resolved
    UTC
    Resolved

    We just resolved the issue, literally days ago! Sorry for the late update, but the wiki is now officially kicking in, albeit with some backend issues, missing pages and stuff.

  • Update
    UTC
    Update

    We're technically back, although an issue with our custom domain has been fixed behind the scenes. Because we recreated the wiki without any backups (while the drives where the database cluster is the wiki's database is under Owen's custody in the meanwhile), this could take longer to make things back into its state before the unexpected drive issues happened.

    Please keep tuning in to the linked GitLab issue for updates regarding the situation, but for now, we downgraded the severity of this incident to `Degraded performance` per this update.

  • Monitoring
    UTC
    Monitoring

    Sorry for this late update, but we're being notified days ago by Miraheze team that our wiki is affected by recent `cloud14` issues, particularly the disk issues on `db141`. We're currently monitoring the situation behind the scenes via this GitLab issue.

  • Identified
    UTC
    Identified

    The sysadmins at Miraheze are working on bringing cloud14 back online, albeit without the databases. Once cloud14 is back, MediaWiki speeds should improve, and mail should work once again. There is no ETA for when wikis will return, however. (update from <https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283340>)

Scheduled database migration for Vaultwarden
  • Completed
    2022-11-24 at 7:39 AMUTC
    Completed
    2022-11-24 at 7:39 AMUTC

    Maintenance has completed successfully days ago. Sorry for this late update, but we're now kicking tires behind the scenes.

  • Update
    2022-11-21 at 10:42 AMUTC
    Update
    2022-11-21 at 10:42 AMUTC

    Sorry for the long downtime, we're still fixing some stuff behind the scenes. We'll be back soon.

  • Update
    2022-11-20 at 5:12 PMUTC
    Update
    2022-11-20 at 5:12 PMUTC

    Maintenance is still in progress, but we're done migrating to a new database hosting provider. You'll be notified one things are ready.

  • In progress
    2022-11-20 at 3:15 PMUTC
    In progress
    2022-11-20 at 3:15 PMUTC

    Maintenance is now in progress, and the server will be offline shortly.

  • Update
    2022-11-20 at 3:15 PMUTC
    Update
    2022-11-20 at 3:15 PMUTC

    Maintenace has been rescheduled to tomorrow at 11:30 PM, Philippine Standard Time.

  • Planned
    2022-11-20 at 3:15 PMUTC
    Planned
    2022-11-20 at 3:15 PMUTC

    We'll conduct a database migration for our Vaultwarden instance on 2022-11-20 at 15:30 (UTC time) which you might experience up to 1-2 hours depending on the activity. This maintenance is needed as we migrate the database into a separate provider for availability reasons (although we're not going full HA yet) and to do an upgrade deployment afterwards.

    For more details, please see https://mau.dev/RecapTime/infra/issue-tracker/issues/1 for up-to-date information.

2022-10

No notices reported this month

2022-09

No notices reported this month

2022-09 to 2022-11

Next