Recap Time Squad's infra Status - Incident history https://recaptimesquad.instatus.com Recap Time Squad's infra Sat, 28 Jan 2023 14:30:00 +0000 Planned email migration to biz.mail.ru Type: Maintenance Duration: 1 hour and 30 minutes Affected Components: Email Jan 28, 16:00:00 GMT+0 - Completed - Maintenance has completed successfully Jan 28, 14:30:00 GMT+0 - Identified - We are planning for a scheduled maintenance, which might affect deliverability and reachability of `@recaptime.eu.org` email addresses due to DNS propagation by the weekend before end of the month. For squad members with existing `@recaptime.eu.org` email address on Yandex.Mail, please coordinate with Andrei Jiroh for migrating the emails to the new provider. We're done migrating mail for our legacy domain (`recaptime.tk`) there earlier this week (2023-01-04). Jan 28, 14:30:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour and 30 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>14:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance, which might affect deliverability and reachability of `@recaptime.eu.org` email addresses due to DNS propagation by the weekend before end of the month. For squad members with existing `@recaptime.eu.org` email address on Yandex.Mail, please coordinate with Andrei Jiroh for migrating the emails to the new provider. We're done migrating mail for our legacy domain (`recaptime.tk`) there earlier this week (2023-01-04)..</p> <p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>14:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Sat, 28 Jan 2023 14:30:00 +0000 https://recaptimesquad.instatus.com/maintenance/clcj7um3w5984hgovo71ikql7 https://recaptimesquad.instatus.com/maintenance/clcj7um3w5984hgovo71ikql7 Miraheze server maintenance might cause Squad Wiki to go read-only or offline Type: Maintenance Affected Components: Wiki (hosted on Miraheze) Dec 18, 21:00:00 GMT+0 - Identified - Miraheze will be doing server maintenance beginning at 21:00 UTC on Sunday, 18 December 2022 lasting until 03:00 UTC on Monday, 19 December 2022. During this time, some wikis may be intermittently unavailable or go into read-only. Dec 18, 21:00:00 GMT+0 - Completed - Maintenance has completed successfully, but a few hiccups were happened. Please see https://meta.miraheze.org/wiki/Tech:SRE_noticeboard for further updates. Type: Maintenance

Affected Components:

<p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>21:00:00</var> GMT+0</small><br><strong>Identified</strong> - Miraheze will be doing server maintenance beginning at 21:00 UTC on Sunday, 18 December 2022 lasting until 03:00 UTC on Monday, 19 December 2022. During this time, some wikis may be intermittently unavailable or go into read-only..</p> <p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>21:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully, but a few hiccups were happened. Please see https://meta.miraheze.org/wiki/Tech:SRE_noticeboard for further updates..</p> ]]> Sun, 18 Dec 2022 21:00:00 +0000 https://recaptimesquad.instatus.com/maintenance/clbk6r88p8746imncq177jqlr https://recaptimesquad.instatus.com/maintenance/clbk6r88p8746imncq177jqlr Squad Wiki temporarily unavailable due to issues on one of Miraheze's servers Type: Incident Duration: 16 days, 20 hours and 54 minutes Affected Components: Wiki (hosted on Miraheze) Nov 18, 18:00:00 GMT+0 - Identified - The sysadmins at Miraheze are working on bringing cloud14 back online, albeit without the databases. Once cloud14 is back, MediaWiki speeds should improve, and mail should work once again. There is no ETA for when wikis will return, however. (update from <https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283340>) Nov 24, 07:47:44 GMT+0 - Monitoring - Sorry for this late update, but we're being notified days ago by Miraheze team that our wiki is [affected by recent `cloud14` issues](https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283778#Cloud14_issues), particularly the disk issues on `db141`. We're currently monitoring the situation behind the scenes via [this GitLab issue](https://mau.dev/RecapTime/infra/issue-tracker/-/issues/2). Dec 9, 14:53:23 GMT+0 - Monitoring - We're technically back, although an issue with our custom domain has been fixed behind the scenes. Because [we recreated the wiki without any backups](https://mau.dev/RecapTime/infra/issue-tracker/-/issues/2#note_395) ([while the drives where the database cluster is the wiki's database](https://meta.miraheze.org/wiki/Tech:SRE_noticeboard) is under [Owen](https://meta.miraheze.org/wiki/User:Owen)'s custody in the meanwhile), this could take longer to make things back into its state before the unexpected drive issues happened. Please keep tuning in to the linked GitLab issue for updates regarding the situation, but for now, we downgraded the severity of this incident to `Degraded performance` per this update. Jan 5, 14:54:14 GMT+0 - Resolved - We just resolved the issue, literally days ago! Sorry for the late update, but the wiki is now officially kicking in, albeit with some backend issues, missing pages and stuff. Type: Incident

Duration: 16 days, 20 hours and 54 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Identified</strong> - The sysadmins at Miraheze are working on bringing cloud14 back online, albeit without the databases. Once cloud14 is back, MediaWiki speeds should improve, and mail should work once again. There is no ETA for when wikis will return, however. (update from <https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283340>).</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>07:47:44</var> GMT+0</small><br><strong>Monitoring</strong> - Sorry for this late update, but we're being notified days ago by Miraheze team that our wiki is [affected by recent `cloud14` issues](https://meta.miraheze.org/w/index.php?title=Tech:SRE_noticeboard&oldid=283778#Cloud14_issues), particularly the disk issues on `db141`. We're currently monitoring the situation behind the scenes via [this GitLab issue](https://mau.dev/RecapTime/infra/issue-tracker/-/issues/2)..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>14:53:23</var> GMT+0</small><br><strong>Monitoring</strong> - We're technically back, although an issue with our custom domain has been fixed behind the scenes. Because [we recreated the wiki without any backups](https://mau.dev/RecapTime/infra/issue-tracker/-/issues/2#note_395) ([while the drives where the database cluster is the wiki's database](https://meta.miraheze.org/wiki/Tech:SRE_noticeboard) is under [Owen](https://meta.miraheze.org/wiki/User:Owen)'s custody in the meanwhile), this could take longer to make things back into its state before the unexpected drive issues happened. Please keep tuning in to the linked GitLab issue for updates regarding the situation, but for now, we downgraded the severity of this incident to `Degraded performance` per this update..</p> <p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>14:54:14</var> GMT+0</small><br><strong>Resolved</strong> - We just resolved the issue, literally days ago! Sorry for the late update, but the wiki is now officially kicking in, albeit with some backend issues, missing pages and stuff..</p> ]]> Thu, 24 Nov 2022 07:47:44 +0000 https://recaptimesquad.instatus.com/incident/claurvbco22758ifnaz2f7vjoe https://recaptimesquad.instatus.com/incident/claurvbco22758ifnaz2f7vjoe Scheduled database migration for Vaultwarden Type: Maintenance Duration: 3 days, 16 hours and 25 minutes Affected Components: Vaultwarden Nov 20, 15:15:00 GMT+0 - Identified - Maintenace has been rescheduled to tomorrow at 11:30 PM, Philippine Standard Time. Nov 20, 15:15:01 GMT+0 - Identified - Maintenance is now in progress, and the server will be offline shortly. Nov 21, 10:42:28 GMT+0 - Identified - Sorry for the long downtime, we're still fixing some stuff behind the scenes. We'll be back soon. Nov 20, 15:15:00 GMT+0 - Identified - We'll conduct a database migration for our Vaultwarden instance on 2022-11-20 at 15:30 (UTC time) which you might experience up to 1-2 hours depending on the activity. This maintenance is needed as we migrate the database into a separate provider for availability reasons (although we're not going full HA yet) and to do an upgrade deployment afterwards. For more details, please see https://mau.dev/RecapTime/infra/issue-tracker/issues/1 for up-to-date information. Nov 24, 07:39:33 GMT+0 - Completed - Maintenance has completed successfully days ago. Sorry for this late update, but we're now kicking tires behind the scenes. Nov 20, 17:12:10 GMT+0 - Identified - Maintenance is still in progress, but we're done migrating to a new database hosting provider. You'll be notified one things are ready. Type: Maintenance

Duration: 3 days, 16 hours and 25 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>15:15:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenace has been rescheduled to tomorrow at 11:30 PM, Philippine Standard Time..</p> <p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>15:15:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress, and the server will be offline shortly..</p> <p><small>Nov <var data-var='date'> 21</var>, <var data-var='time'>10:42:28</var> GMT+0</small><br><strong>Identified</strong> - Sorry for the long downtime, we're still fixing some stuff behind the scenes. We'll be back soon..</p> <p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>15:15:00</var> GMT+0</small><br><strong>Identified</strong> - We'll conduct a database migration for our Vaultwarden instance on 2022-11-20 at 15:30 (UTC time) which you might experience up to 1-2 hours depending on the activity. This maintenance is needed as we migrate the database into a separate provider for availability reasons (although we're not going full HA yet) and to do an upgrade deployment afterwards. For more details, please see https://mau.dev/RecapTime/infra/issue-tracker/issues/1 for up-to-date information..</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>07:39:33</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully days ago. Sorry for this late update, but we're now kicking tires behind the scenes..</p> <p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>17:12:10</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is still in progress, but we're done migrating to a new database hosting provider. You'll be notified one things are ready..</p> ]]> Sun, 20 Nov 2022 15:15:00 +0000 https://recaptimesquad.instatus.com/maintenance/cl9srassg23538i4obiwq09bxa https://recaptimesquad.instatus.com/maintenance/cl9srassg23538i4obiwq09bxa