Surprisingly, this downtime was even longer than that point the datacenter the place a few of our servers had been burned down.
We’re a neighborhood platform, which is solely funded by our neighborhood by means of small charitable donations — so that you, as our beneficiaries and stakeholders, who assist us and encourage us — should get clarification on what occurred, what we did, and what we’ll do sooner or later to mitigate this threat once more.
What occurred
Basically, our predominant server (referred to as manta) misplaced connection to our personal community at 14:54 UTC. Thibault, our founder and predominant developer, and Lucas, our charity President and predominant system administrator, instantly started investigating.
Inside minutes, that they had each tried our regular failsafes and recognized the trigger was doubtless a bodily {hardware} problem in our supplier’s (OVH) knowledge centre. After that problem was lastly fastened by OVH technicians, a secondary problem cropped up on the identical server, this time affecting entry to the web, which may additionally solely be fastened bodily by a technician within the knowledge centre. From begin to end, the interventions lasted roughly 10 hours.
What we did
With Thibault and Lucas making an attempt a number of fixes, the content material crew additionally rapidly stepped into movement. The downtime was unlucky, given the Chess Olympiad was in progress and our every day commentary was reside with hours left.
When it turned clear the downtime can be quite a lot of minutes, they started organising our commentator, GM Illia Nyzhnyk, to a personal sandbox model of Lichess. Illia — streaming for Lichess solo for the primary time — dealt with the state of affairs exceptionally professionally, and continued his glorious commentary virtually seamlessly.
With our personal choices rapidly exhausted on the technical facet, Lucas submitted a assist ticket to our supplier at 15:33 UTC.
There wasn’t rather more we may do aside from await OVH to ship a technician to repair the bodily problem with our server. Thibault, Lucas, and by now a number of different builders and programs directors, had been now discussing more and more outlandish (and exceptionally costly) concepts to get Lichess quickly again on-line.
After a comply with up on our ticket to OVH, round two hours later, OVH tell us {that a} community connector had been changed and the intervention was deemed profitable.
Nevertheless, we had been then virtually instantly hit by a further problem; now it regarded like the opposite community connector of the server was out of order! We then needed to ship one other assist ticket, however with it coming as much as 19:00 UTC, or 21:00 in Europe, and with out getting additional replies from OVH, it appeared our subsequent intervention had been pushed till morning.
By now, just about all of our programs directors and builders had been in the identical channel, with over 1,000 messages being despatched within the infrastructure channel of our Zulip. Sadly, there was as soon as once more nothing extra we may do aside from wait.
Coming as much as 22:00 UTC, and having heard nothing farther from OVH, we assumed the difficulty doubtless would solely be checked out within the morning. Lucas determined to name it an evening (with name alerts arrange) an hour later. Thibault stayed up.
At 00:41 UTC it appeared motion was being taken on the server, judging from the logs and a few bumpy restarts. Ultimately, Lichess was up and steady from 01:25 UTC.
What we’ll do sooner or later
We’re within the strategy of getting additional info from OVH to grasp which element(s) needed to be bodily changed, and why the intervention appeared to take an unusually very long time.
Past that, we’re investigating failsafes we are able to arrange for these points. The problem just isn’t solely with OVH — to some extent, we additionally ought to have additional failsafes arrange. Nevertheless, these failsafes take a substantial period of time and experience to arrange, coping with very delicate elements of Lichess. With the time and experience it takes, and on the finances that now we have, we should ruthlessly prioritise.
Whereas our builders and system directors are extremely proficient, they’re typically part-time volunteers; Lucas has a day job as a system administrator. One other of our sysadmins is a CTO at an modern huge geo startup. One other is in aeronautics for a residing.
This isn’t shared to excuse us, however to focus on that the abilities and assets we already depend on are sometimes stretched skinny, with different obligations — and that these people have all contributed their time as a result of they play on Lichess and like the location.
FAQ
I misplaced score factors — will these be refunded?
Sadly, any score misplaced won’t be refunded. We admire that is unlucky, however technically it is rather troublesome to have the ability to refund score after such a crash.
On the brilliant facet, score just isn’t a foreign money, it robotically adjusts to your talent. It is going to return to regular after only a few extra rated video games.
I’ve solely been getting video games as white / black. Is that linked to the crash?
We just lately made a change the place gamers can now not select to play as black or white in all video games apart from direct challenges. Some gamers have a debt to a sure color, to rebalance the video games they owe with that color. These points will not be linked. Nevertheless, we will probably be including a small change in order that the rebalancing will probably be extra gradual, and never 20+ video games of 1 color in a row.
