Blog de Mageia (English)

We have always had bots visiting our website. They were mostly kind bots, like the crawlers that keep the databases of search engines up-to-date. Those kind bots start by looking at our robots.txt files before doing anything, and respect the restrictions that are set in those files.

However, things have changed. Like other websites, for instance Wikipedia, we are more and more being visited by AI scrapers, bots that scrape the Internet for anything they can find to train AI applications. They are usually extremely hungry for information, so they download much, much more than an ordinary user would do. Moreover, many of them are impolite: they don’t respect the rules set in our robots.txt files, they hide who they really are, they don’t put a little pause in between requests – on the contrary, they hammer our servers with requests from lots and lots of different IP addresses at the same time. The result is that parts of mageia.org, like our Bugzilla, Wiki and Forums, become unreachable.

Below you can see the CPU load of one of our most important servers, where, amongst other things, our forums and wiki are located:

Even if our infra upgrade had already been finished, this would be really hard to mitigate.

Blocking the used IP addresses is useless because they constantly switch to new ones. One of our sysadmins just told me about a big issue: “mobile proxies” where bots proxy their request through unsuspecting users’ phones. That makes the requests look much more legitimate and hard to block without also blocking real users. A lot of that happens without users even knowing their phone is being used like this. Some applications include proxies along with some game or other app and hide it in fine print in the terms of service. Last year, it was reported that Google had removed a bunch of such applications from their store.

Apart from phones, there are IoT devices and also ordinary computers that ended up in botnets, because they were not well protected. They can be used for AI scraping and probably are now.

Our sysadmins do time and again succeed in mitigating the problem, but it is a “cat and mouse game”, so the problem is likely to reoccur.

If you know people working on AI applications which need to be trained, please ask them to make sure their bots read and respect the robots.txt files they encounter. And, of course, please nudge your friends and family, when you think they need that, to make sure their computers and other smart devices get all security updates as soon as they are released.

To do a good job, we need good tools. Some of our servers are old, no longer powerful enough and have limited disk resources to meet the needs of developers. RPM manufacturing takes a long time and this is detrimental to the efficiency of maintaining and evolving the distribution. In short, the machines are well depreciated.

This is why our infrastructure is first getting a makeover. Better adapted to new technologies, it will allow our developers to work faster and more efficiently.

So where is this new infrastructure?

 We received 5 new servers:

– 2 new nodes for building packages: HPE ProLiant DL 360 Gen10 – 2xXeon 6126 (12C/2.6GHz) –

256GB RAM – 2xSSD 3.8TB HW Raid 1 – 2x10Gb/s NICs

– 2 servers to replace sucuk and duvel: HPE ProLiant DL 380 Gen10 – 2 Xeon 6126 (12C/2.6GHz) –

256GB RAM – 2xSSD 3.8TB HW Raid 1 – 10xHDD 12TB HW Raid 5 – 2x10Gb/s NICs

– 1 server for deployment and backup: HPE ProLiant DL80 Gen9 – 2xXeon  E5-2603v4

(6C/1.7GHz) – 256GB RAM – 6xHDD 6TB (donated, with some renewed parts)

– 1 Arista 7120T switch 20xRJ-45 10Gb/s 4xSFP+ 10Gb/s for interconnecting the machines

One of the ideas is to use the latest server to deploy quickly and as automatically as possible the construction nodes and other machines. The method is ready for x86_64 nodes and is being finalized for ARM nodes. The preparation of the servers takes time because the teams anticipate the future and future developments.

Once the preparation part of our servers is finished, the integration part into the Data Center will remain.

We are therefore taking our time to do things well in order to perpetuate the future and future versions of Mageia.

In the meantime, the future version 10 of Mageia continues to bubble in its cauldron! But we are not ready yet to plan a release date for the moment.

Feel free to come and strengthen our teams.