Lector de Feeds

It's time for that rarest of events: a blog post! And it's another debugging adventure. Settle in, folks!

Recently I got interested in improving the time it takes to do a full compose of Fedora. This is when we start from just the packages and a few other inputs (basically image recipes and package groups), and produce a set of repositories and boot trees and images and all that stuff. For a long time this took somewhere between 5 and 10 hours. Recently we've managed to get it down to 3-4, then I figured out a change which has got it under 3 hours.

After that I re-analyzed the process and figured out that the biggest remaining point to attack is something called the 'pkgset' phase, which happens near the start of the process, not in parallel with anything else, and takes 35 minutes or so. So I started digging into that to see if it can be improved.

I fairly quickly found that it spends about 20 minutes in one relatively small codepath. It's created one giant package set (this is just a concept in memory at the time, it gets turned into an actual repo later) with every package in the compose in it. During this 20 minutes, it creates subsets of that package set per architecture, with only the packages relevant to that architecture in it (so packages for that arch, plus noarch packages, plus source packages, plus 'compatible' arches, like including i686 for x86_64).

I poked about at that code a bit and decided I could maybe make it a bit more efficient. The current version works by creating each arch subset one at a time by looping over the big global set. Because every arch includes noarch and src packages, it winds up looping over the noarch and src lists once per arch, which seemed inefficient. So I went ahead and rewrote it to create them all at once, to try and reduce the repeated looping.

Today I was testing that out, which unfortunately has to be done more or less 'in production', so if you like you can watch me here, where you'll see composes appearing and disappearing every fifteen minutes or so. At first of course my change didn't work at all because I'd made the usual few dumb mistakes with wrong variable names and stuff. After fixing all that up, I timed it, and it turned out about 7 minutes faster. Not earth shattering, but hey.

So I started checking it was accurate (i.e. created the same package sets as the old code). It turned out it wasn't quite (a subtle bug with noarch package exclusions). While fixing that, I ran across some lines in the code that had bugged me since the first time I started looking at it:

if i.file_path in self.file_cache: # TODO: test if it really works continue

These were extra suspicious to me because, not much later, they're followed by this:

self.file_cache.file_cache[i.file_path] = i

that is, we check if the thing is in self.file_cache and move on if it is, but if it's not, we add it to self.file_cache.file_cache? That didn't look right at all. But up till now I'd left it alone, because hey, it had been this way for years, right? Must be OK. Well, this afternoon, in passing, I thought "eh, let's try changing it".

Then things got weird.

I was having trouble getting the compose process to actually run exactly as it does in production, but once I did, I was getting what seemed like really bizarre results. The original code was taking 22 minutes in my tests. My earlier test of my branch had taken about 14 minutes. Now it was taking three seconds.

I thought, this can't possibly be right! So I spent a few hours running and re-running the tests, adding debug lines, trying to figure out how (surely) I had completely broken it and it was just bypassing the whole block, or something.

Then I thought...what if I go back to the original code, but change the cache thing?

So I went back to unmodified pungi code, commented out those three lines, ran a compose...and it took three seconds. Tried again with the check corrected to self.file_cache.file_cache instead of self.file_cache...three seconds.

I repeated this enough times that it must be true, but it still bugged me. So I just spent a while digging into it, and I think I know why. These file caches are kobo.pkgset.FileCache instances; see the source code here. So, what's the difference between foo in self.file_cache and foo in self.file_cache.file_cache? Well, a FileCache instance's own file_cache is a dict. FileCache instances also implement __iter__, returning iter(self.file_cache). I think this is why foo in self.file_cache works at all - it actually does do the right thing. But the key is, I think, that it does it inefficiently.

Python's preferred way to do foo in bar is to call bar.__contains__(foo). If that doesn't work, it falls back on iterating over bar until it either hits foo or runs out of iterations. If bar doesn't support iteration it just raises an exception.

Python dictionaries have a very efficient implementation of __contains__. So when we do foo in self.file_cache.file_cache, we hit that efficient algorithm. But FileCache does not implement __contains__, so when we do foo in self.file_cache, we fall back to iteration and wind up using that iterator over the dictionary's keys. This works, but is massively less efficient than the dictionary's __contains__ method would be. And because these package sets are absolutely fracking huge, that makes a very significant difference in the end (because we hit the cache check a huge number of times, and every time it has to iterate over a huge number of dict keys).

So...here's the pull request.

Turns out I could have saved the day and a half it took me to get my rewrite correct. And if anyone had ever got around to TODOing the TODO, we could've saved about 20 minutes out of every Fedora compose for the last nine years...

Add note about MirrorBrain being dead upstream and about the alternative

← Older revision Revision as of 20:53, 16 November 2024 Line 1: Line 1:  [[Category: ProposedFeatureMageia8]] [[Category: ProposedFeatureMageia8]]     +{{Note|Mirrorbrain is dead upstream, a better alternative is [https://github.com/fedora-infra/mirrormanager2 MirrorManager2]. This page has been cloned and adjusted to create [[Feature:Migrate to MirrorManager2]] for Mageia10}}  = Summary = = Summary =    Marja

‎Summary: typo fix

← Older revision Revision as of 20:24, 16 November 2024 Line 6: Line 6:  = Summary = = Summary =    −Migrate mirror management to [MirrorManager2 (after adjusting it for Mageia) and start generating metalinks for DNF to use for intelligent mirror selection.+Migrate mirror management to MirrorManager2 (after adjusting it for Mageia) and start generating metalinks for DNF to use for intelligent mirror selection.     = Owner = = Owner = Marja

Marja moved page User:Marja/Feature:Migrate to MirrorManager2 to Feature:Migrate to MirrorManager2 ready for the wiki

← Older revision Revision as of 20:19, 16 November 2024 (2 intermediate revisions by the same user not shown)Line 10: Line 10:  = Owner = = Owner =    −The owner of [[Feature:Migrate to MirrorBrain]] was:+''The owner of [[Feature:Migrate to MirrorBrain]] was:'' −* '''Name''': Neal Gompa+* ''Name: Neal Gompa'' −* '''Email''': ngompa13@gmail.com+* ''Email: ngompa13@gmail.com''  +   +However, he lacks time nowadays, so a new owner is needed for [[Feature:Migrate to MirrorManager2]]:  +* '''Name''':  +* '''Email''':     = Resources = = Resources = Line 79: Line 83:  = Associated Bugs = = Associated Bugs =  * [https://bugs.mageia.org/show_bug.cgi?id=3166 mga#3166] * [https://bugs.mageia.org/show_bug.cgi?id=3166 mga#3166] −* [https://bugs.mageia.org/show_bug.cgi?id=17400 mga#17400]      = Packager comments = = Packager comments = Marja

Publication date: 16 Nov 2024
Type: bugfix
Affected Mageia releases : 9
Description Fixed a bug which could cause applications using GBM to crash when running with nvidia-drm.modeset=0 References

• https://bugs.mageia.org/show_bug.cgi?id=33704
• https://www.nvidia.com/en-us/drivers/details/233004/

SRPMS 9/nonfree

• nvidia-current-550.127.05-1.mga9.nonfree

‎Advisories for backports

← Older revision Revision as of 00:38, 16 November 2024 Line 431: Line 431:  = Advisories for backports = = Advisories for backports =  We are using the [https://ml.mageia.org/l/info/backports-announce backports-announce mailing list] to publish announcements: We are using the [https://ml.mageia.org/l/info/backports-announce backports-announce mailing list] to publish announcements: −* when new backport candidates are available for testing in the backports_testing repository+* when new backport candidates are available for testing in the backports_testing repository, we recommend start the subject of the mail with <nowiki>[Backport Candidate] package_name version</nowiki> −* and also after testing, once the packages have been moved to the backports repository+* and also after testing, once the packages have been moved to the backports repository, we recommend start the subject of the mail with one of this <nowiki>[New Backport],  [Security Update Backport], [Backport Update], as needed, then include after, package_name version</nowiki> −Any member of [http://people.mageia.org/g/mga-qa-committers.html the mga-qa-committers group] can send mails to the list, see [https://ml.mageia.org/l/arc/backports-announce/2024-07/ the list archive] for examples of the announcements.+Any member of [http://people.mageia.org/g/mga-qa-committers.html the mga-qa-committers group] can send mails to the list, see [https://ml.mageia.org/l/arc/backports-announce the list archive] for examples of the announcements.  Once the second announcement is published and the packages are in the backport repository, the bug for the backport request should be closed as RESOLVED FIXED. Once the second announcement is published and the packages are in the backport repository, the bug for the backport request should be closed as RESOLVED FIXED.    Katnatek

‎References and documents: try to enhance the lay out of the links and add a link to the Debian wiki

← Older revision Revision as of 22:10, 15 November 2024 (2 intermediate revisions by the same user not shown)Line 1: Line 1:    −{{Multi language banner|[[User:Zeldas7777|english]] ;}}+{{Multi language banner|[[Secure boot clarification|english]] ;}}     {{Draft}} {{Draft}}    −= Secure boot clarification =     −=== Overview ===+   += Overview =     Secure boot was created to ensure the protection of the operating system (OS). The Linux community did not like the secure boot upgrade. The end user would have to disable secure boot then install Linux. The reason was because the computers did not have the TPM Linux distribution signatures installed in the computer before manufacturing the computers. The Linux distribution developers would have to sign the bootloader, kernel, and drivers. This also created the need for more documentation for computers without Linux distribution signatures to successfully install Linux distribution. This wiki entry will hopefully bring some clarity from users in the Linux community on the secure boot implementation, and it's impact on Linux. The purpose of enabling secure boot is to ensure the OS is secure from rootkits, keyloggers, and malware. The secure boot is for protecting the end user from any threats to security and/or privacy. The secure boot feature stops the OS from booting upon detection of invalid signatures.   Secure boot was created to ensure the protection of the operating system (OS). The Linux community did not like the secure boot upgrade. The end user would have to disable secure boot then install Linux. The reason was because the computers did not have the TPM Linux distribution signatures installed in the computer before manufacturing the computers. The Linux distribution developers would have to sign the bootloader, kernel, and drivers. This also created the need for more documentation for computers without Linux distribution signatures to successfully install Linux distribution. This wiki entry will hopefully bring some clarity from users in the Linux community on the secure boot implementation, and it's impact on Linux. The purpose of enabling secure boot is to ensure the OS is secure from rootkits, keyloggers, and malware. The secure boot is for protecting the end user from any threats to security and/or privacy. The secure boot feature stops the OS from booting upon detection of invalid signatures.      −=== Who this applies to ===+= Who this applies to =     This wiki does not apply to your computer if it was manufactured before 2009. If your computer was manufactured in 2009 or later, You may have a Trusted Platform Module (TPM) chip. This wiki can apply to your computer. Computers with a TPM chip version 1.0 started to appear in 2009. This chip was soon updated to TPM version 1.1 in 2011. The next major update to TPM chips was version 2.0, which came out in 2014. This has been considered the new standard since 2016. The TPM chip does ensure the boot processes of your PC cannot be modified without your knowledge. If you would like to learn more about TPM, please check out the reference links below. This wiki does not apply to your computer if it was manufactured before 2009. If your computer was manufactured in 2009 or later, You may have a Trusted Platform Module (TPM) chip. This wiki can apply to your computer. Computers with a TPM chip version 1.0 started to appear in 2009. This chip was soon updated to TPM version 1.1 in 2011. The next major update to TPM chips was version 2.0, which came out in 2014. This has been considered the new standard since 2016. The TPM chip does ensure the boot processes of your PC cannot be modified without your knowledge. If you would like to learn more about TPM, please check out the reference links below. Line 18: Line 18:  Wikipedia - https://en.wikipedia.org/wiki/Trusted_Platform_Module<br> Wikipedia - https://en.wikipedia.org/wiki/Trusted_Platform_Module<br>    −=== Why the secure boot mode option was created ===+= Why the secure boot mode option was created =     Extensible Firmware Interface (EFI) was developed in the mid 1990's. In 2004, Intel released the first open source Unified Extensible Firmware Interface (UEFI) implementation. Then EFI was transitioned to Unified Extensible Firmware Interface (UEFI) in 2005. There has been several root vulnerabilities found in the computer BIOS that were exposed that allowed the booting OS to be compromised. This left the OS kernel and hardware drivers exposed. Eventually, even more vulnerabilities to the system were discovered making it hard to keep OS secure and protected from exploits. It was clear that the time had come to improve upon the Unified Extensible Firmware Interface (UEFI) with emphasis on making the process even more secure. Then "Trusted Platform Module (TPM)  was developed. This was not enough and TPM was updated to make secure boot mode possible. Secure boot was implemented in the BIOS using the TPM chip. This would allow authentication of the OS from signed bootloader, kernel, and drivers. This affected the Linux community because the time secure boot came out limited documentation was available at the time. This is why we have so many issues with secure boot. The Linux community has been working hard on this for years now to learn and implement secure boot on the OS.   Extensible Firmware Interface (EFI) was developed in the mid 1990's. In 2004, Intel released the first open source Unified Extensible Firmware Interface (UEFI) implementation. Then EFI was transitioned to Unified Extensible Firmware Interface (UEFI) in 2005. There has been several root vulnerabilities found in the computer BIOS that were exposed that allowed the booting OS to be compromised. This left the OS kernel and hardware drivers exposed. Eventually, even more vulnerabilities to the system were discovered making it hard to keep OS secure and protected from exploits. It was clear that the time had come to improve upon the Unified Extensible Firmware Interface (UEFI) with emphasis on making the process even more secure. Then "Trusted Platform Module (TPM)  was developed. This was not enough and TPM was updated to make secure boot mode possible. Secure boot was implemented in the BIOS using the TPM chip. This would allow authentication of the OS from signed bootloader, kernel, and drivers. This affected the Linux community because the time secure boot came out limited documentation was available at the time. This is why we have so many issues with secure boot. The Linux community has been working hard on this for years now to learn and implement secure boot on the OS.      −=== The secure mode operation design ===+= The secure mode operation design =     Secure boot mode is designed to authenticate the OS from a list of authorized operating systems in the TPM chip. By default, if a signature is in the "blocked" list, The computer will stop booting indicating that an invalid signature has been detected. Secure boot mode operation is meant to validate three areas while booting the OS. Authentication is performed by checking the bootloader, kernel, and kernel drivers on booting. If any of these areas fails authentication, the system will stop booting. This design creates a secure boot environment. If the bootloader, kernel, or its drivers are modified the signature is marked invalid and stops booting. The Invalid signatures are also installed when firmware updates the UEFI firmware. Any OS without a valid signature is also blocked. This presents a challenge during the development of an OS, but is required to maintain OS security. Secure boot mode is designed to authenticate the OS from a list of authorized operating systems in the TPM chip. By default, if a signature is in the "blocked" list, The computer will stop booting indicating that an invalid signature has been detected. Secure boot mode operation is meant to validate three areas while booting the OS. Authentication is performed by checking the bootloader, kernel, and kernel drivers on booting. If any of these areas fails authentication, the system will stop booting. This design creates a secure boot environment. If the bootloader, kernel, or its drivers are modified the signature is marked invalid and stops booting. The Invalid signatures are also installed when firmware updates the UEFI firmware. Any OS without a valid signature is also blocked. This presents a challenge during the development of an OS, but is required to maintain OS security.    −=== The secure mode operation while booting ===+= The secure mode operation while booting =     Secure mode authenticates the system from the installed signatures. Here is how the process works. Secure mode authenticates the system from the installed signatures. Here is how the process works. Line 35: Line 35:  If everything is successful, the OS will boot as expected. If everything is successful, the OS will boot as expected.    −=== CPU board manufacturer requirements ===+= CPU board manufacturer requirements =     CPU board manufacturers are required to follow fair trade laws This means that no company can be biased and that all OS vendors share equal rights. All manufactures have a standard to follow that is strictly monitored. We have a few types of CPU boards on the market that must comply with personal data security. Here are the following types of CPU boards that allow secure boot to be disabled, those that do not allow it, or made optional for custom manufactured computers. CPU board manufacturers are required to follow fair trade laws This means that no company can be biased and that all OS vendors share equal rights. All manufactures have a standard to follow that is strictly monitored. We have a few types of CPU boards on the market that must comply with personal data security. Here are the following types of CPU boards that allow secure boot to be disabled, those that do not allow it, or made optional for custom manufactured computers. Line 45: Line 45:  The documentation for the UEFI firmware is required to be made available to all OS vendors. This documentation shall have all commands required for UEFI firmware updates. The currently installed OS owns the updating of the firmware. If you have a dual-boot or multi-boot system, then each OS shares ownership rights. The documentation for the UEFI firmware is required to be made available to all OS vendors. This documentation shall have all commands required for UEFI firmware updates. The currently installed OS owns the updating of the firmware. If you have a dual-boot or multi-boot system, then each OS shares ownership rights.    −=== The requirement to enable secure boot ===+= The requirement to enable secure boot =     The requirements to successfully enable secure boot mode on an OS are: The requirements to successfully enable secure boot mode on an OS are: Line 55: Line 55:  # Must have a TPM chip.   # Must have a TPM chip.      −=== References and documents ===+= References and documents =    −Uefi.org documents in PDF file format+==Uefi.org documents in PDF file format==    −Uefi Information - https://uefi.org/sites/default/files/resources/UEFI_Secure_Boot_in_Modern_Computer_Security_Solutions_2013.pdf+Uefi Information - [https://uefi.org/sites/default/files/resources/UEFI_Secure_Boot_in_Modern_Computer_Security_Solutions_2013.pdf UEFI Secure Boot in Modern Computer Security Solutions 2013.pdf]     Microsoft KEK expiring because the certificate is expiring on 10/19/2026. This means there will be another secure boot update coming for current and new computers. Microsoft KEK expiring because the certificate is expiring on 10/19/2026. This means there will be another secure boot update coming for current and new computers.  Here is the link to the PDF document: Here is the link to the PDF document:  +[https://uefi.org/sites/default/files/resources/Evolving%20the%20Secure%20Boot%20Ecosystem_Flick%20and%20Sutherland.pdf Evolving the Secure Boot Ecosystem 9/12/2023]    −Evolving the Secure Boot Ecosystem 9/12/2023 - https://uefi.org/sites/default/files/resources/Evolving%20the%20Secure%20Boot%20Ecosystem_Flick%20and%20Sutherland.pdf+==Other links==    −To learn more about the UEFI and secure boot visit uefi.org website.+To learn more about the UEFI and secure boot visit [https://www.uefi.org the uefi.org website.]    −https://www.uefi.org+[https://wiki.debian.org/SecureBoot Secure Boot in the Debian wiki]    −=== Support history from the secure boot upgrade ===+= Support history from the secure boot upgrade =     Visit this link to see recent issues related to secure boot. Visit this link to see recent issues related to secure boot. Line 76: Line 77:  https://forums.mageia.org/en/search.php?keywords=secure+boot&fid%5B0%5D=7 https://forums.mageia.org/en/search.php?keywords=secure+boot&fid%5B0%5D=7    −=== Common issues with TPM ===+= Common issues with TPM =     Dual or multi-boot is harder to work with when you want to boot Windows and Linux. This can be even harder for Windows, Linux, and another OS. If the Linux distro does not support secure boot enabled and you have the TPM on the computer, you would need to enable legacy mode and disable secure boot. This is the only way to dual or multi boot with Windows. This will slow down the boot process and disable all BIOS protection. This will also disable any hardware improved features until the OS has booted. Remember that, if you need this kind of environment, you will need to reinstall Windows and any other operating systems you wish to dual or multi boot. This method is not recommended as it will open a security risk of having malware infecting or modifying your computer. Dual or multi-boot is harder to work with when you want to boot Windows and Linux. This can be even harder for Windows, Linux, and another OS. If the Linux distro does not support secure boot enabled and you have the TPM on the computer, you would need to enable legacy mode and disable secure boot. This is the only way to dual or multi boot with Windows. This will slow down the boot process and disable all BIOS protection. This will also disable any hardware improved features until the OS has booted. Remember that, if you need this kind of environment, you will need to reinstall Windows and any other operating systems you wish to dual or multi boot. This method is not recommended as it will open a security risk of having malware infecting or modifying your computer.    −=== Clarification conclusion ===+= Clarification conclusion =     Again, this article will hopefully bring some clarity to the confusion caused by the secure boot updates and its impact on Linux. I hope you learned the importance of the secure boot and why we need it. We need to maintain stable and secure Linux distributions for all users. I will be creating a "How to" and linking it to this wiki when I am finished. Again, this article will hopefully bring some clarity to the confusion caused by the secure boot updates and its impact on Linux. I hope you learned the importance of the secure boot and why we need it. We need to maintain stable and secure Linux distributions for all users. I will be creating a "How to" and linking it to this wiki when I am finished. Marja

Marja deleted page Mageia wiki:Secure boot clarification wrong location

Marja

Correct the redirect

← Older revision Revision as of 21:35, 15 November 2024 Line 1: Line 1: −#REDIRECT [[Mageia wiki:Secure boot clarification]]+#REDIRECT [[Secure boot clarification]] Marja

Marja moved page User:Zeldas7777 to Mageia wiki:Secure boot clarification This page needs to be more visible, so more people will provide feedback or improvements

← Older revision Revision as of 21:33, 15 November 2024 (One intermediate revision by the same user not shown)Line 1: Line 1:     {{Multi language banner|[[User:Zeldas7777|english]] ;}} {{Multi language banner|[[User:Zeldas7777|english]] ;}}  +  +{{Draft}}     = Secure boot clarification = = Secure boot clarification = Marja

Marja deleted page Template:Meertalige banner-nl wrong page name

Marja