Lector de Feeds

Initial notes on supporting vendored dependencies

New page

A '''vendored dependency''' is an aggregation of code (such as a package, module or library) that is included as part a larger aggregation (usually an application) but which is also available separately standalone (such as a dynamic library). A typical example is an application (e.g. Mariadb) that distributes the source for a separate library within its source tree (e.g. readline). This can be done for a number of reasons, such as licensing issues that prevent using of the module separately, custom code changes made for the application's use that are not/can not be upstreamed, and ease of compilation for the developer.

There are many downsides to this approach. Probably the biggest one is that when a standalone dynamic library is updated to fix a security bug, the vendored versions included in other applications are not automatically updated. These applications must be updated separately, and the upstream developer may not immediately (or even ever) include the security fixes in the vendored copy, leaving the application vulnerable to security issues.

For these reasons, vendored libraries are discouraged in Mageia ''(TBD: point to the policy)''

A closely related issue is using '''dependencies that are downloaded''' at compile time. This are common in languages such as Go, Rust and Javascript (npm) and it ends up with similar problems to vendoring. The result is even worse problems because it can not only be difficult to determine which dependencies have been used, but downright impossible to determine the version numbers actually used at compile time. Without dependency names and version numbers, it becomes impossible to tell when a package is affected by a security issue in a dependency. When a security issue is reported, it can result in the need to recompile many application packages instead of a single one. It is also difficult to ensure that the licenses of all downloaded packages match those allowed by the distribution. And when packages have licenses like the GPL that require source code be supplied with the binary, it becomes mandatory to make a copy of the downloaded source available to users.

For these reasons, files downloaded at compile time are disallowed in Mageia ''(TBD: point to the policy)''

'''Static linking''' is another practice that effectively results in the same problems as the above. Some languages (e.g. Go, Rust) statically link their dependencies so security issues in those dependencies means rebuilding them all against the fixed packages.

For these reasons, static linking is discouraged in Mageia ''(TBD: point to the policy)''

These are really three separate topics but since the effects of all three are very similar, they are discussed here as a block.

== Problems with disallowing vendoring ==

When vendored (bundled) and downloaded packages are disallowed, they must instead be packaged separately. This means extra work for packagers since adding a single new complicated application can require individually packaging dozens or even hundreds of new separate dependency packages. This takes extra time, disk space and the especially rare commodity, packager time. It is simply not scalable and effectively means that new applications written in some of the languages particularly prone to this way of working just aren't available to Mageia users.

== The landscape ==

Two languages becoming more popular these days, Go and Rust, particularly suffer from the issues described and supporting applications using them in Mageia is difficult due to policies designed for the C and C++ applications that were the most popular ones in the past. If we want to support programs in these languages, we need to ease the burden on packagers.

The main reasons for anti-vendoring (and related) policies are:

# to easily identify which packages need to be updated to fix security issues
# to ensure that a known security issue does not unknowingly go unfixed in the distribution
# to reduce the work in updating those packages when necessary
# to reduce time, bandwidth and disk space for users
# to ensure source code is always available to users to fulfill licensing obligations

If we can find a way to satisfy those requirements to a reasonable degree while still allowing vendoring and downloading of modules at compile time, we can ease the burden on our packagers and infrastructure.

== A way forward ==

=== Go ===

The following proposal satisfies the reasons for the anti-vendoring policies above, while allowing applications to be packaged without individually packaging each dependency. The specifics listed in this section are for handling Go applications, but it can be generalized for other languages in the future.

TBD

=== C/C++ ===

Since these languages support dynamic linking and the developer culture does not generally encourage a huge number of small dependencies, no proposal is currently being made to ease the vendoring restrictions in C or C++ applications.

=== Others ===

The proposals for Go should be able to be generalized to work for applications in other compiled languages as well. It is possible to develop infrastructure to support interpreted languages as well, but the benefits may not be as clear.

The languages with the most number of modules in Mageia as of this writing (in approximate decreasing order) are:

# Rust
# Perl
# Python
# Java
# Go
# Ruby
# OCaml
# Nodejs (Javascript)
# PHP
# Erlang

== See Also ==

* [[Packages carrying bundled copies of system libraries]]
* [[Security Updates]]

[[Category:Packaging]] Danf

‎Cancel the build: Run as schedbot

← Older revision Revision as of 06:58, 24 January 2025 Line 11: Line 11:  Find the build at https://pkgsubmit.mageia.org/ and look under ''Status'' for the build's ID (it will look like YYYYMMDDHHMMSS.USER.HOST.ID). Log in to duvel and run: Find the build at https://pkgsubmit.mageia.org/ and look under ''Status'' for the build's ID (it will look like YYYYMMDDHHMMSS.USER.HOST.ID). Log in to duvel and run:    −   cancel_build 20231231211004.pterjan.duvel.1487312+   sudo -u schedbot cancel_build 20231231211004.pterjan.duvel.1487312     providing the build ID you just found. providing the build ID you just found. Danf

Figured I'd post an update on how things are going with the new laptop (HP Omnibook Ultra 14, AMD Ryzen AI 9 365 "Strix Point", for the searchers) and with Silverblue.

I managed to work around the hub issue by swapping out the fancy $300 Thunderbolt hub for a $40 USB-C hub off Amazon. This comes with limitations - you're only going to get a single 4k 60Hz external display, and limited bandwidth for anything else - but it's sufficient for my needs, and makes me regret buying the fancy hub in the first place. It seems to work 100% reliably on startup, reboot and across suspend/resume. There's still clearly something wrong with Thunderbolt handling in the kernel, but it's not my problem any more.

The poor performance of some sites in Firefox turned out to be tied to the hanging problem - I'd disabled graphics acceleration in Firefox, which helped with the hanging, but was causing the appalling performance on Google sites and others. I've now cargo-culted a set of kernel args - amdgpu.dcdebugmask=0x800 amdgpu.lockup_timeout=100000 drm.vblankoffdelay=0 - which seem to be helping; I turned graphics acceleration back on in Firefox and it hasn't started hanging again. At least, I haven't had random hangs for the last few days, and this morning I played a video on youtube and the system has not hung since then. I've no idea how bad they are for battery life, but hey, they seem to be keeping things stable. So, the system is pretty workable at this point. I've been using it full-time, haven't had to go back to the old one.

I'm also feeling better about Silverblue as a main OS this time. A lot of things seem to have got better. The toolbox container experience is pretty smooth now. I managed to get adb working inside a container by putting these udev rules in /etc/udev/rules.d. It seems like I have to kill and re-start the adb server any time the phone disconnects or reboots - usually adb would keep seeing the phone just fine across those events - but it's a minor inconvenience. I had to print something yesterday, was worried for a moment that I'd have to figure out how to get hp-setup to do its thing, but then...Silverblue saw my ancient HP printer on the network, let me print to it, and it worked, all without any manual setup at all. It seems to be working over IPP, but I'm a bit surprised, as the printer is from 2010 or 2011 and I don't think it worked before. But I'm not complaining!

I haven't had any real issues with app availability so far. All the desktop apps I need to use are available as flatpaks, and the toolbox container handles CLI stuff. I'm running Firefox (baked-in version), Evolution, gedit, ptyxis (built-in), liferea, nheko, slack and vesktop (for discord) without any trouble. LibreOffice and GIMP flatpaks also work fine. Everything's really been pretty smooth.

I do have a couple of tweaks in my bashrc (I put them in a file in ~/.bashrc.d, which is a neat invention) that other Atomic users might find useful...

if [ -n "$container" ] then alias gedit="flatpak-spawn --host /var/lib/flatpak/exports/bin/org.gnome.gedit" alias xdg-open=flatpak-xdg-open else alias gedit=/var/lib/flatpak/exports/bin/org.gnome.gedit fi

the gedit aliases let me do gedit somefile either inside or outside a container, and the file just opens in my existing gedit instance. Can't really live without that. You can adapt it for anything that's a flatpak app on the host. The xdg-open alias within containers similar makes xdg-open somefile within the container do the same as it would outside the container.

So it's still early days, but I'm optimistic I'll keep this setup this time. I might try rebasing to the bootc build soon.

Written by Atelier Team

That was fast: we have only just explained why our Mageia Application Database was offline and now papoteur tells us that new MADb is ready to be used.

Open the site and at first glance you might think you have somehow connected to the old site as the differences between it and the new one are hard to spot (the top one is the old site):

However, this is only the outward appearance, as the style sheets (CSS) were re-used with little modification.

The tool itself, previously written in PHP, has been completely rewritten from scratch, using Python, Flask, Jinja2 and DNF5, so the runtime code is entirely new. Papoteur showed two code snippets as an example of what really changed (again, the top one is old):

That is as different as a page from an English book compared to a page from its Indonesian translation!

About 3000 new lines of code were written for this new MADb, which is now live and ready to answer your queries about Mageia applications.

As you use the site, we ask you to think about what questions you feel an integrated help function (not yet available) should answer, and also what you feel we should include in a new wiki article about MADb. Please note your suggestions in the comments to this post, as you are all the Mageia community and this tool is for you.

Thanks to the MLO community for hosting new MADb.

UPDATE: It is now possible to visit the new MADb here https://madb.mageialinux-online.org/. It will take a bit longer for https://madb.mageia.org/ to link to it.

Written by aguador.

In Mageia, MADb, the Mageia App Db, has been an essential tool, particularly for QA (Quality Assurance) testers. It is the goto site for information on applications in our repositories with links to bug reports, priorities for updates, version comparisons and more. Searchable by Mageia version and CPU architecture, the site has not only been key for developers and testers, but many users as well who have found it an alternative to searching with our MCC control center or the command line when looking for a package to do (“whatever”).

But, er, “Houston, we have a problem.” If I go to MADb (https://madb.mageia.org/) all I got was the error message below and now a redirect to this post!

MADb was not affected by the move of Mageia’s servers announced on this blog (https://blog.mageia.org/en/2024/10/08/most-of-our-servers-will-be-offline-because-they-are-relocating/) early this month because it was originally developed by two of our contributors many years ago, and running on a different server. Mageia.Org took over ownership of their rented server a few months ago. Unfortunately, that server passed away and since the technology behind the old MADb is not compatible with newer infrastructure (mostly newer php-version), we cannot bring it back as it was. However, not all was lost!

Back in April, papoteur had submitted his initial work on a new version of MADb for testing…and since then it has undergone numerous revisions and improvements. However, it still remains “under wraps” for most users (like the author of this post!) until everyone, above all papoteur, is satisfied that it is not only a solid db interface for users, but is even better than before.

Since MADb has played such a vital role in testing, the development version is available to the QA team and other testers. It is only fair that they get the first look and use given all the work they do to assure that Mageia remains a quality distro. The rest of us simply need a bit more patience.

Ah, and not to forget the servers, not only was the move successful, with the other services affected now back up and running smoothly, but we expect to announce more good news about our servers, soon. Apart from that, most Mageia mirrors are in a good shape (they are all hosted on external servers, which we do not control).

In Mageia/9/x86_64: Mesa is an OpenGL 4.6 compatible 3D graphics library.

In Mageia/9/aarch64: Mesa is an OpenGL 4.6 compatible 3D graphics library.

In Mageia/9/armv7hl: Mesa is an OpenGL 4.6 compatible 3D graphics library.

In Mageia/9/i586: Mesa is an OpenGL 4.6 compatible 3D graphics library.

In Mageia/cauldron/x86_64: Rachota is a portable application for timetracking different projects. It runs everywhere. It displays time data in diagram form, creates customized reports and invoices or analyses measured data and suggests hints to improve user's time usage. The totally portable yet personal timetracker.

In Mageia/cauldron/i586: Rachota is a portable application for timetracking different projects. It runs everywhere. It displays time data in diagram form, creates customized reports and invoices or analyses measured data and suggests hints to improve user's time usage. The totally portable yet personal timetracker.

In Mageia/cauldron/i586: A program to convert images from PPM format into the control language for the Alps Micro-Dry printers, at various times sold by Citizen, Alps and Okidata. This program drives the Alps Micro-Dry series of printers, including the Citizen Printiva series, Alps MD series, and Oki DP series (but not yet the DP-7000). In the current release, the program drives the standard mode fairly well; the dye sublimation mode very well; and the VPhoto mode reasonably well. It supports all the colours available up to the DP-5000, including the foil colours.

In Mageia/cauldron/x86_64: A program to convert images from PPM format into the control language for the Alps Micro-Dry printers, at various times sold by Citizen, Alps and Okidata. This program drives the Alps Micro-Dry series of printers, including the Citizen Printiva series, Alps MD series, and Oki DP series (but not yet the DP-7000). In the current release, the program drives the standard mode fairly well; the dye sublimation mode very well; and the VPhoto mode reasonably well. It supports all the colours available up to the DP-5000, including the foil colours.

In Mageia/cauldron/x86_64: This tool tries to recover JFIF (JPEG) pictures and MOV movies (using recovermov) from a peripheral. This may be useful if you mistakenly overwrite a partition or if a device such as a digital camera memory card is bogus.

In Mageia/cauldron/i586: This tool tries to recover JFIF (JPEG) pictures and MOV movies (using recovermov) from a peripheral. This may be useful if you mistakenly overwrite a partition or if a device such as a digital camera memory card is bogus.

In Mageia/cauldron/x86_64: Rdfind is a program that finds duplicate files. It is useful for compressing backup directories or just finding duplicate files. It compares files based on their content, NOT on their file names.

In Mageia/cauldron/i586: Rdfind is a program that finds duplicate files. It is useful for compressing backup directories or just finding duplicate files. It compares files based on their content, NOT on their file names.

In Mageia/cauldron/x86_64: Unifont is a Unicode font with a glyph for every visible Unicode Basic Multilingual Plane code point and more, with supporting utilities to modify the font. This package contains tools and glyph descriptions.

In Mageia/cauldron/i586: Unifont is a Unicode font with a glyph for every visible Unicode Basic Multilingual Plane code point and more, with supporting utilities to modify the font. This package contains tools and glyph descriptions.