Fedora Copr stories

(Fedora Nest 2022)

Pavel Raiskup <praiskup@redhat.com>

Miroslav Suchý <msuchy@redhat.com>

Statistics

• Users with projects: 3 171
• Projects: 19 228
• Storage: 13TB of packages in repos (copr-be)
  • AWS EC2 16TB volume (max)
  • Weekly snapshots

Stats #2

• Builds: 7600 builds/day average, +50GB/day
• Aggressive cleanups, one NVR, auto-project removal, otherwise ...

Storage in focus

• We needed a storage usage monitoring
• 2021 year of Packit

Storage in focus

• EOL policy effects

Througput

• Peaks > 50k/day, not on limit!
• Up to 45/35 concurrent builds for user
• On up to 285 concurrent builders *ATM*

Architectures

• native builds needed, but armhfp --forcearch
• x86_64 on-premise, fallback to AWS
• s390x in IBM Cloud
• ppc64le on-premise, fallback to OSUOSL
• aarch64 AWS only

Builders

• History, and now ...
• resalloc
• Plug-in pool-configuration
• Flexibility, new overview page
• No babysitting!

Generating golden images

• Currently complicated
• virt-sysprep for Fedora on RHEL 8
• Poking at Image Builder, but no support yet

Througput considerations

• Many builds generated by a few large projects
• createrepo_c --update is run very frequently
• I/O: createrepo --skip-stat :-)
• I/O: --recycle-pkglist
• CPU: Not enough → Batched createrepo
• CPU: @dralley optimized out 85%

Througput considerations #2

• appstream-builder is slow
• Dnf is slow for large projects

Build timeouts

• Bugs in spec files that hang the build
• What is the best timeout value? Max allowed timeout value?
• Build timeout needed tweaks, 5h default, Blink!

Fun facts

• ivyxxcspcqlaocvjbghawvbdartwsfffurhnqzlwvsbgieweawfntu\ wecdcminmiaunqteqgbrfuxppntjdvyvsswxwepnbfqstnrnsotrhn\ dihkudyahthaxatviwrwtgllwbqhibouqctrxtypac
• Batches and llvm
• SHA-256 signatures, obs-sign

Users' demand

• More powerful builders
• Auto-rebuilds for dep-changes
• Easier Copr deployment, OpenShift attempt

In Copr team we ...

• Develop, well.. Copr, Mock, Tito, DistGit, Resalloc, ..
• Deploy Fedora Copr (with Fedora Infra!)
• ... with interesting limits and quotas
• ... and with Packit integration
• Keep minimal barriers for building and distributing
• Push the packaging stack to the limits
• Talk on #fedora-buildsys (libera.chat)