glibc related cleanup

Extracted from https://github.com/Homebrew/brew/pull/13577
This commit is contained in:
Mike McQuaid 2022-08-23 12:42:02 +01:00
parent aa297cd8d1
commit c294dcc616
No known key found for this signature in database
GPG Key ID: 3338A31AFDB1D829
12 changed files with 62 additions and 35 deletions

View File

@ -172,7 +172,7 @@ class CompilerSelector
def compiler_version(name) def compiler_version(name)
case name.to_s case name.to_s
when "gcc", GNU_GCC_REGEXP when "gcc", GNU_GCC_REGEXP
versions.non_apple_gcc_version(name.to_s) versions.gcc_version(name.to_s)
else else
versions.send("#{name}_build_version") versions.send("#{name}_build_version")
end end

View File

@ -30,12 +30,23 @@ class DependencyCollector
@requirements = Requirements.new @requirements = Requirements.new
end end
def initialize_copy(other)
super
@deps = @deps.dup
@requirements = @requirements.dup
end
def add(spec) def add(spec)
case dep = fetch(spec) case dep = fetch(spec)
when Dependency when Dependency
@deps << dep @deps << dep
when Requirement when Requirement
@requirements << dep @requirements << dep
when nil
# no-op when we have a nil value
nil
else
raise ArgumentError, "DependencyCollector#add passed something that isn't a Dependency or Requirement!"
end end
dep dep
end end
@ -63,7 +74,7 @@ class DependencyCollector
Dependency.new("git", tags) Dependency.new("git", tags)
end end
def brewed_curl_dep_if_needed(tags) def curl_dep_if_needed(tags)
Dependency.new("curl", tags) Dependency.new("curl", tags)
end end
@ -148,7 +159,7 @@ class DependencyCollector
strategy = spec.download_strategy strategy = spec.download_strategy
if strategy <= HomebrewCurlDownloadStrategy if strategy <= HomebrewCurlDownloadStrategy
@deps << brewed_curl_dep_if_needed(tags) @deps << curl_dep_if_needed(tags)
parse_url_spec(spec.url, tags) parse_url_spec(spec.url, tags)
elsif strategy <= CurlDownloadStrategy elsif strategy <= CurlDownloadStrategy
parse_url_spec(spec.url, tags) parse_url_spec(spec.url, tags)

View File

@ -78,8 +78,8 @@ class DevelopmentTools
end end
sig { params(cc: String).returns(Version) } sig { params(cc: String).returns(Version) }
def non_apple_gcc_version(cc) def gcc_version(cc)
(@non_apple_gcc_version ||= {}).fetch(cc) do (@gcc_version ||= {}).fetch(cc) do
path = HOMEBREW_PREFIX/"opt/#{CompilerSelector.preferred_gcc}/bin"/cc path = HOMEBREW_PREFIX/"opt/#{CompilerSelector.preferred_gcc}/bin"/cc
path = locate(cc) unless path.exist? path = locate(cc) unless path.exist?
version = if path && version = if path &&
@ -88,14 +88,14 @@ class DevelopmentTools
else else
Version::NULL Version::NULL
end end
@non_apple_gcc_version[cc] = version @gcc_version[cc] = version
end end
end end
sig { void } sig { void }
def clear_version_cache def clear_version_cache
@clang_version = @clang_build_version = nil @clang_version = @clang_build_version = nil
@non_apple_gcc_version = {} @gcc_version = {}
end end
sig { returns(T::Boolean) } sig { returns(T::Boolean) }

View File

@ -4,8 +4,6 @@
class CompilerSelector class CompilerSelector
sig { returns(String) } sig { returns(String) }
def self.preferred_gcc def self.preferred_gcc
# gcc-5 is the lowest gcc version we support on Linux. OS::LINUX_PREFERRED_GCC_FORMULA
# gcc-5 is the default gcc in Ubuntu 16.04 (used for our CI)
"gcc@5"
end end
end end

View File

@ -435,16 +435,26 @@ class Formula
end end
# If this is a `@`-versioned formula. # If this is a `@`-versioned formula.
sig { returns(T::Boolean) }
def versioned_formula? def versioned_formula?
name.include?("@") name.include?("@")
end end
# Returns any `@`-versioned formulae for any formula (including versioned formulae). # Returns any `@`-versioned formulae names for any formula (including versioned formulae).
def versioned_formulae sig { returns(T::Array[String]) }
Pathname.glob(path.to_s.gsub(/(@[\d.]+)?\.rb$/, "@*.rb")).map do |versioned_path| def versioned_formulae_names
@versioned_formulae_names ||= Pathname.glob(path.to_s.gsub(/(@[\d.]+)?\.rb$/, "@*.rb")).map do |versioned_path|
next if versioned_path == path next if versioned_path == path
Formula[versioned_path.basename(".rb").to_s] versioned_path.basename(".rb").to_s
end.compact.sort
end
# Returns any `@`-versioned Formula objects for any Formula (including versioned formulae).
sig { returns(T::Array[Formula]) }
def versioned_formulae
@versioned_formulae ||= versioned_formulae_names.map do |name|
Formula[name]
rescue FormulaUnavailableError rescue FormulaUnavailableError
nil nil
end.compact.sort_by(&:version).reverse end.compact.sort_by(&:version).reverse

View File

@ -433,11 +433,12 @@ module Homebrew
def audit_glibc def audit_glibc
return unless @core_tap return unless @core_tap
return if formula.name != "glibc" return if formula.name != "glibc"
return if [OS::CI_GLIBC_VERSION, "2.27", "2.31", "2.35"].include?(formula.version.to_s) # Also allow LINUX_GLIBC_NEXT_CI_VERSION for when we're upgrading.
return if [OS::LINUX_GLIBC_CI_VERSION, OS::LINUX_GLIBC_NEXT_CI_VERSION].include?(formula.version.to_s)
problem "The glibc version must be #{OS::CI_GLIBC_VERSION}, as this is the version used by our CI on Linux. " \ problem "The glibc version must be #{OS::LINUX_GLIBC_CI_VERSION}, as needed by our CI on Linux. " \
"Glibc is for users who have a system Glibc with a lower version, " \ "The glibc formula is for users who have a system glibc with a lower version, " \
"which allows them to use our Linux bottles, which were compiled against system Glibc on CI." "which allows them to use our Linux bottles, which were compiled against system glibc on CI."
end end
ELASTICSEARCH_KIBANA_RELICENSED_VERSION = "7.11" ELASTICSEARCH_KIBANA_RELICENSED_VERSION = "7.11"

View File

@ -333,9 +333,9 @@ class GitHubPackages
os_version ||= "macOS #{bottle_tag.to_macos_version}" os_version ||= "macOS #{bottle_tag.to_macos_version}"
when "linux" when "linux"
os_version&.delete_suffix!(" LTS") os_version&.delete_suffix!(" LTS")
os_version ||= OS::CI_OS_VERSION os_version ||= OS::LINUX_CI_OS_VERSION
glibc_version = tab["built_on"]["glibc_version"].presence if tab["built_on"].present? glibc_version = tab["built_on"]["glibc_version"].presence if tab["built_on"].present?
glibc_version ||= OS::CI_GLIBC_VERSION glibc_version ||= OS::LINUX_GLIBC_CI_VERSION
cpu_variant = tab["oldest_cpu_family"] || Hardware::CPU::INTEL_64BIT_OLDEST_CPU.to_s cpu_variant = tab["oldest_cpu_family"] || Hardware::CPU::INTEL_64BIT_OLDEST_CPU.to_s
end end

View File

@ -373,7 +373,7 @@ class Keg
@bottle_dependencies ||= begin @bottle_dependencies ||= begin
formulae = [] formulae = []
gcc = Formulary.factory(CompilerSelector.preferred_gcc) gcc = Formulary.factory(CompilerSelector.preferred_gcc)
formulae << gcc if DevelopmentTools.non_apple_gcc_version("gcc") < gcc.version.to_i formulae << gcc if DevelopmentTools.gcc_version("gcc") < gcc.version.to_i
formulae formulae
end end
end end

View File

@ -45,8 +45,15 @@ module OS
::OS_VERSION = ENV.fetch("HOMEBREW_OS_VERSION").freeze ::OS_VERSION = ENV.fetch("HOMEBREW_OS_VERSION").freeze
CI_GLIBC_VERSION = "2.23" LINUX_CI_OS_VERSION = "Ubuntu 16.04"
CI_OS_VERSION = "Ubuntu 16.04" LINUX_GLIBC_CI_VERSION = "2.23"
LINUX_GCC_CI_VERSION = "5.0"
LINUX_PREFERRED_GCC_FORMULA = "gcc@5"
# Ubuntu 22.04 (see Linux-CI.md)
LINUX_GLIBC_NEXT_CI_VERSION = "2.35"
# LINUX_GCC_CI_VERSION = "11.0"
# LINUX_PREFERRED_GCC_FORMULA = "gcc@11"
if OS.mac? if OS.mac?
require "os/mac" require "os/mac"

View File

@ -18,7 +18,7 @@ describe CompilerSelector do
end end
before do before do
allow(versions).to receive(:non_apple_gcc_version) do |name| allow(versions).to receive(:gcc_version) do |name|
case name case name
when "gcc-7" then Version.create("7.1") when "gcc-7" then Version.create("7.1")
when "gcc-6" then Version.create("6.1") when "gcc-6" then Version.create("6.1")

View File

@ -149,7 +149,7 @@ describe Formula do
end end
end end
it "returns true by default" do it "returns array with versioned formulae" do
FileUtils.touch f.path FileUtils.touch f.path
FileUtils.touch f2.path FileUtils.touch f2.path
allow(Formulary).to receive(:load_formula_from_path).with(f2.name, f2.path).and_return(f2) allow(Formulary).to receive(:load_formula_from_path).with(f2.name, f2.path).and_return(f2)

View File

@ -2,22 +2,22 @@
We currently use Ubuntu 16.04 for bottling in `homebrew/core`. We currently use Ubuntu 16.04 for bottling in `homebrew/core`.
### Ubuntu versus other distributions ## Ubuntu vs. other Linux distributions
As of 2022, around 77% of our users are using Ubuntu. This is the reason why we have chosen this distribution for our base CI image. As of 2022, around 77% of our users are using Ubuntu. This is the reason why we have chosen this distribution for our base CI image.
We have been using Ubuntu for CI since version 14.04 with success. We have successfully used Ubuntu for CI since version 14.04.
The LTS versions have 5 year support. A new LTS version is released every 2 years. The Ubuntu LTS versions are supported for 5 years. A new LTS version is released every 2 years.
Our bottles are compatible with other distributions like Debian/CentOS, even when compiled on Ubuntu. Our bottles are compatible with other distributions like Debian/CentOS, even when compiled on Ubuntu.
### Past and next versions ## Past and next versions
We are currently moving our CI to Ubuntu 22.04. This work will probably be done before end of 2022. We are currently moving our CI to Ubuntu 22.04. This work will probably be done before end of 2022.
Moving from Ubuntu 16.04 to Ubuntu 22.04 (and thus skipping version 18.04 and 20.04) took more time than expected due to other more urgent issues we had to take care of. Moving from Ubuntu 16.04 to Ubuntu 22.04 (and thus skipping version 18.04 and 20.04) took longer than expected.
We plan to proceed with a more regular update from 2022 on. We aim to use the latest Ubuntu LTS version for our CI. We plan to proceed with regular updates from 2022 onwards. We aim to use the latest Ubuntu LTS version for our CI.
We will start using the latest Ubuntu LTS version for our CI 3 months after it's release. Ideally the migration to the newest version will be done within 12 months after the LTS release, depending of course on the maintainer bandwidth to carry out such a migration. We will start using the latest Ubuntu LTS version for our CI no earlier than 3 months after its release and, ideally, no more than 12 months after its release.
| Distribution | Glibc | GCC | Usage | | Distribution | Glibc | GCC | Usage |
|---|---|---|---| |---|---|---|---|
@ -26,10 +26,10 @@ We will start using the latest Ubuntu LTS version for our CI 3 months after it's
| Ubuntu 22.04 | 2.35 | 11 | From 2022 to 2024 | | Ubuntu 22.04 | 2.35 | 11 | From 2022 to 2024 |
| Ubuntu 24.04 | ? | ? | From 2024 to 2026 | | Ubuntu 24.04 | ? | ? | From 2024 to 2026 |
### Why always using the latests version? ## Why always use the latest version?
Homebrew is a rolling-release package manager. We try to ship the newest things as quickest as possible, on macOS and Linux. Homebrew is a rolling-release package manager. We try to ship the newest things as quickly as possible, on macOS and Linux.
When a formula needs a newer GCC because our host GCC in CI is too old, we need to make that formula depend on a newer Homebrew GCC. All C++ dependents of that formula immediately acquire a dependency on Homebrew GCC as well. While we have taken the steps to make sure this no longer holds up GCC updates, it still creates a maintenance headache. This problem is more likely for formula which are very actively maintained and try to use newer features of C++. We decided that we shouldn't be creating maintenance burdens for formulae which are doing the right thing by staying up to date. It makes a lot of sense for Homebrew maintainers to submit upstream fixes when formulae are not working with newer compilers. It makes a lot less sense for Homebrew maintainers to submit fixes because our host compiler is too old. When a formula needs a newer GCC because our host GCC in CI is too old, we needed to make that formula depend on a newer Homebrew GCC. All C++ dependents of that formula immediately acquire a dependency on Homebrew GCC as well. While we have taken the steps to make sure this no longer holds up GCC updates, it still creates a maintenance burden. This problem is more likely for formula which are very actively maintained and try to use newer features of C++. We decided that we shouldn't have a maintenance burden for formulae which are doing the right thing by staying up to date. It makes a lot of sense for Homebrew maintainers to submit upstream fixes when formulae are not working with newer compilers. It makes a lot less sense for Homebrew maintainers to submit fixes because our host compiler is too old.
Note that `glibc` will need to be installed for more users as their `glibc` version will often be too old: disk space is cheap and we have can handle this situation for our users. This situation will often arise when update to a new LTS version and adoption of the new Ubuntu is still low during the first months. For the same reasons as above: we prefer to stay on the bleeding edge and give our users a gentle nudge to think about updating their OS. Note that `glibc` will need to be installed for more users as their `glibc` version will often be too old: disk space is cheap and we have can handle this situation for our users. This situation will often arise when update to a new LTS version and adoption of the new Ubuntu is still low during the first months. For the same reasons as above: we prefer to stay on the bleeding edge and give our users a gentle nudge to think about updating their OS.