Shortened version of the conversation with the Fedora people (both mails @redhat.com), direct copy + pastes: Me: Is there a reason to have local copy of boost for building librados2 and not use the one which is inside the distribution? Them: I package in Fedora what Ceph releases, with little or no modification. I suggest you register your concerns with the Ceph developers directly. Me: But isn't that actually the problem? ... Shouldn't Fedora distribute not the latest Ceph, but the one which will actually work with that Fedora? Them: Then we would never release any recent/current version of Ceph on Fedora because AFAIK, for the foreseeable future, Ceph will use a newer version of boost, rocksdb, etc., than what is in Fedora. ... I'm not a Ceph developer and packaging it for Fedora is not my $dayjob. ... I (we) try to package reasonably current versions of Ceph for the community to use and experiment with, because that's what the community seems to want. Me: You are saying that you will rather have super new version of Ceph library which will not work and cause any software linking to it and other parts of the distribution (boost) to have UB. I still kind of do not really believe this... But anyway, as I mentioned earlier, thank you for your time ... -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Whole conversation, chronologically ordered (oldes to newest). Only changes performed were to cross out names and addresses: -------------------------------------------------------------------------------- From: Pavel Cernohorsky Subject: bugreport: librados2 package build breaks ODR by using different boost To: ANON_DEV_2@redhat.com, ANON_DEV_1@redhat.com Date: Thu, 3 Jan 2019 08:25:20 +0100 Dear Fedora contributors, I write to you as people mentioned at https://apps.fedoraproject.org/packages/librados2/changelog/ as people contributing to the librados2 package. I am not sure who is the actual package maintainer / person to contact, so I hope you will redirect me, if it is not you. I write to report a problem with the librados2 package in Fedora 27. I know that Fedora 27 reached EOL, this is why I do not do normal bug report. However the problem inside Fedora 27 may be of a systematic kind and affect future Fedora releases as well, so it is worth attention. The librados2 seems to use it’s internal copy of boost library during the build (1.66 in Fedora 27). That copy is different than boost inside the distribution (1.64 in Fedora 27): - md5 for /usr/src/debug/ceph-12.2.8-1.fc27.x86_64/build/boost/include/boost/system/error_code.hpp is f465639145c000ca2198325d1072ba92 - md5 for /usr/include/boost/system/error_code.hpp is 8d407caebd0f2a989ac1720038c596c6 Boost does not provide any binary API compatibility guarantees, so this results into breaking of one definition rule and undefined behavior which is very very hard to detect and diagnose (see http://pavel.cernohorsky.name/blog/2019/01/how_i_wrote_simple_cpp_lsda_parser.php for more details). Basically if you write a program which uses boost inside Fedora 27 and links to libceph-common in that very same distribution, you just created undefined behavior and the program will work or not work as if you roll a dice. Is there a reason to have local copy of boost for building librados2 and not use the one which is inside the distribution? Kind regards, Pavel Cernohorsky -------------------------------------------------------------------------------- Subject: Re: bugreport: librados2 package build breaks ODR by using different boost To: Pavel Cernohorsky , ANON_DEV_2@redhat.com From: "ANON_DEV_1" Date: Thu, 3 Jan 2019 09:28:23 -0500 My understanding is that this is a deliberate decision taken by the Ceph developers. There were features in boost-1.66 needed by Ceph that were not in boost-1.64. This is apparently quite common with Ceph and is a practice that continues in newer releases of Ceph. I package in Fedora what Ceph releases, with little or no modification. I suggest you register your concerns with the Ceph developers directly. E.g. by filing a bug report at https://tracker.ceph.com/issues/ Regards, -- ANON_DEV_1 -------------------------------------------------------------------------------- Subject: Re: bugreport: librados2 package build breaks ODR by using different boost To: ANON_DEV_1@redhat.com, ANON_DEV_2@redhat.com From: Pavel Cernohorsky Date: Thu, 3 Jan 2019 15:46:38 +0100 Hello ANON_DEV_1, Thanks for your answer. You say "I package in Fedora what Ceph releases, with little or no modification.". But isn't that actually the problem? That you just package some version which they release, but that thing will break any application using both boost and Ceph running on your system? Shouldn't Fedora distribute not the latest Ceph, but the one which will actually work with that Fedora? Kind regards, Pavel -------------------------------------------------------------------------------- Subject: Re: bugreport: librados2 package build breaks ODR by using different boost To: Pavel Cernohorsky From: "ANON_DEV_1" Cc: ANON_DEV_2@redhat.com Date: Thu, 3 Jan 2019 10:21:25 -0500 Then we would never release any recent/current version of Ceph on Fedora because AFAIK, for the foreseeable future, Ceph will use a newer version of boost, rocksdb, etc., than what is in Fedora. I'm sympathetic, but I don't have a good answer for you. :-/ I'm not a Ceph developer and packaging it for Fedora is not my $dayjob. I'm a NFS-Ganesha developer and my colleagues and I want/need to have a current version of Ceph to develop to without having to jump through a lot of hoops. I (we) try to package reasonably current versions of Ceph for the community to use and experiment with, because that's what the community seems to want. You are certainly welcome to build your own packages for Fedora of older versions of Ceph that meet your criteria, if that's more important than having a current and maintained version. The other choice is to send us patches to make Ceph build with the boost that comes with Fedora. I suspect that's a big job though. And I really believe you should address your concerns to the Ceph developers. They need to hear more of this from their users. They're the ones that make the decision that they need, e.g., a newer version of boost than what's shipping in _any_ Linux release. I kinda suspect that they use newer versions of, e.g., boost, because it has some feature that they would otherwise have to implement themselves, outside of boost, and it's "easier" to just jump to a new boost instead of writing a few (or a few hundred) more lines of code. Maybe that's too much of a generalization, I don't know. Regards, -- ANON_DEV_1 -------------------------------------------------------------------------------- Subject: Re: bugreport: librados2 package build breaks ODR by using different boost To: "ANON_DEV_1" Cc: ANON_DEV_2@redhat.com From: Pavel Cernohorsky Date: Thu, 3 Jan 2019 16:28:02 +0100 Thank you very much for your answer and time, ANON_DEV_1. I will try the Ceph developers then. Kind regards, Pavel -------------------------------------------------------------------------------- From: ANON_DEV_2 Date: Thu, 3 Jan 2019 16:36:15 +0100 Subject: Re: Re: bugreport: librados2 package build breaks ODR by using different boost To: Pavel Cernohorsky Cc: ANON_DEV_1 FWIW: This should not 'break' the applications using both boost and Ceph. IIRC, the boost bits should be compiled statically (the same as rocksdb bits/any other bundled bits) and not linked dynamically. In fact, a simple $ ldd /usr/bin/ceph-osd|grep boost should confirm that for you. There is an build-time option to use the system version of boost but that requires the system to use recent enough version of boost (which is rarely the case). Regards, ANON_DEV_2 -------------------------------------------------------------------------------- Subject: Re: bugreport: librados2 package build breaks ODR by using different boost To: ANON_DEV_2 Cc: ANON_DEV_1 From: Pavel Cernohorsky Date: Thu, 3 Jan 2019 16:54:09 +0100 Hello ANON_DEV_2, But this is exactly the problem! That you have things linked in, which will have different definition if somebody tries to use distribution-wide version of boost together with the Ceph libraries! readelf -aW /usr/lib64/ceph/libceph-common.so | c++filt | grep 'boost::asio::error::get_netdb_category()::instance' Dynamic linker will not really be happy about this breaking of one definition rule and you will get undefined behavior straight away (see my original article I referenced in the first mail). You are saying that you will rather have super new version of Ceph library which will not work and cause any software linking to it and other parts of the distribution (boost) to have UB. I still kind of do not really believe this... But anyway, as I mentioned earlier, thank you for your time, I will take this to the Ceph people, if they have anything to say about it... Kind regards, Pavel -------------------------------------------------------------------------------- From: ANON_DEV_2 Date: Thu, 3 Jan 2019 17:34:21 +0100 Subject: Re: bugreport: librados2 package build breaks ODR by using different boost To: Pavel Cernohorsky Cc: ANON_DEV_1 Hmm, ok, I gave it a read. I wish you luck! Personally, I never liked the behaviour upstream but there seemed to be very little I could do about it. Now that you can point the developers to an actual problem that it causes, they may be more open to your suggestions. Regards, ANON_DEV_2 --------------------------------------------------------------------------------