incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Mahé <>
Subject Re: Packaging concerns
Date Mon, 22 Aug 2011 19:54:45 GMT
On 08/22/2011 10:21 AM, Eric Yang wrote:
> On Aug 22, 2011, at 9:02 AM, Roman Shaposhnik wrote:
>> On Sat, Aug 20, 2011 at 11:10 AM, Eric Yang <> wrote:
>>> Greetings big top developers,
>>> Bigtop has started it's own packaging customization build process
>>> using Linux distributions based packaging tools to fully customize
>>> Hadoop stack packages.
>> Just to be clear -- part of the Bigtop charter is to NOT patch the
>> Apache releases. We are also NOT packaging trunks (or otherwise
>> unreleased upstream bits) and for the duration of incubation we're
>> going to make sure that our packages are clearly labeled as coming
>> from an incubating Apache Bigtop project.
>>> In traditional GPL camp, meta package
>>> to build source and apply patches as part of rpm/deb package construction.
>>> The advantage is that you can apply hot fix to the open source related
>>> source to customize to fit Linux distributions.
>> I'm not sure I follow. Can you, please, elaborate?
> Redhat produces a kernel rpm package, which is stock kernel tar ball with thousands of
patches applied for back port and sustain features.  I thought you were going to patch Apache
software and certify them and release as bigtop distribution.  You said Bigtop is not going
to patch Apache release, hence, bigtop could easily reuse the project based .spec and control
file to be in sync with Apache projects.  If "no patch statement" is valid, then there is
no need to create bigtop's own version of package management for the projects right?

I believe your answer is in the thread titled: "Projects that bundle
packaging infrastructure" started on 08/12/2011 on this mailing list.

>>> In Apache, software are released as tar ball with md5 signature.
>> Yes. And those are source tarballs. My understanding of Apache
>> release process is that everything else (including a functional
>> binaries) is but a convenience artifact. Some of them go into
>> Maven repositories, some of them go into the same source
>> release tarballs, but there's no clear process defined for
>> releasing anything but a source tar ball.
> Apache are flexible in how the project release software to accommodate platform and language
barriers.  For Java projects, jar file md5 signature are important.  Apache maven jar files
are identical in maven repository as well as in the binary tar ball.  This is best practice
to ensure that bits downloaded through maven are really the version that the project released.
>>> Ideally, Apache released rpm/deb packages should be the same
>>> bits that is in the release tar ball.

I am not sure to follow. Bigtop only cares about the source artifacts
and will always rebuild all the projects. So the binary/convenience
artifacts and the resulting packages from bigtop use the same official
Apache release tarball, but resulting binary/convenience artifacts may
differ (for exemple see all the odd versionning checks done through who
build it at what time).

>> Is there such a thing as Apache released rpm/deb packages
>> for any project at all? Does Apache have infrastructure to support
>> those types of packaged releases? I'm very curious to find this out,
>> simply becuase it'll help Bigtop a lot if we can piggyback off of that
>> sort of infrastructure and existing efforts.
>>>  Apache HTTPD project stopped distributing RPM form after a couple short releases.
>> Ok, so you're saying that Apache is not a correct place to have
>> packages released?
>> I'm confused now.
> Apache httpd is not releasing RPM due to shortage in infrastructure and man power to
maintain a decoupled building process.  The previous attempt was using a decoupled build process
from the project, which is the wrong approach.  If there are donated servers to host yum/apt
repositories, it can easily be automated through Jenkins by building as part of the original

Why is it the wrong approach? This is what all GNU/Linux distributions
have been doing since the beginning.

>>> It seems bigtop has chosen to use traditional Redhat/Debian methodology of producing
>>> to fit Linux distributions.  It is a novel goal from a packaging purity perspective.
>> In fact this very subject has been discussed at our meetup last week
>> (I really wished either
>> you or Owen could have participated -- but oh well :-(). You're right
>> that Bigtop has
>> to decide whether to take the route of producing OS-friendly packages
>> or tarball-friendly
>> packages. It seem that the community is currently in favor of the
>> OS-friendly ones,
>> but if you have arguments in favor of the other style -- let us know.
>> Especially if
>> you have customer feedback that would point in that direction.
>> That said, the decision, as you can see, is not tied to the packaging
>> infrastructure
>> decisions made by various projects. I could very well imagine project A meeting
>> their customer requirements with one style of packaging and project B pursuing
>> a different style. That kind of diversity is fine for individual
>> projects. For Bigtop
>> we have to stick with one style consistently.
>>> However, you might want to pay close attention to license and potential pitfalls.
>> Can you be a bit more clear on what is it that you're referring to as
>> "license and potential pitfalls"?
> rpmbuild does jar file repack, you need to disable them otherwise, it will repackage
exist jar files.  This could consider the software was linked to GPL software.  Hence, it
could get you into trouble, if it is not handle properly.  The same applies to debugging symbol
strip for .so files.

I am not sure to follow how GNU/GPL licensed software comes into play.
We are only repackaging Apache software which by definition will not
include any GNU/GPL licensed code (correct me if I am wrong). So how can
we repack GNU/GPL licensed software?

>>> It may be more interesting to focus on testing the community produced packages
in MHO.
>> Where can I get these community produced packages you're referring to? What
>> projects are included?
> There were work done for Hadoop 0.20.204 and 0.23 (HADOOP-6255).  However, trunk version
is removed by mavenization.  Alejandro ask me to work on it again after mavenization work
is done.  HBase/pig rpm were committed a while back, ZooKeeper (ZOOKEEPER-999) package is
done and waiting for commit. HCatalog is also in progress (HCATALOG-63).  Hive is probably
the only one left that does not produce rpm/deb package.  The work are done recently, and
Hadoop is most likely to release rpm/deb packages.  It would be interesting to see the response
from Hadoop community to determine if yum repos on Apache make sense.
> regards,
> Eric

View raw message