hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: i686 support
Date Mon, 20 Jun 2016 21:02:51 GMT
If it's a problem pattern that spans multiple sub-components, then please file it as a HADOOP
issue.  It might end up getting split into related issues assigned to each sub-project, but
HADOOP is a good starting point.

--Chris Nauroth

From: Christopher <ctubbsii@apache.org<mailto:ctubbsii@apache.org>>
Date: Monday, June 20, 2016 at 1:18 PM
To: Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>,
"user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: i686 support

It looks like there might be many other maven-jar-plugin executions without an <id>
specified and without a classifier. In 2.4.1:

**/hadoop-mapreduce-client-app/pom.xml
**/hadoop-mapreduce-client-jobclient/pom.xml
**/hadoop-yarn-applications-distributedshell/pom.xml
**/hadoop-yarn-common/pom.xml
**/hadoop-yarn-server-resourcemanager/pom.xml
**/hadoop-yarn-server-tests/pom.xml
**/hadoop-yarn-server-web-proxy/pom.xml

And some with an extra execution without a separate classifier at all:

hadoop-project-dist/pom.xml

So, it looks like it affects more than just the yarn components (should I file against YARN
still, or a more general component?).
I haven't been able to compare with other versions to see how far this problem persists.



On Mon, Jun 20, 2016 at 3:58 PM Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>
wrote:
I believe the intent there is for the Distributed Shell jar to be fully built and available
for use during a test run, which requires submitting the jar through the YARN machinery as
part of the application submission.  Potential solutions would be the configuration you suggested
to skip the default execution, or possibly move this test into the integration test phase,
which is not something we're currently taking advantage of in the Hadoop build.

Also, I have a small correction on my last email.  If this problem is limited to the YARN
portion of the build (e.g. just hadoop-yarn-applications-distributedshell), then please file
the JIRA against the YARN project.

--Chris Nauroth

From: Christopher <ctubbsii@apache.org<mailto:ctubbsii@apache.org>>
Date: Monday, June 20, 2016 at 12:48 PM

To: Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>,
"user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: i686 support

I'm still in information-gathering mode right now... hoping somebody can tell me what the
*expected* behavior of this module's build is supposed to be, so I can see exactly where there's
a discrepancy and what the right direction is. I can't file a JIRA saying it's not doing what
it's supposed to do, if I don't know what it's supposed to do. All I know right now is that
some portion of the build breaks if you upgrade the version of the maven-jar-plugin.

I *think* the intent is simply to build the main jar earlier in the lifecycle (vs. creating
a second jar). If somebody can confirm that's true, then I'll have more information to file
the JIRA with.

On Mon, Jun 20, 2016 at 3:33 PM Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>
wrote:
Christopher, thank you for the follow-up information.  Please feel free to file an Apache
JIRA in the HADOOP project with your findings.

--Chris Nauroth

From: Christopher <ctubbsii@apache.org<mailto:ctubbsii@apache.org>>
Date: Monday, June 20, 2016 at 12:26 PM

To: Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>,
"user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: i686 support

As it turns out, it looks like it's because the version of maven-jar-plugin in Fedora was
upgraded, and hadoop's build takes advantage of the fact that older maven-jar-plugin versions
don't notice clobbering artifacts with altered ones. This is probably going to bite Hadoop
(if not already) when upgrading to ASF parent pom 18, which specifies the newest maven-jar-plugin.

The solution, which was suggested to me on the Fedora lists, is to skip the default-jar execution
of the maven-jar-plugin, so that only the customized execution would run:

<execution>
  <id>default-jar</id>
  <phase>skip</phase>
</execution>

Unfortunately, maven-jar-plugin doesn't seem to provide a proper "skip" flag (all plugins
should!), so one has got to hack the execution by instructing it to run on a non-existing
phase.

Alternatively, Hadoop should really override the "default-jar" execution with its own configuration
(which as far as I can tell, it just makes the jar build earlier in the lifecycle; side note:
this is the wrong way to go... the "unit test" described in the comment, which requires the
main jar, should probably be an "integration test").

Thoughts?

(I'm not sure why the build worked for me fine in x86_64, vs. i686, but it appears to be a
fluke... maybe I had an older version of maven-jar-plugin installed already?)

On Fri, Jun 17, 2016 at 5:45 PM Christopher <ctubbsii@apache.org<mailto:ctubbsii@apache.org>>
wrote:
It's possible it's a problem with the Fedora-packaged Maven or the jar plugin. However, I
was able to do a mock build (mock in Fedora uses containers) to build it fine on x86_64. I've
been given a hint as to how to tell mock to use a different arch, but haven't yet had a chance
to try it.

Mainly, I just want to make sure it's supported upstream. If it's not, and never will be,
then I can use ExcludeArch in the RPM spec. Otherwise, I've gotta try to figure out why it's
broken.

On Thu, Jun 16, 2016 at 4:31 PM Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>
wrote:
Thank you for sharing that.

I was expecting to see some kind of failure in building our C codebase, where a 32-bit vs.
64-bit portability problem is likely to happen.  Instead, that's a failure in Maven's handling
of a jar dependency.  I tried building from the 2.4.0 source, and I couldn't reproduce the
problem.

Is it possible that this is a problem specific to the Maven version running your build?  The
BUILDING.txt file in the root of the source tree has details on what tools and what versions
are required for the build.

--Chris Nauroth

From: Christopher <ctubbsii@apache.org<mailto:ctubbsii@apache.org>>
Date: Thursday, June 16, 2016 at 12:48 PM
To: Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>,
"user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: i686 support

Well, the error I got was:


[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:3.0.1:jar (default-jar)
on project hadoop-yarn-applications-distributedshell: You have to use a classifier to attach
supplemental artifacts to the project instead of replacing them. -> [Help 1]

https://kojipkgs.fedoraproject.org//work/tasks/1712/14521712/build.log

I'm not sure if this is a Fedora-specific problem (due to its dependency convergence and the
specific version of the maven-jar-plugin, for example), or if it's an upstream problem. This
was version 2.4.1. Right now, I'm just trying to stabilize the currently packaged 2.4 version,
before I begin moving towards an upgrade path for users to migrate to a newer version (2.6,
probably). It's just that I'm a bit overwhelmed with Hadoop's build and quantity of components.

Not sure where to begin looking.

On Thu, Jun 16, 2016 at 3:42 PM Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>
wrote:
Hello Christopher,

While 64-bit is certainly the common case at this point, we also have produced 32-bit builds
in the past.  I'm not aware of any conscious choice by the community to halt 32-bit support,
so if that's not working, then it might be a bug.  Do you have more details on the build failure?
 That might help us confirm if it's a bug, and if so, move to tracking it in an Apache JIRA.

--Chris Nauroth

From: Christopher <ctubbsii@apache.org<mailto:ctubbsii@apache.org>>
Date: Thursday, June 16, 2016 at 12:35 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: i686 support

Hi,

Is i686 a supported arch by the upstream Hadoop developers? I'm trying to package hadoop RPMs
for Fedora (reviving somebody else's previous efforts), and can't get things to build on i686.
Is this normal? Does upstream care about 32-bit users? What about release testing on i686?

I normally wouldn't care about 32-bit, but Fedora packaging standards encourage me to get
it to work for all core architectures supported by Fedora.

Has anybody else done any 32-bit packaging that could help me troubleshoot? I don't even own
32-bit hardware. (Even better if you want to help co-maintain Hadoop in Fedora.. that would
be awesome).


Mime
View raw message