hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher <ctubb...@apache.org>
Subject Re: i686 support
Date Mon, 20 Jun 2016 20:18:45 GMT
It looks like there might be many other maven-jar-plugin executions without
an <id> specified and without a classifier. In 2.4.1:

**/hadoop-mapreduce-client-app/pom.xml
**/hadoop-mapreduce-client-jobclient/pom.xml
**/hadoop-yarn-applications-distributedshell/pom.xml
**/hadoop-yarn-common/pom.xml
**/hadoop-yarn-server-resourcemanager/pom.xml
**/hadoop-yarn-server-tests/pom.xml
**/hadoop-yarn-server-web-proxy/pom.xml

And some with an extra execution without a separate classifier at all:

hadoop-project-dist/pom.xml

So, it looks like it affects more than just the yarn components (should I
file against YARN still, or a more general component?).
I haven't been able to compare with other versions to see how far this
problem persists.



On Mon, Jun 20, 2016 at 3:58 PM Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> I believe the intent there is for the Distributed Shell jar to be fully
> built and available for use during a test run, which requires submitting
> the jar through the YARN machinery as part of the application submission.
> Potential solutions would be the configuration you suggested to skip the
> default execution, or possibly move this test into the integration test
> phase, which is not something we're currently taking advantage of in the
> Hadoop build.
>
> Also, I have a small correction on my last email.  If this problem is
> limited to the YARN portion of the build (e.g. just
> hadoop-yarn-applications-distributedshell), then please file the JIRA
> against the YARN project.
>
> --Chris Nauroth
>
> From: Christopher <ctubbsii@apache.org>
> Date: Monday, June 20, 2016 at 12:48 PM
>
> To: Chris Nauroth <cnauroth@hortonworks.com>, "user@hadoop.apache.org" <
> user@hadoop.apache.org>
> Subject: Re: i686 support
>
> I'm still in information-gathering mode right now... hoping somebody can
> tell me what the *expected* behavior of this module's build is supposed to
> be, so I can see exactly where there's a discrepancy and what the right
> direction is. I can't file a JIRA saying it's not doing what it's supposed
> to do, if I don't know what it's supposed to do. All I know right now is
> that some portion of the build breaks if you upgrade the version of the
> maven-jar-plugin.
>
> I *think* the intent is simply to build the main jar earlier in the
> lifecycle (vs. creating a second jar). If somebody can confirm that's true,
> then I'll have more information to file the JIRA with.
>
> On Mon, Jun 20, 2016 at 3:33 PM Chris Nauroth <cnauroth@hortonworks.com>
> wrote:
>
>> Christopher, thank you for the follow-up information.  Please feel free
>> to file an Apache JIRA in the HADOOP project with your findings.
>>
>> --Chris Nauroth
>>
>> From: Christopher <ctubbsii@apache.org>
>> Date: Monday, June 20, 2016 at 12:26 PM
>>
>> To: Chris Nauroth <cnauroth@hortonworks.com>, "user@hadoop.apache.org" <
>> user@hadoop.apache.org>
>> Subject: Re: i686 support
>>
>> As it turns out, it looks like it's because the version of
>> maven-jar-plugin in Fedora was upgraded, and hadoop's build takes advantage
>> of the fact that older maven-jar-plugin versions don't notice clobbering
>> artifacts with altered ones. This is probably going to bite Hadoop (if not
>> already) when upgrading to ASF parent pom 18, which specifies the newest
>> maven-jar-plugin.
>>
>> The solution, which was suggested to me on the Fedora lists, is to skip
>> the default-jar execution of the maven-jar-plugin, so that only the
>> customized execution would run:
>>
>> <execution>
>>   <id>default-jar</id>
>>   <phase>skip</phase>
>> </execution>
>>
>> Unfortunately, maven-jar-plugin doesn't seem to provide a proper "skip"
>> flag (all plugins should!), so one has got to hack the execution by
>> instructing it to run on a non-existing phase.
>>
>> Alternatively, Hadoop should really override the "default-jar" execution
>> with its own configuration (which as far as I can tell, it just makes the
>> jar build earlier in the lifecycle; side note: this is the wrong way to
>> go... the "unit test" described in the comment, which requires the main
>> jar, should probably be an "integration test").
>>
>> Thoughts?
>>
>> (I'm not sure why the build worked for me fine in x86_64, vs. i686, but
>> it appears to be a fluke... maybe I had an older version of
>> maven-jar-plugin installed already?)
>>
>> On Fri, Jun 17, 2016 at 5:45 PM Christopher <ctubbsii@apache.org> wrote:
>>
>>> It's possible it's a problem with the Fedora-packaged Maven or the jar
>>> plugin. However, I was able to do a mock build (mock in Fedora uses
>>> containers) to build it fine on x86_64. I've been given a hint as to how to
>>> tell mock to use a different arch, but haven't yet had a chance to try it.
>>>
>>> Mainly, I just want to make sure it's supported upstream. If it's not,
>>> and never will be, then I can use ExcludeArch in the RPM spec. Otherwise,
>>> I've gotta try to figure out why it's broken.
>>>
>>> On Thu, Jun 16, 2016 at 4:31 PM Chris Nauroth <cnauroth@hortonworks.com>
>>> wrote:
>>>
>>>> Thank you for sharing that.
>>>>
>>>> I was expecting to see some kind of failure in building our C codebase,
>>>> where a 32-bit vs. 64-bit portability problem is likely to happen.
>>>> Instead, that's a failure in Maven's handling of a jar dependency.  I tried
>>>> building from the 2.4.0 source, and I couldn't reproduce the problem.
>>>>
>>>> Is it possible that this is a problem specific to the Maven version
>>>> running your build?  The BUILDING.txt file in the root of the source tree
>>>> has details on what tools and what versions are required for the build.
>>>>
>>>> --Chris Nauroth
>>>>
>>>> From: Christopher <ctubbsii@apache.org>
>>>> Date: Thursday, June 16, 2016 at 12:48 PM
>>>> To: Chris Nauroth <cnauroth@hortonworks.com>, "user@hadoop.apache.org"
>>>> <user@hadoop.apache.org>
>>>> Subject: Re: i686 support
>>>>
>>>> Well, the error I got was:
>>>>
>>>> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:3.0.1:jar
(default-jar) on project hadoop-yarn-applications-distributedshell: You have to use a classifier
to attach supplemental artifacts to the project instead of replacing them. -> [Help 1]
>>>>
>>>>
>>>> https://kojipkgs.fedoraproject.org//work/tasks/1712/14521712/build.log
>>>>
>>>> I'm not sure if this is a Fedora-specific problem (due to its
>>>> dependency convergence and the specific version of the maven-jar-plugin,
>>>> for example), or if it's an upstream problem. This was version 2.4.1. Right
>>>> now, I'm just trying to stabilize the currently packaged 2.4 version,
>>>> before I begin moving towards an upgrade path for users to migrate to a
>>>> newer version (2.6, probably). It's just that I'm a bit overwhelmed with
>>>> Hadoop's build and quantity of components.
>>>>
>>>> Not sure where to begin looking.
>>>>
>>>> On Thu, Jun 16, 2016 at 3:42 PM Chris Nauroth <cnauroth@hortonworks.com>
>>>> wrote:
>>>>
>>>>> Hello Christopher,
>>>>>
>>>>> While 64-bit is certainly the common case at this point, we also have
>>>>> produced 32-bit builds in the past.  I'm not aware of any conscious choice
>>>>> by the community to halt 32-bit support, so if that's not working, then
it
>>>>> might be a bug.  Do you have more details on the build failure?  That
might
>>>>> help us confirm if it's a bug, and if so, move to tracking it in an Apache
>>>>> JIRA.
>>>>>
>>>>> --Chris Nauroth
>>>>>
>>>>> From: Christopher <ctubbsii@apache.org>
>>>>> Date: Thursday, June 16, 2016 at 12:35 PM
>>>>> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>>>>> Subject: i686 support
>>>>>
>>>>> Hi,
>>>>>
>>>>> Is i686 a supported arch by the upstream Hadoop developers? I'm trying
>>>>> to package hadoop RPMs for Fedora (reviving somebody else's previous
>>>>> efforts), and can't get things to build on i686. Is this normal? Does
>>>>> upstream care about 32-bit users? What about release testing on i686?
>>>>>
>>>>> I normally wouldn't care about 32-bit, but Fedora packaging standards
>>>>> encourage me to get it to work for all core architectures supported by
>>>>> Fedora.
>>>>>
>>>>> Has anybody else done any 32-bit packaging that could help me
>>>>> troubleshoot? I don't even own 32-bit hardware. (Even better if you want
to
>>>>> help co-maintain Hadoop in Fedora.. that would be awesome).
>>>>>
>>>>>

Mime
View raw message