hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: hadoop 3 scripts & classpath setup
Date Sat, 26 Aug 2017 11:35:44 GMT

> On 25 Aug 2017, at 19:49, Allen Wittenauer <aw@effectivemachines.com> wrote:
> 
> 
>> On Aug 25, 2017, at 10:00 AM, Steve Loughran <stevel@hortonworks.com> wrote:
>> 
>> Catching up on this. Looks like I don't have a hadoop-aws profile, which explains
a lot, doesn't it.
> 
> 	Yes. This is exactly the type of failure I'd expect.
> 
>> How do those profiles get created/copied in?
> 
> 	Maven kludgery.
> 
> 	In a hadoop-tools sub-module pom.xml, you'll find an entry like this or similar:
> 
>      <plugin>
>        <groupId>org.apache.maven.plugins</groupId>
>        <artifactId>maven-dependency-plugin</artifactId>
>        <executions>
>          <execution>
>            <id>deplist</id>
>            <phase>compile</phase>
>            <goals>
>              <goal>list</goal>
>            </goals>
>            <configuration>
>              <!-- build a shellprofile -->
>              <outputFile>${project.basedir}/target/hadoop-tools-deps/${project.artifactId}.tools-optional.txt</outputFile>
>            </configuration>
>          </execution>
>        </executions>
>      </plugin>
> 
> 	The files generated by this entry get read by dev-support/bin/dist-tools-hooks-maker.
 That script is run as part of -Pdist in hadoop-dist.  The outputFile name determines what
kind of support hook it makes.  (There were a lot of bad decisions made in nomenclature here.
I take full responsibility for the confusion.  But it makes more sense when one views the
names from the perspective of the code in hadoop-functions.sh)
> 
> 	All/most of this hackery should probably get replaced by something smarter in the hadoop-maven-plugin.
 But for the most part, this does work though and makes the end user experience significantly
better.
> 
>> I know there's an explicit s3guard entry now.
>> 
>> hadoop-tools/hadoop-aws/src/main/shellprofile.d/hadoop-s3guard.sh
>> 
>> ..do you think the presence of that entry is causing problems (i.e stopping a hadoop-aws
profile being created?)
> 
> 	I can confirm that HADOOP-13345 doesn't get a shellprofile.d/hadoop-aws.sh created.
That's not good. I don't have time right now to dig deep, but a few things pop into my head:
> 
> * multiple org.apache.maven.plugins definitions in the pom.xml (do all of them get executed
or just the last one?)
> * dist-tools-hooks-maker may only allowed one of builtin or optional . may need to define
a 3rd type that does a smart version of both
> * -Pdist may only allow one shellprofile.d dir per module ?
> 
> 	If you want, file a jira and assign it to me.  I'll try and dig into it next week.

Let me look at it first; I'd want this dealt with before doing the S3Guard branch merge so
there's no regression in trunk

> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message