Return-Path: Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: (qmail 80557 invoked from network); 6 Oct 2010 10:29:43 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Oct 2010 10:29:43 -0000 Received: (qmail 63460 invoked by uid 500); 6 Oct 2010 10:29:43 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 63329 invoked by uid 500); 6 Oct 2010 10:29:41 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 63321 invoked by uid 99); 6 Oct 2010 10:29:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 10:29:40 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 10:29:35 +0000 Received: by ewy10 with SMTP id 10so4075481ewy.35 for ; Wed, 06 Oct 2010 03:29:13 -0700 (PDT) Received: by 10.213.21.133 with SMTP id j5mr8627389ebb.95.1286360953201; Wed, 06 Oct 2010 03:29:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.31.195 with HTTP; Wed, 6 Oct 2010 03:28:43 -0700 (PDT) In-Reply-To: <1286359035.7134.36.camel@linux-elo4.site> References: <1285324881.20975.29.camel@expat> <1286359035.7134.36.camel@linux-elo4.site> From: Alejandro Abdelnur Date: Wed, 6 Oct 2010 18:28:43 +0800 Message-ID: Subject: Re: Too large class path for map reduce jobs To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0015174c4270ab2b440491f03fd8 --0015174c4270ab2b440491f03fd8 Content-Type: text/plain; charset=ISO-8859-1 1. Classloader business can be done right. Actually it could be done as spec-ed for servlet web-apps. 2. If the issue is strictly 'too large classpath', then a simpler solution would be to sof-link all JARs to the current directory and create the classpath with the JAR names only (no path). Note that the soft-linking business is already supported by the DistributedCache. So the changes would be mostly in the TT to create the JAR names only classpath before starting the child. Alejandro On Wed, Oct 6, 2010 at 5:57 PM, Henning Blohm wrote: > Hi Tom, > > that's exactly it. Thanks! I don't think that I can comment on the issues > in Jira so I will do it here. > > Tricking with class paths and deviating from the default class loading > delegation has never been anything but a short term relieve. Fixing things > by imposing a "better" order of stuff on the class path will not work when > people do actually use child loaders (as the parent win) - like we do. Also > it may easily lead to very confusing situations because the former part of > the class path is not complete and gets other stuff from a latter part etc. > etc.... no good. > > Child loaders are good for module separation but should not be used to > "hide" type visibiliy from the parent. Almost certainly leading to Class > Loader Contraint Violation - once you lose control (which is usually earlier > than expected). > > The suggestion to reduce the Job class path to the required minimum is > the most practical approach. There is some gray area there of course and it > will not be feasible to reach the absolute minimal set of types there - but > something reasonable, i.e. the hadoop core that suffices to run the job. > Certainly jetty & co are not required for job execution (btw. I "hacked" > 0.20.2 to remove anything in "server/" from the classpath before setting the > job class path). > > I would suggest to > > a) introduce some HADOOP_JOB_CLASSPATH var that, if set, is the > additional classpath, added to the "core" classpath (as described above). If > not set, for compatibility, preserve today's behavior. > b) not getting into custom child loaders for jobs as part of hadoop M/R. > It's non-trivial to get it right and feels to be beyond scope. > > I wouldn't mind helping btw. > > Thanks, > Henning > > > > On Tue, 2010-10-05 at 15:59 -0700, Tom White wrote: > > Hi Henning, > > I don't know if you've seenhttps://issues.apache.org/jira/browse/MAPREDUCE-1938 andhttps://issues.apache.org/jira/browse/MAPREDUCE-1700 which have > discussion about this issue. > > Cheers > Tom > > On Fri, Sep 24, 2010 at 3:41 AM, Henning Blohm wrote: > > Short update on the issue: > > > > I tried to find a way to separate class path configurations by modifying the > > scripts in HADOOP_HOME/bin but found that TaskRunner actually copies the > > class path setting from the parent process when starting a local task so > > that I do not see a way of having less on a job's classpath without > > modifying Hadoop. > > > > As that will present a real issue when running our jobs on Hadoop I would > > like to propose to change TaskRunner so that it sets a class path > > specifically for M/R tasks. That class path could be defined in the scipts > > (as for the other processes) using a particular environment variable (e.g. > > HADOOP_JOB_CLASSPATH). It could default to the current VM's class path, > > preserving today's behavior. > > > > Is it ok to enter this as an issue? > > > > Thanks, > > Henning > > > > > > Am Freitag, den 17.09.2010, 16:01 +0000 schrieb Allen Wittenauer: > > > > On Sep 17, 2010, at 4:56 AM, Henning Blohm wrote: > > > >> When running map reduce tasks in Hadoop I run into classpath issues. > >> Contrary to previous posts, my problem is not that I am missing classes on > >> the Task's class path (we have a perfect solution for that) but rather find > >> too many (e.g. ECJ classes or jetty). > > > > The fact that you mention: > > > >> The libs in HADOOP_HOME/lib seem to contain everything needed to run > >> anything in Hadoop which is, I assume, much more than is needed to run a map > >> reduce task. > > > > hints that your perfect solution is to throw all your custom stuff in lib. > > If so, that's a huge mistake. Use distributed cache instead. > > > > > --0015174c4270ab2b440491f03fd8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable 1. Classloader business can be done right. Actually it could be done as spe= c-ed for servlet web-apps.=A0

2. If the issue is strictl= y 'too large classpath', then a simpler solution would be to sof-li= nk all JARs to the current directory and create the classpath with the JAR = names only (no path). Note that the soft-linking business is already suppor= ted by the DistributedCache. So the changes would be mostly in the TT to cr= eate the JAR names only classpath before starting the child.

Alejandro

On Wed, Oct 6, 2010 at 5:57 PM, Henning Blohm <henning.blohm@zfabrik.de><= /span> wrote:
=20 =20
Hi Tom,

=A0 that's exactly it. Thanks! I don't think that I can comment on = the issues in Jira so I will do it here.

=A0 Tricking with class paths and deviating from the default class loading = delegation has never been anything but a short term relieve. Fixing things = by imposing a "better" order of stuff on the class path will not = work when people do actually use child loaders (as the parent win) - like w= e do. Also it may easily lead to very confusing situations because the form= er part of the class path is not complete and gets other stuff from a latte= r part etc. etc.... no good.

=A0 Child loaders are good for module separation but should not be used to = "hide" type visibiliy from the parent. Almost certainly leading t= o Class Loader Contraint Violation - once you lose control (which is usuall= y earlier than expected).

=A0 The suggestion to reduce the Job class path to the required minimum is = the most practical approach. There is some gray area there of course and it= will not be feasible to reach the absolute minimal set of types there - bu= t something reasonable, i.e. the hadoop core that suffices to run the job. = Certainly jetty & co are not required for job execution (btw. I "h= acked" 0.20.2 to remove anything in "server/" from the class= path before setting the job class path).

=A0 I would suggest to

=A0 a) introduce some HADOOP_JOB_CLASSPATH var that, if set, is the additio= nal classpath, added to the "core" classpath (as described above)= . If not set, for compatibility, preserve today's behavior.
=A0 b) not getting into custom child loaders for jobs as part of hadoop M/R= . It's non-trivial to get it right and feels to be beyond scope.

=A0 I wouldn't mind helping btw.

Thanks,
=A0 Henning



On Tue, 2010-10-05 at 15:59 -0700, Tom White wrote: 
Hi Henning,

I don't know if you've seen
https://issues.apache.org/jira/browse/MAPREDUCE-1938 and
https://issues.apache.org/jira/browse/MAPREDUCE-1700 which have
discussion about this issue.

Cheers
Tom

On Fri, Sep 24, 2010 at 3:41 AM, Henning Blohm <henning.blohm@zfabrik.de> wrot=
e:
> Short update on the issue:
>
> I tried to find a way to separate class path configurations by modifyi=
ng the
> scripts in HADOOP_HOME/bin but found that TaskRunner actually copies t=
he
> class path setting from the parent process when starting a local task =
so
> that I do not see a way of having less on a job's classpath withou=
t
> modifying Hadoop.
>
> As that will present a real issue when running our jobs on Hadoop I wo=
uld
> like to propose to change TaskRunner so that it sets a class path
> specifically for M/R tasks. That class path could be defined in the sc=
ipts
> (as for the other processes) using a particular environment variable (=
e.g.
> HADOOP_JOB_CLASSPATH). It could default to the current VM's class =
path,
> preserving today's behavior.
>
> Is it ok to enter this as an issue?
>
> Thanks,
> =A0 Henning
>
>
> Am Freitag, den 17.09.2010, 16:01 +0000 schrieb Allen Wittenauer:
>
> On Sep 17, 2010, at 4:56 AM, Henning Blohm wrote:
>
>> When running map reduce tasks in Hadoop I run into classpath issue=
s.
>> Contrary to previous posts, my problem is not that I am missing cl=
asses on
>> the Task's class path (we have a perfect solution for that) bu=
t rather find
>> too many (e.g. ECJ classes or jetty).
>
> The fact that you mention:
>
>> The libs in HADOOP_HOME/lib seem to contain everything needed to r=
un
>> anything in Hadoop which is, I assume, much more than is needed to=
 run a map
>> reduce task.
>
> hints that your perfect solution is to throw all your custom stuff in =
lib.
> If so, that's a huge mistake.  Use distributed cache instead.
>


--0015174c4270ab2b440491f03fd8--