Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mbautin@gmail.com designates
 72.14.220.159 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=iudmAOX+ybryGmov0xHn+OQTi4py/rYSX/EFfaVzPYeCzQGZHDnq68gYaF35R9OEGoqVhKKVMfaHCp0SK0t4+t/blfTuuCbjub8IEDFsN5nmboS5j3KnWhlZqCGV++JC3lQBDjSE2R1E8gTHati1IM/XPeAwy2hw4Gwqolv+yU8=
Message-ID: <e76221610804181502l6600d1dbrc2c33760a85bdbef@mail.gmail.com>
Date: Fri, 18 Apr 2008 18:02:49 -0400
From: "Mikhail Bautin" <mbautin@gmail.com>
To: core-user@hadoop.apache.org
Subject: jar files on NFS instead of DistributedCache
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_6451_1049215.1208556169174"

------=_Part_6451_1049215.1208556169174
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hello,

We are using Hadoop here at Stony Brook University to power the
next-generation text analytics backend for www.textmap.com.  We also have an
NFS partition that is mounted on all machines of our 100-node cluster.  I
found it much more convenient to store manually created files (e.g.
configuration) on the NFS partition and just use them from my mappers and
reducers rather than copying them to HDFS every time I change them, which is
necessary when using DistributedCache.  Is there a way to do the same for
jars?

Specifically, I just need a way to alter the child JVM's classpath via
JobConf, without having the framework copy anything in and out of HDFS,
because all my files are already accessible from all nodes.  I see how to do
that by adding a couple of lines to TaskRunner's run() method, e.g.:

  classPath.append(sep);
  classPath.append(conf.get("mapred.additional.classpath"));

or something similar.  Is there already such a feature or should I just go
ahead and implement it?

Thanks,

Mikhail Bautin

------=_Part_6451_1049215.1208556169174--