Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of billmcn@gmail.com designates
 209.85.214.48 as permitted sender)
MIME-Version: 1.0
Date: Mon, 9 Jan 2012 10:30:57 -0800
Message-ID: 
 <CAN9z+opXmMzjGLdj4rd5za61vqcOwnV10m0HP+LaiDi-G0TCgw@mail.gmail.com>
Subject: Adding a soft-linked archive file to the distributed cache doesn't
 work as advertised
From: "W.P. McNeill" <billmcn@gmail.com>
To: Hadoop Mailing List <common-user@hadoop.apache.org>
Cc: Shakthi Poornima <pshakthi@intelius.com>
Content-Type: multipart/alternative; boundary=000e0ce0266a7d004104b61c9902

--000e0ce0266a7d004104b61c9902
Content-Type: text/plain; charset=UTF-8

I am trying to add a zip file to the distributed cache and have it unzipped
on the task nodes with a softlink to the unzipped directory placed in the
working directory of my mapper process. I think I'm doing everything the
way the documentation tells me to, but it's not working.

On the client in the run() function while I'm creating the job I first call:

fs.copyFromLocalFile("gate-app.zip", "/tmp/gate-app.zip");

As expected, this copies the archive file gate-app.zip to the HDFS
directory /tmp.

Then I call

DistributedCache.addCacheArchive("/tmp/gate-app.zip#gate-app",
configuration);

I expect this to add "/tmp/gate-app.zip" to the distributed cache and put a
softlink to it called gate-app in the working directory of each task.
However, when I call job.waitForCompletion(), I see the following error:

Exception in thread "main" java.io.FileNotFoundException: File does not
exist: /tmp/gate-app.zip#gate-app.

It appears that the distributed cache mechanism is interpreting the entire
URI as the literal name of the file, instead of treating the fragment as
the name of the softlink.

As far as I can tell, I'm doing this correctly according to the API
documentation:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html
.

The full project in which I'm doing this is up on github:
https://github.com/wpm/Hadoop-GATE.

Can someone tell me what I'm doing wrong?

--000e0ce0266a7d004104b61c9902--