Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAu13zH9z2vwBi+EatURtxQaQHnAZpGK3TzW9RdFp56j9Uv34w@mail.gmail.com>
References: 
 <CAAu13zH9z2vwBi+EatURtxQaQHnAZpGK3TzW9RdFp56j9Uv34w@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Fri, 7 Dec 2012 03:45:19 +0530
Message-ID: 
 <CAOcnVr15peJf6jY-TOa4Vynr=qiqkoS20B+HWhQbwG+J+c5AzQ@mail.gmail.com>
Subject: Re: DFS and the RecordReader
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

Not sure what you're talking about. RecordReaders, or for that matter,
any DFS InputStream, does not pull data locally before reading it.
Non-data-local reads are streamed over the network like how regular
data local reads are streamed over a local disk.

There is no such logic as the one you seek.

On Fri, Dec 7, 2012 at 3:07 AM, Jay Vyas <jayunit100@gmail.com> wrote:
> Hi guys:
>
> Where and how does a Hadoop's record reader decide wether or not it needs=
 to
> copy a file to local disk ?
>
> Clearly, since the InputSplit (which has meta data about file inputs) is =
the
> input to the RecordReader, the RecordReader would have to implement some
> kind of smart decision making ... Im looking for something like
>
> //Psuedocode
> if(! file.existsLocally())
>    copyFileToDisk(filegetPath());
>
> return new InputStream(file);
>
> I've looked here:
>
> http://grepcode.com/file/repo1.maven.org/maven2/org.jvnet.hudson.hadoop/h=
adoop-core/0.19.1-hudson-2/org/apache/hadoop/hdfs/DFSClient.java#DFSClient.=
create%28java.lang.String%2Corg.apache.hadoop.fs.permission.FsPermission%2C=
boolean%2Cshort%2Clong%2Corg.apache.hadoop.util.Progressable%2Cint%29
>
> but don't see anything.
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com


--=20
Harsh J