Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0572F1192E for ; Thu, 21 Aug 2014 08:39:05 +0000 (UTC) Received: (qmail 7180 invoked by uid 500); 21 Aug 2014 08:39:00 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 7051 invoked by uid 500); 21 Aug 2014 08:39:00 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 7039 invoked by uid 99); 21 Aug 2014 08:38:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Aug 2014 08:38:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rabmdu@gmail.com designates 209.85.220.179 as permitted sender) Received: from [209.85.220.179] (HELO mail-vc0-f179.google.com) (209.85.220.179) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Aug 2014 08:38:34 +0000 Received: by mail-vc0-f179.google.com with SMTP id hq11so10367910vcb.38 for ; Thu, 21 Aug 2014 01:38:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ZttWy1mN75RRZv8fZsIY5FxZ5DmKZyxGfNbxwO241LI=; b=XQMwKJbIA256nSWDNwBnSlD75AR3T3QsAazZ0L4i62nXWz2Q6go0IimWDiqsGaM5zB Eyd0tX9Eww8aKrr7NLH1qqd3xgUPnG7EPh9BcguCPtAHUR2NN1cIuXe2xJVXjAQoUjXx D4BOlCp7tnsw9pJ+k9W4BLkExNYpjCE89B07su2/vaU+q+krBObDvKmUkL25NurBweV4 n343mDbphEBL4Ng4f7LdbWZgM0d4F/8wPuQx5wUyl4Yf6IBLfYpZgN/awgHWUx6bgLIu WLbiG0J4d20sVObGwOLRbpuu1DntrJ5krazieKy2q9f+Ij/bpiVgO+3CfWr7AkgneCGJ wkTA== MIME-Version: 1.0 X-Received: by 10.220.105.201 with SMTP id u9mr41207419vco.11.1408610312823; Thu, 21 Aug 2014 01:38:32 -0700 (PDT) Received: by 10.220.173.2 with HTTP; Thu, 21 Aug 2014 01:38:32 -0700 (PDT) Received: by 10.220.173.2 with HTTP; Thu, 21 Aug 2014 01:38:32 -0700 (PDT) In-Reply-To: <83870647-9F12-47AB-9790-7FD3B1806EDF@gmail.com> References: <83870647-9F12-47AB-9790-7FD3B1806EDF@gmail.com> Date: Thu, 21 Aug 2014 14:08:32 +0530 Message-ID: Subject: Re: Hadoop InputFormat - Processing large number of small files From: rab ra To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b33dd1e52a98805011fa5e0 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33dd1e52a98805011fa5e0 Content-Type: text/plain; charset=UTF-8 Thanks for the link. If it is not required for CFinputformat to have contents of the files in the map process but only the filename, what changes need to be done in the code? rab. On 20 Aug 2014 22:59, "Felix Chern" wrote: > I wrote a post on how to use CombineInputFormat: > > http://www.idryman.org/blog/2013/09/22/process-small-files-on-hadoop-using-combinefileinputformat-1/ > In the RecordReader constructor, you can get the context of which file you > are reading in. > In my example, I created FileLineWritable to include the filename in the > mapper input key. > Then you can use the input key as: > > public static class TestMapper extends Mapper Text, IntWritable>{ private Text txt = new Text(); private IntWritable > count = new IntWritable(1); public void map (FileLineWritable key, Text > val, Context context) throws IOException, InterruptedException{ > StringTokenizer st = new StringTokenizer(val.toString()); while (st. > hasMoreTokens()){ txt.set(key.fileName + st.nextToken()); context.write( > txt, count); } } } > > > Cheers, > Felix > > > On Aug 20, 2014, at 8:19 AM, rab ra wrote: > > Thanks for the response. > > Yes, I know wholeFileInputFormat. But i am not sure filename comes to map > process either as key or value. But, I think this file format reads the > contents of the file. I wish to have a inputformat that just gives filename > or list of filenames. > > Also, files are very small. The wholeFileInputFormat spans one map process > per file and thus results huge number of map processes. I wish to span a > single map process per group of files. > > I think I need to tweak CombineFileInputFormat's recordreader() so that it > does not read the entire file but just filename. > > > regards > rab > > regards > Bala > > > On Wed, Aug 20, 2014 at 6:48 PM, Shahab Yunus > wrote: > >> Have you looked at the WholeFileInputFormat implementations? There are >> quite a few if search for them... >> >> >> http://hadoop-sandy.blogspot.com/2013/02/wholefileinputformat-in-java-hadoop.html >> >> https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java >> >> Regards, >> Shahab >> >> >> On Wed, Aug 20, 2014 at 1:46 AM, rab ra wrote: >> >>> Hello, >>> >>> I have a use case wherein i need to process huge set of files stored in >>> HDFS. Those files are non-splittable and they need to be processed as a >>> whole. Here, I have the following question for which I need answers to >>> proceed further in this. >>> >>> 1. I wish to schedule the map process in task tracker where data is >>> already available. How can I do it? Currently, I have a file that contains >>> list of filenames. Each map get one line of it via NLineInputFormat. The >>> map process then accesses the file via FSDataInputStream and work with it. >>> Is there a way to ensure this map process is running on the node where the >>> file is available?. >>> >>> 2. Since the files are not large and it can be called as 'small' files >>> by hadoop standard. Now, I came across CombineFileInputFormat that can >>> process more than one file in a single map process. What I need here is a >>> format that can process more than one files in a single map but does not >>> have to read the files, and either in key or value, it has the filenames. >>> In map process then, I can run a loop to process these files. Any help? >>> >>> 3. Any othe alternatives? >>> >>> >>> >>> regards >>> rab >>> >>> >> > > --047d7b33dd1e52a98805011fa5e0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Thanks for the link. If it is not required for CFinputformat to have con= tents of the files in the map process but only the filename, what changes n= eed to be done in the code?

rab.

On 20 Aug 2014 22:59, "Felix Chern" &l= t;idryman@gmail.com> wrote:
I wrote a post on how to use CombineInp= utFormat:
In the RecordReader constructor, you can get the context of which file= you are reading in.
In my example, I created FileLineWritable to= include the filename in the mapper input key.
Then you can use t= he input key as:



Cheers,
Felix
=

On Aug 20, 2014, at 8:19 AM, rab ra <rabmdu@gmail.com> wr= ote:

Thanks for the response.
Yes, I know wholeFileInputFormat. But i am not sure filenam= e comes to map process either as key or value. But, I think this file forma= t reads the contents of the file. I wish to have a inputformat that just gi= ves filename or list of filenames.

Also, files are very small. The wholeFileInputFormat sp= ans one map process per file and thus results huge number of map processes.= I wish to span a single map process per group of files.=C2=A0
I think I need to tweak CombineFileInputFormat's recordreade= r() so that it does not read the entire file but just filename.
<= br>

regards
rab

regards
Bala


<= div class=3D"gmail_quote">On Wed, Aug 20, 2014 at 6:48 PM, Shahab Yunus <shahab.yunus@gmail.com> wrote:
Have you looked at the Whol= eFileInputFormat implementations? There are quite a few if search for them.= ..


Regards,
Shahab


On Wed, Aug 20, 2014 at= 1:46 AM, rab ra <rabmdu@gmail.com> wrote:
Hello,
<= div>
I have a use case wherein i need to process huge set of files = stored in HDFS. Those files are non-splittable and they need to be processe= d as a whole. Here, I have the following question for which I need answers = to proceed further in this.

1. =C2=A0I wish to schedule the map process in task= tracker where data is already available. How can I do it? Currently, I hav= e a file that contains list of filenames. Each map get one line of it via N= LineInputFormat. The map process then accesses the file via FSDataInputStre= am and work with it. Is there a way to ensure this map process is running o= n the node where the file is available?.=C2=A0

2. =C2=A0Since the files are not large a= nd it can be called as 'small' files by hadoop standard. Now, I cam= e across CombineFileInputFormat that can process more than one file in a si= ngle map process. =C2=A0What I need here is a format that can process more = than one files in a single map but does not have to read the files, and eit= her in key or value, it has the filenames. In map process then, I can run a= loop to process these files. Any help?

3. Any othe alternatives?


regards
rab




--047d7b33dd1e52a98805011fa5e0--
public static class TestMapper <= span style=3D"font-weight:bold">extends Mapper<FileLineWritable, Text, Text, <= span>IntWritable>{
private Text txt =3D new Text();
private IntWrit= able count =3D = new IntWritable(1);
public void map (<= /span>FileLineWritable key, Text val, Context context) throws = IOException, Int= erruptedException{
StringTokenizer st =3D new <= span style=3D"color:rgb(153,0,0);font-weight:bold">StringTokenizer(val.toString());
while (st.hasMoreTokens()){
txt.set(ke= y.fileName + st.nextToken());
context.<= /span>write(txt, = count);
}
}
}