Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=HzX4ElWVSa6nBu44RYcCfecOBXR3clPb4s9I2oBkowSqPwHTTslUhHo3jzWg1AsDK+EU1Nju9Do2RJfdysFjR2HKcM0NdQ3M20ZyxDENneUfvZy9XasmSiJhI4xwZSaJsbBejOWKdZ+e8QffdrzhZG+j2Qmxac8P1OjZWwa/YbU=;
Message-ID: <83669.53128.qm@web54202.mail.re2.yahoo.com>
Date: Sat, 23 Jan 2010 06:54:16 -0800 (PST)
From: Raymond Jennings III <raymondjiii@yahoo.com>
Subject: Re: Passing whole text file to a single map
To: common-user@hadoop.apache.org
In-Reply-To: <27286204.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Not sure if this solves your problem but I had a similar case where there was unique data at the beginning of the file and if that file was split between maps I would lose that for the 2nd and subsequent maps.  I was able to pull the file name from the conf and read the first two lines for every map.

--- On Sat, 1/23/10, stolikp <stolikp@o2.pl> wrote:

> From: stolikp <stolikp@o2.pl>
> Subject: Passing whole text file to a single map
> To: core-user@hadoop.apache.org
> Date: Saturday, January 23, 2010, 9:49 AM
> 
> I've got some text files in my input directory and I want
> to pass each single
> text file (whole file not just a line) to a map (one file
> per one map). How
> can I do this ? TextInputFormat splits text into lines and
> I do not want
> this to happen.
> I tried:
> http://hadoop.apache.org/common/docs/r0.20./streaming.html#How+do+I+process+files%2C+one+per+map%3F
> but it doesn't work for me, compiler doesn't know what
> NonSplitableTextInputFormat.class is.
> I'm using hadoop 0.20.1 
> -- 
> View this message in context: http://old.nabble.com/Passing-whole-text-file-to-a-single-map-tp27286204p27286204.html
> Sent from the Hadoop core-user mailing list archive at
> Nabble.com.
> 
>