Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of cenyongh@gmail.com designates
 209.85.142.184 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=AoLjo/nZe621NM1T0PZfk5BDVxeEanMa+3tBYH1UGIh/79rpRwv6QQCiDbEBmEvQKR
         1L58uSfMROeh2VBlUdnvnamdXZbhen3qKPE7JY9hLdMXiS6qK0uPD3jGSYEzUDYgpPW4
         NGJzEMqAPAfgDT4a4Fb56O5x+RqmX60sMgnyk=
MIME-Version: 1.0
In-Reply-To: <9D634D5D-E49C-4BFA-993F-4778AB786D2D@indiana.edu>
References: <9D634D5D-E49C-4BFA-993F-4778AB786D2D@indiana.edu>
Date: Wed, 6 May 2009 09:06:22 +0800
Message-ID: <f6e1a3de0905051806v683e19a7g62d2238dfb20aed5@mail.gmail.com>
Subject: Re: multi-line records and file splits
From: Nick Cen <cenyongh@gmail.com>
To: core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e652f56af9fcd70469340024

--0016e652f56af9fcd70469340024
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I think your SDFInputFormat should implement the MultiFileInputFormat
instead of the TextInputFormat, which will not splid the file into chunk.

2009/5/6 Rajarshi Guha <rguha@indiana.edu>

> Hi, I have implemented a subclass of RecordReader to handle a plain text
> file format where a record is multi-line and of variable length.
> Schematically each record is of the form
>
> some_title
> foo
> bar
> $$$$
> another_title
> foo
> foo
> bar
> $$$$
>
> where $$$$ is the marker for the end of the record. My code is at
> http://blog.rguha.net/?p=293 and it seems to work fine on my input data.
>
> However, I realized that when I run the program, Hadoop will 'chunk' the
> input file. As a result, the SDFRecordReader might get a chunk of input
> text, such that the last record is actually incomplete (a missing $$$$). Is
> this correct?
>
> If so, how would the RecordReader implementation recover from this
> situation? Or is there a way to indicate to Hadoop that the input file
> should be chunked keeping in mind end of record delimiters?
>
> Thanks
>
> -------------------------------------------------------------------
> Rajarshi Guha  <rguha@indiana.edu>
> GPG Fingerprint: D070 5427 CC5B 7938 929C  DD13 66A1 922C 51E7 9E84
> -------------------------------------------------------------------
> Q:  What's polite and works for the phone company?
> A:  A deferential operator.
>
>
>


-- 
http://daily.appspot.com/food/

--0016e652f56af9fcd70469340024--