hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Problem with large .lzo files
Date Mon, 15 Feb 2010 22:51:04 GMT
On Mon, Feb 15, 2010 at 2:48 PM, Steve Kuo <kuosenhao@gmail.com> wrote:
> You should be able to downcast the InputSplit to FileSplit, if you're
>> using the new API. From there you can get the start and length of the
>> split.
>> Cool, let me give it a shot.
>> Interesting. If you can somehow make a reproducible test case I'd be
>> happy to look into this.
>> This sounds great.  As the input file is 1G, let me do some work on my side
> to see if I can pinpoint it so as not have to transfer a 1G file around.

Yea, if you could generate a 1G file programatically that experiences
the problem, that would work too.


View raw message