hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Oberman <ober...@civicscience.com>
Subject gz containing null chars?
Date Mon, 10 Jun 2013 17:23:21 GMT
I posted this to the pig mailing list, but it might be more related to
hadoop itself, I'm not sure.

Quick recap: I had a file of "\n" separated lines of JSON.  I decided to
compress it to save on storage costs.  After compression I got a different
answer for a pig query that basically == "count lines".

After a lot of digging, I found an input file that had a line that is a
huge block of null characters followed by a "\n".  I wrote scripts to
examine the file directly, and if I stop counting at the weird line, I get
the same count as what pig claims for that file.   If I count all lines
(e.g. don't fail at the corrupt line) I get the "uncompressed" count pig

I don't know how to debug hadoop/pig quite as well, though I'm trying now.
 But, my working theory is that some combination of pig/hadoop aborts
processing the gz stream on a null character (or something like that), but
keeps chugging on a non-gz stream.  Does that sound familiar or make sense
to anyone?


View raw message