crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Lewi (JIRA)" <>
Subject [jira] [Created] (CRUNCH-369) Crunch doesn't use custom getSplits functions of FileInputFormat subclasses
Date Sun, 23 Mar 2014 03:33:42 GMT
Jeremy Lewi created CRUNCH-369:

             Summary: Crunch doesn't use custom getSplits functions of FileInputFormat subclasses
                 Key: CRUNCH-369
             Project: Crunch
          Issue Type: Bug
          Components: IO
    Affects Versions: 0.9.0
            Reporter: Jeremy Lewi
            Priority: Minor

Suppose I create a source for a custom InputFormat which is a subclass of FileInputFormat;

TableSource<LongWritable, FastQWritable> source = From.formattedFile(
     inputFile, FastQInputFormatNew.class, LongWritable.class,

where FastQInputFormat is a subclass of FileInputFormat.

This won't work as expected because by default CrunchInputFormat.getSplits will end up using
CrunchCombineFileInputFormat to split the file. This doesn't work because my custom FIleInputFormat
uses a custom file splitter.

I can work around this by explicitly disabling the combining: e.g

source.inputConf(RuntimeParameters.DISABLE_COMBINE_FILE, Boolean.TRUE.toString());

but this doesn't strike me as the best solution. If I tell Crunch to use a custom InputFormat
I shouldn't have to specify a second config option in order to tell Crunch to respect the
getSplits function in my custom InputFormat.

I think CrunchInputFormat.getSplits should check that the format class exactly matches FileInputFormat;
i.e. it isn't a subclass. For subclasses Crunch should use the getsplits function in the custom
InputFormat class. I think changing the check to the following might work

if (format.getClass().equals(FileInputFormat.class) &&
    !conf.getBoolean(RuntimeParameters.DISABLE_COMBINE_FILE, true) 

This message was sent by Atlassian JIRA

View raw message