hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Bloniarz (JIRA)" <>
Subject [jira] [Created] (HIVE-3198) StorageHandler properties not passed to InputFormat (?)
Date Mon, 25 Jun 2012 19:13:44 GMT
Brian Bloniarz created HIVE-3198:

             Summary: StorageHandler properties not passed to InputFormat (?)
                 Key: HIVE-3198
             Project: Hive
          Issue Type: Bug
         Environment: trunk r1352973
            Reporter: Brian Bloniarz

I'm working on a custom StorageHandler implementation. I use configureTableJobProperties to
pass properties onto a serde & InputFormat, but it looks to me like the properties aren't
present inside the InputFormat.

I found the following code which looks like it's supposed to propagate JobProperties:
public class HiveInputFormat<K extends WritableComparable, V extends Writable>
  public RecordReader getRecordReader(InputSplit split, JobConf job,
      Reporter reporter) throws IOException {

    HiveInputSplit hsplit = (HiveInputSplit) split;
    boolean nonNative = false;
    PartitionDesc part = pathToPartitionInfo.get(hsplit.getPath().toString());
    if ((part != null) && (part.getTableDesc() != null)) {
      Utilities.copyTableJobPropertiesToConf(part.getTableDesc(), cloneJobConf);
      nonNative = part.getTableDesc().isNonNative();

In the debugger, I see that part==null so copyTableJobPropertiesToConf doesn't get called.
I see that for this table:
create external table test3 () STORED BY 'foo' location '/data/bar';
The InputSplit path is the *file* (i.e. "/data/bar/part-00000") but pathToPartitionInfo has
an entry for the *dir* (i.e "/data/bar").

I attached a patch which fixes the problem for me; it makes things explicit by passing along
the directory name inside the HiveInputSplit; this mean we don't have to figure out which
files are a part of which partition.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message