hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Yongqiang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1801) HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice
Date Mon, 22 Nov 2010 18:12:14 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934536#action_12934536
] 

He Yongqiang commented on HIVE-1801:
------------------------------------

can you put "if (recordReader instanceof RCFileRecordReader)" at the same level with "else
if (inputFormatClass.getName().contains("RCFile")) {"?

     } else if (inputFormatClass.getName().contains("RCFile")) {
-      RCFile.Reader in = new RCFile.Reader(fs, path, job);
       blockPointer = true;
-      in.sync(fileSplit.getStart());
-      blockStart = in.getPosition();
-      in.close();
+
+      if (recordReader instanceof RCFileRecordReader) {
+        blockStart = ((RCFileRecordReader)recordReader).getStart();
+      } else {
+        RCFile.Reader in = new RCFile.Reader(fs, path, job);
+        in.sync(fileSplit.getStart());
+        blockStart = in.getPosition();
+        in.close();
+      }


> HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-1801
>                 URL: https://issues.apache.org/jira/browse/HIVE-1801
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Siying Dong
>            Assignee: Siying Dong
>         Attachments: HIVE-1801.1.patch
>
>
> HiveInputFormat or CombineHiveInputFormat RCFile.Reader.sync() twice. One in getReader()
and one in initIOContext(). We can avoid the latter one by read the sync() position of the
former one.
> We also sync() twice for SequenceFile but since SequenceFileReader is not a part of Hive
code, maybe we should be careful when depending on the implementation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message