hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-3234) getting the reporter in the recordwriter
Date Tue, 13 Nov 2012 21:16:13 GMT

     [ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Phabricator updated HIVE-3234:
------------------------------

    Attachment: HIVE-3234.D6699.1.patch

omalley requested code review of "HIVE-3234 [jira] getting the reporter in the recordwriter".
Reviewers: JIRA

  hive-3234. pass reporter down to outputfileformat

  We would like to generate some custom statistics and report back to map/reduce later wen
implement the
   FileSinkOperator.RecordWriter interface. However, the current interface design doesn't
allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter
interface so that it's close() method passes in a map reduce reporter object.

  For the same reason, please also extend the RecordReader interface too to include a reporter
object so that users can passes in custom map reduce  counters.

TEST PLAN
  Added a unit test that uses a custom input output format, which
  ensures the Progressable is a non-null Reporter.

REVISION DETAIL
  https://reviews.facebook.net/D6699

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
  ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13InputFormat.java
  ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
  ql/src/test/queries/clientpositive/custom_input_output_format.q
  ql/src/test/results/clientpositive/custom_input_output_format.q.out

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/15939/

To: JIRA, omalley

                
> getting the reporter in the recordwriter
> ----------------------------------------
>
>                 Key: HIVE-3234
>                 URL: https://issues.apache.org/jira/browse/HIVE-3234
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>    Affects Versions: 0.9.1
>         Environment: any
>            Reporter: Jimmy Hu
>            Assignee: Owen O'Malley
>              Labels: newbie
>             Fix For: 0.9.1
>
>         Attachments: HIVE-3234.D6699.1.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> We would like to generate some custom statistics and report back to map/reduce later
wen implement the 
>  FileSinkOperator.RecordWriter interface. However, the current interface design doesn't
allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter
interface so that it's close() method passes in a map reduce reporter object. 
> For the same reason, please also extend the RecordReader interface too to include a reporter
object so that users can passes in custom map reduce  counters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message