incubator-hcatalog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HCATALOG-373) ProgressReporter should work with both old and new MR API
Date Fri, 11 May 2012 23:43:52 GMT

    [ https://issues.apache.org/jira/browse/HCATALOG-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273727#comment-13273727
] 

jiraposter@reviews.apache.org commented on HCATALOG-373:
--------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4971/#review7816
-----------------------------------------------------------


Looks good to me. Just a concern with the change in the test output directory. If you agree
with any of my proposals, I can change that on commit. Will run e2e tests.


src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
<https://reviews.apache.org/r/4971/#comment17147>

    The warehouse directory is set in the ant test target. If you're doing this for runs in
your IDE then it'd be better to have this check if the sysproperty hive.metastore.warehouse.dir
is set to support both cases.
    
    It'll probably better if you put the dir in /tmp similar to ant the config "/tmp/hcat_junit_warehouse".
Or add ant into the name {user.home}/hive_build/...


- Francis


On 2012-05-05 01:23:50, Travis Crawford wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4971/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-05-05 01:23:50)
bq.  
bq.  
bq.  Review request for hcatalog and Francis Liu.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Update ProgressReporter to work with both old and new mapreduce API. Delay creating the
base record reader so we have a StatusReporter and can use counters.
bq.  
bq.  
bq.  This addresses bug HCATALOG-373.
bq.      https://issues.apache.org/jira/browse/HCATALOG-373
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/java/org/apache/hcatalog/mapreduce/HCatBaseInputFormat.java 268167e 
bq.    src/java/org/apache/hcatalog/mapreduce/HCatRecordReader.java 65f96f4 
bq.    src/java/org/apache/hcatalog/mapreduce/InternalUtil.java 1837081 
bq.    src/java/org/apache/hcatalog/mapreduce/ProgressReporter.java fb379cd 
bq.    src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java f3d07a0 
bq.  
bq.  Diff: https://reviews.apache.org/r/4971/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  "ant clean test" passes
bq.  
bq.  I can run pig+hcatalog queries using Elephant-Bird deprecated API wrappers, which is
why this issue originally came up.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Travis
bq.  
bq.


                
> ProgressReporter should work with both old and new MR API
> ---------------------------------------------------------
>
>                 Key: HCATALOG-373
>                 URL: https://issues.apache.org/jira/browse/HCATALOG-373
>             Project: HCatalog
>          Issue Type: Bug
>            Reporter: Travis Crawford
>            Assignee: Travis Crawford
>         Attachments: HCATALOG-373_progress_reporter.diff, HCATALOG-373_progress_reporter_2.diff,
HCATALOG-373_progress_reporter_3.diff, HCATALOG-373_progress_reporter_4.diff
>
>
> {{org.apache.hcatalog.mapreduce.ProgressReporter}} currently implements {{org.apache.hadoop.mapred.Reporter}}.
It should also extend {{org.apache.hadoop.mapreduce.StatusReporter}} so it works with code
expecting either an old or new API reporter.
> The use case is using a wrapper so a serde works with a new-API input format.
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/mapred/input/DeprecatedInputFormatWrapper.java#L163

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message