incubator-hcatalog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HCATALOG-353) Random Reads for random-accessible-tables in HCatalog
Date Fri, 06 Apr 2012 18:11:26 GMT

    [ https://issues.apache.org/jira/browse/HCATALOG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248558#comment-13248558
] 

jiraposter@reviews.apache.org commented on HCATALOG-353:
--------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4548/#review6745
-----------------------------------------------------------


Ill review the rest later.


http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccess.java
<https://reviews.apache.org/r/4548/#comment14758>

    .mapreduce. would be best since this depends on mapreduce interfaces.  If this is planned
to go into the mapreduce package then it should loose all references to hcat, like hcat's
serialization and deserialization methods.
    
    For now I feel its fine to keep it in hcatalog.data.randomaccess until it matures enough
to file a patch for MR.



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccess.java
<https://reviews.apache.org/r/4548/#comment14759>

    all caps denotes static, so single uppercase letter would be best.



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccess.java
<https://reviews.apache.org/r/4548/#comment14763>

    Why not store each alias separately in their own field in the config?  Seems costly if
you will have to serialize and deserialize the same giant object for each random access when
really you only need to do it for the one you care about



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessCommitter.java
<https://reviews.apache.org/r/4548/#comment14765>

    remove the abstract



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessCommitter.java
<https://reviews.apache.org/r/4548/#comment14768>

    Do you need both setupJob and prepareJob?  They seem to cover the same thing.  RandomAccess
is like MR's OutputFormat and RandomAccessCommiter is like MR's OutputCommiter, so if this
method is needed why not put it in RandomAccess?



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessOutputFormat.java
<https://reviews.apache.org/r/4548/#comment14773>

    Should only catch what is thrown by Class.forName and ReflrecionUtils.newInstance



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessible.java
<https://reviews.apache.org/r/4548/#comment14774>

    This should be the same package as everything else.  Shouldn't go into storagehandler



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessible.java
<https://reviews.apache.org/r/4548/#comment14775>

    I dont see this referenced here anyplace?



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/mapreduce/HCatRandomAccess.java
<https://reviews.apache.org/r/4548/#comment14776>

    HCatSchema should be @Nullable and if null fetch the table's schema for me.  An overloaded
method should be created to reflect this (without hcatschema).
    
    Need to file a jira for HcatOutput to do the same...



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/mapreduce/HCatRandomAccess.java
<https://reviews.apache.org/r/4548/#comment14777>

    So what happens when HCatStorageHandler goes away in favor of hive's?



http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/mapreduce/HCatRandomAccess.java
<https://reviews.apache.org/r/4548/#comment14779>

    Is this needed?  the only reference is a comment.


- David


On 2012-03-29 23:24:37, Mithun Radhakrishnan wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4548/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-03-29 23:24:37)
bq.  
bq.  
bq.  Review request for Francis Liu, Vandana Ayyalasomayajula and David Capwell.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Modified version of RandomAccess.
bq.  
bq.  Now the user is not exposed to HBase, and can code directly against HCat.
bq.  
bq.  Also, *SnapshotReader's filter-functionality has been moved to a separate class, to be
used from HBaseRandomAccess as well.
bq.  
bq.  
bq.  This addresses bug HCATALOG-353.
bq.      https://issues.apache.org/jira/browse/HCATALOG-353
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/common/HCatConstants.java
1306669 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccess.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessCommitter.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessOutputFormat.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/data/randomaccess/RandomAccessible.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/mapreduce/HCatRandomAccess.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/gen-java/org/apache/hcatalog/hbase/SnapshotFilter.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java
1306669 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseRandomAccess.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseRandomAccessCommitter.java
PRE-CREATION 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HbaseSnapshotRecordReader.java
1306669 
bq.    http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseRandomAccess.java
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4548/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Unit-tests, integration.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mithun
bq.  
bq.


                
> Random Reads for random-accessible-tables in HCatalog
> -----------------------------------------------------
>
>                 Key: HCATALOG-353
>                 URL: https://issues.apache.org/jira/browse/HCATALOG-353
>             Project: HCatalog
>          Issue Type: New Feature
>          Components: mapreduce
>            Reporter: Mithun Radhakrishnan
>            Assignee: Mithun Radhakrishnan
>             Fix For: 0.5
>
>         Attachments: HCATALOG-353.patch
>
>
> Implement Random-read for Random-accessible tables in HCatalog (E.g. backed by HBase.)
> Design-specs for generic random-access for MR are here:
> https://cwiki.apache.org/confluence/display/HCATALOG/Generic+MR+Random+Access+Framework
> Design-specs for HCat-specific implementation are here:
> https://cwiki.apache.org/confluence/display/HCATALOG/Random+Access+Framework

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message