incubator-hcatalog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francis Liu" <tof...@apache.org>
Subject Re: Review Request: [HCAT-367] Pig writes data but partition information isn't getting updated
Date Thu, 12 Apr 2012 02:42:53 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4702/#review6866
-----------------------------------------------------------

Ship it!


Looks good. Unit tests and e2e tests (except hbase, need to fix my env) passes.

- Francis


On 2012-04-12 00:50:11, Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4702/
> -----------------------------------------------------------
> 
> (Updated 2012-04-12 00:50:11)
> 
> 
> Review request for hcatalog.
> 
> 
> Summary
> -------
> 
> Cause:
> FosterStorageHandler sorts the partition keys and creates the output directories in that
order instead of the actual order. Setting file permissions in FileOutputCommitterContainer
was failing as the path constructed was in the order partition keys were defined and that
path did not exist.
> for(FieldSchema partKey : table.getPartitionKeys()){
>             partPath = constructPartialPartPath(partPath, partKey.getName().toLowerCase(),
partKVs);
>             fs.setPermission(partPath, perms);
>           
>         }
> 
> Because of the exception, the partitions were never registered with hcat server. When
running through PIG this exception was gobbled because of PIG-2568 (PigOutputCommitter hide
exception in commitJob) and so the map job was successful. The only stack trace that appears
in the job is HCAT-236 which is kind of misleading as it is just a logged message. 
> 
> So basically if the partition keys were not defined in the sorted order, it would fail.
>  
> Fix:
>    Changed FosterStorageHandler to construct the output path same way as FileOutputCommitterContainer.
> 
> 
> This addresses bug HCATALOG-367.
>     https://issues.apache.org/jira/browse/HCATALOG-367
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
1325041 
>   http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
1325041 
>   http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java
1325041 
> 
> Diff: https://reviews.apache.org/r/4702/diff
> 
> 
> Testing
> -------
> 
> Unit tests pass. Did an integration test issuing pig commands. e2e tests not run.
> 
> 
> Thanks,
> 
> Rohini
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message