tajo-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi" <hyun...@apache.org>
Subject Re: Review Request 19821: TAJO-717: Improve file splitting for large number of splits
Date Fri, 04 Apr 2014 05:57:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19821/#review39516
-----------------------------------------------------------

Ship it!


+1

Your patch seem to make getSplit() more efficient by reducing the remote call to HDFS. The
part becomes much better. Ship it!

- Hyunsik Choi


On April 4, 2014, 2:07 p.m., Jinho Kim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19821/
> -----------------------------------------------------------
> 
> (Updated April 4, 2014, 2:07 p.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-717
>     https://issues.apache.org/jira/browse/TAJO-717
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> In currently, The storageManager invoke the getFileBlockStorageLocations() per input
path, it occurred too many rpc to the associated datanodes
> 
> 
> Diffs
> -----
> 
>   tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
b2adaa4a2623b49022f4dac716dbf5457e93b71d 
>   tajo-storage/pom.xml b9a162ad3bcfa46b42ff70a47972b991f5ed4283 
>   tajo-storage/src/main/java/org/apache/tajo/storage/AbstractStorageManager.java a7ed9817749787bfddcb719913ab03021aa19c6c

>   tajo-storage/src/main/java/org/apache/tajo/storage/fragment/FileFragment.java ea8bf9f16b93ac5d8f32bcbadd5e4359dac8ad9b

>   tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java bbb9df1e5cce58fc236a8f372f654ec78aa24b85

>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorageManager.java 083670a27545c95b56dd2bece8d1c0620593a6f8

> 
> Diff: https://reviews.apache.org/r/19821/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jinho Kim
> 
>


Mime
View raw message