drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5379) Set Hdfs Block Size based on Parquet Block Size
Date Fri, 05 May 2017 17:23:04 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998610#comment-15998610
] 

ASF GitHub Bot commented on DRILL-5379:
---------------------------------------

GitHub user ppadma opened a pull request:

    https://github.com/apache/drill/pull/826

    DRILL-5379: Set Hdfs Block Size based on Parquet Block Size

    Provide an option to specify blocksize during file creation.
    This will help create parquet files with single block on HDFS, helping improve
    performance when we read those files.
    See DRILL-5379 for details.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ppadma/drill DRILL-5379

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/826.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #826
    
----
commit ae77a26aa950e401f2ca40488021431ebfde7156
Author: Padma Penumarthy <ppenumar97@yahoo.com>
Date:   2017-04-20T00:25:20Z

    DRILL-5379: Set Hdfs Block Size based on Parquet Block Size
    Provide an option to specify blocksize during file creation.
    This will help create parquet files with single block on HDFS, helping improve
    performance when we read those files.
    See DRILL-5379 for details.

----


> Set Hdfs Block Size based on Parquet Block Size
> -----------------------------------------------
>
>                 Key: DRILL-5379
>                 URL: https://issues.apache.org/jira/browse/DRILL-5379
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Parquet
>    Affects Versions: 1.9.0
>            Reporter: F M├ęthot
>             Fix For: Future
>
>
> It seems there a way to force Drill to store CTAS generated parquet file as a single
block when using HDFS. Java HDFS API allows to do that, files could be created with the Parquet
block-size set in a session or system config.
> Since it is ideal  to have single parquet file per hdfs block.
> Here is the HDFS API that allow to do that:
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%20long)
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/FileSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%20long)
> Drill uses the hadoop ParquetFileWriter (https://github.com/Parquet/parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/ParquetFileWriter.java).
> This is where the file creation occurs so it might be tricky.
> However, ParquetRecordWriter.java (https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java)
in Drill creates the ParquetFileWriter with an hadoop configuration object.
> something to explore: Could the block size be set as a property within the Configuration
object before passing it to ParquetFileWriter constructor?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message