Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Wed, 17 May 2017 21:18:04 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13058894.1490362970000.234904.1495055884207@Atlassian.JIRA>
In-Reply-To: <JIRA.13058894.1490362970000@Atlassian.JIRA>
References: <JIRA.13058894.1490362970000@Atlassian.JIRA> <JIRA.13058894.1490362970150@jira-lw-us.apache.org>
Subject: [jira] [Commented] (DRILL-5379) Set Hdfs Block Size based on
 Parquet Block Size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 17 May 2017 21:18:10 -0000


    [ https://issues.apache.org/jira/browse/DRILL-5379?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1601=
4792#comment-16014792 ]=20

ASF GitHub Bot commented on DRILL-5379:
---------------------------------------

Github user ppadma commented on a diff in the pull request:

    https://github.com/apache/drill/pull/826#discussion_r117112378
 =20
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parq=
uet/ParquetRecordWriter.java ---
    @@ -380,14 +384,21 @@ public void endRecord() throws IOException {
    =20
           // since ParquetFileWriter will overwrite empty output file (app=
end is not supported)
           // we need to re-apply file permission
    -      parquetFileWriter =3D new ParquetFileWriter(conf, schema, path, =
ParquetFileWriter.Mode.OVERWRITE);
    +      if (useConfiguredBlockSize) {
    +        // Round up blockSize to multiple of 64K.
    +        long writeBlockSize =3D ((long) ceil((double)blockSize/BLOCKSI=
ZE_MULTIPLE)) * BLOCKSIZE_MULTIPLE;
    +        // Passing writeBlockSize creates files with this blockSize in=
stead of filesystem default blockSize.
    +        // Currently, this is supported only by filesystems included i=
n
    +        // BLOCK_FS_SCHEMES (ParquetFileWriter.java in parquet-mr), wh=
ich includes HDFS.
    +        // For other filesystems, it uses default blockSize configured=
 for the file system.
    +        parquetFileWriter =3D new ParquetFileWriter(conf, schema, path=
, ParquetFileWriter.Mode.OVERWRITE, writeBlockSize, 0);
    +      } else {
    +        parquetFileWriter =3D new ParquetFileWriter(conf, schema, path=
, ParquetFileWriter.Mode.OVERWRITE);
    --- End diff --
   =20
    No. It does not. If user had previously changed the parquet block size,=
 that will be used in checkBlockSizeReached as before. So, parquet rowGroup=
 size will be same as configured parquet blockSize (not default). This opti=
on only changes the behavior of how we write that parquet row group onto di=
sk i.e. single block or not.=20


> Set Hdfs Block Size based on Parquet Block Size
> -----------------------------------------------
>
>                 Key: DRILL-5379
>                 URL: https://issues.apache.org/jira/browse/DRILL-5379
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Parquet
>    Affects Versions: 1.9.0
>            Reporter: F M=C3=A9thot
>             Fix For: Future
>
>
> It seems there a way to force Drill to store CTAS generated parquet file =
as a single block when using HDFS. Java HDFS API allows to do that, files c=
ould be created with the Parquet block-size set in a session or system conf=
ig.
> Since it is ideal  to have single parquet file per hdfs block.
> Here is the HDFS API that allow to do that:
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/Fi=
leSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%=
20long)
> http://archive.cloudera.com/cdh4/cdh/4/hadoop/api/org/apache/hadoop/fs/Fi=
leSystem.html#create(org.apache.hadoop.fs.Path,%20boolean,%20int,%20short,%=
20long)
> Drill uses the hadoop ParquetFileWriter (https://github.com/Parquet/parqu=
et-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/ParquetFileWr=
iter.java).
> This is where the file creation occurs so it might be tricky.
> However, ParquetRecordWriter.java (https://github.com/apache/drill/blob/m=
aster/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Parq=
uetRecordWriter.java) in Drill creates the ParquetFileWriter with an hadoop=
 configuration object.
> something to explore: Could the block size be set as a property within th=
e Configuration object before passing it to ParquetFileWriter constructor?


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)