drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From François Méthot <fmetho...@gmail.com>
Subject Single Hdfs block per parquet file
Date Wed, 22 Mar 2017 18:54:56 GMT
Hi,

Is there a way to force Drill to store CTAS generated parquet file as a
single block when using HDFS? Java HDFS API allows to do that, files could
be created with the Parquet block-size.

We are using Drill on hdfs configured with block size of 128MB. Changing
this size is not an option at this point.

It would be ideal for us to have single parquet file per hdfs block, setting
store.parquet.block-size to 128MB would fix our issue but we end up with a
lot more files to deal with.

Thanks
Francois

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message