impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [48/50] [abbrv] incubator-impala git commit: IMPALA-3453: S3: Uneven split sizes are generated for Parquet causing execution skew
Date Thu, 12 May 2016 22:10:23 GMT
IMPALA-3453: S3: Uneven split sizes are generated for Parquet causing execution skew

Previously the Parquet file format was considered by us as a
non-splittable file format. However, we have since done some work on
our parquet scanner that will assign row groups based on the split
that contains them. This allows for us to chop up a parquet file into
multiple splits and still have the file be scanned reliably.

This patch changes our perception of Parquet as a splittable file
format, which now allows synthesizeBlockMetadata() to split a parquet
file on S3 into multiple "blocks" instead of assigning one scan range
per file, so that there is an even distribution of scan ranges across
the cluster, hence minimizing skew greatly.

P.S: To control the size of scan ranges for splittable files on S3,
you can change the default "block" size for the S3A filesystem which
is governed by "fs.s3a.block.size". Its default value is 32MB.

Change-Id: Ib1518ad0c89ef35a3b0567c3902e85a41e34bc3d
Reviewed-by: Sailesh Mukil <>
Tested-by: Internal Jenkins


Branch: refs/heads/master
Commit: 02d3e93774d49b2170079bbe5172d326e03be012
Parents: 1c704f3
Author: Sailesh Mukil <>
Authored: Thu May 5 14:32:16 2016 -0700
Committer: Tim Armstrong <>
Committed: Thu May 12 14:18:04 2016 -0700

 fe/src/main/java/com/cloudera/impala/catalog/ | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fe/src/main/java/com/cloudera/impala/catalog/ b/fe/src/main/java/com/cloudera/impala/catalog/
index 3670aa5..7fa1041 100644
--- a/fe/src/main/java/com/cloudera/impala/catalog/
+++ b/fe/src/main/java/com/cloudera/impala/catalog/
@@ -225,9 +225,8 @@ public enum HdfsFileFormat {
       case RC_FILE:
       case SEQUENCE_FILE:
       case AVRO:
-        return true;
       case PARQUET:
-        return false;
+        return true;
         throw new RuntimeException("Unknown HdfsFormat: "
             + this + " - should never happen!");

View raw message