Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201710100130.v9A1UXg2004116@ip-10-146-233-104.ec2.internal>
Date: Tue, 10 Oct 2017 01:30:33 +0000
From: "Impala Public Jenkins (Code Review)" <gerrit@cloudera.org>
To: Quanlong Huang <huangquanlong@gmail.com>, impala-cr@cloudera.com,
        reviews@impala.incubator.apache.org
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5448=3A_fix_invalid_number_of_splits_reported_in_Parquet_scan_node=0A?=
In-Reply-To: <gerrit.1506469606000.Iaacc2d775032f5707061e704f12e0a63cde695d1@gerrit.cloudera.org>
References: <gerrit.1506469606000.Iaacc2d775032f5707061e704f12e0a63cde695d1@gerrit.cloudera.org>
Reply-To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org,
        huangquanlong@gmail.com
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.14.2
Content-Type: multipart/alternative; boundary="DJm+2vtOg4Y="; charset=UTF-8
archived-at: Tue, 10 Oct 2017 01:30:44 -0000

--DJm+2vtOg4Y=
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Impala Public Jenkins has submitted this change and it was merged=2E ( http=
://gerrit=2Ecloudera=2Eorg:8080/8147 )

Change subject: IMPALA-5448: fix in=
valid number of splits reported in Parquet scan node
=2E=2E=2E=2E=2E=2E=2E=
=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=
=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=
=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E

IMPALA-5448: fix invalid number of=
 splits reported in Parquet scan node

Parquet splits with multi columns ar=
e marked as completed by using
HdfsScanNodeBase::RangeComplete()=2E It dupl=
icately counts the file types
as column codec types=2E Thus the number of p=
arquet splits are the real count
multiplies number of materialized columns=
=2E

Furthermore, according to the Parquet definition, it allows mixed comp=
ression
codecs on different columns=2E This's handled in this patch as well=
=2E A parquet file
using gzip and snappy compression codec will be reported=
 as:
	FileFormats: PARQUET/(GZIP,SNAPPY):1

This patch introduces a compres=
sion types set for the above cases=2E

Testing:
Add end-to-end tests handli=
ng parquet files with all columns compressed in
snappy, and handling parque=
t files with multi compression codec=2E

Change-Id: Iaacc2d775032f5707061e7=
04f12e0a63cde695d1
Reviewed-on: http://gerrit=2Ecloudera=2Eorg:8080/8147
Re=
viewed-by: Tim Armstrong <tarmstrong@cloudera=2Ecom>
Tested-by: Impala Publ=
ic Jenkins
---
M be/src/exec/hdfs-scan-node-base=2Ecc
M be/src/exec/hdfs-sc=
an-node-base=2Eh
A testdata/multi_compression_parquet_data/README
A testdat=
a/multi_compression_parquet_data/tinytable_0_gzip_snappy=2Eparq
A testdata/=
multi_compression_parquet_data/tinytable_1_snappy_gzip=2Eparq
A testdata/wo=
rkloads/functional-query/queries/QueryTest/hdfs_parquet_scan_node_profile=
=2Etest
M tests/query_test/test_scanners=2Epy
7 files changed, 132 insertio=
ns(+), 13 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, appro=
ved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit=2Ec=
loudera=2Eorg:8080/8147
To unsubscribe, visit http://gerrit=2Ecloudera=2Eor=
g:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Me=
ssageType: merged
Gerrit-Change-Id: Iaacc2d775032f5707061e704f12e0a63cde695=
d1
Gerrit-Change-Number: 8147
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Hua=
ng <huangquanlong@gmail=2Ecom>
Gerrit-Reviewer: Impala Public Jenkins
Gerri=
t-Reviewer: Mostafa Mokhtar <mmokhtar@cloudera=2Ecom>
Gerrit-Reviewer: Quan=
long Huang <huangquanlong@gmail=2Ecom>
Gerrit-Reviewer: Tim Armstrong <tarm=
strong@cloudera=2Ecom>

--DJm+2vtOg4Y=--