Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8EAC2200C85 for ; Tue, 30 May 2017 21:04:13 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8D2A5160BC9; Tue, 30 May 2017 19:04:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D4D36160BB1 for ; Tue, 30 May 2017 21:04:12 +0200 (CEST) Received: (qmail 861 invoked by uid 500); 30 May 2017 19:04:12 -0000 Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.incubator.apache.org Received: (qmail 846 invoked by uid 99); 30 May 2017 19:04:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 May 2017 19:04:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 66640CA764 for ; Tue, 30 May 2017 19:04:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id nnLDdFOf5uL4 for ; Tue, 30 May 2017 19:04:10 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 9743B5FC7A for ; Tue, 30 May 2017 19:04:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id v4UJ49KU026345; Tue, 30 May 2017 19:04:10 GMT Date: Tue, 30 May 2017 19:04:09 +0000 From: "Sailesh Mukil (Code Review)" To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org Message-ID: Reply-To: sailesh@cloudera.com X-Gerrit-MessageType: newchange Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5383=3A_Fix_PARQUET_FILE_SIZE_option_for_ADLS=0A?= X-Gerrit-Change-Id: I474a913b0ff9b2709f397702b58cb1c74251c25b X-Gerrit-ChangeURL: X-Gerrit-Commit: af23a97f7b5467c92c838794e756dc101ed5a0dc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 archived-at: Tue, 30 May 2017 19:04:13 -0000 Sailesh Mukil has uploaded a new change for review. http://gerrit.cloudera.org:8080/7018 Change subject: IMPALA-5383: Fix PARQUET_FILE_SIZE option for ADLS ...................................................................... IMPALA-5383: Fix PARQUET_FILE_SIZE option for ADLS PARQUET_FILE_SIZE query option doesn't work with ADLS because the AdlFileSystem doesn't have a notion of block sizes. And impala depends on the filesystem remembering the block size which is then used as the target parquet file size (this is done for Hdfs so that the parquet file size and block size match even if the parquet_file_size isn't a valid blocksize). We special case for Adls just like we do for S3 to bypass the FileSystem block size, and instead just use the requested PARQUET_FILE_SIZE as the output partitions block_size (and consequently the parquet file target size). Testing: Re-enabled test_insert_parquet_verify_size() for ADLS. Also fixed a miscellaneous bug with the ADLS client listing helper function. Change-Id: I474a913b0ff9b2709f397702b58cb1c74251c25b --- M be/src/exec/hdfs-table-sink.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M tests/query_test/test_insert_parquet.py M tests/util/adls_util.py 5 files changed, 15 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/7018/1 -- To view, visit http://gerrit.cloudera.org:8080/7018 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I474a913b0ff9b2709f397702b58cb1c74251c25b Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Sailesh Mukil