Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 46C8E200B39 for ; Fri, 24 Jun 2016 18:01:17 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 45520160A2E; Fri, 24 Jun 2016 16:01:17 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 91F5A160A58 for ; Fri, 24 Jun 2016 18:01:16 +0200 (CEST) Received: (qmail 65564 invoked by uid 500); 24 Jun 2016 16:01:15 -0000 Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list dev@impala.incubator.apache.org Received: (qmail 65519 invoked by uid 99); 24 Jun 2016 16:01:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2016 16:01:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 215161A5DB4 for ; Fri, 24 Jun 2016 16:01:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 2wIMCDkZhhyq for ; Fri, 24 Jun 2016 16:01:11 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 712E25FB60 for ; Fri, 24 Jun 2016 16:01:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id u5OG194g026689; Fri, 24 Jun 2016 16:01:09 GMT Message-Id: <201606241601.u5OG194g026689@ip-10-146-233-104.ec2.internal> Date: Fri, 24 Jun 2016 16:01:09 +0000 From: "Tim Armstrong (Code Review)" To: anujphadke , impala-cr@cloudera.com, dev@impala.incubator.apache.org Reply-To: tarmstrong@cloudera.com X-Gerrit-MessageType: comment Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-3766=3A__Applying_LZ4_compression_on_buffers_before_spilling=0A?= X-Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488 X-Gerrit-ChangeURL: X-Gerrit-Commit: c3f748a410ff9ce3024d1149212214c45707bebf In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.2 archived-at: Fri, 24 Jun 2016 16:01:17 -0000 Tim Armstrong has posted comments on this change. Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling ...................................................................... Patch Set 1: It would be good to get some numbers on how this compares to the uncompressed version in terms of bytes written to disk and end-to-end performance, I think we should get an idea of the gains before doing the detailed review. * Total uncompressed bytes versus compressed bytes written - you could add an extra profile counter for this since we already have bytes_written_counter_. * End-to-end query time on a release build. * Disk space saved. This is a little tricky since we allocate 8mb file ranges, but don't write the whole 8mb. The files will be sparse so we will save disk space, but you need to do something like this to see it: http://extrabright.com/blog/2010/03/30/how-to-know-if-a-file-on-linux-is-sparse/ -- To view, visit http://gerrit.cloudera.org:8080/3478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: anujphadke Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke Gerrit-HasComments: No