Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201710060420.v964KKZ7014581@ip-10-146-233-104.ec2.internal>
Date: Fri, 6 Oct 2017 04:20:20 +0000
From: "Alex Behm (Code Review)" <gerrit@cloudera.org>
To: John Russell <jrussell@cloudera.com>, impala-cr@cloudera.com,
        reviews@impala.incubator.apache.org
CC: Silvius Rus <srus@cloudera.com>, Mostafa Mokhtar <mmokhtar@cloudera.com>,
        Greg Rahn <grahn@cloudera.com>
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_=5BDOCS=5D_Tighten_up_advice_about_first_COMPUTE_INCREMENTAL_STATS=0A?=
In-Reply-To: <gerrit.1504808141000.Ia53a6518ce5541e5c9a2cd896856ce042a599b03@gerrit.cloudera.org>
References: <gerrit.1504808141000.Ia53a6518ce5541e5c9a2cd896856ce042a599b03@gerrit.cloudera.org>
Reply-To: alex.behm@cloudera.com, jrussell@cloudera.com,
        impala-cr@cloudera.com, marcelk@gmail.com, srus@cloudera.com,
        mmokhtar@cloudera.com, reviews@impala.incubator.apache.org,
        grahn@cloudera.com
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.14.2
Content-Type: multipart/alternative; boundary="3hTa2t8mnQw="; charset=UTF-8
archived-at: Fri, 06 Oct 2017 04:20:29 -0000

--3hTa2t8mnQw=
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Alex Behm has posted comments on this change=2E ( http://gerrit=2Ecloudera=
=2Eorg:8080/7999 )

Change subject: [DOCS] Tighten up advice about first CO=
MPUTE INCREMENTAL STATS
=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=
=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=
=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=
=2E=2E=2E


Patch Set 1:

(4 comments)

http://gerrit=2Ecloudera=2Eorg:8080=
/#/c/7999/1/docs/shared/impala_common=2Exml
File docs/shared/impala_common=
=2Exml:

http://gerrit=2Ecloudera=2Eorg:8080/#/c/7999/1/docs/shared/impala_=
common=2Exml@1226
PS1, Line 1226:         and the statistics are computed a=
gain from the beginning=2E Therefore, expect a one-time
from scratch


http=
://gerrit=2Ecloudera=2Eorg:8080/#/c/7999/1/docs/shared/impala_common=2Exml@=
1241
PS1, Line 1241: -- by -1 under #Rows and false under Incremental stats=
=2E
I suggest you leave out the -1 under #Rows part since that may be confu=
sing=2E The reason is that DROP INCREMENTAL STATS will *not* modify the #Ro=
ws=2E

Here's how you can think about incremental stats:
COMPUTE INCREMENTA=
L STATS populates the "regular" stats such as the #rows and column NDVs tha=
t COMPUTE STATS also does, but in addition it also stores "incremental stat=
s" to speed up the next COMPUTE INCREMENTAL STATS=2E So the "incremental" p=
art is really this extra information which you can drop separately from the=
 "regular" stats=2E

One nice thing is that you can safely DROP INCREMENTAL=
 STATS everywhere to reduce the size of table metadata without impacting qu=
ery plans because the "regular" stats are preserved=2E


http://gerrit=2Ecl=
oudera=2Eorg:8080/#/c/7999/1/docs/topics/impala_partitioning=2Exml
File doc=
s/topics/impala_partitioning=2Exml:

http://gerrit=2Ecloudera=2Eorg:8080/#/=
c/7999/1/docs/topics/impala_partitioning=2Exml@611
PS1, Line 611:         B=
ecause the <codeph>COMPUTE STATS</codeph> statement can be resource-intensi=
ve to run frequently
This advice isn't prescriptive enough for my taste=2E =
We should state very clearly that you should use either COMPUTE STATS xor C=
OMPUTE INCREMENTAL STATS but never both=2E Switching during the lifetime of=
 a table is *not* recommended, but if you really must do so then we recomme=
nd you first drop all stats before the switch (using DROP STATS and DROP IN=
CREMENTAL STATS)=2E


http://gerrit=2Ecloudera=2Eorg:8080/#/c/7999/1/docs/t=
opics/impala_partitioning=2Exml@613
PS1, Line 613:         that is optimize=
d for processing partitioned tables=2E
I wouldn't say that incremental stat=
s is "optimized" for partitioned tables=2E Foremost, incremental stats allo=
w you to compute stats in a partition-by-partition fashion which might be a=
 better fit for a user's data ingestion pattern=2E However, we should be ve=
ry clear about the cost of incremental stats=2E Incremental stats need ~400=
bytes per column per partition in the table metadata (which gets disseminat=
ed and cached everywhere), so incremental stats it not a good fit for table=
s with a huge number of columns and partitions=2E If you have a partitioned=
 table and only a few of the partitions are "active" then you can compute i=
ncremental stats for new partitions coming in and drop incremental stats fo=
r those partitions "phased" out to limit your exposure to the metadata size=
 problems=2E

You can even state that the huge table metadata can crash the=
 catalog and/or impalads due to the Java 2GB array size limit=2E (We're wor=
king on fixing that)

Basically I want to be sure that users understand the=
 cost of incremental stats and the impact (crash) of when they go overboard=
 with incremental stats=2E There is no graceful degradation here=2E


-- 
=
To view, visit http://gerrit=2Ecloudera=2Eorg:8080/7999
To unsubscribe, vis=
it http://gerrit=2Ecloudera=2Eorg:8080/settings

Gerrit-Project: Impala-ASF=

Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia53a6=
518ce5541e5c9a2cd896856ce042a599b03
Gerrit-Change-Number: 7999
Gerrit-Patch=
Set: 1
Gerrit-Owner: John Russell <jrussell@cloudera=2Ecom>
Gerrit-Reviewer=
: Alex Behm <alex=2Ebehm@cloudera=2Ecom>
Gerrit-Reviewer: Greg Rahn <grahn@=
cloudera=2Ecom>
Gerrit-Reviewer: Mostafa Mokhtar <mmokhtar@cloudera=2Ecom>
=
Gerrit-Reviewer: Silvius Rus <srus@cloudera=2Ecom>
Gerrit-Comment-Date: Fri=
, 06 Oct 2017 04:20:20 +0000
Gerrit-HasComments: Yes

--3hTa2t8mnQw=--