From user-return-1270-archive-asf-public=cust-asf.ponee.io@kudu.apache.org Wed Feb 28 15:19:22 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 738E6180657 for ; Wed, 28 Feb 2018 15:19:21 +0100 (CET) Received: (qmail 15357 invoked by uid 500); 28 Feb 2018 14:19:20 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 15328 invoked by uid 99); 28 Feb 2018 14:19:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Feb 2018 14:19:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 18495C18A7 for ; Wed, 28 Feb 2018 14:19:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.199 X-Spam-Level: * X-Spam-Status: No, score=1.199 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=boristyukin.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id IHyH3Yq0TJjL for ; Wed, 28 Feb 2018 14:19:15 +0000 (UTC) Received: from mx36-out26.antispamcloud.com (mx36-out26.antispamcloud.com [209.126.121.74]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6B13A5F520 for ; Wed, 28 Feb 2018 14:19:15 +0000 (UTC) Received: from s2.fcomet.com ([99.198.101.250]) by mx61.antispamcloud.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1er2ZR-0008LA-0B for user@kudu.apache.org; Wed, 28 Feb 2018 15:19:07 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=boristyukin.com; s=default; h=Content-Type:To:Subject:Message-ID:Date:From: References:In-Reply-To:MIME-Version:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Y2iVIWG/ag//OgMiD8mRiTszVUxWVRmFPzdVyHjac5o=; b=lOfCP3G2a0GLfEK4XTsi5l17L C0aQNkyxUe0gbGLyHgHWiiFzefyrMywMTItF5Ppp/FwDwNqEGQRGOEwJBBVJ3TWnPkoiYgxkA9Ajs FtqIZBpgrxoiUW/Ih9ezY3ps2HdoZwLH7LLwOJFhVeyFBLvLybCI7NTWFA40x4TXxYB4k=; Received: from mail-it0-f50.google.com ([209.85.214.50]:39958) by s2.fcomet.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89_1) (envelope-from ) id 1er2Yi-003njB-K3 for user@kudu.apache.org; Wed, 28 Feb 2018 08:18:16 -0600 Received: by mail-it0-f50.google.com with SMTP id v186so3663892itc.5 for ; Wed, 28 Feb 2018 06:18:16 -0800 (PST) X-Gm-Message-State: APf1xPAG4g6HPwhIdEZ/azTdpSPwEOC4UG16IcQXVTl5yZEU79KaaHQi UGuw52CpcGxLbvh5GCrAV13sFjJCnyRnUX55Q7o= X-Google-Smtp-Source: AG47ELus7LlzZmT4rck2S0rSi+xVE++0MDeujhm4jXI0rdYZk/MLBNy4G+H3CXIDoI8h3xXLcPU6HBHK1/OoxnG57vU= X-Received: by 10.36.43.80 with SMTP id h77mr4700895ita.103.1519827496484; Wed, 28 Feb 2018 06:18:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.88.11 with HTTP; Wed, 28 Feb 2018 06:17:36 -0800 (PST) In-Reply-To: References: From: Boris Tyukin Date: Wed, 28 Feb 2018 09:17:36 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Impala Parquet to Kudu 1.5 - severe ingest performance degradation To: user@kudu.apache.org Content-Type: multipart/alternative; boundary="001a114741380c46bf0566466ac1" X-AuthUser: boris@boristyukin.com X-Originating-IP: 99.198.101.250 X-AntiSpamCloud-Domain: s2.fcomet.com X-AntiSpamCloud-Username: 99.198.101.250 Authentication-Results: antispamcloud.com; auth=pass smtp.auth=99.198.101.250@s2.fcomet.com X-AntiSpamCloud-Outgoing-Class: unsure X-AntiSpamCloud-Outgoing-Evidence: Combined (0.16) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5gfqn6utgtFo3n8i08iv1wp602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO2HFEYQDlNqPthodLGs7Ym7sT+bYMn6GaM87VISxooGyvLGbhKpjeqpkN003J3zku1pr jQPFk8m4tSTfORUp3ynEm+h0A2koB3qKN5bbUQlCvB1aHZYTYX2JaePLXtK5ho32ycKuYR+eCujo ZgTPFnZKenuD+fJRvZgsOGa/86DNKY4i2I4KjPTYrQ+5jVlmW+/cSY+GknWNdoEa0JVhYAcTmEFB zGJ4I3iI+cUBLpxHZqMsFXHkY4b0tMjYHlbEsQ6yCIj+j+sW7DnHSTh9wIB9qzbFFetd0V4Svjqc FPolF8VpymDAm51vSeaku+X2qYfWdHzoB2yW92yX6zvRr8FsMyewd83qhRHUN79OaH7QZM3NuHht 4ah3jKpVe9Q5dQwPf1ekvTsJu+noz87G5EthgB37cSRdX+S8eSGMVvuO++DuIQUs/5JJj4C/n4CI Li+gVoH4Y4mEYjqEfXsul64b0UPBAS8omC79k+qsNnQT+/po5FWskOUwhXZy7/fjwbxPEfEFvbTk +Lir7OrW4hW+spcE8FQWDXbSazBD3hvFP37mOUOnOTSuGuehGCC+F1vUrq1BBIjpFV+o6NlIzT+L MAF7QM+hm1BxMug426x0SGG/BQd7r2NVEaNhWos5pqKVl+ajqst6AzvfqvgNh/AccBIk1Sag4dKi qCrF8eZZVZC1Qaw0ZJEo8H7sex4RXic1CG8nCiB/V3E6bvwJj+gbMBZLEYWlTN+FJ0vSnADQAmuT GVQUI3zHVNdy4u/JHkvkSJLqhrljfvPvZqI3Fx6J+cv73CChOPjKA0/DVd83FIoahTu4NEG1Ryos iFcSDsdBzArngWMzAt8YZn+EfbautnwEZDsHYjWxVhSN2Caf X-Report-Abuse-To: spam@quarantine1.antispamcloud.com --001a114741380c46bf0566466ac1 Content-Type: text/plain; charset="UTF-8" first of all, thanks for a very quick response. It means a lot to know you guys stand behind Kudu and work with users like myself. In fact, we get a better support here than through the official channels :) That said, I needed to vent a bit :) Granted this should be probably directed at Impala devs but I think Kudu has been impacted the most. we ran a cluster health check command and saw some warning in there. But we proceeded with running insert statement as Todd suggested with hints and now we are back to our time we've used to get with Kudu 1.3 / Impala 2.8 it is a bit frustrating I must say that these changes impacted performance so dramatically. In my opinion, Kudu and CDH release notes must have stuff like that in BOLD RED colors so it does not catch users by surprise. And not everyone so organized like us - we knew exactly how much time it took with Kudu 1.3 /Impala 2.8. I've tracked down all the related JIRAs and I did not see a word about a dramatic performance degradation. I did see words like 'optimized' and 'improved'. Since it is part of the official CDH distro, I would expect a little bit more proactive warning. If you want to open a JIRA on this, would be happy to do it...We see this degradation all across our tables and as you can see from my example query, it is a really straight select from a table - no joins, no predicates and no complex calculations. Thanks again, Boris On Thu, Feb 22, 2018 at 2:44 PM, Todd Lipcon wrote: > In addition to what Hao suggests, I think it's worth noting that the > insert query plan created by Impala changed a bit over time. > > It sounds like you upgraded from Impala 2.8 (in CDH 5.11), which used a > very straightforward insert plan - each node separately inserted rows in > whatever order the rows were consumed. This plan worked well for smaller > inserts but could cause timeouts with larger workloads. > > In Impala 2.9, the plan was changed so that Impala performs some shuffling > and sorting before inserting into Kudu. This makes the Kudu insert pattern > more reliable and efficient, but could cause a degradation for some > workloads since Impala's sorts are single-threaded. > > Impala 2.10 (which I guess you are running) improved a bit over 2.9 in > ensuring that the sorts can be "partial" which resolved some of the > performance degradation, but it's possible your workload is still affected > negatively. > > To disable the new behavior you can use the insert hints 'noshuffle' > and/or 'noclustered', such as: > > upsert into my_table /* +noclustered,noshuffle */ select * from > my_other_table; > > > Hope that helps > -Todd > > On Thu, Feb 22, 2018 at 11:02 AM, Hao Hao wrote: > >> Did you happen to check the health of the cluster after the upgrade by 'kudu >> cluster ksck'? >> >> Best, >> Hao >> >> On Thu, Feb 22, 2018 at 6:31 AM, Boris Tyukin >> wrote: >> >>> Hello, >>> >>> we just upgraded our dev cluster from Kudu 1.3 to kudu 1.5.0-cdh5.13.1 >>> and noticed quite severe performance degradation. We did CTAS from Impala >>> parquet table which has not changed a bit since the upgrade (even the same >>> # of rows) to Kudu using the follow query below. >>> >>> It used to take 11-11.5 hours on Kudu 1.3 and now taking 50-55 hours. >>> >>> Of course Impala version was also bumped with CDH 5.13. >>> >>> Any clue why it takes so much time now? >>> >>> Table has 5.5B rows.. >>> >>> create TABLE kudutest_ts.clinical_event_nots >>> >>> PRIMARY KEY (clinical_event_id) >>> >>> PARTITION BY HASH(clinical_event_id) PARTITIONS 120 >>> >>> STORED AS KUDU >>> >>> AS >>> >>> SELECT >>> >>> clinical_event_id, >>> >>> encntr_id, >>> >>> person_id, >>> >>> encntr_financial_id, >>> >>> event_id, >>> >>> event_title_text, >>> >>> CAST(view_level as string) as view_level, >>> >>> order_id, >>> >>> catalog_cd, >>> >>> series_ref_nbr, >>> >>> accession_nbr, >>> >>> contributor_system_cd, >>> >>> reference_nbr, >>> >>> parent_event_id, >>> >>> event_reltn_cd, >>> >>> event_class_cd, >>> >>> event_cd, >>> >>> event_tag, >>> >>> CAST(event_end_dt_tm_os as BIGINT) as event_end_dt_tm_os, >>> >>> result_val, >>> >>> result_units_cd, >>> >>> result_time_units_cd, >>> >>> task_assay_cd, >>> >>> record_status_cd, >>> >>> result_status_cd, >>> >>> CAST(authentic_flag as STRING) authentic_flag, >>> >>> CAST(publish_flag as STRING) publish_flag, >>> >>> qc_review_cd, >>> >>> normalcy_cd, >>> >>> normalcy_method_cd, >>> >>> inquire_security_cd, >>> >>> resource_group_cd, >>> >>> resource_cd, >>> >>> CAST(subtable_bit_map as STRING) subtable_bit_map, >>> >>> collating_seq, >>> >>> verified_prsnl_id, >>> >>> performed_prsnl_id, >>> >>> updt_id, >>> >>> CAST(updt_task as STRING) updt_task, >>> >>> updt_cnt, >>> >>> CAST(updt_applctx as STRING) updt_applctx, >>> >>> normal_low, >>> >>> normal_high, >>> >>> critical_low, >>> >>> critical_high, >>> >>> CAST(event_tag_set_flag as STRING) event_tag_set_flag, >>> >>> CAST(note_importance_bit_map as STRING) note_importance_bit_map, >>> >>> CAST(order_action_sequence as STRING) order_action_sequence, >>> >>> entry_mode_cd, >>> >>> source_cd, >>> >>> clinical_seq, >>> >>> CAST(event_end_tz as STRING) event_end_tz, >>> >>> CAST(event_start_tz as STRING) event_start_tz, >>> >>> CAST(performed_tz as STRING) performed_tz, >>> >>> CAST(verified_tz as STRING) verified_tz, >>> >>> task_assay_version_nbr, >>> >>> modifier_long_text_id, >>> >>> ce_dynamic_label_id, >>> >>> CAST(nomen_string_flag as STRING) nomen_string_flag, >>> >>> src_event_id, >>> >>> CAST(last_utc_ts as BIGINT) last_utc_ts, >>> >>> device_free_txt, >>> >>> CAST(trait_bit_map as STRING) trait_bit_map, >>> >>> CAST(clu_subkey1_flag as STRING) clu_subkey1_flag, >>> >>> CAST(clinsig_updt_dt_tm as BIGINT) clinsig_updt_dt_tm, >>> >>> CAST(event_end_dt_tm as BIGINT) event_end_dt_tm, >>> >>> CAST(event_start_dt_tm as BIGINT) event_start_dt_tm, >>> >>> CAST(expiration_dt_tm as BIGINT) expiration_dt_tm, >>> >>> CAST(verified_dt_tm as BIGINT) verified_dt_tm, >>> >>> CAST(src_clinsig_updt_dt_tm as BIGINT) src_clinsig_updt_dt_tm, >>> >>> CAST(updt_dt_tm as BIGINT) updt_dt_tm, >>> >>> CAST(valid_from_dt_tm as BIGINT) valid_from_dt_tm, >>> >>> CAST(valid_until_dt_tm as BIGINT) valid_until_dt_tm, >>> >>> CAST(performed_dt_tm as BIGINT) performed_dt_tm, >>> >>> txn_id_text, >>> >>> CAST(ingest_dt_tm as BIGINT) ingest_dt_tm >>> >>> FROM v500.clinical_event >>> >> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera > --001a114741380c46bf0566466ac1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
first of all, thanks for a very quick response. It means a= lot to know you guys stand behind Kudu and work with users like myself. In= fact, we get a better support here than through the official channels :)
That said, I needed to vent a bit :) Granted this should = be probably directed at Impala devs but I think Kudu has been impacted the = most.

we ran a cluster health check command and saw = some warning in there. But we proceeded with running insert statement as To= dd suggested with hints and now we are back to our time we've used to g= et with Kudu 1.3 / Impala 2.8

it is a bit frustrat= ing I must say that these changes impacted performance so dramatically. In = my opinion, Kudu and CDH release notes must have stuff like that in BOLD RE= D colors so it does not catch users by surprise. And not everyone so organi= zed like us - we knew exactly how much time it took with Kudu 1.3 /Impala 2= .8.

I've tracked down all the related JIRAs an= d I did not see a word about a dramatic performance degradation. I did see = words like 'optimized' and 'improved'.

=
Since it is part of the official CDH distro, I would expect a little b= it more proactive warning.

If you want to open a J= IRA on this, would be happy to do it...We see this degradation all across o= ur tables and as you can see from my example query, it is a really straight= select from a table - no joins, no predicates and no complex calculations.=

Thanks again,
Boris

On Thu, Feb 22, 201= 8 at 2:44 PM, Todd Lipcon <todd@cloudera.com> wrote:
In addition to what Hao suggest= s, I think it's worth noting that the insert query plan created by Impa= la changed a bit over time.

It sounds like you upgraded = from Impala 2.8 (in CDH 5.11), which used a very straightforward insert pla= n - each node separately inserted rows in whatever order the rows were cons= umed. This plan worked well for smaller inserts but could cause timeouts wi= th larger workloads.

In Impala 2.9, the plan was c= hanged so that Impala performs some shuffling and sorting before inserting = into Kudu. This makes the Kudu insert pattern more reliable and efficient, = but could cause a degradation for some workloads since Impala's sorts a= re single-threaded.

Impala 2.10 (which I guess you= are running) improved a bit over 2.9 in ensuring that the sorts can be &qu= ot;partial" which resolved some of the performance degradation, but it= 's possible your workload is still affected negatively.

<= /div>
To disable the new behavior you can use the insert hints 'nos= huffle' and/or 'noclustered', such as:

upsert into my_table /* +noclustered,noshuffle */ select * from my_ot= her_table;


Hope that helps
-Todd
=
On Thu, Feb 22, 2018 at 11:02 AM, Hao Hao <hao.hao@cloudera.com> wrote:
Did you happen to check the health of the cluster aft= er the upgrade by 'kudu cluster=C2=A0<= /span>ksck'?

Best,
Hao

On Thu, Feb 22, = 2018 at 6:31 AM, Boris Tyukin <boris@boristyukin.com> wr= ote:
Hello,

we just upgraded our dev cluster from Kudu 1.3 to=C2=A0kudu 1.5= .0-cdh5.13.1 and noticed quite severe performance degradation. We did CTAS = from Impala parquet table which has not changed a bit since the upgrade (ev= en the same # of rows) to Kudu using the follow query below.

It used to take 11-11.5 hours on Kudu 1.3 an= d now taking 50-55 hours.

Of c= ourse Impala version was also bumped with CDH 5.13.
Any clue why it takes so much time now?
<= div>
Table has 5.5B rows..
<= br>

create TABLE kudutest_ts.clinic= al_event_nots

PRIMARY KEY (clinical_event_id)

PARTITION BY HASH(clinical_event_id) PARTITIONS 120<= span>

STORED AS KUDU

AS

SELECT

=C2=A0 clinical_event_id,

=C2=A0 encntr_id,

=C2=A0 person_id,

=C2=A0 encntr_financial_id,

=C2=A0 event_id,

=C2=A0 event_title_text,

=C2=A0 CAST(view_level as string) as view_level,

=C2=A0 order_id,

=C2=A0 catalog_cd,

=C2=A0 series_ref_nbr,

=C2=A0 accession_nbr,

=C2=A0 contributor_system_cd,

=C2=A0 reference_nbr,

=C2=A0 parent_event_id,

=C2=A0 event_reltn_cd,

=C2=A0 event_class_cd,

=C2=A0 event_cd,

=C2=A0 event_tag,

=C2=A0 CAST(event_end_dt_tm_os as BIGINT) as event_e= nd_dt_tm_os,

=C2=A0 result_val,

=C2=A0 result_units_cd,

=C2=A0 result_time_units_cd,

=C2=A0 task_assay_cd,

=C2=A0 record_status_cd,

=C2=A0 result_status_cd,

=C2=A0 CAST(authentic_flag as STRING) authentic_flag= ,

=C2=A0 CAST(publish_flag as STRING) publish_flag,

=C2=A0 qc_review_cd,

=C2=A0 normalcy_cd,

=C2=A0 normalcy_method_cd,

=C2=A0 inquire_security_cd,

=C2=A0 resource_group_cd,

=C2=A0 resource_cd,

=C2=A0 CAST(subtable_bit_map as STRING) subtable_bit= _map,

=C2=A0 collating_seq,

=C2=A0 verified_prsnl_id,

=C2=A0 performed_prsnl_id,

=C2=A0 updt_id,

=C2=A0 CAST(updt_task as STRING) updt_task,

=C2=A0 updt_cnt,

=C2=A0 CAST(updt_applctx as STRING) updt_applctx,

=C2=A0 normal_low,

=C2=A0 normal_high,

=C2=A0 critical_low,

=C2=A0 critical_high,

=C2=A0 CAST(event_tag_set_flag as STRING) event_tag_= set_flag,

=C2=A0 CAST(note_importance_bit_map as STRING) note_= importance_bit_map,

=C2=A0 CAST(order_action_sequence as STRING) order_a= ction_sequence,

=C2=A0 entry_mode_cd,

=C2=A0 source_cd,

=C2=A0 clinical_seq,

=C2=A0 CAST(event_end_tz as STRING) event_end_tz,

=C2=A0 CAST(event_start_tz as STRING) event_start_tz= ,

=C2=A0 CAST(performed_tz as STRING) performed_tz,

=C2=A0 CAST(verified_tz as STRING) verified_tz,

=C2=A0 task_assay_version_nbr,

=C2=A0 modifier_long_text_id,

=C2=A0 ce_dynamic_label_id,

=C2=A0 CAST(nomen_string_flag as STRING) nomen_strin= g_flag,

=C2=A0 src_event_id,

=C2=A0 CAST(last_utc_ts as BIGINT) last_utc_ts,

=C2=A0 device_free_txt,

=C2=A0 CAST(trait_bit_map as STRING) trait_bit_map,<= span>

=C2=A0 CAST(clu_subkey1_flag as STRING) clu_subkey1_= flag,

=C2=A0 CAST(clinsig_updt_dt_tm as BIGINT) clinsig_up= dt_dt_tm,

=C2=A0 CAST(event_end_dt_tm as BIGINT) event_end_dt_= tm,

=C2=A0 CAST(event_start_dt_tm as BIGINT) event_start= _dt_tm,

=C2=A0 CAST(expiration_dt_tm as BIGINT) expiration_d= t_tm,

=C2=A0 CAST(verified_dt_tm as BIGINT) verified_dt_tm= ,

=C2=A0 CAST(src_clinsig_updt_dt_tm as BIGINT) src_cl= insig_updt_dt_tm,

=C2=A0 CAST(updt_dt_tm as BIGINT) updt_dt_tm,<= /span>

=C2=A0 CAST(valid_from_dt_tm as BIGINT) valid_from_d= t_tm,

=C2=A0 CAST(valid_until_dt_tm as BIGINT) valid_until= _dt_tm,

=C2=A0 CAST(performed_dt_tm as BIGINT) performed_dt_= tm,

=C2=A0 txn_id_text,

=C2=A0 CAST(ingest_dt_tm as BIGINT) ingest_dt_tm

FROM v500.clinical_event





<= /div>--
Todd= Lipcon
Software Engineer, Cloudera

--001a114741380c46bf0566466ac1--