From user-return-1703-archive-asf-public=cust-asf.ponee.io@kudu.apache.org Fri Jul 19 17:26:50 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9FB29180665 for ; Fri, 19 Jul 2019 19:26:49 +0200 (CEST) Received: (qmail 84639 invoked by uid 500); 19 Jul 2019 17:26:48 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 84628 invoked by uid 99); 19 Jul 2019 17:26:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jul 2019 17:26:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id EE7AA1A4190 for ; Fri, 19 Jul 2019 17:26:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.803 X-Spam-Level: * X-Spam-Status: No, score=1.803 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, NUMERIC_HTTP_ADDR=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id CeKJcPrIwhF8 for ; Fri, 19 Jul 2019 17:26:45 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::831; helo=mail-qt1-x831.google.com; envelope-from=aserbin@cloudera.com; receiver= Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 250837DC5D for ; Fri, 19 Jul 2019 17:26:44 +0000 (UTC) Received: by mail-qt1-x831.google.com with SMTP id k10so31871928qtq.1 for ; Fri, 19 Jul 2019 10:26:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=MhKBLFWsge/Q4qOj3kw2BX9ZAPpDYkpmjs2CIa4+Dm4=; b=kOBnvZwHrWQo7wNTl5sCd3QS81kseLF4WZBH1I3xdqPeF7OGPNyhB2KDUkYy3jNLUb YOCZiSzQMe4h+xh2rhljFc41XEEyv59GWaXv20+LGao82sC5PMpwSEbvBoKbsim6063c /q1kcv5p+2Fgz6xFEkD0aJi8CEkiUXw3DhUNBHCwqWJ/7lEuMrhDU30Sn28iwuDiF8Hx pPokp7gLoZm8/UlaPhPipG4CwFF2LwYmIRo2bjJu4DniNu+ZBzrfIWNR6jjyGcUW9Nhg g9TDT5MFa2cQcSccAdwTq65rYxPz+2SSl4hATu9JHmbadhhkvUTZM+9b2d2fMVA3IQdn dDWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=MhKBLFWsge/Q4qOj3kw2BX9ZAPpDYkpmjs2CIa4+Dm4=; b=WUQlW3Qvs+mal/nntOKtBLz2h59Ajz9E5w+jcywqEnbMjboUm29VDeeOQvH5C6gEkq nh60xBCZqiuN+Bl4A9oZFXKJGif8R0khX5RpRYoha3Yumqx+/HbzSEWvaalMGx1+lKu5 MRu8O4hEYOaU/Y9u6WDHQZWZaYczl8NukY7uxp12P+33sRAHe/E2yeyfxHOfNT5knOEy eIR408TB0Spjj1DE18FEDwVT0qhsJjWPxxxwd1cAOpTGv2SJhRtgZHdmMRwyxR0cc5NU jTlNx0797/rbpy3+ZKXt0Qy9C72DC/LNTooKNxC7MXxzGs4H5rsE6kJp/qvC0GhC7QGh Geww== X-Gm-Message-State: APjAAAWy48mEa7phWBO5jqyZLZjGdFOsDqOSEHau8xlKKMMGhLf6Z2ck Hq8zvj1PK7FijG9OEiTavW1yIas9wR69N64eoAqqZK1o X-Google-Smtp-Source: APXvYqxbUhUsb6NBygqb2OPVGH6fDEGZUIIjOwMX/qoMoMEAdnNXkO6Qz6xMiS8jsrhWR4iWVGQd5+g8aet20ES/dv4= X-Received: by 2002:aed:3535:: with SMTP id a50mr39319262qte.237.1563557202506; Fri, 19 Jul 2019 10:26:42 -0700 (PDT) MIME-Version: 1.0 References: <7fff374e.110ba8c.16c059d951f.Coremail.lk_hadoop@163.com> In-Reply-To: From: Alexey Serbin Date: Fri, 19 Jul 2019 10:26:18 -0700 Message-ID: Subject: Re: impala with kudu write become very slow To: user@kudu.apache.org Cc: user@impala.apache.org Content-Type: multipart/alternative; boundary="000000000000a458c6058e0c07da" --000000000000a458c6058e0c07da Content-Type: text/plain; charset="UTF-8" Hi, It's hard to say what might be the problem without additional information. Could you clarify on the following questions: 1. What was the rate of write operations for the 270M rows you mentioned? Was that regular 50K rows/sec or something else? 2. Do you still observe the slowness or it's already gone? (if that was just a spike in ops rate, Kudu tablet servers might be rejecting writes due to the memory pressure when not being able to flush the data on disk fast enough). 3. What was special about those 270M rows? Maybe, the order of keys changed somehow? Kudu ingests data much faster if row keys come in kind of sequential order: so if the prior write batches had that property, but the recent 270M chunk didn't, it might be the case. 4. Was there any additional concurrent activity on the nodes where Kudu tablet servers run? Thanks, Alexey On Thu, Jul 18, 2019 at 7:48 PM Tim Armstrong wrote: > Also including the Kudu list in case someone there recognises the problem. > > On Thu, Jul 18, 2019 at 8:05 AM lk_hadoop wrote: > >> I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on 5 >> backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000 >> I0718 18:42:22.679605 12873 impala-internal-service.cc:44] >> ExecQueryFInstances(): query_id=2e4a3fbec0d7d721:2ec73c1c00000000 >> I0718 18:42:22.679620 12873 query-exec-mgr.cc:46] StartQueryFInstances() >> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 >> coord=realtimeanalysis-kudu-04-10-8-50-58:22000 >> I0718 18:42:22.679625 12873 query-state.cc:178] Buffer pool limit for >> 2e4a3fbec0d7d721:2ec73c1c00000000: 17179869184 >> I0718 18:42:22.679675 12873 initial-reservations.cc:60] Successfully >> claimed initial reservations (4.00 MB) for query >> 2e4a3fbec0d7d721:2ec73c1c00000000 >> I0718 18:42:22.679769 51332 query-state.cc:309] StartFInstances(): >> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 #instances=2 >> I0718 18:42:22.680196 51332 query-state.cc:322] descriptor table for >> query=2e4a3fbec0d7d721:2ec73c1c00000000 >> tuples: >> Tuple(id=2 size=567 slots=[Slot(id=52 type=INT col_path=[] offset=464 >> null=(offset=563 mask=20) slot_idx=29 field_idx=-1), Slot(id=53 type=STRING >> col_path=[] offset=0 null=(offset=560 mask=1) slot_idx=0 field_idx=-1), >> Slot(id=54 type=STRING col_path=[] offset=16 null=(offset=560 mask=2) >> slot_idx=1 field_idx=-1), Slot(id=55 type=STRING col_path=[] offset=32 >> null=(offset=560 mask=4) slot_idx=2 field_idx=-1), Slot(id=56 type=STRING >> col_path=[] offset=48 null=(offset=560 mask=8) slot_idx=3 field_idx=-1), >> Slot(id=57 type=STRING col_path=[] offset=64 null=(offset=560 mask=10) >> slot_idx=4 field_idx=-1), Slot(id=58 type=STRING col_path=[] offset=80 >> null=(offset=560 mask=20) slot_idx=5 field_idx=-1), Slot(id=59 type=STRING >> col_path=[] offset=96 null=(offset=560 mask=40) slot_idx=6 field_idx=-1), >> Slot(id=60 type=STRING col_path=[] offset=112 null=(offset=560 mask=80) >> slot_idx=7 field_idx=-1), Slot(id=61 type=STRING col_path=[] offset=128 >> null=(offset=561 mask=1) slot_idx=8 field_idx=-1), Slot(id=62 type=STRING >> col_path=[] offset=144 null=(offset=561 mask=2) slot_idx=9 field_idx=-1), >> Slot(id=63 type=STRING col_path=[] offset=160 null=(offset=561 mask=4) >> slot_idx=10 field_idx=-1), Slot(id=64 type=INT col_path=[] offset=468 >> null=(offset=563 mask=40) slot_idx=30 field_idx=-1), Slot(id=65 type=INT >> col_path=[] offset=472 null=(offset=563 mask=80) slot_idx=31 field_idx=-1), >> Slot(id=66 type=INT col_path=[] offset=476 null=(offset=564 mask=1) >> slot_idx=32 field_idx=-1), Slot(id=67 type=INT col_path=[] offset=480 >> null=(offset=564 mask=2) slot_idx=33 field_idx=-1), Slot(id=68 type=STRING >> col_path=[] offset=176 null=(offset=561 mask=8) slot_idx=11 field_idx=-1), >> Slot(id=69 type=STRING col_path=[] offset=192 null=(offset=561 mask=10) >> slot_idx=12 field_idx=-1), Slot(id=70 type=STRING col_path=[] offset=208 >> null=(offset=561 mask=20) slot_idx=13 field_idx=-1), Slot(id=71 type=STRING >> col_path=[] offset=224 null=(offset=561 mask=40) slot_idx=14 field_idx=-1), >> Slot(id=72 type=STRING col_path=[] offset=240 null=(offset=561 mask=80) >> slot_idx=15 field_idx=-1), Slot(id=73 type=STRING col_path=[] offset=256 >> null=(offset=562 mask=1) slot_idx=16 field_idx=-1), Slot(id=74 type=INT >> col_path=[] offset=484 null=(offset=564 mask=4) slot_idx=34 field_idx=-1), >> Slot(id=75 type=INT col_path=[] offset=488 null=(offset=564 mask=8) >> slot_idx=35 field_idx=-1), Slot(id=76 type=INT col_path=[] offset=492 >> null=(offset=564 mask=10) slot_idx=36 field_idx=-1), Slot(id=77 type=INT >> col_path=[] offset=496 null=(offset=564 mask=20) slot_idx=37 field_idx=-1), >> Slot(id=78 type=INT col_path=[] offset=500 null=(offset=564 mask=40) >> slot_idx=38 field_idx=-1), Slot(id=79 type=INT col_path=[] offset=504 >> null=(offset=564 mask=80) slot_idx=39 field_idx=-1), Slot(id=80 type=INT >> col_path=[] offset=508 null=(offset=565 mask=1) slot_idx=40 field_idx=-1), >> Slot(id=81 type=STRING col_path=[] offset=272 null=(offset=562 mask=2) >> slot_idx=17 field_idx=-1), Slot(id=82 type=STRING col_path=[] offset=288 >> null=(offset=562 mask=4) slot_idx=18 field_idx=-1), Slot(id=83 type=INT >> col_path=[] offset=512 null=(offset=565 mask=2) slot_idx=41 field_idx=-1), >> Slot(id=84 type=STRING col_path=[] offset=304 null=(offset=562 mask=8) >> slot_idx=19 field_idx=-1), Slot(id=85 type=STRING col_path=[] offset=320 >> null=(offset=562 mask=10) slot_idx=20 field_idx=-1), Slot(id=86 type=STRING >> col_path=[] offset=336 null=(offset=562 mask=20) slot_idx=21 field_idx=-1), >> Slot(id=87 type=STRING col_path=[] offset=352 null=(offset=562 mask=40) >> slot_idx=22 field_idx=-1), Slot(id=88 type=STRING col_path=[] offset=368 >> null=(offset=562 mask=80) slot_idx=23 field_idx=-1), Slot(id=89 type=INT >> col_path=[] offset=516 null=(offset=565 mask=4) slot_idx=42 field_idx=-1), >> Slot(id=90 type=INT col_path=[] offset=520 null=(offset=565 mask=8) >> slot_idx=43 field_idx=-1), Slot(id=91 type=INT col_path=[] offset=524 >> null=(offset=565 mask=10) slot_idx=44 field_idx=-1), Slot(id=92 type=INT >> col_path=[] offset=528 null=(offset=565 mask=20) slot_idx=45 field_idx=-1), >> Slot(id=93 type=INT col_path=[] offset=532 null=(offset=565 mask=40) >> slot_idx=46 field_idx=-1), Slot(id=94 type=INT col_path=[] offset=536 >> null=(offset=565 mask=80) slot_idx=47 field_idx=-1), Slot(id=95 type=INT >> col_path=[] offset=540 null=(offset=566 mask=1) slot_idx=48 field_idx=-1), >> Slot(id=96 type=INT col_path=[] offset=544 null=(offset=566 mask=2) >> slot_idx=49 field_idx=-1), Slot(id=97 type=INT col_path=[] offset=548 >> null=(offset=566 mask=4) slot_idx=50 field_idx=-1), Slot(id=98 type=INT >> col_path=[] offset=552 null=(offset=566 mask=8) slot_idx=51 field_idx=-1), >> Slot(id=99 type=INT col_path=[] offset=556 null=(offset=566 mask=10) >> slot_idx=52 field_idx=-1), Slot(id=100 type=STRING col_path=[] offset=384 >> null=(offset=563 mask=1) slot_idx=24 field_idx=-1), Slot(id=101 type=STRING >> col_path=[] offset=400 null=(offset=563 mask=2) slot_idx=25 field_idx=-1), >> Slot(id=102 type=STRING col_path=[] offset=416 null=(offset=563 mask=4) >> slot_idx=26 field_idx=-1), Slot(id=103 type=DECIMAL(30,6) col_path=[] >> offset=432 null=(offset=563 mask=8) slot_idx=27 field_idx=-1), Slot(id=104 >> type=DECIMAL(30,6) col_path=[] offset=448 null=(offset=563 mask=10) >> slot_idx=28 field_idx=-1)] tuple_path=[]) >> Tuple(id=1 size=0 slots=[] tuple_path=[]) >> Tuple(id=0 size=563 slots=[Slot(id=0 type=STRING col_path=[0] offset=0 >> null=(offset=556 mask=1) slot_idx=0 field_idx=-1), Slot(id=1 type=STRING >> col_path=[29] offset=16 null=(offset=556 mask=2) slot_idx=1 field_idx=-1), >> Slot(id=2 type=STRING col_path=[1] offset=32 null=(offset=556 mask=4) >> slot_idx=2 field_idx=-1), Slot(id=3 type=STRING col_path=[30] offset=48 >> null=(offset=556 mask=8) slot_idx=3 field_idx=-1), Slot(id=4 type=STRING >> col_path=[51] offset=64 null=(offset=556 mask=10) slot_idx=4 field_idx=-1), >> Slot(id=5 type=STRING col_path=[7] offset=80 null=(offset=556 mask=20) >> slot_idx=5 field_idx=-1), Slot(id=6 type=STRING col_path=[2] offset=96 >> null=(offset=556 mask=40) slot_idx=6 field_idx=-1), Slot(id=7 type=STRING >> col_path=[3] offset=112 null=(offset=556 mask=80) slot_idx=7 field_idx=-1), >> Slot(id=8 type=STRING col_path=[4] offset=128 null=(offset=557 mask=1) >> slot_idx=8 field_idx=-1), Slot(id=9 type=STRING col_path=[5] offset=144 >> null=(offset=557 mask=2) slot_idx=9 field_idx=-1), Slot(id=10 type=STRING >> col_path=[6] offset=160 null=(offset=557 mask=4) slot_idx=10 field_idx=-1), >> Slot(id=11 type=INT col_path=[8] offset=464 null=(offset=559 mask=20) >> slot_idx=29 field_idx=-1), Slot(id=12 type=INT col_path=[9] offset=468 >> null=(offset=559 mask=40) slot_idx=30 field_idx=-1), Slot(id=13 type=INT >> col_path=[10] offset=472 null=(offset=559 mask=80) slot_idx=31 >> field_idx=-1), Slot(id=14 type=INT col_path=[11] offset=476 >> null=(offset=560 mask=1) slot_idx=32 field_idx=-1), Slot(id=15 type=STRING >> col_path=[12] offset=176 null=(offset=557 mask=8) slot_idx=11 >> field_idx=-1), Slot(id=16 type=STRING col_path=[13] offset=192 >> null=(offset=557 mask=10) slot_idx=12 field_idx=-1), Slot(id=17 type=STRING >> col_path=[14] offset=208 null=(offset=557 mask=20) slot_idx=13 >> field_idx=-1), Slot(id=18 type=STRING col_path=[15] offset=224 >> null=(offset=557 mask=40) slot_idx=14 field_idx=-1), Slot(id=19 type=STRING >> col_path=[16] offset=240 null=(offset=557 mask=80) slot_idx=15 >> field_idx=-1), Slot(id=20 type=STRING col_path=[17] offset=256 >> null=(offset=558 mask=1) slot_idx=16 field_idx=-1), Slot(id=21 type=INT >> col_path=[18] offset=480 null=(offset=560 mask=2) slot_idx=33 >> field_idx=-1), Slot(id=22 type=INT col_path=[19] offset=484 >> null=(offset=560 mask=4) slot_idx=34 field_idx=-1), Slot(id=23 type=INT >> col_path=[20] offset=488 null=(offset=560 mask=8) slot_idx=35 >> field_idx=-1), Slot(id=24 type=INT col_path=[21] offset=492 >> null=(offset=560 mask=10) slot_idx=36 field_idx=-1), Slot(id=25 type=INT >> col_path=[22] offset=496 null=(offset=560 mask=20) slot_idx=37 >> field_idx=-1), Slot(id=26 type=INT col_path=[23] offset=500 >> null=(offset=560 mask=40) slot_idx=38 field_idx=-1), Slot(id=27 type=INT >> col_path=[24] offset=504 null=(offset=560 mask=80) slot_idx=39 >> field_idx=-1), Slot(id=28 type=STRING col_path=[25] offset=272 >> null=(offset=558 mask=2) slot_idx=17 field_idx=-1), Slot(id=29 type=STRING >> col_path=[26] offset=288 null=(offset=558 mask=4) slot_idx=18 >> field_idx=-1), Slot(id=30 type=INT col_path=[27] offset=508 >> null=(offset=561 mask=1) slot_idx=40 field_idx=-1), Slot(id=31 type=STRING >> col_path=[31] offset=304 null=(offset=558 mask=8) slot_idx=19 >> field_idx=-1), Slot(id=32 type=STRING col_path=[32] offset=320 >> null=(offset=558 mask=10) slot_idx=20 field_idx=-1), Slot(id=33 type=STRING >> col_path=[33] offset=336 null=(offset=558 mask=20) slot_idx=21 >> field_idx=-1), Slot(id=34 type=STRING col_path=[34] offset=352 >> null=(offset=558 mask=40) slot_idx=22 field_idx=-1), Slot(id=35 type=STRING >> col_path=[35] offset=368 null=(offset=558 mask=80) slot_idx=23 >> field_idx=-1), Slot(id=36 type=INT col_path=[36] offset=512 >> null=(offset=561 mask=2) slot_idx=41 field_idx=-1), Slot(id=37 type=INT >> col_path=[37] offset=516 null=(offset=561 mask=4) slot_idx=42 >> field_idx=-1), Slot(id=38 type=INT col_path=[38] offset=520 >> null=(offset=561 mask=8) slot_idx=43 field_idx=-1), Slot(id=39 type=INT >> col_path=[39] offset=524 null=(offset=561 mask=10) slot_idx=44 >> field_idx=-1), Slot(id=40 type=INT col_path=[40] offset=528 >> null=(offset=561 mask=20) slot_idx=45 field_idx=-1), Slot(id=41 type=INT >> col_path=[41] offset=532 null=(offset=561 mask=40) slot_idx=46 >> field_idx=-1), Slot(id=42 type=INT col_path=[42] offset=536 >> null=(offset=561 mask=80) slot_idx=47 field_idx=-1), Slot(id=43 type=INT >> col_path=[43] offset=540 null=(offset=562 mask=1) slot_idx=48 >> field_idx=-1), Slot(id=44 type=INT col_path=[44] offset=544 >> null=(offset=562 mask=2) slot_idx=49 field_idx=-1), Slot(id=45 type=INT >> col_path=[45] offset=548 null=(offset=562 mask=4) slot_idx=50 >> field_idx=-1), Slot(id=46 type=INT col_path=[46] offset=552 >> null=(offset=562 mask=8) slot_idx=51 field_idx=-1), Slot(id=47 type=STRING >> col_path=[47] offset=384 null=(offset=559 mask=1) slot_idx=24 >> field_idx=-1), Slot(id=48 type=STRING col_path=[48] offset=400 >> null=(offset=559 mask=2) slot_idx=25 field_idx=-1), Slot(id=49 type=STRING >> col_path=[28] offset=416 null=(offset=559 mask=4) slot_idx=26 >> field_idx=-1), Slot(id=50 type=DECIMAL(30,6) col_path=[49] offset=432 >> null=(offset=559 mask=8) slot_idx=27 field_idx=-1), Slot(id=51 >> type=DECIMAL(30,6) col_path=[50] offset=448 null=(offset=559 mask=10) >> slot_idx=28 field_idx=-1)] tuple_path=[]) >> I0718 18:42:22.680297 51333 query-state.cc:404] Executing instance. >> instance_id=2e4a3fbec0d7d721:2ec73c1c00000009 fragment_idx=0 >> per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=1 >> I0718 18:42:22.680315 51139 coordinator.cc:370] started execution on 5 >> backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000 >> I0718 18:42:22.680457 51334 query-state.cc:404] Executing instance. >> instance_id=2e4a3fbec0d7d721:2ec73c1c00000004 fragment_idx=1 >> per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=2 >> I0718 18:42:22.680583 51334 hdfs-scan-node.cc:190] Max row batch queue >> size for scan node '0' in fragment instance >> '2e4a3fbec0d7d721:2ec73c1c00000004': 40 >> I0718 18:42:22.687943 51332 query-exec-mgr.cc:155] ReleaseQueryState(): >> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 refcnt=4 >> I0718 18:42:22.689262 51338 coordinator.cc:789] Coordinator waiting for >> backends to finish, 5 remaining. query_id=2e4a3fbec0d7d721:2ec73c1c00000000 >> I0718 18:51:07.797159 51597 webserver.cc:362] Webserver: error reading: >> Resource temporarily unavailable >> I0718 19:22:57.687764 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 >> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} >> blocked reactor thread for 47602.1us >> I0718 19:23:27.786094 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 >> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} >> blocked reactor thread for 49243.8us >> I0718 19:27:28.426831 180697 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050 >> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}} >> blocked reactor thread for 47536.6us >> I0718 19:29:28.859411 232946 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request >> call id 5453218) took 518541ms. Request Metrics: {} >> I0718 19:29:28.859459 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request >> call id 5432432) took 518541ms. Request Metrics: {} >> I0718 19:29:28.859504 232946 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request >> call id 5500254) took 518534ms. Request Metrics: {} >> I0718 19:29:28.859417 232945 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request >> call id 5513959) took 518536ms. Request Metrics: {} >> I0718 19:29:28.859589 232941 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request >> call id 5485666) took 518524ms. Request Metrics: {} >> I0718 20:09:09.275442 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 >> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} >> blocked reactor thread for 52958.3us >> I0718 20:09:29.338707 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 >> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} >> blocked reactor thread for 49366.4us >> I0718 20:14:50.273746 180697 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050 >> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}} >> blocked reactor thread for 47568.8us >> I0718 20:15:20.342495 180697 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050 >> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}} >> blocked reactor thread for 51994us >> I0718 20:15:56.003931 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request >> call id 5647298) took 510394ms. Request Metrics: {} >> I0718 20:15:56.004006 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request >> call id 5626571) took 510394ms. Request Metrics: {} >> I0718 20:15:56.004089 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request >> call id 5661461) took 510386ms. Request Metrics: {} >> I0718 20:15:56.004143 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request >> call id 5601664) took 510383ms. Request Metrics: {} >> I0718 20:15:56.004218 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request >> call id 5577825) took 510328ms. Request Metrics: {} >> I0718 21:56:51.345875 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 >> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} >> blocked reactor thread for 61974.1us >> I0718 21:57:41.519013 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 >> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} >> blocked reactor thread for 48016.9us >> I0718 22:07:02.248558 180696 outbound_call.cc:286] RPC callback for RPC >> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.56:7050 >> (realtimeanalysis-kudu-02-10-8-50-56), user_credentials={real_user=hive}} >> blocked reactor thread for 61139.4us >> I0718 22:07:56.005770 232949 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request >> call id 5804566) took 839901ms. Request Metrics: {} >> I0718 22:07:56.005787 232937 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request >> call id 5749547) took 839899ms. Request Metrics: {} >> I0718 22:07:56.005823 232934 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request >> call id 5820240) took 839850ms. Request Metrics: {} >> I0718 22:07:56.005831 232940 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request >> call id 5712080) took 839898ms. Request Metrics: {} >> I0718 22:07:56.005977 232939 rpcz_store.cc:255] Call >> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request >> call id 5768794) took 839898ms. Request Metrics: {} >> >> *I will write 632 million rows to kudu, normally it's about 50 thousand/s >> , but today after write about 270 million , then write speed become very >> slow , I don't know what was wrong.* >> >> 2019-07-18 >> ------------------------------ >> lk_hadoop >> > --000000000000a458c6058e0c07da Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

<= div class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif= ;font-size:small">It's hard to say what might be the problem without ad= ditional information.=C2=A0 Could you clarify on the following questions:

1.=C2=A0 What was the r= ate of write operations for the 270M rows you mentioned?=C2=A0 Was that reg= ular 50K rows/sec or something else?
2.=C2=A0 Do yo= u still observe the slowness or it's already gone?=C2=A0 (if that was j= ust a spike in ops rate, Kudu tablet servers might be rejecting writes due = to the memory pressure when not being able to flush the data on disk fast e= nough).=C2=A0
3.=C2=A0 What was special about those= 270M rows?=C2=A0 Maybe, the order of keys changed somehow?=C2=A0 Kudu inge= sts data much faster if row keys come in kind of sequential order: so if th= e prior write batches had that property, but the recent 270M chunk didn'= ;t, it might be the case.
4.=C2=A0 Was there any ad= ditional concurrent activity on the nodes where Kudu tablet servers run?


Thanks,

Alexey

On Thu, Jul 18, 2019 at 7:48 PM Tim Armstrong <tarmstrong@cloudera.com> wrote:
Also including= the Kudu list in case someone there recognises the problem.

<= div class=3D"gmail_quote">
On Thu, Jul= 18, 2019 at 8:05 AM lk_hadoop <lk_hadoop@163.com> wrote:
I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on = 5=20 backends for query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000
I0718 18:42:22= .679605=20 12873 impala-internal-service.cc:44] ExecQueryFInstances():=20 query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000
I0718 18:42:22.679620 12873= =20 query-exec-mgr.cc:46] StartQueryFInstances()=20 query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000=20 coord=3Drealtimeanalysis-kudu-04-10-8-50-58:22000
I0718 18:42:22.679625 = 12873=20 query-state.cc:178] Buffer pool limit for 2e4a3fbec0d7d721:2ec73c1c00000000= :=20 17179869184
I0718 18:42:22.679675 12873 initial-reservations.cc:60]=20 Successfully claimed initial reservations (4.00 MB) for query=20 2e4a3fbec0d7d721:2ec73c1c00000000
I0718 18:42:22.679769 51332=20 query-state.cc:309] StartFInstances():=20 query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000 #instances=3D2
I0718 18:42:= 22.680196=20 51332 query-state.cc:322] descriptor table for=20 query=3D2e4a3fbec0d7d721:2ec73c1c00000000
tuples:
Tuple(id=3D2 size= =3D567=20 slots=3D[Slot(id=3D52 type=3DINT col_path=3D[] offset=3D464 null=3D(offset= =3D563 mask=3D20)=20 slot_idx=3D29 field_idx=3D-1), Slot(id=3D53 type=3DSTRING col_path=3D[] off= set=3D0=20 null=3D(offset=3D560 mask=3D1) slot_idx=3D0 field_idx=3D-1), Slot(id=3D54 t= ype=3DSTRING=20 col_path=3D[] offset=3D16 null=3D(offset=3D560 mask=3D2) slot_idx=3D1 field= _idx=3D-1),=20 Slot(id=3D55 type=3DSTRING col_path=3D[] offset=3D32 null=3D(offset=3D560 m= ask=3D4) slot_idx=3D2=20 field_idx=3D-1), Slot(id=3D56 type=3DSTRING col_path=3D[] offset=3D48 null= =3D(offset=3D560=20 mask=3D8) slot_idx=3D3 field_idx=3D-1), Slot(id=3D57 type=3DSTRING col_path= =3D[] offset=3D64=20 null=3D(offset=3D560 mask=3D10) slot_idx=3D4 field_idx=3D-1), Slot(id=3D58 = type=3DSTRING=20 col_path=3D[] offset=3D80 null=3D(offset=3D560 mask=3D20) slot_idx=3D5 fiel= d_idx=3D-1),=20 Slot(id=3D59 type=3DSTRING col_path=3D[] offset=3D96 null=3D(offset=3D560 m= ask=3D40)=20 slot_idx=3D6 field_idx=3D-1), Slot(id=3D60 type=3DSTRING col_path=3D[] offs= et=3D112=20 null=3D(offset=3D560 mask=3D80) slot_idx=3D7 field_idx=3D-1), Slot(id=3D61 = type=3DSTRING=20 col_path=3D[] offset=3D128 null=3D(offset=3D561 mask=3D1) slot_idx=3D8 fiel= d_idx=3D-1),=20 Slot(id=3D62 type=3DSTRING col_path=3D[] offset=3D144 null=3D(offset=3D561 = mask=3D2)=20 slot_idx=3D9 field_idx=3D-1), Slot(id=3D63 type=3DSTRING col_path=3D[] offs= et=3D160=20 null=3D(offset=3D561 mask=3D4) slot_idx=3D10 field_idx=3D-1), Slot(id=3D64 = type=3DINT=20 col_path=3D[] offset=3D468 null=3D(offset=3D563 mask=3D40) slot_idx=3D30 fi= eld_idx=3D-1),=20 Slot(id=3D65 type=3DINT col_path=3D[] offset=3D472 null=3D(offset=3D563 mas= k=3D80) slot_idx=3D31=20 field_idx=3D-1), Slot(id=3D66 type=3DINT col_path=3D[] offset=3D476 null=3D= (offset=3D564=20 mask=3D1) slot_idx=3D32 field_idx=3D-1), Slot(id=3D67 type=3DINT col_path= =3D[] offset=3D480=20 null=3D(offset=3D564 mask=3D2) slot_idx=3D33 field_idx=3D-1), Slot(id=3D68 = type=3DSTRING=20 col_path=3D[] offset=3D176 null=3D(offset=3D561 mask=3D8) slot_idx=3D11 fie= ld_idx=3D-1),=20 Slot(id=3D69 type=3DSTRING col_path=3D[] offset=3D192 null=3D(offset=3D561 = mask=3D10)=20 slot_idx=3D12 field_idx=3D-1), Slot(id=3D70 type=3DSTRING col_path=3D[] off= set=3D208=20 null=3D(offset=3D561 mask=3D20) slot_idx=3D13 field_idx=3D-1), Slot(id=3D71= type=3DSTRING=20 col_path=3D[] offset=3D224 null=3D(offset=3D561 mask=3D40) slot_idx=3D14 fi= eld_idx=3D-1),=20 Slot(id=3D72 type=3DSTRING col_path=3D[] offset=3D240 null=3D(offset=3D561 = mask=3D80)=20 slot_idx=3D15 field_idx=3D-1), Slot(id=3D73 type=3DSTRING col_path=3D[] off= set=3D256=20 null=3D(offset=3D562 mask=3D1) slot_idx=3D16 field_idx=3D-1), Slot(id=3D74 = type=3DINT=20 col_path=3D[] offset=3D484 null=3D(offset=3D564 mask=3D4) slot_idx=3D34 fie= ld_idx=3D-1),=20 Slot(id=3D75 type=3DINT col_path=3D[] offset=3D488 null=3D(offset=3D564 mas= k=3D8) slot_idx=3D35=20 field_idx=3D-1), Slot(id=3D76 type=3DINT col_path=3D[] offset=3D492 null=3D= (offset=3D564=20 mask=3D10) slot_idx=3D36 field_idx=3D-1), Slot(id=3D77 type=3DINT col_path= =3D[] offset=3D496=20 null=3D(offset=3D564 mask=3D20) slot_idx=3D37 field_idx=3D-1), Slot(id=3D78= type=3DINT=20 col_path=3D[] offset=3D500 null=3D(offset=3D564 mask=3D40) slot_idx=3D38 fi= eld_idx=3D-1),=20 Slot(id=3D79 type=3DINT col_path=3D[] offset=3D504 null=3D(offset=3D564 mas= k=3D80) slot_idx=3D39=20 field_idx=3D-1), Slot(id=3D80 type=3DINT col_path=3D[] offset=3D508 null=3D= (offset=3D565=20 mask=3D1) slot_idx=3D40 field_idx=3D-1), Slot(id=3D81 type=3DSTRING col_pat= h=3D[] offset=3D272=20 null=3D(offset=3D562 mask=3D2) slot_idx=3D17 field_idx=3D-1), Slot(id=3D82 = type=3DSTRING=20 col_path=3D[] offset=3D288 null=3D(offset=3D562 mask=3D4) slot_idx=3D18 fie= ld_idx=3D-1),=20 Slot(id=3D83 type=3DINT col_path=3D[] offset=3D512 null=3D(offset=3D565 mas= k=3D2) slot_idx=3D41=20 field_idx=3D-1), Slot(id=3D84 type=3DSTRING col_path=3D[] offset=3D304 null= =3D(offset=3D562=20 mask=3D8) slot_idx=3D19 field_idx=3D-1), Slot(id=3D85 type=3DSTRING col_pat= h=3D[] offset=3D320=20 null=3D(offset=3D562 mask=3D10) slot_idx=3D20 field_idx=3D-1), Slot(id=3D86= type=3DSTRING=20 col_path=3D[] offset=3D336 null=3D(offset=3D562 mask=3D20) slot_idx=3D21 fi= eld_idx=3D-1),=20 Slot(id=3D87 type=3DSTRING col_path=3D[] offset=3D352 null=3D(offset=3D562 = mask=3D40)=20 slot_idx=3D22 field_idx=3D-1), Slot(id=3D88 type=3DSTRING col_path=3D[] off= set=3D368=20 null=3D(offset=3D562 mask=3D80) slot_idx=3D23 field_idx=3D-1), Slot(id=3D89= type=3DINT=20 col_path=3D[] offset=3D516 null=3D(offset=3D565 mask=3D4) slot_idx=3D42 fie= ld_idx=3D-1),=20 Slot(id=3D90 type=3DINT col_path=3D[] offset=3D520 null=3D(offset=3D565 mas= k=3D8) slot_idx=3D43=20 field_idx=3D-1), Slot(id=3D91 type=3DINT col_path=3D[] offset=3D524 null=3D= (offset=3D565=20 mask=3D10) slot_idx=3D44 field_idx=3D-1), Slot(id=3D92 type=3DINT col_path= =3D[] offset=3D528=20 null=3D(offset=3D565 mask=3D20) slot_idx=3D45 field_idx=3D-1), Slot(id=3D93= type=3DINT=20 col_path=3D[] offset=3D532 null=3D(offset=3D565 mask=3D40) slot_idx=3D46 fi= eld_idx=3D-1),=20 Slot(id=3D94 type=3DINT col_path=3D[] offset=3D536 null=3D(offset=3D565 mas= k=3D80) slot_idx=3D47=20 field_idx=3D-1), Slot(id=3D95 type=3DINT col_path=3D[] offset=3D540 null=3D= (offset=3D566=20 mask=3D1) slot_idx=3D48 field_idx=3D-1), Slot(id=3D96 type=3DINT col_path= =3D[] offset=3D544=20 null=3D(offset=3D566 mask=3D2) slot_idx=3D49 field_idx=3D-1), Slot(id=3D97 = type=3DINT=20 col_path=3D[] offset=3D548 null=3D(offset=3D566 mask=3D4) slot_idx=3D50 fie= ld_idx=3D-1),=20 Slot(id=3D98 type=3DINT col_path=3D[] offset=3D552 null=3D(offset=3D566 mas= k=3D8) slot_idx=3D51=20 field_idx=3D-1), Slot(id=3D99 type=3DINT col_path=3D[] offset=3D556 null=3D= (offset=3D566=20 mask=3D10) slot_idx=3D52 field_idx=3D-1), Slot(id=3D100 type=3DSTRING col_p= ath=3D[]=20 offset=3D384 null=3D(offset=3D563 mask=3D1) slot_idx=3D24 field_idx=3D-1), = Slot(id=3D101=20 type=3DSTRING col_path=3D[] offset=3D400 null=3D(offset=3D563 mask=3D2) slo= t_idx=3D25=20 field_idx=3D-1), Slot(id=3D102 type=3DSTRING col_path=3D[] offset=3D416 nul= l=3D(offset=3D563=20 mask=3D4) slot_idx=3D26 field_idx=3D-1), Slot(id=3D103 type=3DDECIMAL(30,6)= col_path=3D[]=20 offset=3D432 null=3D(offset=3D563 mask=3D8) slot_idx=3D27 field_idx=3D-1), = Slot(id=3D104=20 type=3DDECIMAL(30,6) col_path=3D[] offset=3D448 null=3D(offset=3D563 mask= =3D10) slot_idx=3D28=20 field_idx=3D-1)] tuple_path=3D[])
Tuple(id=3D1 size=3D0 slots=3D[]=20 tuple_path=3D[])
Tuple(id=3D0 size=3D563 slots=3D[Slot(id=3D0 type=3DSTR= ING col_path=3D[0]=20 offset=3D0 null=3D(offset=3D556 mask=3D1) slot_idx=3D0 field_idx=3D-1), Slo= t(id=3D1=20 type=3DSTRING col_path=3D[29] offset=3D16 null=3D(offset=3D556 mask=3D2) sl= ot_idx=3D1=20 field_idx=3D-1), Slot(id=3D2 type=3DSTRING col_path=3D[1] offset=3D32 null= =3D(offset=3D556=20 mask=3D4) slot_idx=3D2 field_idx=3D-1), Slot(id=3D3 type=3DSTRING col_path= =3D[30] offset=3D48=20 null=3D(offset=3D556 mask=3D8) slot_idx=3D3 field_idx=3D-1), Slot(id=3D4 ty= pe=3DSTRING=20 col_path=3D[51] offset=3D64 null=3D(offset=3D556 mask=3D10) slot_idx=3D4 fi= eld_idx=3D-1),=20 Slot(id=3D5 type=3DSTRING col_path=3D[7] offset=3D80 null=3D(offset=3D556 m= ask=3D20)=20 slot_idx=3D5 field_idx=3D-1), Slot(id=3D6 type=3DSTRING col_path=3D[2] offs= et=3D96=20 null=3D(offset=3D556 mask=3D40) slot_idx=3D6 field_idx=3D-1), Slot(id=3D7 t= ype=3DSTRING=20 col_path=3D[3] offset=3D112 null=3D(offset=3D556 mask=3D80) slot_idx=3D7 fi= eld_idx=3D-1),=20 Slot(id=3D8 type=3DSTRING col_path=3D[4] offset=3D128 null=3D(offset=3D557 = mask=3D1)=20 slot_idx=3D8 field_idx=3D-1), Slot(id=3D9 type=3DSTRING col_path=3D[5] offs= et=3D144=20 null=3D(offset=3D557 mask=3D2) slot_idx=3D9 field_idx=3D-1), Slot(id=3D10 t= ype=3DSTRING=20 col_path=3D[6] offset=3D160 null=3D(offset=3D557 mask=3D4) slot_idx=3D10 fi= eld_idx=3D-1),=20 Slot(id=3D11 type=3DINT col_path=3D[8] offset=3D464 null=3D(offset=3D559 ma= sk=3D20)=20 slot_idx=3D29 field_idx=3D-1), Slot(id=3D12 type=3DINT col_path=3D[9] offse= t=3D468=20 null=3D(offset=3D559 mask=3D40) slot_idx=3D30 field_idx=3D-1), Slot(id=3D13= type=3DINT=20 col_path=3D[10] offset=3D472 null=3D(offset=3D559 mask=3D80) slot_idx=3D31 = field_idx=3D-1),=20 Slot(id=3D14 type=3DINT col_path=3D[11] offset=3D476 null=3D(offset=3D560 m= ask=3D1)=20 slot_idx=3D32 field_idx=3D-1), Slot(id=3D15 type=3DSTRING col_path=3D[12] o= ffset=3D176=20 null=3D(offset=3D557 mask=3D8) slot_idx=3D11 field_idx=3D-1), Slot(id=3D16 = type=3DSTRING=20 col_path=3D[13] offset=3D192 null=3D(offset=3D557 mask=3D10) slot_idx=3D12 = field_idx=3D-1),=20 Slot(id=3D17 type=3DSTRING col_path=3D[14] offset=3D208 null=3D(offset=3D55= 7 mask=3D20)=20 slot_idx=3D13 field_idx=3D-1), Slot(id=3D18 type=3DSTRING col_path=3D[15] o= ffset=3D224=20 null=3D(offset=3D557 mask=3D40) slot_idx=3D14 field_idx=3D-1), Slot(id=3D19= type=3DSTRING=20 col_path=3D[16] offset=3D240 null=3D(offset=3D557 mask=3D80) slot_idx=3D15 = field_idx=3D-1),=20 Slot(id=3D20 type=3DSTRING col_path=3D[17] offset=3D256 null=3D(offset=3D55= 8 mask=3D1)=20 slot_idx=3D16 field_idx=3D-1), Slot(id=3D21 type=3DINT col_path=3D[18] offs= et=3D480=20 null=3D(offset=3D560 mask=3D2) slot_idx=3D33 field_idx=3D-1), Slot(id=3D22 = type=3DINT=20 col_path=3D[19] offset=3D484 null=3D(offset=3D560 mask=3D4) slot_idx=3D34 f= ield_idx=3D-1),=20 Slot(id=3D23 type=3DINT col_path=3D[20] offset=3D488 null=3D(offset=3D560 m= ask=3D8)=20 slot_idx=3D35 field_idx=3D-1), Slot(id=3D24 type=3DINT col_path=3D[21] offs= et=3D492=20 null=3D(offset=3D560 mask=3D10) slot_idx=3D36 field_idx=3D-1), Slot(id=3D25= type=3DINT=20 col_path=3D[22] offset=3D496 null=3D(offset=3D560 mask=3D20) slot_idx=3D37 = field_idx=3D-1),=20 Slot(id=3D26 type=3DINT col_path=3D[23] offset=3D500 null=3D(offset=3D560 m= ask=3D40)=20 slot_idx=3D38 field_idx=3D-1), Slot(id=3D27 type=3DINT col_path=3D[24] offs= et=3D504=20 null=3D(offset=3D560 mask=3D80) slot_idx=3D39 field_idx=3D-1), Slot(id=3D28= type=3DSTRING=20 col_path=3D[25] offset=3D272 null=3D(offset=3D558 mask=3D2) slot_idx=3D17 f= ield_idx=3D-1),=20 Slot(id=3D29 type=3DSTRING col_path=3D[26] offset=3D288 null=3D(offset=3D55= 8 mask=3D4)=20 slot_idx=3D18 field_idx=3D-1), Slot(id=3D30 type=3DINT col_path=3D[27] offs= et=3D508=20 null=3D(offset=3D561 mask=3D1) slot_idx=3D40 field_idx=3D-1), Slot(id=3D31 = type=3DSTRING=20 col_path=3D[31] offset=3D304 null=3D(offset=3D558 mask=3D8) slot_idx=3D19 f= ield_idx=3D-1),=20 Slot(id=3D32 type=3DSTRING col_path=3D[32] offset=3D320 null=3D(offset=3D55= 8 mask=3D10)=20 slot_idx=3D20 field_idx=3D-1), Slot(id=3D33 type=3DSTRING col_path=3D[33] o= ffset=3D336=20 null=3D(offset=3D558 mask=3D20) slot_idx=3D21 field_idx=3D-1), Slot(id=3D34= type=3DSTRING=20 col_path=3D[34] offset=3D352 null=3D(offset=3D558 mask=3D40) slot_idx=3D22 = field_idx=3D-1),=20 Slot(id=3D35 type=3DSTRING col_path=3D[35] offset=3D368 null=3D(offset=3D55= 8 mask=3D80)=20 slot_idx=3D23 field_idx=3D-1), Slot(id=3D36 type=3DINT col_path=3D[36] offs= et=3D512=20 null=3D(offset=3D561 mask=3D2) slot_idx=3D41 field_idx=3D-1), Slot(id=3D37 = type=3DINT=20 col_path=3D[37] offset=3D516 null=3D(offset=3D561 mask=3D4) slot_idx=3D42 f= ield_idx=3D-1),=20 Slot(id=3D38 type=3DINT col_path=3D[38] offset=3D520 null=3D(offset=3D561 m= ask=3D8)=20 slot_idx=3D43 field_idx=3D-1), Slot(id=3D39 type=3DINT col_path=3D[39] offs= et=3D524=20 null=3D(offset=3D561 mask=3D10) slot_idx=3D44 field_idx=3D-1), Slot(id=3D40= type=3DINT=20 col_path=3D[40] offset=3D528 null=3D(offset=3D561 mask=3D20) slot_idx=3D45 = field_idx=3D-1),=20 Slot(id=3D41 type=3DINT col_path=3D[41] offset=3D532 null=3D(offset=3D561 m= ask=3D40)=20 slot_idx=3D46 field_idx=3D-1), Slot(id=3D42 type=3DINT col_path=3D[42] offs= et=3D536=20 null=3D(offset=3D561 mask=3D80) slot_idx=3D47 field_idx=3D-1), Slot(id=3D43= type=3DINT=20 col_path=3D[43] offset=3D540 null=3D(offset=3D562 mask=3D1) slot_idx=3D48 f= ield_idx=3D-1),=20 Slot(id=3D44 type=3DINT col_path=3D[44] offset=3D544 null=3D(offset=3D562 m= ask=3D2)=20 slot_idx=3D49 field_idx=3D-1), Slot(id=3D45 type=3DINT col_path=3D[45] offs= et=3D548=20 null=3D(offset=3D562 mask=3D4) slot_idx=3D50 field_idx=3D-1), Slot(id=3D46 = type=3DINT=20 col_path=3D[46] offset=3D552 null=3D(offset=3D562 mask=3D8) slot_idx=3D51 f= ield_idx=3D-1),=20 Slot(id=3D47 type=3DSTRING col_path=3D[47] offset=3D384 null=3D(offset=3D55= 9 mask=3D1)=20 slot_idx=3D24 field_idx=3D-1), Slot(id=3D48 type=3DSTRING col_path=3D[48] o= ffset=3D400=20 null=3D(offset=3D559 mask=3D2) slot_idx=3D25 field_idx=3D-1), Slot(id=3D49 = type=3DSTRING=20 col_path=3D[28] offset=3D416 null=3D(offset=3D559 mask=3D4) slot_idx=3D26 f= ield_idx=3D-1),=20 Slot(id=3D50 type=3DDECIMAL(30,6) col_path=3D[49] offset=3D432 null=3D(offs= et=3D559 mask=3D8)=20 slot_idx=3D27 field_idx=3D-1), Slot(id=3D51 type=3DDECIMAL(30,6) col_path= =3D[50]=20 offset=3D448 null=3D(offset=3D559 mask=3D10) slot_idx=3D28 field_idx=3D-1)]= =20 tuple_path=3D[])
I0718 18:42:22.680297 51333 query-state.cc:404] Executi= ng=20 instance. instance_id=3D2e4a3fbec0d7d721:2ec73c1c00000009 fragment_idx=3D0= =20 per_fragment_instance_idx=3D4 coord_state_idx=3D3 #in-flight=3D1
I0718= =20 18:42:22.680315 51139 coordinator.cc:370] started execution on 5 backends f= or=20 query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000
I0718 18:42:22.680457 51334= =20 query-state.cc:404] Executing instance.=20 instance_id=3D2e4a3fbec0d7d721:2ec73c1c00000004 fragment_idx=3D1=20 per_fragment_instance_idx=3D4 coord_state_idx=3D3 #in-flight=3D2
I0718= =20 18:42:22.680583 51334 hdfs-scan-node.cc:190] Max row batch queue size for s= can=20 node '0' in fragment instance '2e4a3fbec0d7d721:2ec73c1c0000000= 4': 40
I0718=20 18:42:22.687943 51332 query-exec-mgr.cc:155] ReleaseQueryState():=20 query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000 refcnt=3D4
I0718 18:42:22.6= 89262=20 51338 coordinator.cc:789] Coordinator waiting for backends to finish, 5=20 remaining. query_id=3D2e4a3fbec0d7d721:2ec73c1c00000000
I0718 18:51:07.7= 97159=20 51597 webserver.cc:362] Webserver: error reading: Resource temporarily=20 unavailable
I0718 19:22:57.687764 180696 outbound_call.cc:286] RPC callb= ack=20 for RPC call kudu.tserver.TabletServerService.Write ->=20 {remote=3D10.8.50.58:7= 050 (realtimeanalysis-kudu-04-10-8-50-58),=20 user_credentials=3D{real_user=3Dhive}} blocked reactor thread for 47602.1us=
I0718=20 19:23:27.786094 180696 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.58:7050=20 (realtimeanalysis-kudu-04-10-8-50-58), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 49243.8us
I0718 19:27:28.426831 180697=20 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.59:7050=20 (realtimeanalysis-kudu-05-10-8-50-59), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 47536.6us
I0718 19:29:28.859411 232946=20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.59:36460 = (request call id 5453218) took 518541ms. Request Metrics:=20 {}
I0718 19:29:28.859459 232949 rpcz_store.cc:255] Call=20 impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call id=20 5432432) took 518541ms. Request Metrics: {}
I0718 19:29:28.859504 232946= =20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.56:49430 = (request call id 5500254) took 518534ms. Request Metrics:=20 {}
I0718 19:29:28.859417 232945 rpcz_store.cc:255] Call=20 impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call id=20 5513959) took 518536ms. Request Metrics: {}
I0718 19:29:28.859589 232941= =20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.58:52438 = (request call id 5485666) took 518524ms. Request Metrics:=20 {}
I0718 20:09:09.275442 180696 outbound_call.cc:286] RPC callback for R= PC=20 call kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.58:7050=20 (realtimeanalysis-kudu-04-10-8-50-58), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 52958.3us
I0718 20:09:29.338707 180696=20 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.58:7050=20 (realtimeanalysis-kudu-04-10-8-50-58), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 49366.4us
I0718 20:14:50.273746 180697=20 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.59:7050=20 (realtimeanalysis-kudu-05-10-8-50-59), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 47568.8us
I0718 20:15:20.342495 180697=20 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.59:7050=20 (realtimeanalysis-kudu-05-10-8-50-59), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 51994us
I0718 20:15:56.003931 232949=20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.56:49430 = (request call id 5647298) took 510394ms. Request Metrics:=20 {}
I0718 20:15:56.004006 232949 rpcz_store.cc:255] Call=20 impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call id=20 5626571) took 510394ms. Request Metrics: {}
I0718 20:15:56.004089 232949= =20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.57:60524 = (request call id 5661461) took 510386ms. Request Metrics:=20 {}
I0718 20:15:56.004143 232949 rpcz_store.cc:255] Call=20 impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call id=20 5601664) took 510383ms. Request Metrics: {}
I0718 20:15:56.004218 232949= =20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.55:34302 = (request call id 5577825) took 510328ms. Request Metrics:=20 {}
I0718 21:56:51.345875 180696 outbound_call.cc:286] RPC callback for R= PC=20 call kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.58:7050=20 (realtimeanalysis-kudu-04-10-8-50-58), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 61974.1us
I0718 21:57:41.519013 180696=20 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.58:7050=20 (realtimeanalysis-kudu-04-10-8-50-58), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 48016.9us
I0718 22:07:02.248558 180696=20 outbound_call.cc:286] RPC callback for RPC call=20 kudu.tserver.TabletServerService.Write -> {remote=3D10.8.50.56:7050=20 (realtimeanalysis-kudu-02-10-8-50-56), user_credentials=3D{real_user=3Dhive= }}=20 blocked reactor thread for 61139.4us
I0718 22:07:56.005770 232949=20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.56:49430 = (request call id 5804566) took 839901ms. Request Metrics:=20 {}
I0718 22:07:56.005787 232937 rpcz_store.cc:255] Call=20 impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call id=20 5749547) took 839899ms. Request Metrics: {}
I0718 22:07:56.005823 232934= =20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.57:60524 = (request call id 5820240) took 839850ms. Request Metrics:=20 {}
I0718 22:07:56.005831 232940 rpcz_store.cc:255] Call=20 impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call id=20 5712080) took 839898ms. Request Metrics: {}
I0718 22:07:56.005977 232939= =20 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from=20 10.8.50.58:52438 = (request call id 5768794) took 839898ms. Request Metrics:=20 {}
=C2=A0
I will write 632 million rows to kudu, normally it's about= 50=20 thousand/s , but today after write about 270 million=C2=A0 , then write spe= ed=20 become very slow , I don't know what was wrong.
=C2=A0
2019-07-18
lk_hadoop
--000000000000a458c6058e0c07da--