Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9A183200C8F for ; Fri, 26 May 2017 02:53:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 98DD9160BD8; Fri, 26 May 2017 00:53:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BF43B160BCA for ; Fri, 26 May 2017 02:53:57 +0200 (CEST) Received: (qmail 10847 invoked by uid 500); 26 May 2017 00:53:51 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 10832 invoked by uid 99); 26 May 2017 00:53:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 May 2017 00:53:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A8D1CC0040 for ; Fri, 26 May 2017 00:53:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.541 X-Spam-Level: *** X-Spam-Status: No, score=3.541 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, HTML_OBFUSCATE_10_20=1.162, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id e-AOQPVjOz-e for ; Fri, 26 May 2017 00:53:47 +0000 (UTC) Received: from mail-wm0-f48.google.com (mail-wm0-f48.google.com [74.125.82.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B37765F36F for ; Fri, 26 May 2017 00:53:46 +0000 (UTC) Received: by mail-wm0-f48.google.com with SMTP id e127so2494106wmg.1 for ; Thu, 25 May 2017 17:53:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=zUXSgAQ9kn04aDz1JFaGhrKtJoaekWUSjCFzhjrBxUU=; b=J/dM0vMH50hv94/tNpEfM8y39rxiCDLvbGXIWUAPHht4WTB6fllyfe5pmgKyojlvFJ tHNQlVnE/+xAMLkTtV0f7ZDj95ymVZ64+B2r9SXDPquYg8cg5UXTlU0S+QRe42FY1QOd CQhCjRP/1UtAnLJbTih/IueARuxD/RLXFKTq7cElStmOgHeQHMFAvr1A+wNFTYslhYJL s+2x4W/CtIN7fjwc92vez92IxRSo03h9CVDrUVCivAgpj9dLGjEqNUo/vPo6TaGJ0w6T 8H53KsPLtVvG1NY0Fssb+DaThAy7SldwTKHx9Mhmv25hgM2UJbhPiEHDB5jN7Kbf53Ab w9MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=zUXSgAQ9kn04aDz1JFaGhrKtJoaekWUSjCFzhjrBxUU=; b=CtwbaSi/xgjwSO3v878SSWBnMTrWOoEiF1/ARaX2W0KTBtolLNMcZCqZ0G7i53NwGZ xoOnkoBcpJvMSLYDysK+oeJjTcbQfTGs3k4JaWx8GH+WVy8I+JrDtG9TcN5+D8Byqpo9 3UdCGeKwhrZ4esIAx+LuMv6vNmGFKilSAXBg3q8oX6rFEmSmMg8Y7IocXixP5dCpIT3z exlbhgWNoezY2uGBiEUnQ3fLyBJPfTU+balgnBEY47Qxuuf/7lJxhDl3ZPBNjKKnJaEc fmNxzhrmfuzwzMvLAfuFOLy3RXhOsmSnh7d1KvxG9+zicoKwdDfUcbVInTow8x+PLIg7 RoWw== X-Gm-Message-State: AODbwcBAwvlk8g9BRhXz5ui9jMO53JJuhYCttAuQUbp47LavmzZDuxTx bHFJtxnheGaKsfR2ZGYV2wJKk7tmQg== X-Received: by 10.223.151.6 with SMTP id r6mr25386774wrb.189.1495760026256; Thu, 25 May 2017 17:53:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.156.11 with HTTP; Thu, 25 May 2017 17:53:25 -0700 (PDT) From: anil gupta Date: Thu, 25 May 2017 17:53:25 -0700 Message-ID: Subject: mapreduce.LoadIncrementalHFiles: Split occured while grouping HFiles To: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary="f403045f50b608cc1e055062c571" archived-at: Fri, 26 May 2017 00:53:58 -0000 --f403045f50b608cc1e055062c571 Content-Type: text/plain; charset="UTF-8" Cross posting since this seems to be an HBase issue. I think completeBulkLoad step is failing. Please refer to the mail below. ---------- Forwarded message ---------- From: anil gupta Date: Thu, May 25, 2017 at 4:38 PM Subject: [IndexTool NOT working] mapreduce.LoadIncrementalHFiles: Split occured while grouping HFiles To: "user@phoenix.apache.org" Hi, We are using HDP2.3.2(Phoenix 4.4 and HBase 1.1), we created a secondary index on an already existing table. We paused all writes to Primary table. Then we invoked IndexTool to populate secondary index table. We have tried same steps many times but we keep on getting following error(we have also tried drop the index and adding it again): 2017-05-24 18:00:10,281 WARN [LoadIncrementalHFiles-2] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-24 18:00:10,340 WARN [LoadIncrementalHFiles-12] util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2017-05-24 18:00:10,342 INFO [LoadIncrementalHFiles-11] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ c79ae6d27824424f99523dad586e86b1 first=JF2GPADC8GH331037\x00\ x80\x00\x1A0\x80\x00\x01Wj\x03r1defc4d301e4ec172b49be4a7ea33c2f7 last=JTHBK1GG4E2122477\x00\x80\x00$\xE4\x80\x00\x01[\xAD`{\ x17901d036d588292854ac5b1d4c29d8e1e 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-14] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ f0e97b218aed4abf9949cf49a57e559b first=5NPEB4AC3DH620091\x00\ x80\x00\xE0\x16\x80\x00\x01X\xE5g\xD6\x0B81d210ac753ed281e8627e5edb7eb59f last=JF2GPADC8GH331037\x00\x80\x00\x1A0\x80\x00\x01W]&\ xE54f37d636104f6cd916b2b07bf3aa94d3f 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-2] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 27c0a905ee174c9898d324acf1554bf9 first=WMWZP3C58FT708786\x00\ x80\x00\xE0\x16\x80\x00\x01Y\xB8\x95U\xA0d21d32aed18af976dd53735705c728cd last=`1GCRCPE05BZ430377\x00\x80\x00}\x05\x80\x00\x01[\xDEE\ x91L383768c6ac5f306fa99f68964b4f18aa 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-12] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ d0a6c4b727bb416f840ed254658f3982 first=1N4BZ0CP4GC308715\x00\ x80\x01T\xFC\x80\x00\x01U\xE3\x7FL\x9A37b77d47941e99e430fcb0e0657f5558 last=2GKALMEK7H6220949\x00\x80\x00!\x1A\x80\x00\x01Y\x18\xE6\xB3\ xB42e72036f7e7e03078f41fc82712c5de7 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 11627a2861e3446e9d6f684ab534563e first=3C7WRNAJ6GG313342\x00\ x80\x00NB\x80\x00\x01V}\xFD\xE4+65bbebdd06dedd8466a31ebd33841a51 last=3N1CE2CP2FL407481\x00\x80\x00\xE0\x16\x80\x00\x01W\x1B\x0A\x02\ xC1fc95d4114d5e91197a5e41bf37c9e8c7 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 23df78bafd304ff887385a2b6becf06d first=1C4RJFLT6HC742023\x00\ x80\x00x\xFF\x80\x00\x01[J8\x0Ac8b65a80fe1662fb25d80798a66cc83dc last=1FMCU9J90EUB68140\x00\x80\x01X\xA4\x80\x00\x01[\x1B\xDD\xB2\ x1C577502512ec987844b0108738a9ec6ba 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-3] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 39dc73882bec49a0bdd5d787b06ac032 first=1G1JD5SB5H4136951\x00\ x80\x00!\x8A\x80\x00\x01Z%\xF6\x7Ffef0b8faeeeb4a10103e1a67ea5ebdbec last=1GNKVHKD7HJ275239\x00\x80\x00$\x87\x80\x00\x01Z%\xF6s\ xDC0961566a370af3b7da440e9705bc4c8c 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-8] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ a37a2a56ff5c48399cf1abd92f99662f first=###########239824\x00\ x80\x01(\xFE\x80\x00\x01Z\xAE\xD6\xE0Xe5a45a2beab337228bdba90c06f34a12 last=1C4RJFLT6HC742023\x00\x80\x00x\xFF\x80\x00\x01[H\xF9w\ x8D60edb518c27ef80f8a751701926d9174 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-10] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ b5843a62a6bd47fbbfc29303bee158e3 first=5FNRL5H4XFB033259\x00\ x80\x00\x1EZ\x80\x00\x01\x5C"\x87s\xF5ce24ec7e2a3698836386bccabc1265af last=5NPEB4AC3DH620091\x00\x80\x00\xE0\x16\x80\x00\x01X\xE4\x1B\ x9Dq95568f371c1ebd06c497df7129f248a2 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-5] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 595e89b2fae8494d8a878bc6ba306e2f first=JTHBK1GG4E2122477\x00\ x80\x00$\xE4\x80\x00\x01[\xAF\xF0]%d306ddb81ea3bc093c40efe9f198f03a last=KNDPMCAC1H7201793\x00\x80\x00\x1E\xC7\x80\x00\x01X\xD5\xF5\xEA\ xD183ff28e7e1aeeff315b167d3d869b797 2017-05-24 18:00:10,348 INFO [LoadIncrementalHFiles-9] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ ac4edbfdf7394e88aaf84f5489511b7e first=1GNKVHKD7HJ275239\x00\ x80\x00$\x87\x80\x00\x01Z+\x1C\x87\xB65ab1c776dde09cf3257821e420288075 last=1N4BZ0CP4GC308715\x00\x80\x01T\xFC\x80\x00\x01U\xE2\xA1 \ xB743fe1c3b744fae3c2f9e6a896003869f 2017-05-24 18:00:10,348 INFO [LoadIncrementalHFiles-4] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 5580694921e846949f49c1426162da6e first=2GKALMEK7H6220949\x00\ x80\x00!\x1A\x80\x00\x01Y!\xE5\xCF\xC110e1f23860ff6a21fe848bb85e2e0969 last=3C7WRNAJ6GG313342\x00\x80\x00NB\x80\x00\x01V}\x1D\xF5\ x10620faf6227fb4181a7ee99fa07a52f08 2017-05-24 18:00:10,349 INFO [LoadIncrementalHFiles-7] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 7bc03c5b15304ec083e18c2376789562 first=1FMCU9J90EUB68140\x00\ x80\x01X\xA4\x80\x00\x01[\x1D&\x9D\x12f80b9cac0af584642ecbd6e1b1bbd617 last=1G1JD5SB5H4136951\x00\x80\x00!\x8A\x80\x00\x01Z$\xB0\xAC\ x1625dc1ae357cb3e9332d823210102de85 2017-05-24 18:00:10,349 INFO [LoadIncrementalHFiles-13] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ d28c4d2668cc4169b201b349b2522f9a first=3N1CE2CP2FL407531\x00\ x80\x00\x0A\xC9\x80\x00\x01X\xD1\xBC\xD195d73fa7cd13082d2e6f590ca1578b453 last=5FNRL5H4XFB033259\x00\x80\x00\x1EZ\x80\x00\x01\x5C!8\xA7. dd776deeea36c2c5b132fafed7013b16 2017-05-24 18:00:10,350 INFO [LoadIncrementalHFiles-6] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 5d90de564af645f084319a03f64b0a3d first=KNDPMCAC1H7201793\x00\ x80\x00\x1E\xC7\x80\x00\x01X\xD7F#\xEBcd2872ac4f258a091a79a55d4dbfef9e last=WMWZP3C58FT708786\x00\x80\x00\xE0\x16\x80\x00\x01Y\xB7?\xCA\ xADb28da871b5581e409561c813ed9c0805 2017-05-24 18:00:11,780 WARN [LoadIncrementalHFiles-15] mapreduce.LoadIncrementalHFiles: Attempt to bulk load region containing into table BI.VIN_IDX with files [family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ d28c4d2668cc4169b201b349b2522f9a, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/23df78bafd304ff887385a2b6becf06d, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 39dc73882bec49a0bdd5d787b06ac032, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/a37a2a56ff5c48399cf1abd92f99662f, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 11627a2861e3446e9d6f684ab534563e, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/ac4edbfdf7394e88aaf84f5489511b7e, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ f0e97b218aed4abf9949cf49a57e559b, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/7bc03c5b15304ec083e18c2376789562, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 5d90de564af645f084319a03f64b0a3d, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/5580694921e846949f49c1426162da6e, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ 595e89b2fae8494d8a878bc6ba306e2f, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/27c0a905ee174c9898d324acf1554bf9, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ c79ae6d27824424f99523dad586e86b1, family:I path:hdfs://HDFS1/user/hbase/ idx_test_7/BI.VIN_IDX/I/b5843a62a6bd47fbbfc29303bee158e3, family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/ d0a6c4b727bb416f840ed254658f3982] failed. This is recoverable and they will be retried. 2017-05-24 18:00:11,793 INFO [main] mapreduce.LoadIncrementalHFiles: Split occured while grouping HFiles, retry attempt 1 with 15 files remaining to group or split -- Thanks & Regards, Anil Gupta -- Thanks & Regards, Anil Gupta --f403045f50b608cc1e055062c571--