From user-return-55023-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Mon Mar 19 21:47:14 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6724A180647 for ; Mon, 19 Mar 2018 21:47:13 +0100 (CET) Received: (qmail 47015 invoked by uid 500); 19 Mar 2018 20:47:11 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 46998 invoked by uid 99); 19 Mar 2018 20:47:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Mar 2018 20:47:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id AA30C1A16D6 for ; Mon, 19 Mar 2018 20:47:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id AwOPsrTt5LtH for ; Mon, 19 Mar 2018 20:47:07 +0000 (UTC) Received: from mail-qk0-f169.google.com (mail-qk0-f169.google.com [209.85.220.169]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E67A75F47B for ; Mon, 19 Mar 2018 20:47:06 +0000 (UTC) Received: by mail-qk0-f169.google.com with SMTP id g184so19842174qkd.10 for ; Mon, 19 Mar 2018 13:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=lEl2JGyqwwfdIpM/nEdDyU4JURuExComnRj0nOBR304=; b=CpNs5AjKLc9XMn+N+RLnwSWU9Yz707EPsC1HKeo8CC1Bpd5rZME49Rot5sfHqTtX6s MiE7jN5o5Cdce81h/kh5uvd/Si+ZNgpzCn2IViEPy+eJpKO9ST4IBALUmbH0cA+Ldm2o U2uYKZVyoHYO93VbT+bp4miJMHOWml9Dlhimzbn67rG52zXZwcG8IF58ER6BR8jl7ZYF EDM9xCClJ9qOnv+0Zz36yj68MZvC0JPhveL3XkdffxFJRX45LzyJGM8IwVIwH5yeLonE BmK0hHUi48M+CP3qpccfRTLil3Di4oKH5M0PFNtA1LzGoYtFxSmzCx5j690V1oZfhucy yjFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=lEl2JGyqwwfdIpM/nEdDyU4JURuExComnRj0nOBR304=; b=PDX85azUJ9e9nXQ715yPVYoruzIeE/kzhFEL0X0aJzgK8lRMUdH5n0llD0U7RhXBrG z58NGb1RAxkxckr88CPU34djrFUnkyq0jzMl+DR4UXn6N7PKurWnWbQOsE2L1qUZCMMx zazJVrq4MIeb2RzyY4NuF97Qlw1x2BdUIqbCbSrE9LF4IZXctvnOuJ1mUPkiSp6mVyOZ 3m3a9Gxd38lfvqKGTfFnVEOak4USs0s8vapYPuwX2higsMJNNPW74kM72kSzlUtgCSey bsuvwddeE9tMxbBb9TOoeY3ueY+8Vk2wREPdSMXSpMx4sspIuw4b42gLUCxz8p+Snw5Y /b7A== X-Gm-Message-State: AElRT7HNAcqjAPxxRAjWrTlQqTK5FYdEBoOcF7eUchTIRuIEc3LI862r iwECfkEJFx6aJaovpP75TrPrTj+HcEj8nm4e8Bp0QQ== X-Google-Smtp-Source: AG47ELvbBDM5XwrjTyuMXeNTWLjpIBBfzYd69zr4EjRvT0w0lqorZE+8AvbOfSTJr7r2rM6KCYC4rJx5jKBuDnDh684= X-Received: by 10.55.209.216 with SMTP id o85mr19707132qkl.217.1521492425508; Mon, 19 Mar 2018 13:47:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.174.12 with HTTP; Mon, 19 Mar 2018 13:47:04 -0700 (PDT) In-Reply-To: References: <3FDDDDF2-9B9C-43E9-8A21-E0AF086D7718@cloudera.com> From: Saad Mufti Date: Mon, 19 Mar 2018 16:47:04 -0400 Message-ID: Subject: Re: CorruptedSnapshotException Taking Snapshot Of Table With Large Number Of Files To: user@hbase.apache.org Content-Type: multipart/alternative; boundary="001a1147991e8d0af50567ca0fc8" --001a1147991e8d0af50567ca0fc8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks, I tried briefly but maybe I didn't do quite the right search. In any case, thanks for the help. ---- Saad On Mon, Mar 19, 2018 at 2:50 PM, Huaxiang Sun wrote: > You can google search the exception stack and mostly it will find the JIR= A. > > Regards, > > Huaxiang > > > On Mar 19, 2018, at 10:52 AM, Saad Mufti wrote: > > > > Thanks!!! Wish that was documented somewhere in the manual. > > > > Cheers. > > > > ---- > > Saad > > > > > > On Mon, Mar 19, 2018 at 1:38 PM, Huaxiang Sun wrote= : > > > >> Mostly it is due to HBASE-15430 >> jira/browse/HBASE-15430>, =E2=80=9Csnapshot.manifest.size.limit=E2=80= =9D needs to be > >> configured as 64MB or 128MB. > >> > >> Regards, > >> > >> Huaxiang Sun > >> > >> > >>> On Mar 19, 2018, at 10:16 AM, Saad Mufti wrote= : > >>> > >>> Hi, > >>> > >>> We are running on HBase 1.4.0 on an AWS EMR/HBase cluster. > >>> > >>> We have started seeing the following stacktrace when trying to take a > >>> snapshot of a table with a very large number of files (12000 regions > and > >>> roughly 360000 - 400000 files). The number of files should go down as > we > >>> haven't been compacting for a while for other operational reasons and > are > >>> now running it. But I'd to understand why our snapshots are failing > with > >>> the following: > >>> > >>> 2018-03-19 16:05:56,948 ERROR > >>>> [MASTER_TABLE_OPERATIONS-ip-10-194-208-6:16000-0] > >>>> snapshot.TakeSnapshotHandler: Failed taking snapshot { > >>>> ss=3Dpgs-device.03-19-2018-15 table=3Dpgs-device type=3DSKIPFLUSH } = due to > >>>> exception:unable to parse data manifest Protocol message was too > >> large. May > >>>> be malicious. Use CodedInputStream.setSizeLimit() to increase the > size > >>>> limit. > >>>> > >>>> org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: unable > to > >>>> parse data manifest Protocol message was too large. May be maliciou= s. > >> Use > >>>> CodedInputStream.setSizeLimit() to increase the size limit. > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest( > >> SnapshotManifest.java:468) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest. > >> load(SnapshotManifest.java:297) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest. > >> open(SnapshotManifest.java:129) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier. > >> verifySnapshot(MasterSnapshotVerifier.java:108) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process( > >> TakeSnapshotHandler.java:203) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.executor.EventHandler.run( > >> EventHandler.java:129) > >>>> > >>>> at > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1149) > >>>> > >>>> at > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:624) > >>>> > >>>> at java.lang.Thread.run(Thread.java:748) > >>>> > >>>> Caused by: com.google.protobuf.InvalidProtocolBufferException: > Protocol > >>>> message was too large. May be malicious. Use > >>>> CodedInputStream.setSizeLimit() to increase the size limit. > >>>> > >>>> at > >>>> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded= ( > >> InvalidProtocolBufferException.java:110) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.refillBuffer( > >> CodedInputStream.java:755) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readRawBytes( > >> CodedInputStream.java:811) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readBytes( > >> CodedInputStream.java:329) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1313) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1263) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile$1.parsePartialFrom( > >> SnapshotProtos.java:1364) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile$1.parsePartialFrom( > >> SnapshotProtos.java:1359) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readMessage( > >> CodedInputStream.java:309) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2161) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2103) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom( > >> SnapshotProtos.java:2197) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom( > >> SnapshotProtos.java:2192) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readMessage( > >> CodedInputStream.java:309) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest.(SnapshotProtos.java:1165) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest.(SnapshotProtos.java:1094) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readMessage( > >> CodedInputStream.java:309) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest.(SnapshotProtos.java:3858) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest.(SnapshotProtos.java:3792) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889) > >>>> > >>>> at > >>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89) > >>>> > >>>> at > >>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95) > >>>> > >>>> at > >>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest.parseFrom(SnapshotProtos.java:4115) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest( > >> SnapshotManifest.java:464) > >>>> > >>>> : > >>>> > >>>> ... 8 more > >>>> > >>>> > >>> Thanks. > >>> > >>> ---- > >>> Saad > >> > >> > > --001a1147991e8d0af50567ca0fc8--