hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matteo Bertozzi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
Date Mon, 14 Mar 2016 00:30:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192584#comment-15192584
] 

Matteo Bertozzi commented on HBASE-15430:
-----------------------------------------

that's a good point. It wasn't particularly slow on my machine ~5sec but yeah it may be faster.
I think TEST_UTIL.createLocalHRegion() is writing on fs anyway, less than the SnapshotMock
because with the createLocalHRegion we end up with a snapshot of empty regions.
It will be nice to exercise the normal write path as SnapshotMock does. An InMemoryFilesystem
for testing will do the trick. but I don't think we have one, so that needs to be added (and
it will probably be useful in many other places). 
any other alternatives? even changes to the SnapshotManifest class to allow testing without
writing as much are welcome.
if no other option we can do the Manifest manually. the createLocalHRegion() is not needed
since that is used only for the HRegionInfo, which can be create alone. and then use ProtobufUtil
toTableName() for tableName conversion. 

> Failed taking snapshot - Manifest proto-message too large
> ---------------------------------------------------------
>
>                 Key: HBASE-15430
>                 URL: https://issues.apache.org/jira/browse/HBASE-15430
>             Project: HBase
>          Issue Type: Bug
>          Components: snapshots
>    Affects Versions: 0.98.11
>            Reporter: JunHo Cho
>            Assignee: JunHo Cho
>            Priority: Critical
>             Fix For: 0.98.18
>
>         Attachments: HBASE-15430-v4.patch, hbase-15430-v1.patch, hbase-15430-v2.patch,
hbase-15430-v3.branch.0.98.patch, hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot meta is over
64MB. 
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed taking snapshot
{ ss=snapshot_xxx table=xxx type=FLUSH } due to exception:Protocol message was too large.
 May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.:com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to
increase the size limit.
>         at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
>         at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
>         at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
>         ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
>         at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
>         at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
>         at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
>         at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
>         at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.<init>(HBaseProtos.java:3767)
>         at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.<init>(HBaseProtos.java:3699)
>         at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
>         at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.<init>(SnapshotProtos.java:1152)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.<init>(SnapshotProtos.java:1094)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.<init>(SnapshotProtos.java:3858)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.<init>(SnapshotProtos.java:3792)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
>         at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>         at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
>         at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>         at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>         at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
>         at org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
>         at org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
>         at org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
>         at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message