hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16567) Upgrade to protobuf3
Date Wed, 07 Sep 2016 01:12:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469159#comment-15469159
] 

stack commented on HBASE-16567:
-------------------------------

Just a note to say that upgrading our pb to pb3 w/o running protoc3, we do pretty good but
fail reading trailers on hfiles. We can't find CellComparator. TODO: check why.

{code}
2016-09-06 18:06:42,856 ERROR [RS_OPEN_META-localhost:55129-0] handler.OpenRegionHandler:
Failed open of region=hbase:meta,,1.1588230740, starting to roll back the global memstore
size.
java.io.IOException: java.io.IOException: org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
Problem reading HFile Trailer from file file:/var/folders/cj/jgfy62h13vz019xgz681df_r0000gp/T/hbase-stack/hbase/data/hbase/meta/1588230740/info/47a6d1f4adb24f6e92de8dbe9e9c144f
       	at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:831)
       	at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:732)
       	at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:705)
       	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4750)
       	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4721)
       	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4693)
       	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4649)
       	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4600)
       	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:276)
       	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:103)
       	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
       	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file file:/var/folders/cj/jgfy62h13vz019xgz681df_r0000gp/T/hbase-stack/hbase/data/hbase/meta/1588230740/info/47a6d1f4adb24f6e92de8dbe9e9c144f
       	at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:544)
       	at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:499)
       	at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:267)
       	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3652)
       	at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:805)
       	at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:802)
       	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       	... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer
from file file:/var/folders/cj/jgfy62h13vz019xgz681df_r0000gp/T/hbase-stack/hbase/data/hbase/meta/1588230740/info/47a6d1f4adb24f6e92de8dbe9e9c144f
       	at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:477)
       	at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:505)
       	at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1033)
       	at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:241)
       	at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:365)
       	at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:462)
       	at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:629)
       	at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:123)
       	at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
       	at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
       	... 6 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.CellComparator$MetaCellComparator
       	at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:581)
       	at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserializeFromPB(FixedFileTrailer.java:300)
       	at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserialize(FixedFileTrailer.java:242)
       	at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:407)
       	at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:462)
       	... 15 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.CellComparator$MetaCellComparator
       	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
       	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
       	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
       	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
       	at java.lang.Class.forName0(Native Method)
       	at java.lang.Class.forName(Class.java:264)
       	at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:579)
       	... 19 more
{code}

> Upgrade to protobuf3
> --------------------
>
>                 Key: HBASE-16567
>                 URL: https://issues.apache.org/jira/browse/HBASE-16567
>             Project: HBase
>          Issue Type: Task
>          Components: Protobufs
>    Affects Versions: 2.0.0
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>         Attachments: HBASE-16567.master.001.patch
>
>
> Move master branch on to protobuf3. See https://github.com/google/protobuf/releases We'd
do it because pb3 saves some on byte copies can work with offheap buffers -- needed for the
off-heap write path project -- thought read-time is still a TODO.
> HBASE-15638 has us first shading protobufs before upgrading. Let us list here issues
just going to pb3 without shading if only for completeness sake; i.e. do we have to shade?
>  * pb3 is by default wire compatible with pb2.
>  * protoc3 run against our .protos works fine except pb3 breaks our HBaseZeroCopyLiteralByteString
hack.
>  * Starting up a cluster that is all pb3'd seems to work fine.
>  * A pb2 branch-1 can read and write against the pb3 master cluster.
> What will break if we just upgrade to pb3?
>  * We should be able to write HDFS messages on our AsyncWAL using pb3; the pb2 HDFS should
be able to  read them (not tested). Or maybe not. See policy here: https://github.com/google/protobuf/issues/1852
>  * Core Coprocessor Endpoints such as AccessControl seem to just work (their protos will
have been protoc3'd). I did simple test with a server from master branch up on pb3 and then
going against it with a branch-1 client on pb2. I was able to add grants.
>  * For non-core CPEPs where the protos are pb2 still, it might just work. To test. It
would not be the end-of-the-world if they did not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message