hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matteo Bertozzi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14783) Proc-V2: Master aborts when downgrading from 1.3 to 1.1
Date Fri, 06 Nov 2015 22:42:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14994605#comment-14994605
] 

Matteo Bertozzi commented on HBASE-14783:
-----------------------------------------

there are 3 problems here
 - on a not clean shutdown your cluster is in a non consistent state, starting the master
and ignoring the exception is a bad idea. you have run hbck or you can restart the cluster
and do a clean shutdown before the downgrade.
 - on a clean shutdown and restart with lower version you will not have procedure running.
so nothing to load and you'll not hit this problem

the 3rd is how do we load, completed procedures (but not yet deleted) are loaded using convert()
which creates a Procedure instance, but we don't really need that. in fact we remove that
instance later when we realize that we just need the result. 

> Proc-V2: Master aborts when downgrading from 1.3 to 1.1
> -------------------------------------------------------
>
>                 Key: HBASE-14783
>                 URL: https://issues.apache.org/jira/browse/HBASE-14783
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>            Assignee: Stephen Yuan Jiang
>
> I was running ITBLL with 1.3 deployed on a 6 node cluster.
> Then I stopped the cluster, deployed 1.1 release and tried to start cluster.
> However, master failed to start due to:
> {code}
> 2015-11-06 00:58:40,351 FATAL [eval-test-2:20000.activeMasterManager] master.HMaster:
Failed to become active master
> java.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure
must be accessible and have an empty constructor
>   at org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:548)
>   at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:640)
>   at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:105)
>   at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:82)
>   at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:298)
>   at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:275)
>   at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:434)
>   at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1208)
>   at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1107)
>   at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:694)
>   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:186)
>   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1713)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:191)
>   at org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:536)
>   ... 12 more
> {code}
> The cause was that ServerCrashProcedure, written in some WAL file under MasterProcWALs
from first run, was absent in 1.1 release.
> After a brief discussion with Stephen, I am logging this JIRA to solicit discussion on
how customer experience can be improved if downgrade of hbase is performed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message