hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: HBase exception after upgrading from hbase-0.90-branch to trunk
Date Thu, 18 Aug 2011 17:45:47 GMT
Can you redo with logs at DEBUG level?

This seems like a migration issue -- i.e. migrating the catalog tables
from old format to new -- but from what I can see in the logs, its
more an issue w/ state transitions in zk... I need the DEBUG logging
to tell for sure.

St.Ack

On Thu, Aug 18, 2011 at 8:27 AM, Sebastian Bauer <admin@ugame.net.pl> wrote:
> Hi, comments inline:
>
> On 16.08.2011 06:42, Stack wrote:
>>
>> Oh, are you running on hbase trunk?  If so, did you write all your
>> data with TRUNK or did you start up 0.92 on a data that was written w/
>> 0.90 (It should work but you may have hit an issue).
>
> I had branch-0.90 and after proper shutdown i had updated hbase to trunk, it
> looks like -ROOT- table was migrated but .META. had problem with saving
> information about migration in logs i had informations:
>
> 2011-08-18 17:16:09,334 INFO org.apache.hadoop.hbase.master.HMaster: .META.
> assigned=2, rit=false, location=lala.golden,60020,1313680556932
> 2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster:
> metaUpdated = NULL.
> 2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster: Meta
> updated status = false
> 2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster: Meta
> has HRI with HTDs. Updating meta now.
> 2011-08-18 17:16:17,535 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
> Updated -ROOT- row with metaMigrated status = true
> 2011-08-18 17:16:17,536 INFO org.apache.hadoop.hbase.master.HMaster: Meta
> updated with new HRI.
>
> and even after restarting i saw this in logs.
>
> ./hbase shell show all one regions, but any operation on tables show
> information that .META. table doesn't exist.
> Full logs from first and second run after upgrade are here:
>  http://s2.e-back.net/hbase_logs.tar.gz
>
> -ROOT- have this rows:
> hbase(main):005:0> scan '-ROOT-'
> ROW
> COLUMN+CELL
>  -ROOT-,,0
>  column=info:metamigrated, timestamp=1313680577531, value=\xFF
>  .META.,,1
>  column=info:regioninfo, timestamp=1278560826938, value=REGION => {NAME =>
> '.META.,,1 TableName => ', STARTKEY => '', ENDKEY => '', ENCODED =>
> 1028785192,}
>  .META.,,1
>  column=info:server, timestamp=1313680569288, value=lala.golden:60020
>  .META.,,1
>  column=info:serverstartcode, timestamp=1313680569288, value=1313680556932
> 2 row(s) in 0.0500 seconds
>
>
> On 16.08.2011 06:40, Stack wrote:
>>
>> What version of hbase is this Sebastian?
>>
>> What you mean by remove -ROOT- and .META. (that doesn't sound good but
>> you probably mean something else though looking at the exception
>> below, maybe this is what you did).
>
> Long time ago when i smashed my hbase instalation when i removed .META. and
> -ROOT- (./bin/hadoop fs -rmr /hbase/.META.) and run add_table.rb script all
> tables were online again.
>>
>> St.Ack
>>
>> On Thu, Aug 11, 2011 at 11:54 AM, Sebastian Bauer<admin@ugame.net.pl>
>>  wrote:
>>>
>>> 2011-08-11 20:53:55,889 WARN
>>> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
>>>
>>> Golden_ATU,m_ef06511a814155022686d50cebb0dba5_2010_9_40CC7C8170915ABC43432E524F0E6D2B,1301066006080.f656ca602b8638ad8329c36af85ae43e.
>>> to db2a.goldenline.pl,60020,1313088827048, trying to assign elsewhere
>>> instead; retry=
>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>>> java.lang.IllegalArgumentException: Can not create a Path from an empty
>>> string
>>>        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
>>>        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>>>        at org.apache.hadoop.fs.Path.<init>(Path.java:50)
>>>        at
>>> org.apache.hadoop.hbase.util.FSUtils.getTablePath(FSUtils.java:845)
>>>        at
>>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoPath(FSUtils.java:818)
>>>        at
>>>
>>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:832)
>>>        at
>>>
>>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127)
>>>        at
>>>
>>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
>>>        at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2329)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at
>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
>>>        at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1222)
>>>
>>>        at
>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:871)
>>>        at
>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141)
>>>        at $Proxy7.openRegion(Unknown Source)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:404)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1239)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1034)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1014)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:101)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:458)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:389)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:355)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:296)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:243)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:467)
>>>        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:300)
>>>
>>> 2011-08-11 20:53:55,895 WARN
>>> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
>>>
>>> Golden_ATU,m_ef06511a814155022686d50cebb0dba5_2010_9_40CC7C8170915ABC43432E524F0E6D2B,1301066006080.f656ca602b8638ad8329c36af85ae43e.
>>> to db2b.goldenline.pl,60020,1313088826553, trying to assign elsewhere
>>> instead; retry=
>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>>> java.lang.IllegalArgumentException: Can not create a Path from an empty
>>> string
>>>        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
>>>        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>>>        at org.apache.hadoop.fs.Path.<init>(Path.java:50)
>>>        at
>>> org.apache.hadoop.hbase.util.FSUtils.getTablePath(FSUtils.java:845)
>>>        at
>>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoPath(FSUtils.java:818)
>>>        at
>>>
>>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:832)
>>>        at
>>>
>>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127)
>>>        at
>>>
>>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
>>>        at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2329)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at
>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
>>>        at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1222)
>>>
>>>        at
>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:871)
>>>        at
>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141)
>>>        at $Proxy7.openRegion(Unknown Source)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:404)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1239)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1034)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1014)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:101)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:458)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:389)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:355)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:296)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:243)
>>>        at
>>>
>>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:467)
>>>        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:300)
>>>
>>> --
>>>
>>> Pozdrawiam
>>> Sebastian Bauer
>>> -----------------------------------------------------
>>> http://tikecik.pl
>>>
>>>
>

Mime
View raw message