hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erdem Agaoglu <erdem.agao...@gmail.com>
Subject Re: Node failure causes weird META data?
Date Tue, 02 Nov 2010 16:02:54 GMT
We reproduced the problem by kill -9ing the region server that hosts -ROOT-
again. Results are the same with what i have explained before. After log
splitting metaScanner complained that .META. is not valid and reassigned it
to somewhere else. As a result, the number of rows in .META. reduced from 24
to 16 meaning we lost access to some regions and some table definitions. To
support what you have said about the data in the memstore, we seem to have
lost only recent regions.

status 'detailed' showed no regionsInTransition, and showed a total of 39
regions assigned to region servers, but as i've said before, we have access
to only 16 of them. No need to mention our queries fail.

Any idea we could try is greatly appreciated. Thanks in advance.

PS. Something probably irrelevant happened this time, interface 60010 lists
all our tables while 'list' command in shell lists only a few. We get
TableNotFoundExceptions on tables shown in web interface but missing in
'list' command output. Seems like web interface is able to list inaccessible
tables.

On Tue, Nov 2, 2010 at 9:56 AM, Erdem Agaoglu <erdem.agaoglu@gmail.com>wrote:

> By "lost tables" i mean no user table definitions were listed in interface
> 60010 and my queries got me TableNotFoundExceptions. Routine BaseScanner
> logged like
>
> # RegionManager.metaScanner scan of 0 row(s) of meta region ... complete
>
> so i guess my .META. was empty. But unfortunately I don't know if any
> regions were stuck in transition, i'll be sure to check this out while i try
> to reproduce.
>
> I rechecked the logs, specifically after the splitting completed and it
> contains lines like "Current assignment is not valid..." so i guess this is
> something unexpected. But in hopes of some configuration error you may spot,
> whole log goes like that (stripped to be more readable):
>
> # hlog file splitting completed in 80372 millis
> # Log split complete, meta reassignment and scanning:
> # ProcessServerShutdown reassigning ROOT region
> # -ROOT- region unset (but not set to be reassigned)
> # ROOT inserted into regionsInTransition
> # Assigning for serverName=C...  total nregions to assign=1, regions to
> give other servers than this=0, isMetaAssign=true
> # Assigning serverName=C... 1 regions
> # Assigning region -ROOT-... to C...
> # Processing MSG_REPORT_OPEN: -ROOT-... from C...; 1 of 1
> # RegionManager.rootScanner scanning meta region {server: C ...
> # Current assignment of .META.... is not valid;  serverAddress=A,
> startCode=blah unknown; checking once more!
> # Current assignment of .META.... is not valid;  serverAddress=A,
> startCode=blah unknown.
> # RegionManager.rootScanner scan of 1 row(s) of meta region {server: C ...
> complete
> # RegionManager.rootScanner scanning meta region {server: C ...
> # RegionManager.rootScanner scan of 1 row(s) of meta region {server: C ...
> complete
> # Assigning for serverName=E...  total nregions to assign=-1, regions to
> give other servers than this=2, isMetaAssign=true
> # Assigning serverName=E... -1 regions
> # Assigning region .META.... to E...
> # Processing MSG_REPORT_OPEN: .META.... from E...; 1 of 1
> # Processing todo: PendingOpenOperation from E...
> # .META.... open on E
> # Updated row .META.... in region -ROOT- ...
> # Adding to onlineMetaRegions: {server: E ...
> # RegionManager.metaScanner scanning meta region {server: E ...
> # RegionManager.metaScanner scan of 0 row(s) of meta region {server: E ...
> # All 1 .META. region(s) scanned
> 10 secs later
> # Processing todo: ProcessServerShutdown of F
> # Process shutdown of server F...: logSplit: true, rootRescanned: false,
> numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> # Log split complete, meta reassignment and scanning
> # Process server shutdown scanning root region on C
> # Process server shutdown scanning root region on C finished master
> # process server shutdown scanning .META.,,1 on E
> # process server shutdown finished scanning .META.,,1 on E
> # Removed F... from deadservers Map
>
> Thanks again.
>
> On Mon, Nov 1, 2010 at 6:58 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> The fact that the tables are "revived" is a clue here IMO, but let's
>> go back to more basic questions...
>>
>> So when you say that during step 1 you lost tables, what do you mean
>> by "lost"? Were the rows of those tables still in .META.? Were the
>> regions stuck in transition (in the shell, do: status 'detailed')? Or
>> when you tried to query them you just got a TableNotFoundException?
>>
>> Also, the fact that only -ROOT- and not .META. was on this region
>> server means that if there was any data lost, it would be .META.'s
>> location and it would have been assigned somewhere else (E), but still
>> stayed assigned on A. Since the data is in the memstore, recent data
>> couldn't be read by this second assignment of .META. but... it could
>> also be reassigned for a "normal" reason like rebalancing. The best
>> way to confirm that is when the -ROOT- region gets reassigned at the
>> end of step 1 (so this is after the message that goes like "...file
>> splitting completed in 80372..."), do so see something like this in
>> the master's log: "Current assignment of .META.,,some_timestamp  is
>> not valid; serverAddress=blah, startcode=blah unknown."? If so, then
>> it seems that data was lost and this is really unexpected.
>>
>> J-D
>>
>> On Mon, Nov 1, 2010 at 1:36 AM, Erdem Agaoglu <erdem.agaoglu@gmail.com>
>> wrote:
>> > Hi again,
>> >
>> > I have re-checked our configuration to confirm that we have
>> > dfs.support.append enabled and saw "Using syncFs -- HDFS-200" in logs. I
>> > inspected logs around log splits to find something, but i'm not sure
>> that's
>> > what we are looking for. In the first step of the scenario i mentioned
>> > before (where we kill -9ed everything on the node that hosts the ROOT
>> > region), HLog says (stripping hdfs:// prefixes and hostnames for
>> clarity)
>> >
>> > # Splitting 7 hlog(s) in .logs/F,60020,1287491528908
>> >
>> > Then it goes over every single one like
>> >
>> > # Splitting hlog 1 of 7
>> > # Splitting hlog 2 of 7
>> > # ...
>> > # Splitting hlog 7 of 7
>> >
>> > On the 7th hlog it WARNs with two lines
>> >
>> > # File .logs/F,60020,1287491528908/10.1.10.229%3A60020.1288021443546
>> might
>> > be still open, length is 0
>> > # Could not open .logs/F,60020,1287491528908/10.1.10.229
>> %3A60020.1288021443546
>> > for reading. File is emptyjava.io.EOFException
>> >
>> > And completes with
>> >
>> > # log file splitting completed in 80372 millis for
>> > .logs/F,60020,1287491528908
>> >
>> > This might be it, but on the sixth step (where we kill -9ed the
>> RegionServer
>> > that hosts the only META region), it splits 2 hlogs without any empty
>> file
>> > problems nor any log above INFO, but as i told before, our testtable
>> still
>> > got lost.
>> >
>> > I'll try to reproduce the problem in a cleaner way, but in the meantime,
>> any
>> > kind of pointers to any problems we might have is greatly appreciated.
>> >
>> >
>> > On Fri, Oct 29, 2010 at 9:25 AM, Erdem Agaoglu <erdem.agaoglu@gmail.com
>> >wrote:
>> >
>> >> Thanks for the answer.
>> >>
>> >> I am pretty sure we have dfs.support.append enabled. I remember both
>> the
>> >> conf file and the logs, and don't recall seeing any errors on 60010. I
>> >> crawled through logs all yesterday but don't remember anything
>> indicating a
>> >> specific error too. But i'm not sure about that. Let me check that and
>> get
>> >> back here on monday.
>> >>
>> >> On Thu, Oct 28, 2010 at 7:30 PM, Jean-Daniel Cryans <
>> jdcryans@apache.org>wrote:
>> >>
>> >>> First thing I'd check is if your configuration has dfs.support.append,
>> >>> you can confirm this by looking at your region server logs. When a RS
>> >>> starts, it creates an HLog and will print out: "Using syncFs --
>> >>> HDFS-200" if it's configured, else you'll see "syncFs -- HDFS-200 --
>> >>> not available, dfs.support.append=false". Also the master web ui (on
>> >>> port 60010) will print an error message regarding that.
>> >>>
>> >>> If it's all ok, then you should take a look at the master log when it
>> >>> does the log splitting and see if it contains any obvious errors.
>> >>>
>> >>> J-D
>> >>>
>> >>> On Thu, Oct 28, 2010 at 12:58 AM, Erdem Agaoglu <
>> erdem.agaoglu@gmail.com>
>> >>> wrote:
>> >>> > Hi all,
>> >>> >
>> >>> > We have a testing cluster of 6 nodes which we try to run an
>> >>> HBase/MapReduce
>> >>> > application on. In order to simulate a power failure we kill -9ed
>> all
>> >>> things
>> >>> > hadoop related on one of the slave nodes (DataNode, RegionServer,
>> >>> > TaskTracker, ZK quorum peer and i think SecondaryNameNode was on
>> this
>> >>> node
>> >>> > too). We were expecting a smooth transition on all services but
were
>> >>> unable
>> >>> > to get on HBase end. While our regions seemed intact (not
>> confirmed), we
>> >>> > lost table definitions that pointed some kind of META region fail.
>> So
>> >>> our
>> >>> > application failed with several TableNotFoundExceptions. Simulation
>> was
>> >>> > conducted with no-load and extremely small data (like 10 rows in
3
>> >>> tables).
>> >>> >
>> >>> > On our setup, HBase is 0.89.20100924, r1001068 while Hadoop
>> >>> > runs 0.20.3-append-r964955-1240, r960957. Most of the configuration
>> >>> > parameters are in default.
>> >>> >
>> >>> > If we did something wrong up to this point, please ignore the rest
>> of
>> >>> the
>> >>> > message as i'll try to explain what we did to reproduce it and
might
>> be
>> >>> > irrelevant.
>> >>> >
>> >>> > Say the machines are named A, B, C, D, E, F; where A is master-like
>> >>> node,
>> >>> > others are slaves and power fail is on F. Since we have little
data,
>> we
>> >>> have
>> >>> > one ROOT and only one META region. I'll try to sum up the whole
>> >>> scenario.
>> >>> >
>> >>> > A: NN, DN, JT, TT, HM, RS
>> >>> > B: DN, TT, RS, ZK
>> >>> > C: DN, TT, RS, ZK
>> >>> > D: DN, TT, RS, ZK
>> >>> > E: DN, TT, RS, ZK
>> >>> > F: SNN, DN, TT, RS, ZK
>> >>> >
>> >>> > 0. Initial state -> ROOT: F, META: A
>> >>> > 1. Power fail on F -> ROOT: C, META: E -> lost tables, waited
for
>> about
>> >>> half
>> >>> > an hour to get nothing BTW
>> >>> > 2. Put F back online -> No effect
>> >>> > 3. Create a table 'testtable' to see if we lose it
>> >>> > 4. Kill -9ed DataNode on F -> No effect -> Start it again
>> >>> > 5. Kill -9ed RegionServer on F -> No effect -> Start it again
>> >>> > 6. Kill -9ed RegionServer on E -> ROOT: C, META: A -> We
lost
>> >>> 'testtable'
>> >>> > but get our tables from before the simulation. It seemed like
>> because A
>> >>> had
>> >>> > META before the simulation, the table definitions were revived.
>> >>> > 7. Restarted the whole cluster -> ROOT: A, META: F -> We
lost 2 out
>> of
>> >>> our
>> >>> > original 6 tables, 'testtable' revived. That small data seems
>> corrupted
>> >>> too
>> >>> > as our Scans don't finish.
>> >>> > 8. Run to mailing-list.
>> >>> >
>> >>> > First of all thanks for reading up to this point. From what we
are
>> now,
>> >>> we
>> >>> > are not even sure if this is the expected behavior, like if ROOT
or
>> META
>> >>> > region dies we lose data and must do sth like hbck, or if we are
>> missing
>> >>> a
>> >>> > configuration, or if this is a bug. No need to mention that we
are
>> >>> > relatively new to HBase so the last possibility is that if we didn't
>> >>> > understand it at all.
>> >>> >
>> >>> > Thanks in advance for any ideas.
>> >>> >
>> >>> > --
>> >>> > erdem agaoglu
>> >>> >
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> erdem agaoglu
>> >>
>> >
>> >
>> >
>> > --
>> > erdem agaoglu
>> >
>>
>
>
>
> --
> erdem agaoglu
>



-- 
erdem agaoglu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message