hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhoushuaifeng <zhoushuaif...@huawei.com>
Subject Re: "NOT flushing memstore for region" keep on printing for half an hour
Date Tue, 26 Apr 2011 04:15:26 GMT
I think I may find out the problem:

1, When split region, it will close parent region, and set writestate.writesEnabled = false:

  private List<StoreFile> doClose(final boolean abort)
  throws IOException {
    synchronized (writestate) {
      // Disable compacting and flushing by background threads for this
      // region.
      writestate.writesEnabled = false;

2, If the memstore is large enouth, preflush will happen:

    if (!abort && !wasFlushing && worthPreFlushing()) {
      LOG.info("Running close preflush of " + this.getRegionNameAsString());
      internalFlushcache();
    }
    this.closing.set(true);
    lock.writeLock().lock();

3, IOException happened, and preflushing failed, and closing parent failed:
    createSplitDir(this.parent.getFilesystem(), this.splitdir);
    this.journal.add(JournalEntry.CREATE_SPLIT_DIR);

    List<StoreFile> hstoreFilesToSplit = this.parent.close(false);
if (hstoreFilesToSplit == null) {


4, roll back split is calling, but split state stay in "CREATE_SPLIT_DIR", so , only clenupSplitDir
will happen.
    while (iterator.hasPrevious()) {
      JournalEntry je = iterator.previous();
      switch(je) {
      case CREATE_SPLIT_DIR:
        cleanupSplitDir(fs, this.splitdir);
        break;

      case CLOSED_PARENT_REGION:

5, what about writestate.writesEnabled? it stayed in false, no one handle it. So, even split
is roll back, but no flush can success in parent region.


Zhou Shuaifeng(Frank)


-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2011年4月26日 11:41
收件人: dev@hbase.apache.org
抄送: Yanlijun
主题: Re: "NOT flushing memstore for region" keep on printing for half an hour

2011/4/25 Zhoushuaifeng <zhoushuaifeng@huawei.com>:
> Thanks St,
> I found that when running closing parent region to prepare for split, preflush occurred.
But for some reason, an IOException throwed from hdfsclient, and caused preflush failed. Then
split failed.
> We have made some change on HDFS and will check why Exception happened.
> But I don't know details on how hbase handle the exception. When rolling back the split
operation, will it reset writestate.writesEnabled to true?
> If not, it will hanging writestate.writesEnabled to false and causing all other flush
operations false.
>

If we fail a split, we rollback.  Usually the rollback is clean (There
is a unit test that exercises various failures splitting verifying
rollback works) but for sure there could be bugs in here.

I would be interested in the log of that regionserver.

Thanks,
St.Ack
Mime
View raw message