hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Frank Chow" <zhoushuaif...@gmail.com>
Subject bulkLoadHFiles failed because write lock on HRegion
Date Sun, 18 May 2014 06:30:40 GMT
Hi guys,

Do bulkload Hfiles sometimes failed on my cluster because rpc time out.
I found that when doing bulkload hfiles, it require wirtelock on the region when there are
multi-families. But the compact operation also require read lock on the region. When the compacting
files are big, it need more time to finish (some times cost half an hour or more). So when
the region is compacting, the bulkloadHfiles operation will not be able to get the writelock,
and causing the bulkloading hfiles timeout.

Code is below:
HRegion:
public boolean bulkLoadHFiles(List<Pair<byte[], String>> familyPaths,
      BulkLoadListener bulkLoadListener) throws IOException {
    Preconditions.checkNotNull(familyPaths);
    // we need writeLock for multi-family bulk load
    startBulkRegionOperation(hasMultipleColumnFamilies(familyPaths));

private void startBulkRegionOperation(boolean writeLockNeeded)
      throws NotServingRegionException, RegionTooBusyException, InterruptedIOException {
    if (this.closing.get()) {
      throw new NotServingRegionException(regionInfo.getRegionNameAsString() +
          " is closing");
    }
    if (writeLockNeeded) lock(lock.writeLock());
    else lock(lock.readLock());

My question is: why write lock is needed when there is multi-families ? Can readlock also
work? If it only need a readlock, there will be no conflict and loading hfiles will not timeout.

Thanks.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message