hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bijieshan <bijies...@huawei.com>
Subject Re: A concurrency issue on SoftValueSortedMap?
Date Wed, 21 Dec 2011 11:17:01 GMT
Sorry, something important I missed in my last email.
SoftValueSortedMap uses TreeMap. It has some synchronized methods to edit the data from this
map. If we use these methods, it’s ok. But in HConnectionManager#getCachedLocation:

      // Cut the cache so that we only get the part that could contain
      // regions that match our key
      SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
        tableLocations.headMap(row);

      // if that portion of the map is empty, then we're done. otherwise,
      // we need to examine the cached location to verify that it is
      // a match by end key as well.
      if (!matchingRegions.isEmpty()) {
        HRegionLocation possibleRegion = null;
        try {
          possibleRegion = matchingRegions.get(matchingRegions.lastKey());
        } catch (NoSuchElementException nsee) {
          LOG.warn("checkReferences() might have removed the key", nsee);
        }
SoftValueSortedMap#get:
  public synchronized V get(Object key) {

      checkReferences();
    SoftValue<K,V> value = this.internalMap.get(key);
    if (value == null) {
      return null;
    }
    if (value.get() == null) {
      this.internalMap.remove(key);
      return null;
    }
    return value.get();
  }

In this method, it can remove the element from internalMap which is a heapMap from the backed
TreeMap. A concurrency issue may happen from here.
Correct me if am wrong.
Thank you.

Jieshan.

发件人: bijieshan
发送时间: 2011年12月21日 18:48
收件人: dev@hbase.apache.org; user@hbase.apache.org
抄送: wenzaohua; Chenjian
主题: A concurrency issue on SoftValueSortedMap?

Hi all,

We use thousand of threads doing the scan operations. One thread got blocked at the position
of “java.util.TreeMap.fixAfterDeletion”. It can’t come out of the loop of “TreeMap.fixAfterDeletion”:

So I think it maybe a concurrency issue. Has someone encountered this?
Thank you.

Jieshan.

Here’s the thread dump:

"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable [0x00007f3d05c1b000]
   java.lang.Thread.State: RUNNABLE
                at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
                at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
                at java.util.TreeMap.remove(TreeMap.java:585)
                at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
                at org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
                - locked <0x00007f3d92937748> (a org.apache.hadoop.hbase.util.SoftValueSortedMap)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
                at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
                at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
                at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
                at com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
                at com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
                at framework.QueryThread.run(QueryThread.java:47)

"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable [0x00007f3d05c1a000]
   java.lang.Thread.State: RUNNABLE
                at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
                at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
                at java.util.TreeMap.remove(TreeMap.java:585)
                at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
                at org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
                - locked <0x00007f3d94f24f70> (a org.apache.hadoop.hbase.util.SoftValueSortedMap)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
                at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
                at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
                at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
                at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
                at com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
                at com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
                at framework.QueryThread.run(QueryThread.java:47)


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message