Return-Path: 
 <common-issues-return-62603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D47641115B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>;
 Thu, 12 Jun 2014 09:06:02 +0000 (UTC)
Received: (qmail 53850 invoked by uid 500); 12 Jun 2014 09:06:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53731 invoked by uid 500); 12 Jun 2014 09:06:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53720 invoked by uid 99); 12 Jun 2014 09:06:01 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jun 2014 09:06:01 +0000
Date: Thu, 12 Jun 2014 09:06:01 +0000 (UTC)
From: "Liang Xie (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12720822.1402563866139.116045.1402563961550@arcas>
In-Reply-To: <JIRA.12720822.1402563866139@arcas>
References: <JIRA.12720822.1402563866139@arcas>
Subject: [jira] [Created] (HADOOP-10687) improve Configuration.CACHE_CLASSES
 concurrentcy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Liang Xie created HADOOP-10687:
----------------------------------

             Summary: improve Configuration.CACHE_CLASSES concurrentcy
                 Key: HADOOP-10687
                 URL: https://issues.apache.org/jira/browse/HADOOP-10687
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
    Affects Versions: 2.4.0, 3.0.0
            Reporter: Liang Xie
            Assignee: Liang Xie


In one of my high concurrent HBase in-memory read only testing scenario, the following strace traces show CACHE_CLASSES becoming hotspot, since there's a "synchronized" or "synchronizedMap", but for most of hadoop applications, when goes into getClassByNameOrNull(), most of code paths should expect there's a cached class already, that means it's a read mostly scenario, the write should be rare enough. So one possible optimization here is using concurrent weakHashMap, the MapMaker could do this.
ps: i added more debug logging above "cache hit" and "Class.forName" and reran my case, shows an expected result (almost all of them go to "cache hit" path), and TestConfiguration also passed in my box.


{code}
"IPC Reader 3 on port 11600" daemon prio=10 tid=0x00007ff2fcb41190 nid=0x5db4 runnable [0x00007ff238943000]

java.lang.Thread.State: RUNNABLE
     at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
     - locked <0x00000002788ab4a0> (a java.util.Collections$SynchronizedMap)
     at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
     - locked <0x00000002784211e8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
"IPC Reader 1 on port 11600" daemon prio=10 tid=0x00007ff2fc8cb2b0 nid=0x5db2 waiting for monitor entry [0x00007ff29006c000]

java.lang.Thread.State: BLOCKED (on object monitor)
     at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
     - waiting to lock <0x00000002788ab4a0> (a java.util.Collections$SynchronizedMap)
     at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
     - locked <0x0000000278434980> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
"IPC Reader 0 on port 11600" daemon prio=10 tid=0x00007ff2fc9b7630 nid=0x5db1 waiting for monitor entry [0x00007ff2900ad000]

java.lang.Thread.State: BLOCKED (on object monitor)
     at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
     - waiting to lock <0x00000002788ab4a0> (a java.util.Collections$SynchronizedMap)
     at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
     - locked <0x00000002784349e0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
"IPC Reader 8 on port 11600" daemon prio=10 tid=0x00007ff2fcbc5290 nid=0x5db9 waiting for monitor entry [0x00007ff2387fe000]

java.lang.Thread.State: BLOCKED (on object monitor)
     at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
     - waiting to lock <0x00000002788ab4a0> (a java.util.Collections$SynchronizedMap)
     at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
     - locked <0x0000000278424268> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
"IPC Reader 6 on port 11600" daemon prio=10 tid=0x00007ff2fcb905c0 nid=0x5db7 waiting for monitor entry [0x00007ff238880000]

java.lang.Thread.State: BLOCKED (on object monitor)
     at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
     - locked <0x00000002788ab4a0> (a java.util.Collections$SynchronizedMap)
     at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:94)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
     at org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:58)
     at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:688)
     at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
     - locked <0x00000002784456d0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
"IPC Reader 5 on port 11600" daemon prio=10 tid=0x00007ff2fcb75d70 nid=0x5db6 waiting for monitor entry [0x00007ff2388c1000]

java.lang.Thread.State: BLOCKED (on object monitor)
     at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
     - waiting to lock <0x00000002788ab4a0> (a java.util.Collections$SynchronizedMap)
     at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
     at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
     - locked <0x00000002784242c8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
     at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)