Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 052D4106D6 for ; Fri, 8 Nov 2013 02:43:18 +0000 (UTC) Received: (qmail 58318 invoked by uid 500); 8 Nov 2013 02:43:17 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 58296 invoked by uid 500); 8 Nov 2013 02:43:17 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 58288 invoked by uid 99); 8 Nov 2013 02:43:17 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Nov 2013 02:43:17 +0000 Date: Fri, 8 Nov 2013 02:43:17 +0000 (UTC) From: "Josh Elser (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-1833) MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple threads MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816944#comment-13816944 ] Josh Elser commented on ACCUMULO-1833: -------------------------------------- I noticed that I'm seeing the ShellServerTest run after the MultiTableBatchWriterTest. The finalizer appears to be trying to call close on a MTBW who has a lock on the TabletLocator class (static synchronized method on the class) and there are a bunch of other tests which are locked trying to get this lock. I wonder if I just need to be more aggressive with closing resources in my test. {noformat} "Finalizer" daemon prio=5 tid=0x00007fedbc051800 nid=0x3903 in Object.wait() [0x000000011a021000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache) at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:146) - locked <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache) at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:233) - locked <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache) at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:188) - locked <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache) at org.apache.accumulo.core.client.ZooKeeperInstance.getInstanceID(ZooKeeperInstance.java:151) at org.apache.accumulo.core.client.impl.TabletLocator.getInstance(TabletLocator.java:96) - locked <0x00000007aecd5928> (a java.lang.Class for org.apache.accumulo.core.client.impl.TabletLocator) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.getLocator(TabletServerBatchWriter.java:636) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:648) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:696) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:232) - locked <0x00000007aeb31408> (a org.apache.accumulo.core.client.impl.TabletServerBatchWriter) at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:345) - locked <0x00000007aeb31408> (a org.apache.accumulo.core.client.impl.TabletServerBatchWriter) at org.apache.accumulo.core.client.impl.MultiTableBatchWriterImpl.close(MultiTableBatchWriterImpl.java:126) at org.apache.accumulo.core.client.impl.MultiTableBatchWriterImpl.finalize(MultiTableBatchWriterImpl.java:138) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101) at java.lang.ref.Finalizer.access$100(Finalizer.java:32) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190) Locked ownable synchronizers: - None {noformat} > MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple threads > --------------------------------------------------------------------------------- > > Key: ACCUMULO-1833 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1833 > Project: Accumulo > Issue Type: Improvement > Affects Versions: 1.5.0, 1.6.0 > Reporter: Chris McCubbin > Attachments: ACCUMULO-1833-test.patch, ZooKeeperThreadUtilization.png > > > This issue comes from profiling our application. We have a MultiTableBatchWriter created by normal means. I am attempting to write to it with multiple threads by doing things like the following: > {code} > batchWriter.getBatchWriter(table).addMutations(mutations); > {code} > In my test with 4 threads writing to one table, this call is quite inefficient and results in a large performance degradation over a single BatchWriter. > I believe the culprit is the fact that the call is synchronized. Also there is the possibility that the zookeeper call to Tables.getTableState on every call is negatively affecting performance: > {code} > @Override > public synchronized BatchWriter getBatchWriter(String tableName) throws AccumuloException, AccumuloSecurityException, TableNotFoundException { > ArgumentChecker.notNull(tableName); > String tableId = Tables.getNameToIdMap(instance).get(tableName); > if (tableId == null) > throw new TableNotFoundException(tableId, tableName, null); > > if (Tables.getTableState(instance, tableId) == TableState.OFFLINE) > throw new TableOfflineException(instance, tableId); > > BatchWriter tbw = tableWriters.get(tableId); > if (tbw == null) { > tbw = new TableBatchWriter(tableId); > tableWriters.put(tableId, tbw); > } > return tbw; > } > {code} > I recommend moving the synchronized block to happen only if the batchwriter is not present, and also only checking if the table is online at that time: > {code} > @Override > public BatchWriter getBatchWriter(String tableName) throws AccumuloException, AccumuloSecurityException, TableNotFoundException { > ArgumentChecker.notNull(tableName); > String tableId = Tables.getNameToIdMap(instance).get(tableName); > if (tableId == null) > throw new TableNotFoundException(tableId, tableName, null); > BatchWriter tbw = tableWriters.get(tableId); > if (tbw == null) { > if (Tables.getTableState(instance, tableId) == TableState.OFFLINE) > throw new TableOfflineException(instance, tableId); > tbw = new TableBatchWriter(tableId); > synchronized(tableWriters){ > //only create a new table writer if we haven't been beaten to it. > if (tableWriters.get(tableId) == null) > tableWriters.put(tableId, tbw); > } > } > return tbw; > } > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)