accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1833) MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple threads
Date Fri, 08 Nov 2013 02:43:17 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816944#comment-13816944
] 

Josh Elser commented on ACCUMULO-1833:
--------------------------------------

I noticed that I'm seeing the ShellServerTest run after the MultiTableBatchWriterTest. The
finalizer appears to be trying to call close on a MTBW who has a lock on the TabletLocator
class (static synchronized method on the class) and there are a bunch of other tests which
are locked trying to get this lock. I wonder if I just need to be more aggressive with closing
resources in my test.

{noformat}
"Finalizer" daemon prio=5 tid=0x00007fedbc051800 nid=0x3903 in Object.wait() [0x000000011a021000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache)
	at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:146)
	- locked <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache)
	at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:233)
	- locked <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache)
	at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:188)
	- locked <0x00000007aeb30878> (a org.apache.accumulo.fate.zookeeper.ZooCache)
	at org.apache.accumulo.core.client.ZooKeeperInstance.getInstanceID(ZooKeeperInstance.java:151)
	at org.apache.accumulo.core.client.impl.TabletLocator.getInstance(TabletLocator.java:96)
	- locked <0x00000007aecd5928> (a java.lang.Class for org.apache.accumulo.core.client.impl.TabletLocator)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.getLocator(TabletServerBatchWriter.java:636)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:648)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:696)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:232)
	- locked <0x00000007aeb31408> (a org.apache.accumulo.core.client.impl.TabletServerBatchWriter)
	at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:345)
	- locked <0x00000007aeb31408> (a org.apache.accumulo.core.client.impl.TabletServerBatchWriter)
	at org.apache.accumulo.core.client.impl.MultiTableBatchWriterImpl.close(MultiTableBatchWriterImpl.java:126)
	at org.apache.accumulo.core.client.impl.MultiTableBatchWriterImpl.finalize(MultiTableBatchWriterImpl.java:138)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)

   Locked ownable synchronizers:
	- None
{noformat}

> MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple threads
> ---------------------------------------------------------------------------------
>
>                 Key: ACCUMULO-1833
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1833
>             Project: Accumulo
>          Issue Type: Improvement
>    Affects Versions: 1.5.0, 1.6.0
>            Reporter: Chris McCubbin
>         Attachments: ACCUMULO-1833-test.patch, ZooKeeperThreadUtilization.png
>
>
> This issue comes from profiling our application. We have a MultiTableBatchWriter created
by normal means. I am attempting to write to it with multiple threads by doing things like
the following:
> {code}
> batchWriter.getBatchWriter(table).addMutations(mutations);
> {code}
> In my test with 4 threads writing to one table, this call is quite inefficient and results
in a large performance degradation over a single BatchWriter.
> I believe the culprit is the fact that the call is synchronized. Also there is the possibility
that the zookeeper call to Tables.getTableState on every call is negatively affecting performance:
> {code}
>   @Override
>   public synchronized BatchWriter getBatchWriter(String tableName) throws AccumuloException,
AccumuloSecurityException, TableNotFoundException {
>     ArgumentChecker.notNull(tableName);
>     String tableId = Tables.getNameToIdMap(instance).get(tableName);
>     if (tableId == null)
>       throw new TableNotFoundException(tableId, tableName, null);
>     
>     if (Tables.getTableState(instance, tableId) == TableState.OFFLINE)
>       throw new TableOfflineException(instance, tableId);
>     
>     BatchWriter tbw = tableWriters.get(tableId);
>     if (tbw == null) {
>       tbw = new TableBatchWriter(tableId);
>       tableWriters.put(tableId, tbw);
>     }
>     return tbw;
>   }
> {code}
> I recommend moving the synchronized block to happen only if the batchwriter is not present,
and also only checking if the table is online at that time:
> {code}
>   @Override
>   public BatchWriter getBatchWriter(String tableName) throws AccumuloException, AccumuloSecurityException,
TableNotFoundException {
>     ArgumentChecker.notNull(tableName);
>     String tableId = Tables.getNameToIdMap(instance).get(tableName);
>     if (tableId == null)
>       throw new TableNotFoundException(tableId, tableName, null);
>     BatchWriter tbw = tableWriters.get(tableId);
>     if (tbw == null) {
>       if (Tables.getTableState(instance, tableId) == TableState.OFFLINE)
>           throw new TableOfflineException(instance, tableId);
>       tbw = new TableBatchWriter(tableId);
>       synchronized(tableWriters){
>           //only create a new table writer if we haven't been beaten to it.
>           if (tableWriters.get(tableId) == null)      
>               tableWriters.put(tableId, tbw);
>       }
>     }
>     return tbw;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message