Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0433610EF8 for ; Mon, 18 Nov 2013 03:36:33 +0000 (UTC) Received: (qmail 98924 invoked by uid 500); 18 Nov 2013 03:35:50 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 98770 invoked by uid 500); 18 Nov 2013 03:35:35 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 98737 invoked by uid 99); 18 Nov 2013 03:35:33 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Nov 2013 03:35:33 +0000 Date: Mon, 18 Nov 2013 03:35:33 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-1833) MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple threads MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825077#comment-13825077 ] ASF subversion and git services commented on ACCUMULO-1833: ----------------------------------------------------------- Commit 6b87c870d9475f024911649deb6eeb614325d00a in branch refs/heads/master from [~elserj] [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6b87c87 ] ACCUMULO-1833 Squashed merge of multiple commits that let MTBW work much more efficiently with concurrent access. Squashed commit of the following: commit 58d61759cdc673cc5ee86ad1176b7db3b2955679 Author: Josh Elser Date: Fri Nov 15 14:26:29 2013 -0800 ACCUMULO-1833 Recommended changes from Keith regarding previous fixes. Guava Cache Exception throwing is covered in tests. Added additional test to exercise table rename. Updated state check to be more active and be less susceptible to a paused thread. commit dd73f52180ca00623469850c4b2d4b03c3768837 Author: Josh Elser Date: Tue Nov 12 18:00:07 2013 -0800 ACCUMULO-1833 Change out the AtomicInteger to AtomicLong to make it slightly more robust. commit 9f7916db23adfd561254b432e5f5a5c4e9b02e54 Author: Josh Elser Date: Fri Nov 8 11:36:06 2013 -0500 ACCUMULO-1833 Simple usage of AtomicInteger to catch table cache invalidations and propagate them through MTBW's cache. commit e8cb6c8ef53afaf41eb9e574607cb03093eec1e8 Author: Josh Elser Date: Fri Nov 8 10:55:01 2013 -0500 ACCUMULO-1833 Remove Connector client methods, but leave constructor on MTBW in place for testing purposes. commit b6c6c0270a8bf52d99e0463b2acc98910c4087ca Author: Josh Elser Date: Thu Nov 7 22:22:19 2013 -0500 ACCUMULO-1833 Ensure that we close the MTBW at the end of the test to avoid it getting GC'ed later and trying to flush when ZK and the instance is already gone. commit a11883e62de57eaacf0aba6a5019b7abe79563ec Author: Josh Elser Date: Thu Nov 7 22:21:29 2013 -0500 ACCUMULO-1833 Update MTBW close method to match what TSBW is doing (update internal boolean then perform the close) commit e634ca03f326070a42a811d1ed9a181df5214a03 Author: Josh Elser Date: Thu Nov 7 20:54:20 2013 -0500 ACCUMULO-1833 Another instance of primitive without synchronization being used instead of AtomicBoolean with expected concurrent access. commit ffe8c243dec4d7c7947cc6512394e9a70a29bc77 Author: Josh Elser Date: Thu Nov 7 20:37:28 2013 -0500 ACCUMULO-1833 Tests for expected functionality in the face of table operations. commit 721616e3ff6a4200fd326b7f1ce4be6e1298a7ec Author: Josh Elser Date: Thu Nov 7 16:49:41 2013 -0500 ACCUMULO-1833 Rework the getBatchWriter method on MTBW to remove zookeeper lock contention and get better concurrent throughput. > MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple threads > --------------------------------------------------------------------------------- > > Key: ACCUMULO-1833 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1833 > Project: Accumulo > Issue Type: Improvement > Affects Versions: 1.5.0, 1.6.0 > Reporter: Chris McCubbin > Assignee: Josh Elser > Fix For: 1.5.1, 1.6.0 > > Attachments: ACCUMULO-1833-test.patch, ZooKeeperThreadUtilization.png > > > This issue comes from profiling our application. We have a MultiTableBatchWriter created by normal means. I am attempting to write to it with multiple threads by doing things like the following: > {code} > batchWriter.getBatchWriter(table).addMutations(mutations); > {code} > In my test with 4 threads writing to one table, this call is quite inefficient and results in a large performance degradation over a single BatchWriter. > I believe the culprit is the fact that the call is synchronized. Also there is the possibility that the zookeeper call to Tables.getTableState on every call is negatively affecting performance: > {code} > @Override > public synchronized BatchWriter getBatchWriter(String tableName) throws AccumuloException, AccumuloSecurityException, TableNotFoundException { > ArgumentChecker.notNull(tableName); > String tableId = Tables.getNameToIdMap(instance).get(tableName); > if (tableId == null) > throw new TableNotFoundException(tableId, tableName, null); > > if (Tables.getTableState(instance, tableId) == TableState.OFFLINE) > throw new TableOfflineException(instance, tableId); > > BatchWriter tbw = tableWriters.get(tableId); > if (tbw == null) { > tbw = new TableBatchWriter(tableId); > tableWriters.put(tableId, tbw); > } > return tbw; > } > {code} > I recommend moving the synchronized block to happen only if the batchwriter is not present, and also only checking if the table is online at that time: > {code} > @Override > public BatchWriter getBatchWriter(String tableName) throws AccumuloException, AccumuloSecurityException, TableNotFoundException { > ArgumentChecker.notNull(tableName); > String tableId = Tables.getNameToIdMap(instance).get(tableName); > if (tableId == null) > throw new TableNotFoundException(tableId, tableName, null); > BatchWriter tbw = tableWriters.get(tableId); > if (tbw == null) { > if (Tables.getTableState(instance, tableId) == TableState.OFFLINE) > throw new TableOfflineException(instance, tableId); > tbw = new TableBatchWriter(tableId); > synchronized(tableWriters){ > //only create a new table writer if we haven't been beaten to it. > if (tableWriters.get(tableId) == null) > tableWriters.put(tableId, tbw); > } > } > return tbw; > } > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)