Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0AD919318 for ; Sun, 20 Mar 2016 18:06:33 +0000 (UTC) Received: (qmail 23269 invoked by uid 500); 20 Mar 2016 18:06:33 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 23230 invoked by uid 500); 20 Mar 2016 18:06:33 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 23114 invoked by uid 99); 20 Mar 2016 18:06:33 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2016 18:06:33 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 95FFE2C14F6 for ; Sun, 20 Mar 2016 18:06:33 +0000 (UTC) Date: Sun, 20 Mar 2016 18:06:33 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-4148) Native map should increment counter for every cell MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203403#comment-15203403 ] ASF GitHub Bot commented on ACCUMULO-4148: ------------------------------------------ GitHub user mjwall opened a pull request: https://github.com/apache/accumulo/pull/82 4148 inmemorymap counter Tests and fixes for ACCUMULO-4148 You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjwall/accumulo 4148-inmemorymap-counter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/accumulo/pull/82.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #82 ---- commit a7a738f3d92d071ac2507fbfce2e71695b82cc81 Author: Michael Wall Date: 2016-02-26T02:20:15Z ACCUMULO-4148 Explaination of problem, with tests The problem here is that InMemoryMap$DefaultMap increments the mutationCount or kvCount for every key value pair in every Mutation that is passed in. The NativeMap, which is used by the InMemoryMap$NativeMapWrapper does not. This causes 2 different issues in the NativeMap. 1) When a single Mutation has duplicate key value pairs, only the last is recorded, because they all have the same mutationCount and the earlier ones are hidden. 2 ) When multiple Mutations are passed in at the same time, the mutationCount or kvCount starts over for each Mutation. This can also lead to hiding of key value pairs. The tests added here expose both the issues as well as do some asserts on simple Mutations. A few tweaks were made to expose information to these tests. 1) Made MemKey public, made it's kvCount private and exposed that via a getter so we can inspect directly instead of parsing the toString. Required changing some calls to kvCount to use the getter. 2) Added a final String to the InMemoryMap which is set during construction. This allows you to see what kind of SimpleMap was setup in the InMemoryMap. Typically, mulitple asserts in one test are not the best design. But in this case so much setup was required and I wanted to compare how different InMemoryMaps treated the same collection of mutations. Here is the output from running these tests Failed tests: InMemoryMapIT.testMultipleMutationsMultipleKeysSomeSame:192->assertEquivalentMutate:221->assertMutatesEquivalent:247 Not all key value pairs included: InMemoryMap type NativeMapWrapper a 8cf:8cq [] 0 false mc=2 a 8cf:8cq [] 0 false mc=1 a 8cf1:8cq1 [] 0 false mc=2 a 8cf1:8cq1 [] 0 false mc=1 a 8cf2:8cq2 [] 0 false mc=2 a 8cf2:8cq2 [] 0 false mc=1 a 8cf3:8cq3 [] 0 false mc=2 b 8cf1:8cq1 [] 0 false mc=3 b 8cf2:8cq2 [] 0 false mc=3 expected:<11> but was:<9> InMemoryMapIT.testMultipleMutationsMultipleSameKeys:165->assertEquivalentMutate:221->assertMutatesEquivalent:247 Not all key value pairs included: InMemoryMap type NativeMapWrapper a 7cf:7cq [] 0 false mc=2 a 7cf:7cq [] 0 false mc=1 expected:<5> but was:<2> InMemoryMapIT.testOneMutationManyKeys:106->assertEquivalentMutate:196->assertEquivalentMutate:221->assertMutatesEquivalent:250 InMemoryMap did not have distinct kvCounts InMemoryMap type NativeMapWrapper a 2cf1:2cq1 [] 0 false mc=1 a 2cf2:2cq2 [] 0 false mc=1 a 2cf3:2cq3 [] 0 false mc=1 a 2cf4:2cq4 [] 0 false mc=1 a 2cf5:2cq5 [] 0 false mc=1 expected:<5> but was:<1> InMemoryMapIT.testOneMutationManySameKeys:117->assertEquivalentMutate:196->assertEquivalentMutate:221->assertMutatesEquivalent:247 Not all key value pairs included: InMemoryMap type NativeMapWrapper a 3cf:3cq [] 0 false mc=1 expected:<5> but was:<1> InMemoryMapIT.testMutlipleMutationsMultipleKeys:151->assertEquivalentMutate:221->assertMutatesEquivalent:250 InMemoryMap did not have distinct kvCounts InMemoryMap type NativeMapWrapper a 6cf1:6cq1 [] 0 false mc=1 a 6cf2:6cq2 [] 0 false mc=1 a 6cf3:6cq3 [] 0 false mc=1 a 6cf4:6cq4 [] 0 false mc=1 a 6cf5:6cq5 [] 0 false mc=1 b 6cf1:6cq1 [] 0 false mc=2 b 6cf2:6cq2 [] 0 false mc=2 expected:<7> but was:<2> Tests run: 8, Failures: 5, Errors: 0, Skipped: 0 commit 3da5412de0cc08133dec7215ce8fe861f73219e4 Author: Michael Wall Date: 2016-03-20T15:04:06Z ACCUMULO-4148 Fix one Mutation with duplicate keys Make NativeMap increment the kvCount each time a key pair is mutated. This is what happens in the mutate method of InMemoryMap$DefaultMap, the kvCount++. Still have failures though when multiple mutations are passed at the same time. Here is the output: Failed tests: InMemoryMapIT.testMultipleMutationsMultipleKeysSomeSame:192->assertEquivalentMutate:221->assertMutatesEquivalent:247 Not all key value pairs included: InMemoryMap type NativeMapWrapper a 8cf:8cq [] 0 false mc=3 a 8cf:8cq [] 0 false mc=2 a 8cf:8cq [] 0 false mc=1 a 8cf1:8cq1 [] 0 false mc=4 a 8cf1:8cq1 [] 0 false mc=2 a 8cf2:8cq2 [] 0 false mc=5 a 8cf2:8cq2 [] 0 false mc=3 a 8cf3:8cq3 [] 0 false mc=6 b 8cf1:8cq1 [] 0 false mc=3 b 8cf2:8cq2 [] 0 false mc=4 expected:<11> but was:<10> InMemoryMapIT.testMultipleMutationsMultipleSameKeys:165->assertEquivalentMutate:221->assertMutatesEquivalent:247 Not all key value pairs included: InMemoryMap type NativeMapWrapper a 7cf:7cq [] 0 false mc=4 a 7cf:7cq [] 0 false mc=3 a 7cf:7cq [] 0 false mc=2 a 7cf:7cq [] 0 false mc=1 expected:<5> but was:<4> InMemoryMapIT.testMutlipleMutationsMultipleKeys:151->assertEquivalentMutate:221->assertMutatesEquivalent:250 InMemoryMap did not have distinct kvCounts InMemoryMap type NativeMapWrapper a 6cf1:6cq1 [] 0 false mc=1 a 6cf2:6cq2 [] 0 false mc=2 a 6cf3:6cq3 [] 0 false mc=3 a 6cf4:6cq4 [] 0 false mc=4 a 6cf5:6cq5 [] 0 false mc=5 b 6cf1:6cq1 [] 0 false mc=2 b 6cf2:6cq2 [] 0 false mc=3 expected:<7> but was:<5> commit 80206d5e4c7669321b2b9d7971dd45f60d4c45ae Author: Michael Wall Date: 2016-03-20T15:14:12Z ACCUMULO-4148 Fix for multiple Mutation objects This is the fix Keith original recommended in the email thread. Make NativeMap._mutate return the current kvCount so it can be used for every Mutation that is passed in. Also, there were 2 different code paths for NativeMap methods public void mutate(Mutation mutation, int mutationCount) and public void mutate(List mutations, int mutationCount) so let's make the first call the second. All the InMemoryMapIT tests are passing. ---- > Native map should increment counter for every cell > -------------------------------------------------- > > Key: ACCUMULO-4148 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4148 > Project: Accumulo > Issue Type: Bug > Affects Versions: 1.6.5, 1.7.0 > Reporter: Keith Turner > Assignee: Michael Wall > Fix For: 1.6.6, 1.7.2, 1.8.0 > > > Dan Blum uncovered a bug in native map outlined in a [mailing list thread|http://mail-archives.apache.org/mod_mbox/accumulo-user/201602.mbox/%3C019b01d16b55%2400c15f00%2402441d00%24%40bbn.com%3E]. > If the link to the thread stops working, below is the header for the first message in the thread to use for searching. > {noformat} > From "Dan Blum" > Subject Bug in either InMemoryMap or NativeMap > Date Fri, 19 Feb 2016 20:34:59 GMT > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)