Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 24CD1100E3 for ; Mon, 24 Mar 2014 04:07:51 +0000 (UTC) Received: (qmail 77955 invoked by uid 500); 24 Mar 2014 04:07:45 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 77702 invoked by uid 500); 24 Mar 2014 04:07:44 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 77579 invoked by uid 500); 24 Mar 2014 04:07:43 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 77574 invoked by uid 99); 24 Mar 2014 04:07:43 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Mar 2014 04:07:43 +0000 Date: Mon, 24 Mar 2014 04:07:43 +0000 (UTC) From: "Harish Butani (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-6707) Lazy maps are broken (LazyMap and LazyBinaryMap) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944725#comment-13944725 ] Harish Butani commented on HIVE-6707: ------------------------------------- +1 for 0.13 > Lazy maps are broken (LazyMap and LazyBinaryMap) > ------------------------------------------------ > > Key: HIVE-6707 > URL: https://issues.apache.org/jira/browse/HIVE-6707 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers > Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0 > Reporter: Prasanth J > Assignee: Prasanth J > Priority: Critical > Labels: serde > Fix For: 0.13.0, 0.14.0 > > Attachments: HIVE-6707.1.patch > > > LazyPrimitive and LazyBinaryPrimitive overrides hashcode method in HIVE-949. But it failed to override equals() method. As a result, LazyMap and LazyBinaryMap will end up having multiple values for the same key. Both LazyMap and LazyBinaryMap uses LinkedHashMap, so the expected behaviour is to have a single value per unique key. > In the following code from LazyMap (LazyBinaryMap also has same code segment) > {code} > LazyPrimitive lazyKey = uncheckedGetKey(i); > if (lazyKey == null) { > continue; > } > Object key = lazyKey.getObject(); > if (key != null && !cachedMap.containsKey(key)) { > {code} > lazyKey.hashcode() returns the writable object's hashcode. The containsKeys() method of hash map (http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/HashMap.java#366) checks if the hashcode are same, if so then it uses equals() method to verify if the key already exists. Since LazyPrimitive does not override equals() method it falls back to use Object equals(). Object equals() will return true only if both object are exactly the same (this == obj). > So in the above code segment, even if the key already exists, the new value will be inserted with hash collision resulting in more number of map entries. -- This message was sent by Atlassian JIRA (v6.2#6252)