From issues-return-116622-archive-asf-public=cust-asf.ponee.io@hive.apache.org Sat May 5 03:25:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 538F7180675 for ; Sat, 5 May 2018 03:25:05 +0200 (CEST) Received: (qmail 37649 invoked by uid 500); 5 May 2018 01:25:04 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 37631 invoked by uid 99); 5 May 2018 01:25:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 May 2018 01:25:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D6390C05EA for ; Sat, 5 May 2018 01:25:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Zj8SRA-K73dL for ; Sat, 5 May 2018 01:25:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 69DC25FBA2 for ; Sat, 5 May 2018 01:25:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5A03BE12D9 for ; Sat, 5 May 2018 01:25:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 6298B212A1 for ; Sat, 5 May 2018 01:25:00 +0000 (UTC) Date: Sat, 5 May 2018 01:25:00 +0000 (UTC) From: "Vihang Karajgaonkar (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-19041) Thrift deserialization of Partition objects should intern fields MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464550#comment-16464550 ] Vihang Karajgaonkar commented on HIVE-19041: -------------------------------------------- v3 of the patch includes additional thrift generated files which operate on list of partitions like {{AddPartitionsRequest}}, {{DropPartitionsRequest}}. Also found that some of the statistics related thrift classes do not intern repeated fields. I wish there was some way to tell thrift to do this instead of us keeping up with newly added thrift classes. > Thrift deserialization of Partition objects should intern fields > ---------------------------------------------------------------- > > Key: HIVE-19041 > URL: https://issues.apache.org/jira/browse/HIVE-19041 > Project: Hive > Issue Type: Improvement > Components: Metastore > Affects Versions: 3.0.0, 2.3.2 > Reporter: Vihang Karajgaonkar > Assignee: Vihang Karajgaonkar > Priority: Major > Attachments: HIVE-19041.01.patch, HIVE-19041.02.patch, HIVE-19041.03.patch > > > When a client is creating large number of partitions, the thrift objects are deserialized into Partition objects. The read method of these objects does not intern the inputformat, location, outputformat which cause large number of duplicate Strings in the HMS memory. We should intern these objects while deserialization to reduce memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)