Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E45B7200B42 for ; Sun, 10 Jul 2016 16:41:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E2D86160A66; Sun, 10 Jul 2016 14:41:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1356A160A58 for ; Sun, 10 Jul 2016 16:41:11 +0200 (CEST) Received: (qmail 65447 invoked by uid 500); 10 Jul 2016 14:41:11 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 65437 invoked by uid 99); 10 Jul 2016 14:41:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jul 2016 14:41:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id EF0432C02A1 for ; Sun, 10 Jul 2016 14:41:10 +0000 (UTC) Date: Sun, 10 Jul 2016 14:41:10 +0000 (UTC) From: "Svetozar Ivanov (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Sun, 10 Jul 2016 14:41:13 -0000 [ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: ----------------------------------- Description: Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key and values. It is necessary when given HBase table is used for persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible with binary sortable format used in Phoenix. It turns out the major difference is in all numeric data types accordingly officially provided documentation - https://phoenix.apache.org/language/datatypes.html. That's how I'm using it in my code with applied patch: {code} private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) { Map serdePropertiesMap = new HashMap<>(); serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName()); serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE); serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor)); serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName()); /* Use different key factory for simple and composite primary key */ if (tableDescriptor.getPkDescriptors().size() == 1) { serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName()); } else { serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName()); } String serDeProperties = serdePropertiesMap.entrySet().stream() .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue())) .collect(Collectors.joining(COLUMNS_SEPARATOR)); logger.debug("SERDEPROPERTIES are [{}]", serDeProperties); return serDeProperties; } {code} was: Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key and values. It is necessary when given HBase table is used for persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible with binary sortable format used in Phoenix. It turns out the major difference is in all numeric data types accordingly officially provided documentation - https://phoenix.apache.org/language/datatypes.html. That's how I'm using it in my code: {code} private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) { Map serdePropertiesMap = new HashMap<>(); serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName()); serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE); serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor)); serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName()); /* Use different key factory for simple and composite primary key */ if (tableDescriptor.getPkDescriptors().size() == 1) { serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName()); } else { serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName()); } String serDeProperties = serdePropertiesMap.entrySet().stream() .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue())) .collect(Collectors.joining(COLUMNS_SEPARATOR)); logger.debug("SERDEPROPERTIES are [{}]", serDeProperties); return serDeProperties; } {code} > Include Apache Phoenix support in HBaseStorageHandler > ----------------------------------------------------- > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler > Affects Versions: 1.2.1, 2.0.0 > Reporter: Svetozar Ivanov > Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch, HIVE-11233.3.patch, HIVE-11233.4.patch, HIVE-11233.5.patch, HIVE-11233.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key and values. It is necessary when given HBase table is used for persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible with binary sortable format used in Phoenix. > It turns out the major difference is in all numeric data types accordingly officially provided documentation - https://phoenix.apache.org/language/datatypes.html. > That's how I'm using it in my code with applied patch: > {code} > private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) { > Map serdePropertiesMap = new HashMap<>(); > serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName()); > serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE); > serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor)); > serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName()); > /* Use different key factory for simple and composite primary key */ > if (tableDescriptor.getPkDescriptors().size() == 1) { > serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName()); > } else { > serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName()); > } > String serDeProperties = serdePropertiesMap.entrySet().stream() > .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue())) > .collect(Collectors.joining(COLUMNS_SEPARATOR)); > logger.debug("SERDEPROPERTIES are [{}]", serDeProperties); > return serDeProperties; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)