hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Svetozar Ivanov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
Date Sun, 10 Jul 2016 14:41:10 GMT

     [ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Svetozar Ivanov updated HIVE-11233:
-----------------------------------
    Description: 
Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key
and values. It is necessary when given HBase table is used for persistence by Apache Hive
and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible
with binary sortable format used in Phoenix.

It turns out the major difference is in all numeric data types accordingly officially provided
documentation - https://phoenix.apache.org/language/datatypes.html.

That's how I'm using it in my code with applied patch:
{code}
    private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) {
        Map<String, String> serdePropertiesMap = new HashMap<>();

        serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName());
        serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE);
        serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor));


        serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName());
        /* Use different key factory for simple and composite primary key */
        if (tableDescriptor.getPkDescriptors().size() == 1) {
            serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName());
        } else {
            serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName());
        }

        String serDeProperties = serdePropertiesMap.entrySet().stream()
                .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue()))
                .collect(Collectors.joining(COLUMNS_SEPARATOR));

        logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
        return serDeProperties;
    }
{code}

  was:
Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key
and values. It is necessary when given HBase table is used for persistence by Apache Hive
and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible
with binary sortable format used in Phoenix.

It turns out the major difference is in all numeric data types accordingly officially provided
documentation - https://phoenix.apache.org/language/datatypes.html.

That's how I'm using it in my code:
{code}
    private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) {
        Map<String, String> serdePropertiesMap = new HashMap<>();

        serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName());
        serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE);
        serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor));


        serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName());
        /* Use different key factory for simple and composite primary key */
        if (tableDescriptor.getPkDescriptors().size() == 1) {
            serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName());
        } else {
            serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName());
        }

        String serDeProperties = serdePropertiesMap.entrySet().stream()
                .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue()))
                .collect(Collectors.joining(COLUMNS_SEPARATOR));

        logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
        return serDeProperties;
    }
{code}


> Include Apache Phoenix support in HBaseStorageHandler
> -----------------------------------------------------
>
>                 Key: HIVE-11233
>                 URL: https://issues.apache.org/jira/browse/HIVE-11233
>             Project: Hive
>          Issue Type: New Feature
>          Components: HBase Handler
>    Affects Versions: 1.2.1, 2.0.0
>            Reporter: Svetozar Ivanov
>            Assignee: Svetozar Ivanov
>              Labels: Binary, Hbase, Numeric, Phoenix, Sortable
>         Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch,
HIVE-11233.2.patch, HIVE-11233.3.patch, HIVE-11233.4.patch, HIVE-11233.5.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable
key and values. It is necessary when given HBase table is used for persistence by Apache Hive
and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible
with binary sortable format used in Phoenix.
> It turns out the major difference is in all numeric data types accordingly officially
provided documentation - https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code with applied patch:
> {code}
>     private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) {
>         Map<String, String> serdePropertiesMap = new HashMap<>();
>         serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName());
>         serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE);
>         serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor));
>         serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName());
>         /* Use different key factory for simple and composite primary key */
>         if (tableDescriptor.getPkDescriptors().size() == 1) {
>             serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName());
>         } else {
>             serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName());
>         }
>         String serDeProperties = serdePropertiesMap.entrySet().stream()
>                 .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue()))
>                 .collect(Collectors.joining(COLUMNS_SEPARATOR));
>         logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
>         return serDeProperties;
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message