hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ratandeep Ratti (JIRA)" <>
Subject [jira] [Commented] (HIVE-4734) Use custom ObjectInspectors for AvroSerde
Date Wed, 08 Jul 2015 08:08:05 GMT


Ratandeep Ratti commented on HIVE-4734:

I found some minor issues with the patch. Adding notes below.

In class, while constructing an {{AvroMapObjectInspector}}
we are using  a {{JavaStringObjectInspector}}

   case MAP:
        MapTypeInfo mti = (MapTypeInfo)ti;
        result = new AvroMapObjectInspector(
            createObjectInspectorWorker(mti.getMapValueTypeInfo(), subSchema.getValueType()));

Whereas Avro string could be of type UTF8 or java.lang.String . Default being UTF8.

Also, {{AvroStringObjectInspector}} assumes that avro strings are always {{Utf8}} which may
not be the case. Avro strings can either be UFT-8 or String type.

{{AvroMapObjectInspector}} also assumes that Map keys will always be {{String}} s. Further
the below methods in MapObjectInspector  assume the second argument ('key') passed to them
is of type {{String}} whereas it could be either {{utf8}} or {{String}}

  public Object getMapValueElement(Object data, Object key) {
    Utf8 utf8key = new Utf8((String) key);
    return super.getMapValueElement(data, utf8key);

  public Object put(Object map, Object key, Object value) {
    Utf8 utf8key = new Utf8((String) key);
    return super.put(map, utf8key, value);

  public Object remove(Object map, Object key) {
    Utf8 utf8key = new Utf8((String) key);
    return super.remove(map, utf8key);

> Use custom ObjectInspectors for AvroSerde
> -----------------------------------------
>                 Key: HIVE-4734
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>            Reporter: Mark Wagner
>            Assignee: Mark Wagner
>              Labels: Avro, AvroSerde, Performance
>         Attachments: HIVE-4734.1.patch, HIVE-4734.2.patch, HIVE-4734.3.patch, HIVE-4734.4.patch,
> Currently, the AvroSerde recursively copies all fields of a record from the GenericRecord
to a List row object and provides the standard ObjectInspectors. Performance can be improved
by providing ObjectInspectors to the Avro record itself.

This message was sent by Atlassian JIRA

View raw message