hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohammad Kamrul Islam (JIRA)" <>
Subject [jira] [Commented] (HIVE-4732) Reduce or eliminate the expensive Schema equals() check for AvroSerde
Date Thu, 11 Jul 2013 22:33:48 GMT


Mohammad Kamrul Islam commented on HIVE-4732:

New patch is uploaded in RB:

Description copied from RB:
>From our performance analysis, we found AvroSerde's schema.equals() call consumed a substantial
amount ( nearly 40%) of time. This patch intends to minimize the number schema.equals() calls
by pushing the check as late/fewer as possible.

At first, we added a unique id for each record reader which is then included in every AvroGenericRecordWritable.
Then, we introduce two new data structures (one hashset and one hashmap) to store intermediate
data to avoid duplicates checkings. Hashset contains all the record readers' IDs that don't
need any re-encoding. On the other hand, HashMap contains the already used re-encoders. It
works as cache and allows re-encoders reuse. With this change, our test shows nearly 40% reduction
in Avro record reading time.

> Reduce or eliminate the expensive Schema equals() check for AvroSerde
> ---------------------------------------------------------------------
>                 Key: HIVE-4732
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>            Reporter: Mark Wagner
>            Assignee: Mohammad Kamrul Islam
>         Attachments: HIVE-4732.1.patch, HIVE-4732.v1.patch
> The AvroSerde spends a significant amount of time checking schema equality. Changing
to compare hashcodes (which can be computed once then reused) will improve performance.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message