hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <>
Subject [jira] [Created] (HIVE-13479) Relax sorting requirement in ACID tables
Date Mon, 11 Apr 2016 17:47:25 GMT
Eugene Koifman created HIVE-13479:

             Summary: Relax sorting requirement in ACID tables
                 Key: HIVE-13479
             Project: Hive
          Issue Type: New Feature
          Components: Transactions
    Affects Versions: 1.2.0
            Reporter: Eugene Koifman
            Assignee: Eugene Koifman

Currently ACID tables require data to be sorted according to internal primary key.  This is
that base + delta files can be efficiently sort/merged to produce the snapshot for current

This prevents the user to make the table sorted based on any other criteria which can be useful.
 One example is using dynamic partition insert (which also occurs for update/delete SQL).
 This may create lots of writers (buckets*partitions) and tax cluster resources.
The usual solution is hive.optimize.sort.dynamic.partition=true which won't be honored for
ACID tables.

We could rely on hash table based algorithm to merge delta files and then not require any
particular sort on Acid tables.  One way to do that is to treat each update event as an Insert
(new internal PK) + delete (old PK).  Delete events are very small since they just need to
contain PKs.  So the hash table would just need to contain Delete events and be reasonably
memory efficient.

This is a significant amount of work but worth doing.

This message was sent by Atlassian JIRA

View raw message