hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <>
Subject [jira] [Created] (HIVE-10381) Allow pushing of property-key-value based predicate filter to Metastore dropPartitions
Date Fri, 17 Apr 2015 18:33:59 GMT
Sushanth Sowmyan created HIVE-10381:

             Summary: Allow pushing of property-key-value based predicate filter to Metastore
                 Key: HIVE-10381
             Project: Hive
          Issue Type: Sub-task
            Reporter: Sushanth Sowmyan

After HIVE-10228, we now have a case wherein we assume, based on our knowledge of how replication
will work, that DROP_PARTITION replication will act on a per-partition level.

In order to be robust, however, we do handle the case where this might not be the case, wherein
the DROP_PARTITION replication command generated would work across multiple partitions, it's
just that that handling is very inefficient if ever used, since it has to fetch each of those
partitions from the metastore, decide whether or not it matches the ReplicationSpec.allowEventReplacementInto
filter, and then drop it.

Ideally, we should allow pushing this filter to the metastore, and let the metastore do a
smart drop based on that.

    ReplicationSpec replicationSpec = dropTbl.getReplicationSpec();
    if (replicationSpec.isInReplicationScope()){
      // TODO: Current implementation of replication will result in DROP_PARTITION under replication
      // scope being called per-partition instead of multiple partitions. However, to be robust,
      // must still handle the case of multiple partitions in case this assumption changes
in the
      // future. However, if this assumption changes, we will not be very performant if we
      // each partition one-by-one, and then decide on inspection whether or not this is a
      // for dropping. Thus, we need a way to push this filter (replicationSpec.allowEventReplacementInto)
      // to the  metastore to allow it to do drop a partition or not, depending on a Predicate
on the
      // parameter key values.
      for (DropTableDesc.PartSpec partSpec : dropTbl.getPartSpecs()){
        try {
          for (Partition p : Iterables.filter(
              db.getPartitionsByFilter(tbl, partSpec.getPartSpec().getExprString()),
        } catch (NoSuchObjectException e){
          // ignore NSOE because that means there's nothing to drop.
        } catch (Exception e) {
          throw new HiveException(e.getMessage(), e);

This message was sent by Atlassian JIRA

View raw message