Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Fri, 17 Apr 2015 18:33:59 +0000 (UTC)
From: "Sushanth Sowmyan (JIRA)" <jira@apache.org>
To: dev@hive.apache.org
Message-ID: <JIRA.12821955.1429295604000.33378.1429295639104@Atlassian.JIRA>
In-Reply-To: <JIRA.12821955.1429295604000@Atlassian.JIRA>
References: <JIRA.12821955.1429295604000@Atlassian.JIRA>
 <JIRA.12821955.1429295604378@arcas>
Subject: [jira] [Created] (HIVE-10381) Allow pushing of property-key-value
 based predicate filter to Metastore dropPartitions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Sushanth Sowmyan created HIVE-10381:
---------------------------------------

             Summary: Allow pushing of property-key-value based predicate filter to Metastore dropPartitions
                 Key: HIVE-10381
                 URL: https://issues.apache.org/jira/browse/HIVE-10381
             Project: Hive
          Issue Type: Sub-task
            Reporter: Sushanth Sowmyan


After HIVE-10228, we now have a case wherein we assume, based on our knowledge of how replication will work, that DROP_PARTITION replication will act on a per-partition level.

In order to be robust, however, we do handle the case where this might not be the case, wherein the DROP_PARTITION replication command generated would work across multiple partitions, it's just that that handling is very inefficient if ever used, since it has to fetch each of those partitions from the metastore, decide whether or not it matches the ReplicationSpec.allowEventReplacementInto filter, and then drop it.

Ideally, we should allow pushing this filter to the metastore, and let the metastore do a smart drop based on that.

{code}
    ReplicationSpec replicationSpec = dropTbl.getReplicationSpec();
    if (replicationSpec.isInReplicationScope()){
      // TODO: Current implementation of replication will result in DROP_PARTITION under replication
      // scope being called per-partition instead of multiple partitions. However, to be robust, we
      // must still handle the case of multiple partitions in case this assumption changes in the
      // future. However, if this assumption changes, we will not be very performant if we fetch
      // each partition one-by-one, and then decide on inspection whether or not this is a candidate
      // for dropping. Thus, we need a way to push this filter (replicationSpec.allowEventReplacementInto)
      // to the  metastore to allow it to do drop a partition or not, depending on a Predicate on the
      // parameter key values.
      for (DropTableDesc.PartSpec partSpec : dropTbl.getPartSpecs()){
        try {
          for (Partition p : Iterables.filter(
              db.getPartitionsByFilter(tbl, partSpec.getPartSpec().getExprString()),
              replicationSpec.allowEventReplacementInto())){
            db.dropPartition(tbl.getDbName(),tbl.getTableName(),p.getValues(),true);
          }
        } catch (NoSuchObjectException e){
          // ignore NSOE because that means there's nothing to drop.
        } catch (Exception e) {
          throw new HiveException(e.getMessage(), e);
        }
      }
      return;
    }
{code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)