Return-Path: <issues-return-27814-archive-asf-public=cust-asf.ponee.io@carbondata.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 33B30200D5D
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Wed, 20 Dec 2017 11:37:59 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 294DB160C15; Wed, 20 Dec 2017 10:37:59 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 6DBC2160C0A
	for <archive-asf-public@cust-asf.ponee.io>; Wed, 20 Dec 2017 11:37:58 +0100 (CET)
Received: (qmail 43472 invoked by uid 500); 20 Dec 2017 10:37:57 -0000
Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@carbondata.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@carbondata.apache.org>
List-Post: <mailto:issues@carbondata.apache.org>
List-Id: <issues.carbondata.apache.org>
Reply-To: dev@carbondata.apache.org
Delivered-To: mailing list issues@carbondata.apache.org
Received: (qmail 43463 invoked by uid 99); 20 Dec 2017 10:37:57 -0000
Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Dec 2017 10:37:57 +0000
Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33)
	id 1DD30DFF59; Wed, 20 Dec 2017 10:37:55 +0000 (UTC)
From: ravipesala <git@git.apache.org>
To: issues@carbondata.apache.org
Reply-To: issues@carbondata.apache.org
References: <git-pr-1674-carbondata@git.apache.org>
In-Reply-To: <git-pr-1674-carbondata@git.apache.org>
Subject: [GitHub] carbondata pull request #1674: [CARBONDATA-1859][CARBONDATA-1861][PARTITION]...
Content-Type: text/plain
Message-Id: <20171220103756.1DD30DFF59@git1-us-west.apache.org>
Date: Wed, 20 Dec 2017 10:37:55 +0000 (UTC)
archived-at: Wed, 20 Dec 2017 10:37:59 -0000

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1674#discussion_r157988312
  
    --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/PartitionMapFileStore.java ---
    @@ -176,13 +179,87 @@ public PartitionMapper readPartitionMap(String partitionMapPath) {
       public void readAllPartitionsOfSegment(String segmentPath) {
         CarbonFile[] partitionFiles = getPartitionFiles(segmentPath);
         if (partitionFiles != null && partitionFiles.length > 0) {
    +      partionedSegment = true;
           for (CarbonFile file : partitionFiles) {
             PartitionMapper partitionMapper = readPartitionMap(file.getAbsolutePath());
             partitionMap.putAll(partitionMapper.getPartitionMap());
           }
         }
       }
     
    +  public boolean isPartionedSegment() {
    +    return partionedSegment;
    +  }
    +
    +  /**
    +   * Drops the partitions from the partition mapper file of the segment and writes to a new file.
    +   * @param segmentPath
    +   * @param partitionsToDrop
    +   * @param uniqueId
    +   * @throws IOException
    +   */
    +  public void dropPartitions(String segmentPath, List<String> partitionsToDrop, String uniqueId)
    +      throws IOException {
    +    readAllPartitionsOfSegment(segmentPath);
    +    List<String> indexesToDrop = new ArrayList<>();
    +    for (Map.Entry<String, List<String>> entry: partitionMap.entrySet()) {
    +      for (String partition: partitionsToDrop) {
    +        if (entry.getValue().contains(partition)) {
    +          indexesToDrop.add(entry.getKey());
    +        }
    +      }
    +    }
    +    if (indexesToDrop.size() > 0) {
    +      // Remove the indexes from partition map
    +      for (String indexToDrop : indexesToDrop) {
    +        partitionMap.remove(indexToDrop);
    +      }
    +      PartitionMapper mapper = new PartitionMapper();
    +      mapper.setPartitionMap(partitionMap);
    +      String path = segmentPath + "/" + uniqueId + CarbonTablePath.PARTITION_MAP_EXT;
    +      writePartitionFile(mapper, path);
    +    }
    +  }
    +
    +  /**
    +   * It deletes the old partition mapper files in case of success. And in case of failure it removes
    +   * the old new file.
    +   * @param segmentPath
    +   * @param uniqueId
    +   * @param success
    +   */
    +  public void commitPartitions(String segmentPath, final String uniqueId, boolean success) {
    +    CarbonFile carbonFile = FileFactory.getCarbonFile(segmentPath);
    --- End diff --
    
    I think it is better not to take any lock inside processMetadata as some users run this command in different machine . So lock only taken in processData and if anything wrong happens we add back partition in undoMetadata command


---