Return-Path: <issues-return-14880-archive-asf-public=cust-asf.ponee.io@carbondata.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 7271E200CCA
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Wed, 19 Jul 2017 14:10:08 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id 6F228168D69; Wed, 19 Jul 2017 12:10:08 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id B3890168D62
	for <archive-asf-public@cust-asf.ponee.io>; Wed, 19 Jul 2017 14:10:07 +0200 (CEST)
Received: (qmail 10455 invoked by uid 500); 19 Jul 2017 12:10:06 -0000
Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@carbondata.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@carbondata.apache.org>
List-Post: <mailto:issues@carbondata.apache.org>
List-Id: <issues.carbondata.apache.org>
Reply-To: dev@carbondata.apache.org
Delivered-To: mailing list issues@carbondata.apache.org
Received: (qmail 10446 invoked by uid 99); 19 Jul 2017 12:10:06 -0000
Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jul 2017 12:10:06 +0000
Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33)
	id C57A3E5E44; Wed, 19 Jul 2017 12:10:06 +0000 (UTC)
From: ravipesala <git@git.apache.org>
To: issues@carbondata.apache.org
Reply-To: issues@carbondata.apache.org
References: <git-pr-1185-carbondata@git.apache.org>
In-Reply-To: <git-pr-1185-carbondata@git.apache.org>
Subject: [GitHub] carbondata pull request #1185: [WIP]Fixed Concurrent table data loading unsa...
Content-Type: text/plain
Message-Id: <20170719121006.C57A3E5E44@git1-us-west.apache.org>
Date: Wed, 19 Jul 2017 12:10:06 +0000 (UTC)
archived-at: Wed, 19 Jul 2017 12:10:08 -0000

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1185#discussion_r128227396
  
    --- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
    @@ -75,76 +79,67 @@
     
       private MemoryAllocator allocator;
     
    -  private long minimumMemory;
    -
    -  // for debug purpose
    -  private Set<MemoryBlock> set = new HashSet<>();
    -
       private UnsafeMemoryManager(long totalMemory, MemoryAllocator allocator) {
         this.totalMemory = totalMemory;
         this.allocator = allocator;
    -    long numberOfCores = CarbonProperties.getInstance().getNumberOfCores();
    -    long sortMemoryChunkSize = CarbonProperties.getInstance().getSortMemoryChunkSizeInMB();
    -    sortMemoryChunkSize = sortMemoryChunkSize * 1024 * 1024;
    -    long totalWorkingMemoryForAllThreads = sortMemoryChunkSize * numberOfCores;
    -    if (totalWorkingMemoryForAllThreads >= totalMemory) {
    -      throw new RuntimeException("Working memory should be less than total memory configured, "
    -          + "so either reduce the loading threads or increase the memory size. "
    -          + "(Number of threads * number of threads) should be less than total unsafe memory");
    -    }
    -    minimumMemory = totalWorkingMemoryForAllThreads;
    -    LOGGER.info("Memory manager is created with size " + totalMemory + " with " + allocator
    -        + " and minimum reserve memory " + minimumMemory);
    +    LOGGER
    +        .info("Working Memory manager is created with size " + totalMemory + " with " + allocator);
       }
     
    -  private synchronized MemoryBlock allocateMemory(long memoryRequested) {
    +  private synchronized MemoryBlock allocateMemory(long taskId, long memoryRequested) {
         if (memoryUsed + memoryRequested <= totalMemory) {
           MemoryBlock allocate = allocator.allocate(memoryRequested);
           memoryUsed += allocate.size();
    -      if (LOGGER.isDebugEnabled()) {
    -        set.add(allocate);
    -        LOGGER.error("Memory block (" + allocate + ") is created with size "  + allocate.size() +
    -            ". Total memory used " + memoryUsed + "Bytes, left " + getAvailableMemory() + "Bytes");
    +      Set<MemoryBlock> listOfMemoryBlock = taskIdToMemoryBlockMap.get(taskId);
    +      if (null == listOfMemoryBlock) {
    +        listOfMemoryBlock = new HashSet<>();
    +        taskIdToMemoryBlockMap.put(taskId, listOfMemoryBlock);
           }
    +      listOfMemoryBlock.add(allocate);
           return allocate;
         }
         return null;
       }
     
    -  public synchronized void freeMemory(MemoryBlock memoryBlock) {
    +  public synchronized void freeMemory(long taskId,MemoryBlock memoryBlock) {
    +    taskIdToMemoryBlockMap.get(taskId).remove(memoryBlock);
    --- End diff --
    
    Please add log


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---