Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
From: paul-rogers <git@git.apache.org>
To: dev@drill.apache.org
Reply-To: dev@drill.apache.org
References: <git-pr-822-drill@git.apache.org>
In-Reply-To: <git-pr-822-drill@git.apache.org>
Subject: [GitHub] drill pull request #822: DRILL-5457: Spill implementation for Hash Aggregate
Content-Type: text/plain
Message-Id: <20170527053809.AA2E1F2184@git1-us-west.apache.org>
Date: Sat, 27 May 2017 05:38:09 +0000 (UTC)
archived-at: Sat, 27 May 2017 05:38:17 -0000

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/822#discussion_r118813857
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java ---
    @@ -400,114 +782,411 @@ public IterOutcome getOutcome() {
     
       @Override
       public int getOutputCount() {
    -    // return outputCount;
         return lastBatchOutputCount;
       }
     
       @Override
       public void cleanup() {
    -    if (htable != null) {
    -      htable.clear();
    -      htable = null;
    -    }
    +      if ( schema == null ) { return; } // not set up; nothing to clean
    +      for ( int i = 0; i < numPartitions; i++) {
    +          if (htables[i] != null) {
    +              htables[i].clear();
    +              htables[i] = null;
    +          }
    +          if ( batchHolders[i] != null) {
    +              for (BatchHolder bh : batchHolders[i]) {
    +                    bh.clear();
    +              }
    +              batchHolders[i].clear();
    +              batchHolders[i] = null;
    +          }
    +
    +          // delete any (still active) output spill file
    +          if ( outputStream[i] != null && spillFiles[i] != null) {
    +            try {
    +              spillSet.delete(spillFiles[i]);
    --- End diff --
    
    This code makes me wonder... How many output streams are open at any one time? Drill is a highly concurrent system: we could have 1000s of fragments. If each has, say, a hash agg with 16 partitions, do we run the risk of 16,000 open file handles? Or, is the file handle opened only when needed for reading or writing?
    
    In general, as we add more spilling, we may need a global file handle cache that controls the number of open files. The same issue arises in the sort merge phase: all spill files maintain an open file handle; we might exceed some limit.
    
    For this PR, consider how long the file handle is open. Perhaps we need to file a JIRA about managing the total number of open files.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---