phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alok Singh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (PHOENIX-2685) SpoolingResultIterator opening a large number of files, hitting process max open file limit
Date Mon, 15 Feb 2016 23:41:18 GMT
Alok Singh created PHOENIX-2685:
-----------------------------------

             Summary: SpoolingResultIterator opening a large number of files, hitting process
max open file limit
                 Key: PHOENIX-2685
                 URL: https://issues.apache.org/jira/browse/PHOENIX-2685
             Project: Phoenix
          Issue Type: Bug
    Affects Versions: 4.6.0, 4.5.1, 4.7.0
            Reporter: Alok Singh


During one of out load tests, in which we execute 5-10 concurrent requests over a few hours,
we have found that eventually the jvm process runs out of file handles due to 1000s of tmp
files (/tmp/ResultSpoolerxxxxxx.bin). 

Upon further investigation, it looks like that under high concurrent load, with the default
'phoenix.query.spoolThresholdBytes' value, SpoolingResultIterator will almost always run into
this issue.
{code}
final MemoryChunk chunk = mm.allocate(0, thresholdBytes);
long waitTime = System.currentTimeMillis() - startTime;
GLOBAL_MEMORY_WAIT_TIME.update(waitTime);
memoryMetrics.getMemoryWaitTimeMetric().change(waitTime);
DeferredFileOutputStream spoolTo = null;
try {
    // Can't be bigger than int, since it's the max of the above allocation
    int size = (int)chunk.getSize();
    spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new File(spoolDirectory))
{
        @Override
        protected void thresholdReached() throws IOException {
            try {
                super.thresholdReached();
            } finally {
                chunk.close();
            }
        }
    };
{code}

The memory chunks successfully created here will not be released until a GC cycle invokes
the MemoryChunk finalizer. Since the default behavoir is to allocate the full "spoolThresholdBytes"
and only resize the chunk after iterating over the scan results, if there is sufficient concurrent
load the GlobablMemoryManager will run out of freememory and cause the SpoolingIterators to
always spool to disk.

{code}
int size = (int)chunk.getSize();
spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new File(spoolDirectory))
{code}

The "size" ends up being 0, causing the "thresholdReached()" method to be invoked when an
attempt is made to write to the SpoolingIterator. 

One way to mitigate this is to reduce the size of "phoenix.query.spoolThresholdBytes". We
ended up reducing it to 10Mb. 

Another would be to use a "Memory Usage" aware outputstream that has a reference to the memory
manager and switches to a spooling only when allocation request to the memory manager fail.
A patch with a simple MemoryManager based outputstream/SpoolingResultIterator is attached.
 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message