drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5273) CompliantTextReader exhausts 4 GB memory when reading 5000 small files
Date Fri, 17 Feb 2017 07:49:41 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871363#comment-15871363
] 

Paul Rogers commented on DRILL-5273:
------------------------------------

As it turns out, the solution is simple once we recognize the problem. The {{CompliantTextRecordReader}}
simply allocates normal, non-managed buffers, then releases them in its {{close()}} call.
With this fix, the 5000 file query runs to completion as expected with no OOM.

Note that an even better solution would be to use normal heap buffers for working buffers.
There is really no gain in using a single 1 MB direct buffer per scan: working with the buffer
is expensive, and so is allocating and managing it.

> CompliantTextReader exhausts 4 GB memory when reading 5000 small files
> ----------------------------------------------------------------------
>
>                 Key: DRILL-5273
>                 URL: https://issues.apache.org/jira/browse/DRILL-5273
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>             Fix For: 1.10
>
>
> A test case was created that consists of 5000 text files, each with a single line with
the file number: 1 to 5001. Each file has a single record, and at most 4 characters per record.
> Run the following query:
> {code}
> SELECT * FROM `dfs.data`.`5000files/text
> {code}
> The query will fail with an OOM in the scan batch on around record 3700 on a Mac with
4GB of direct memory.
> The code to read records in {ScanBatch} is complex. The following appears to occur:
> * Iterate over the record readers for each file.
> * For each, call setup
> The setup code is:
> {code}
>   public void setup(OperatorContext context, OutputMutator outputMutator) throws ExecutionSetupException
{
>     oContext = context;
>     readBuffer = context.getManagedBuffer(READ_BUFFER);
>     whitespaceBuffer = context.getManagedBuffer(WHITE_SPACE_BUFFER);
> {code}
> The two buffers are in direct memory. There is no code that releases the buffers.
> The sizes are:
> {code}
>   private static final int READ_BUFFER = 1024*1024;
>   private static final int WHITE_SPACE_BUFFER = 64*1024;
> = 1,048,576 + 65536 = 1,114,112
> {code}
> This is exactly the amount of memory that accumulates per call to {{ScanBatch.next()}}
> {code}
> Ctor: 0  -- Initial memory in constructor
> Init setup: 1114112  -- After call to first record reader setup
> Entry Memory: 1114112  -- first next() call, returns one record
> Entry Memory: 1114112  -- second next(), eof and start second reader
> Entry Memory: 2228224 -- third next(), second reader returns EOF
> ...
> {code}
> If we leak 1 MB per file, with 5000 files we would leak 5 GB of memory, which would explain
the OOM when given only 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message