orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruve Ashar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ORC-476) Make SearchAgument kryo buffer size configurable
Date Wed, 06 Mar 2019 23:48:00 GMT
Dhruve Ashar created ORC-476:
--------------------------------

             Summary: Make SearchAgument kryo buffer size configurable
                 Key: ORC-476
                 URL: https://issues.apache.org/jira/browse/ORC-476
             Project: ORC
          Issue Type: Improvement
          Components: MapReduce
            Reporter: Dhruve Ashar


The kryo output buffer size for SArg is hardcoded right now to 100000. The hive implementation
sets this to an initial size of 4096 and max size to 10485760.

 

Spark started using the apache orc implementation from 2.3 compared to the hive implementation.
Spark jobs are failing with buffer overflow error due to the small size of the buffer here.
We should make this configurable so that frameworks using the orc implementation can pass
the size configs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message