impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Quanlong Huang" <huang_quanl...@126.com>
Subject Re:Re: How to control max memory usage in JNI
Date Mon, 11 Dec 2017 03:13:22 GMT
+dev


Catalogd will still crash when loading catalog of huge tables with many partitions and files.
Could we add an option 'catalog_mem_limit' just like 'exec_mem_limit' to prevent this? When
memory used in catalog of a loading table exceed this limit, the catalogd should cancel the
loading to protect itself from OOM.
I think it's doable since I simply find this: https://stackoverflow.com/questions/52353/in-java-what-is-the-best-way-to-determine-the-size-of-an-object


Thanks,
Quanlong

On 2017-12-11 10:56:21,"Vincent Tran" <vttran@cloudera.com> wrote:

Hi Yu Feng,


You can try to start catalogd with the "-jvm_args" argument.
Ref: https://github.com/apache/impala/blob/master/bin/start-catalogd.sh#L45



i.e.
bin/start-catalogd.sh  -jvm_args="-Xms4294967296 -Xmx4294967296"




On Dec 10, 2017 9:18 PM, "yu feng" <olaptestyu@gmail.com> wrote:

how impala control the max memory to be allocated in JNICatalogd and JniFronted, I get java
OOM in Catalogd, and doubt how to slove it。


Is there  a way to change it in impala ? maybe set JAVA_TOOL_OPTIONS environment variable
is a choose, But how to do it in impala without modify code.
Mime
View raw message