impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Armstrong <tarmstr...@cloudera.com>
Subject Re: Re: How to control max memory usage in JNI
Date Mon, 11 Dec 2017 16:44:55 GMT
The catalog is cached on all Impala daemons so you also need to increase
the impalad heap size too. There's a brief explanation in the docs:
https://impala.apache.org/docs/build/html/topics/impala_scalability.html
(see the "Impact of Many Tables or Partitions on Impala Catalog Performance
and Memory Usage" section).

On Sun, Dec 10, 2017 at 7:13 PM, Quanlong Huang <huang_quanlong@126.com>
wrote:

> +dev
>
> Catalogd will still crash when loading catalog of huge tables with many
> partitions and files.
> Could we add an option 'catalog_mem_limit' just like 'exec_mem_limit' to
> prevent this? When memory used in catalog of a loading table exceed this
> limit, the catalogd should cancel the loading to protect itself from OOM.
> I think it's doable since I simply find this: https://stackoverflow.
> com/questions/52353/in-java-what-is-the-best-way-to-
> determine-the-size-of-an-object
>
> Thanks,
> Quanlong
>
> On 2017-12-11 10:56:21,"Vincent Tran" <vttran@cloudera.com> wrote:
>
> Hi Yu Feng,
>
> You can try to start catalogd with the "-jvm_args" argument.
> Ref: https://github.com/apache/impala/blob/master/bin/start-
> catalogd.sh#L45
>
> i.e.
> bin/start-catalogd.sh  -jvm_args="-Xms4294967296 -Xmx4294967296"
>
>
> On Dec 10, 2017 9:18 PM, "yu feng" <olaptestyu@gmail.com> wrote:
>
>> how impala control the max memory to be allocated in JNICatalogd and
>> JniFronted, I get java OOM in Catalogd, and doubt how to slove it。
>>
>> Is there  a way to change it in impala ? maybe set JAVA_TOOL_OPTIONS
>> environment variable is a choose, But how to do it in impala without modify
>> code.
>>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message