impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 廖松博 <liaoson...@gridsum.com>
Subject 答复: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7
Date Sun, 25 Sep 2016 23:46:06 GMT
Thanks Dimitris,  is there a formula to compute the size of the serialized metadata of one
table ? So we can see if it is over 2G.

Thanks again.

发件人: Dimitris Tsirogiannis [mailto:dtsirogiannis@cloudera.com]
发送时间: 2016年9月23日 11:33
收件人: user@impala.incubator.apache.org
抄送: Matthew Jacobs
主题: Re: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

That error message indicates that the size of the serialized metadata of some tables exceed
the maximum allowed size of a byte array (2GB) and doesn't have anything to do with the size
of the heap. The logs indicate a large number of refresh tables that could trigger this kind
of errors, especially if these tables have large metadata and/or incremental stats. You may
want to see if there are any changes in the workload (e.g. more frequent REFRESH statements
than before) and if the value of the statestore_frequency_update_ms was changed after the
upgrade.

On Thu, Sep 22, 2016 at 7:24 PM, 廖松博 <liaosongbo@gridsum.com<mailto:liaosongbo@gridsum.com>>
wrote:
Hi,

         The event is the upgrading from CDH 5.5 -> 5.7.  And the Operation is the regular
operations with Impala 2.3 which we work well with for a long time , but after the upgrading,
it consume so much memory.
         It seems a memory leak, because as follow chart:
[cid:image001.jpg@01D21582.157D2320]
when we use impala 2.3,  it use 20G memory more or less, but when we upgrade to 2.5 and set
Java Heap Size of Catalog Server in Bytes = 32 G, but the catalog server consume the 32G rapidly,
and we reset Java Heap Size of Catalog Server in Bytes to 64G , it also increase to 64G in
a not long time.
So I double if it is a memory leak and I see the issue https://issues.cloudera.org/browse/IMPALA-3568
, is it the root cause ?
And the attached is the catalog server log and it contains the : java.lang.OutOfMemoryError:
Requested array size exceeds VM limit.

1.       Is the root cause of the memory leak?

2.       Any way to find the table which cause the exception?

Thanks.

Songbo

发件人: Dimitris Tsirogiannis [mailto:dtsirogiannis@cloudera.com<mailto:dtsirogiannis@cloudera.com>]
发送时间: 2016年9月23日 8:49
收件人: user@impala.incubator.apache.org<mailto:user@impala.incubator.apache.org>
抄送: Matthew Jacobs
主题: Re: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7

There seem to be some events after sept 21 that trigger a spike in memory usage. Can you identify
from CM what these events might be? For example, refreshing tables with lots of metadata.
Looking at the catalog logs might also give you some hints on what the system is doing during
those memory spikes.

Dimitris

On Thursday, September 22, 2016, 廖松博 <liaosongbo@gridsum.com<mailto:liaosongbo@gridsum.com>>
wrote:

Hi guys,



         We upgrade out CDH cluster from 5.5 to 5.7 at Sep 20, and from the following figure:

         [cid:image001.jpg@01D21582.157D2320]

         You can see the catalog server in impala 2.5 use much more memory than what in impala
2.3 and causes the out of memory error.

         We set the Load Catalog In Background = false and restart the service but not help.

         We set Java Heap Size of Catalog Server in Bytes from 32G to 64G ,but the memory
usage also increase to 64G.



         Is there any bugs in catalog server for impala 2.5? Any suggestion for us?



         Thanks.



Songbo

Mime
View raw message