impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dimitris Tsirogiannis <dtsirogian...@cloudera.com>
Subject Re: 答复: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7
Date Fri, 23 Sep 2016 03:33:10 GMT
That error message indicates that the size of the serialized metadata of
some tables exceed the maximum allowed size of a byte array (2GB) and
doesn't have anything to do with the size of the heap. The logs indicate a
large number of refresh tables that could trigger this kind of errors,
especially if these tables have large metadata and/or incremental stats.
You may want to see if there are any changes in the workload (e.g. more
frequent REFRESH statements than before) and if the value of the
statestore_frequency_update_ms was changed after the upgrade.

On Thu, Sep 22, 2016 at 7:24 PM, 廖松博 <liaosongbo@gridsum.com> wrote:

> Hi,
>
>
>
>          The event is the upgrading from CDH 5.5 -> 5.7.  And the
> Operation is the regular operations with Impala 2.3 which we work well with
> for a long time , but after the upgrading, it consume so much memory.
>
>          It seems a memory leak, because as follow chart:
>
> [image: cid:image001.jpg@01D21582.157D2320]
>
> when we use impala 2.3,  it use 20G memory more or less, but when we
> upgrade to 2.5 and set Java Heap Size of Catalog Server in Bytes = 32 G,
> but the catalog server consume the 32G rapidly, and we reset Java Heap Size
> of Catalog Server in Bytes to 64G , it also increase to 64G in a not long
> time.
>
> So I double if it is a memory leak and I see the issue
> https://issues.cloudera.org/browse/IMPALA-3568 , is it the root cause ?
>
> And the attached is the catalog server log and it contains the : java.lang.OutOfMemoryError:
> Requested array size exceeds VM limit.
>
> 1.       Is the root cause of the memory leak?
>
> 2.       Any way to find the table which cause the exception?
>
>
>
> Thanks.
>
>
>
> Songbo
>
>
>
> *发件人:* Dimitris Tsirogiannis [mailto:dtsirogiannis@cloudera.com]
> *发送时间:* 2016年9月23日 8:49
> *收件人:* user@impala.incubator.apache.org
> *抄送:* Matthew Jacobs
> *主题:* Re: Catalog Server Out of memory when upgrading from CDH 5.5 -> 5.7
>
>
>
> There seem to be some events after sept 21 that trigger a spike in memory
> usage. Can you identify from CM what these events might be? For
> example, refreshing tables with lots of metadata. Looking at the catalog
> logs might also give you some hints on what the system is doing during
> those memory spikes.
>
>
>
> Dimitris
>
> On Thursday, September 22, 2016, 廖松博 <liaosongbo@gridsum.com> wrote:
>
> Hi guys,
>
>
>
>          We upgrade out CDH cluster from 5.5 to 5.7 at Sep 20, and from
> the following figure:
>
>          [image: cid:image001.jpg@01D21582.157D2320]
>
>          You can see the catalog server in impala 2.5 use much more memory
> than what in impala 2.3 and causes the out of memory error.
>
>          We set the Load Catalog In Background = false and restart the
> service but not help.
>
>          We set Java Heap Size of Catalog Server in Bytes from 32G to 64G
> ,but the memory usage also increase to 64G.
>
>
>
>          Is there any bugs in catalog server for impala 2.5? Any
> suggestion for us?
>
>
>
>          Thanks.
>
>
>
> Songbo
>
>

Mime
View raw message