impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prahalad kothwal <kothwal...@gmail.com>
Subject Re: Impala Metadata cache limits
Date Tue, 08 May 2018 09:12:11 GMT
Thanks for your response, does Impala community have any plans to overcome
this limitation ? its sounds like we can scale the data and not metadata,
may be make RPC calls to Hive metastore than cache the metadata ?

Thanks,
Prahalad

On Thu, May 3, 2018 at 9:35 PM, Alexander Behm <alex.behm@cloudera.com>
wrote:

> I'd recommend staying below 1GB to avoid OOMing the catalogd or impalads.
> Going up to 2GB is probably ok but is definitely approaching the danger
> zone. The main problem here is the JVM 2GB array limit. When serializing
> the metadata we write to a stream that's backed by a byte array. If that
> byte array goes beyond 2GB then the JVM will OOM and take down the process.
> You can hit this limit in various ways, and it can crash the catalogd and
> impalads.
>
> This 2GB limit applies to the uncompressed thrift-serialized size of the
> metadata.
>
> On Thu, May 3, 2018 at 2:26 AM, Prahalad kothwal <kothwaldev@gmail.com>
> wrote:
>
>> Thanks for your response, we are running 2.8.0 and in the process of
>> upgrading to 2.11.0, we have hundreds of partitioned Impala tables .
>>
>> Thanks,
>> Prahalad
>>
>> On Mon, Apr 30, 2018 at 9:35 PM, Alexander Behm <alex.behm@cloudera.com>
>> wrote:
>>
>>> What version of Impala are you running?
>>>
>>> On Sun, Apr 29, 2018 at 11:48 PM, Prahalad kothwal <kothwaldev@gmail.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> Is there a limit to amount of metadata Impala can cache or is there a
>>>> recommendation from Impala community ? we were told not to have more than
>>>> 1gb of metadata we have 350gb of ram on each host.
>>>>
>>>> Thanks,
>>>> Prahalad
>>>>
>>>>
>>>
>>
>

Mime
View raw message