impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Apple <jbap...@cloudera.com>
Subject Re: Impala Metadata cache limits
Date Tue, 08 May 2018 13:45:18 GMT
Yes, there are many plans to improve metadata scalability. I expect you can
see some of them by looking through the JIRAs:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20Catalog%20ORDER%20BY%20priority%20DESC

On Tue, May 8, 2018 at 2:12 AM, Prahalad kothwal <kothwaldev@gmail.com>
wrote:

> Thanks for your response, does Impala community have any plans to overcome
> this limitation ? its sounds like we can scale the data and not metadata,
> may be make RPC calls to Hive metastore than cache the metadata ?
>
> Thanks,
> Prahalad
>
> On Thu, May 3, 2018 at 9:35 PM, Alexander Behm <alex.behm@cloudera.com>
> wrote:
>
>> I'd recommend staying below 1GB to avoid OOMing the catalogd or impalads.
>> Going up to 2GB is probably ok but is definitely approaching the danger
>> zone. The main problem here is the JVM 2GB array limit. When serializing
>> the metadata we write to a stream that's backed by a byte array. If that
>> byte array goes beyond 2GB then the JVM will OOM and take down the process.
>> You can hit this limit in various ways, and it can crash the catalogd and
>> impalads.
>>
>> This 2GB limit applies to the uncompressed thrift-serialized size of the
>> metadata.
>>
>> On Thu, May 3, 2018 at 2:26 AM, Prahalad kothwal <kothwaldev@gmail.com>
>> wrote:
>>
>>> Thanks for your response, we are running 2.8.0 and in the process of
>>> upgrading to 2.11.0, we have hundreds of partitioned Impala tables .
>>>
>>> Thanks,
>>> Prahalad
>>>
>>> On Mon, Apr 30, 2018 at 9:35 PM, Alexander Behm <alex.behm@cloudera.com>
>>> wrote:
>>>
>>>> What version of Impala are you running?
>>>>
>>>> On Sun, Apr 29, 2018 at 11:48 PM, Prahalad kothwal <
>>>> kothwaldev@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Is there a limit to amount of metadata Impala can cache or is there a
>>>>> recommendation from Impala community ? we were told not to have more
than
>>>>> 1gb of metadata we have 350gb of ram on each host.
>>>>>
>>>>> Thanks,
>>>>> Prahalad
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message