hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi Ozawa <oz...@apache.org>
Subject Re: FileSystem Vs ZKStateStore for RM recovery
Date Fri, 13 Feb 2015 06:59:08 GMT
Hi Suma,

I think ZooKeeper can handle thousands of updates, so thousands of
jobs can be "launched" at the same time.
More jobs can be running at the same time since the number of updates
against ZooKeeper is less than the number of jobs. Please free to ask
us if you face the scalability or performance issue when you test. We
can tackle the issue.

Thanks,
- Tsuyoshi

On Wed, Feb 11, 2015 at 6:08 PM, Suma Shivaprasad
<sumasai.shivaprasad@gmail.com> wrote:
> Can ZKStateStore scale for large clusters. Any idea on the number of
> concurrent jobs that can be supported on top of these ?
>
> Thanks
> Suma
>
> On Wed, Feb 11, 2015 at 1:45 PM, Karthik Kambatla <kasha@cloudera.com>
> wrote:
>>
>> We recommend ZK-store, particularly if you plan to deploy multiple
>> ResourceManagers with failover. ZK-store ensures a single RM has write
>> access and thus is better protected against split-brain cases where both RMs
>> think they are active.
>>
>> On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad
>> <sumasai.shivaprasad@gmail.com> wrote:
>>>
>>> We are planning to deploy Hadoop 2.6.0 with a default configuration to
>>> cache 10000 entries in the state store. With a workload of 150-250
>>> concurrent applications at any time , which state store is better to use
>>> and for what reasons ?
>>>
>>> Thanks
>>> Suma
>>
>>
>>
>>
>> --
>> Karthik Kambatla
>> Software Engineer, Cloudera Inc.
>> --------------------------------------------
>> http://five.sentenc.es
>>
>

Mime
View raw message