kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Alves <davidral...@gmail.com>
Subject Re: Question about memory_limit_hard_bytes and
Date Tue, 25 Apr 2017 15:26:11 GMT
Hi Jason

  Yeah that's not expected to work.
  The tserver might accept that config but eventually the block cache will
consume all available memory up to 4GB and everything else will be starved
for memory.

-david

On Tue, Apr 25, 2017 at 3:53 AM, 기준 <0ctopus13prime@gmail.com> wrote:

> @David Alves
>
> So it is not working if i set memory limit as 4G but set 10G as block
> cache mb?
>
> 2017-04-25 2:45 GMT+09:00 David Alves <davidralves@gmail.com>:
>
>> Hi Jason
>>
>>   memory_limit_hard_bytes refers to all the memory consumed by the tablet
>> server. It has an effect in many things but arguably the biggest effect is
>> that the tablet server will reject all writes when memory consumption
>> reaches this limit.
>>   block_cache_capacity_mb dictates how much memory should be saved for
>> the block cache. The block cache caches blocks that are read from disk,
>> thus speeding up scans that hit cached data. The block cache also has some
>> effect on writes, by allowing to cache bloomfilter blocks, which we lookup
>> to make sure that rows are unique. The page cache caches previously fetched
>> disk pages, but is managed by the OS and thus out of Kudu's control.
>>
>> HTH
>> David
>>
>> On Mon, Apr 24, 2017 at 8:24 AM, Jason Heo <jason.heo.sde@gmail.com>
>> wrote:
>>
>>> Hello.
>>>
>>> I'm using Apache Kudu 1.3.0.
>>>
>>> I found some explanation in manual and cdh about memory_limit_hard and
>>> block_cache_capacity But I couldn't understand what are they exactly used
>>> for, what is stored at block_cache, and what's the meaning of hard limit of
>>> memory?
>>>
>>> In Kudu Configuration Reference:
>>>
>>>    - *memory_limit_hard_bytes
>>>    <https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_memory_limit_hard_bytes>*:
>>>    Maximum amount of memory this daemon should use, in bytes. A value of 0
>>>    autosizes based on the total system memory. A value of -1 disables all
>>>    memory limiting.
>>>    - *block_cache_capacity_mb
>>>    <https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_block_cache_capacity_mb>*:
>>>    block cache capacity in MB
>>>
>>>
>>> In Cloudera Manager's help:
>>>
>>>    - *memory_limit_hard_bytes*: Maximum amount of memory that the Kudu
>>>    Tablet Server will use before it starts rejecting all incoming writes.
>>>    - *block_cache_capacity_mb*: Maximum amount of memory allocated to
>>>    the Kudu Tablet Server's block cache.
>>>
>>> Is hard_limit related with insert/update/delete and is not related to
>>> performance of read? Does block_cache store tablet data? Then what's the
>>> relationship with page cache.
>>>
>>> After watching Comcast presentation, I was surprised because they are
>>> using memory_limit_hard=4G, block_cache=512MB.
>>>
>>> Regards,
>>>
>>> Jason
>>>
>>
>>
>

Mime
View raw message