ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Ignite Durable Memory & Persistent Store introduction for developers
Date Fri, 30 Mar 2018 16:48:52 GMT
Hi Ivan,

Should we update the write-throttling section on readme.io? It makes sense
to do if any recommendations have changed:
https://apacheignite.readme.io/docs/durable-memory-tuning#section-pages-writes-throttling

The compaction technique will be introduced and described on readme:
https://issues.apache.org/jira/browse/IGNITE-8090

--
Denis

On Fri, Mar 30, 2018 at 1:38 AM, Ivan Rakov <ivan.glukos@gmail.com> wrote:

> Igniters,
>
> I've added some info about Pages Write Throttling and WAL compaction to
> the "Ignite Persistent Store - under the hood" document.
> Please let me know if anything is not clear enough.
>
> Best Regards,
> Ivan Rakov
>
>
> On 26.03.2018 0:18, Dmitry Pavlov wrote:
>
>> Hi Igniters,
>>
>> I've done several updates in section
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>> thehood-Regionandsegmentstructure
>>
>> These updates releated to optimizations of LoadedPageTable in Ignite 2.5
>> (Replacement of storage algorithm corresponding to the ID in the page, and
>> the location of a physical address within the segment algorithm is used
>> Robin Hood with backward shift. Previous implementation was not resistant
>> to the ‘page rotation’, a lot of the cells were flooded with Removed
>> state,
>> and new algorithm does not have ‘removed’ state of cell.)
>>
>> New section has now picture of segments structure, and describes locking
>> optimisations in 2.5.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <dpavlov.spb@gmail.com>:
>>
>> Hi Denis,
>>>
>>> I've updated the section in relation to your questions, please see
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>> thehood-IgniteDurableMemory
>>>
>>> I’ve also moved paragraph related to page offset calculation and usage in
>>> files into second article (PDS)
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <dmagda@apache.org>:
>>>
>>> Dmitriy,
>>>>
>>>
>>> First of all, thanks for putting this low level details on paper! Current
>>>> and future community members will be grateful to you.
>>>>
>>>> Second, let me be a reviewer of the page and suggest some improvements.
>>>> Until the review is over I would encourage us to hold on with any public
>>>> announcements such as blog post.
>>>>
>>>> I’ve reviewed and edited Ignite Durable Memory section (
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>> thehood-IgniteDurableMemory
>>>> <
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood#IgniteDurableMemory-under
>>>> thehood-IgniteDurableMemory>).
>>>> Please address the questions below and I’ll move on with the new
>>>> sections
>>>> afterwards.
>>>>
>>>> 1) "Different caches and its partitions bring more complex addressing
>>>> scheme but it is still possible to map from page ID to position in
>>>> particular file: pageId => file + offset in this file”.
>>>>
>>>> Probably this appears too early in the doc because above we just
>>>> introduced page ID notion with durable memory and now we’re talking
>>>> about
>>>> some files? Are these files of ignite persistence? What if the
>>>> persistence
>>>> is disabled? Honestly, the sentence gets out of the story context.
>>>>
>>>>
>>>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>>>> used.”
>>>>
>>>> 50% of the whole memory available? Please clarify.
>>>>
>>>>
>>>> 3) "There is one page ID table per memory segment.”
>>>>
>>>> It’s the same segment as it’s described in the main doc, right?
>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>> ion-memory-segment
>>>> <
>>>> https://apacheignite.readme.io/docs/memory-architecture#sect
>>>> ion-memory-segment
>>>>
>>>> 4) "within unsafe segment”
>>>>
>>>> If we want to mention “usafe” here we need to explain what that is.
>>>>
>>>> 5) "If memory amount less than whole index size, Ignite still can
>>>> operate.”
>>>>
>>>> How do we define “index size”? Is this the page size?
>>>>
>>>>
>>>> —
>>>> Denis
>>>>
>>>>
>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>>>>>
>>>> wrote:
>>>>
>>>>> Hi Igniters,
>>>>>
>>>>>
>>>>> I am glad to introduce two new wiki articles. These articles contain
>>>>> high-level description of
>>>>>
>>>>> - Ignite Durable Memory (
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Durable+Memory+-+under+the+hood
>>>>
>>>>> )
>>>>>
>>>>> - and Ignite Persistent Store (
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+
>>>> Persistent+Store+-+under+the+hood
>>>>
>>>>> )
>>>>>
>>>>> These articles cover basic principles and internal design. Pages may
be
>>>>> useful for new contributors during start of working with persistent
>>>>>
>>>> store
>>>>
>>>>> code base.
>>>>>
>>>>>
>>>>> I would like to thank all contributors who helped me with my questions.
>>>>> Their answers and support made it possible to start this description.
>>>>>
>>>>>
>>>>> I will appreciate any feedback and questions. Suggestions and edits are
>>>>> welcome, of course.
>>>>>
>>>>>
>>>>> Sincerely,
>>>>>
>>>>> Dmitriy Pavlov
>>>>>
>>>>
>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message