ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Rakov <ivan.glu...@gmail.com>
Subject Re: Ignite Durable Memory & Persistent Store introduction for developers
Date Fri, 30 Mar 2018 08:38:15 GMT
Igniters,

I've added some info about Pages Write Throttling and WAL compaction to 
the "Ignite Persistent Store - under the hood" document.
Please let me know if anything is not clear enough.

Best Regards,
Ivan Rakov

On 26.03.2018 0:18, Dmitry Pavlov wrote:
> Hi Igniters,
>
> I've done several updates in section
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Regionandsegmentstructure
>
> These updates releated to optimizations of LoadedPageTable in Ignite 2.5
> (Replacement of storage algorithm corresponding to the ID in the page, and
> the location of a physical address within the segment algorithm is used
> Robin Hood with backward shift. Previous implementation was not resistant
> to the ‘page rotation’, a lot of the cells were flooded with Removed state,
> and new algorithm does not have ‘removed’ state of cell.)
>
> New section has now picture of segments structure, and describes locking
> optimisations in 2.5.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 19 окт. 2017 г. в 22:33, Dmitry Pavlov <dpavlov.spb@gmail.com>:
>
>> Hi Denis,
>>
>> I've updated the section in relation to your questions, please see
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
>>
>> I’ve also moved paragraph related to page offset calculation and usage in
>> files into second article (PDS)
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пт, 1 сент. 2017 г. в 19:50, Denis Magda <dmagda@apache.org>:
>>
>>> Dmitriy,
>>
>>> First of all, thanks for putting this low level details on paper! Current
>>> and future community members will be grateful to you.
>>>
>>> Second, let me be a reviewer of the page and suggest some improvements.
>>> Until the review is over I would encourage us to hold on with any public
>>> announcements such as blog post.
>>>
>>> I’ve reviewed and edited Ignite Durable Memory section (
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory
>>> <
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-IgniteDurableMemory>).
>>> Please address the questions below and I’ll move on with the new sections
>>> afterwards.
>>>
>>> 1) "Different caches and its partitions bring more complex addressing
>>> scheme but it is still possible to map from page ID to position in
>>> particular file: pageId => file + offset in this file”.
>>>
>>> Probably this appears too early in the doc because above we just
>>> introduced page ID notion with durable memory and now we’re talking about
>>> some files? Are these files of ignite persistence? What if the persistence
>>> is disabled? Honestly, the sentence gets out of the story context.
>>>
>>>
>>> 2) "Also, the tree pages can be merged into one if < 50% of space is
>>> used.”
>>>
>>> 50% of the whole memory available? Please clarify.
>>>
>>>
>>> 3) "There is one page ID table per memory segment.”
>>>
>>> It’s the same segment as it’s described in the main doc, right?
>>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
>>> <
>>> https://apacheignite.readme.io/docs/memory-architecture#section-memory-segment
>>>
>>> 4) "within unsafe segment”
>>>
>>> If we want to mention “usafe” here we need to explain what that is.
>>>
>>> 5) "If memory amount less than whole index size, Ignite still can
>>> operate.”
>>>
>>> How do we define “index size”? Is this the page size?
>>>
>>>
>>> —
>>> Denis
>>>
>>>
>>>> On Aug 30, 2017, at 8:24 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>>> wrote:
>>>> Hi Igniters,
>>>>
>>>>
>>>> I am glad to introduce two new wiki articles. These articles contain
>>>> high-level description of
>>>>
>>>> - Ignite Durable Memory (
>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
>>>> )
>>>>
>>>> - and Ignite Persistent Store (
>>>>
>>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>>>> )
>>>>
>>>> These articles cover basic principles and internal design. Pages may be
>>>> useful for new contributors during start of working with persistent
>>> store
>>>> code base.
>>>>
>>>>
>>>> I would like to thank all contributors who helped me with my questions.
>>>> Their answers and support made it possible to start this description.
>>>>
>>>>
>>>> I will appreciate any feedback and questions. Suggestions and edits are
>>>> welcome, of course.
>>>>
>>>>
>>>> Sincerely,
>>>>
>>>> Dmitriy Pavlov
>>>


Mime
View raw message