ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evgeniy Ignatiev <yevgeniy.ignat...@gmail.com>
Subject Re: Losing data during restarting cluster with persistence enabled
Date Fri, 24 Nov 2017 09:23:07 GMT
Sorry linked the wrong page, the latter url is not the example.


On 11/24/2017 1:12 PM, Evgeniy Ignatiev wrote:
> By the way I remembered that there is an annotation CacheLocalStore 
> for marking exactly the CacheStore that is not distributed - 
> http://apache-ignite-developers.2346864.n4.nabble.com/CacheLocalStore-td734.html 
> - here is short explanation and this - 
> https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/localstore/LocalRecoverableStoreExample.java

> - is example implementation.
>
>
> On 11/23/2017 4:42 PM, Dmitry Pavlov wrote:
>> Hi Evgeniy,
>>
>> Technically it is, of course, possible, but still
>> - it is not simple at all
>> - IgniteCacheOffheapManager & IgniteWriteAheadLogManager are internal 
>> APIs,
>> and community can change any APIs here in any time.
>>
>> Vyacheslav,
>>
>> Why Ignite Native Persistence is not suitable for this case?
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> чт, 23 нояб. 2017 г. в 11:01, Evgeniy Ignatiev 
>> <yevgeniy.ignatyev@gmail.com
>>> :
>>> As far as I remember, last webinar I heard on Ignite Native Persistence
>>> - it actually exposes some interfaces like IgniteWriteAheadLogManager,
>>> PageStore, PageStoreManager, etc., with the file-based implementation
>>> provided by Ignite being only one possible approach, and users can
>>> create their own Native Persistence variations. At least that what has
>>> been said by Denis Magda at that time.
>>>
>>> May be creating own implementation of Ignite Native Persistence rather
>>> than CacheStore based persistence is an option here?
>>>
>>> On 11/23/2017 2:23 AM, Valentin Kulichenko wrote:
>>>> Vyacheslav,
>>>>
>>>> There is no way to do this and I'm not sure why you want to do this.
>>> Ignite
>>>> persistence was developed to solve exactly the problems you're
>>> describing.
>>>> Just use it :)
>>>>
>>>> -Val
>>>>
>>>> On Wed, Nov 22, 2017 at 12:36 AM, Vyacheslav Daradur <
>>> daradurvs@gmail.com>
>>>> wrote:
>>>>
>>>>> Valentin, Evgeniy thanks for your help!
>>>>>
>>>>> Valentin, unfortunately, you are right.
>>>>>
>>>>> I've tested that behavior in the following scenario:
>>>>> 1. Started N nodes and filled it with data
>>>>> 2. Shutdown one node
>>>>> 3. Called rebalance directly and waited to finish
>>>>> 4. Stopped all other (N-1) nodes
>>>>> 5. Started N-1 nodes and validated data
>>>>>
>>>>> Validation didn't pass - data consistency was broken. As you say it
>>>>> works only on stable topology.
>>>>> As far as I understand Ignite doesn't manage to rebalance in
>>>>> underlying storage, it became clear from tests and your description
>>>>> that CacheStore design assumes that the underlying storage is shared
>>>>> by all the
>>>>> nodes in the topology.
>>>>>
>>>>> I understand that PDS is the best option in case of distributing
>>>>> persistence.
>>>>> However, could you point me the best way to override default 
>>>>> rebalance
>>>>> behavior?
>>>>> Maybe it's possible to extend it by a custom plugin?
>>>>>
>>>>> On Wed, Nov 22, 2017 at 1:35 AM, Valentin Kulichenko
>>>>> <valentin.kulichenko@gmail.com> wrote:
>>>>>> Vyacheslav,
>>>>>>
>>>>>> If you want the persistence storage to be *distributed*, then using
>>>>> Ignite
>>>>>> persistence would be the easiest thing to do anyway, even if you

>>>>>> don't
>>>>> need
>>>>>> all its features.
>>>>>>
>>>>>> CacheStore indeed can be updated from different nodes with different
>>>>> nodes,
>>>>>> but the problem is in coordination. If instances of the store are

>>>>>> not
>>>>> aware
>>>>>> of each other, it's really hard to handle all rebalancing cases.

>>>>>> Such
>>>>>> solution will work only on stable topology.
>>>>>>
>>>>>> Having said that, if you can have one instance of RocksDB (or any

>>>>>> other
>>>>> DB
>>>>>> for that matter) that is accessed via network by all nodes, then

>>>>>> it's
>>>>> also
>>>>>> an option. But in this case storage is not distributed.
>>>>>>
>>>>>> -Val
>>>>>>
>>>>>> On Tue, Nov 21, 2017 at 4:37 AM, Vyacheslav Daradur <
>>> daradurvs@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> Valentin,
>>>>>>>
>>>>>>>>> Why don't you use Ignite persistence [1]?
>>>>>>> I have a use case for one of the projects that need the RAM on
disk
>>>>>>> replication only. All PDS features aren't needed.
>>>>>>> During the first assessment, persist to RocksDB works faster.
>>>>>>>
>>>>>>>>> CacheStore design assumes that the underlying storage
is 
>>>>>>>>> shared by
>>>>> all
>>>>>>> the nodes in topology.
>>>>>>> This is the very important note.
>>>>>>> I'm a bit confused because I've thought that each node in cluster
>>>>>>> persists partitions for which the node is either primary or backup
>>>>>>> like in PDS.
>>>>>>>
>>>>>>> My RocksDB implementation supports working with one DB instance

>>>>>>> which
>>>>>>> shared by all the nodes in the topology, but it would make no

>>>>>>> sense of
>>>>>>> using embedded fast storage.
>>>>>>>
>>>>>>> Is there any link to a detailed description of CacheStorage 
>>>>>>> design or
>>>>>>> any other advice?
>>>>>>> Thanks in advance.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 17, 2017 at 9:07 PM, Valentin Kulichenko
>>>>>>> <valentin.kulichenko@gmail.com> wrote:
>>>>>>>> Vyacheslav,
>>>>>>>>
>>>>>>>> CacheStore design assumes that the underlying storage is
shared by
>>> all
>>>>>>> the
>>>>>>>> nodes in topology. Even if you delay rebalancing on node
stop 
>>>>>>>> (which
>>>>> is
>>>>>>>> possible via CacheConfiguration#rebalanceDelay), I doubt
it will
>>>>> solve
>>>>>>> all
>>>>>>>> your consistency issues.
>>>>>>>>
>>>>>>>> Why don't you use Ignite persistence [1]?
>>>>>>>>
>>>>>>>> [1] 
>>>>>>>> https://apacheignite.readme.io/docs/distributed-persistent-store
>>>>>>>>
>>>>>>>> -Val
>>>>>>>>
>>>>>>>> On Fri, Nov 17, 2017 at 4:24 AM, Vyacheslav Daradur <
>>>>> daradurvs@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Andrey! Thank you for answering.
>>>>>>>>>
>>>>>>>>>>> Key to partition mapping shouldn't depends on
topology, and
>>>>> shouldn't
>>>>>>>>> changed unstable topology.
>>>>>>>>> Key to partition mapping doesn't depend on topology in
my test
>>>>>>>>> affinity function. It only depends on partitions number.
>>>>>>>>> But partition to node mapping depends on topology and
at cluster
>>>>> stop,
>>>>>>>>> when one node left topology, some partitions may be moved
to 
>>>>>>>>> other
>>>>>>>>> nodes.
>>>>>>>>>
>>>>>>>>>>> Does all nodes share same RockDB database or
each node has 
>>>>>>>>>>> its own
>>>>>>> copy?
>>>>>>>>> Each Ignite node has own RocksDB instance.
>>>>>>>>>
>>>>>>>>>>> Would you please share configuration?
>>>>>>>>> It's pretty simple:
>>>>>>>>>           IgniteConfiguration cfg = new IgniteConfiguration();
>>>>>>>>>           cfg.setIgniteInstanceName(instanceName);
>>>>>>>>>
>>>>>>>>>           CacheConfiguration<Integer, String>
cacheCfg = new
>>>>>>>>> CacheConfiguration<>();
>>>>>>>>>           cacheCfg.setName(TEST_CACHE_NAME);
>>>>>>>>> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
>>>>>>>>>           cacheCfg.setWriteSynchronizationMode(
>>>>>>>>> CacheWriteSynchronizationMode.PRIMARY_SYNC);
>>>>>>>>>           cacheCfg.setBackups(1);
>>>>>>>>>           cacheCfg.setAffinity(new
>>>>>>>>> TestAffinityFunction(partitionsNumber, backupsNumber));
>>>>>>>>>           cacheCfg.setWriteThrough(true);
>>>>>>>>>           cacheCfg.setReadThrough(true);
>>>>>>>>> cacheCfg.setRebalanceMode(CacheRebalanceMode.SYNC);
>>>>>>>>>           cacheCfg.setCacheStoreFactory(new
>>>>>>>>> RocksDBCacheStoreFactory<>("/test/path/to/persistence",
>>>>>>>>> TEST_CACHE_NAME, cfg));
>>>>>>>>>
>>>>>>>>>           cfg.setCacheConfiguration(cacheCfg);
>>>>>>>>>
>>>>>>>>> Could you give me advice on places which I need to pay
attention?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 15, 2017 at 3:02 PM, Andrey Mashenkov
>>>>>>>>> <andrey.mashenkov@gmail.com> wrote:
>>>>>>>>>> Hi Vyacheslav,
>>>>>>>>>>
>>>>>>>>>> Key to partition mapping shouldn't depends on topology,
and
>>>>> shouldn't
>>>>>>>>>> changed unstable topology.
>>>>>>>>>> Looks like you've missed smth.
>>>>>>>>>>
>>>>>>>>>> Would you please share configuration?
>>>>>>>>>> Does all nodes share same RockDB database or each
node has 
>>>>>>>>>> its own
>>>>>>> copy?
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 15, 2017 at 12:22 AM, Vyacheslav Daradur
<
>>>>>>>>> daradurvs@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>>
>>>>>>>>>>> I’m using partitioned Ignite cache with RocksDB
as 3rd party
>>>>>>> persistence
>>>>>>>>>>> store.
>>>>>>>>>>> I've got an issue: if cache rebalancing is switched
on, then 
>>>>>>>>>>> it’s
>>>>>>>>>>> possible to lose some data.
>>>>>>>>>>>
>>>>>>>>>>> Basic scenario:
>>>>>>>>>>> 1) Start Ignite cluster and fill a cache with
RocksDB 
>>>>>>>>>>> persistence;
>>>>>>>>>>> 2) Stop all nodes
>>>>>>>>>>> 3) Start Ignite cluster and validate data
>>>>>>>>>>>
>>>>>>>>>>> This works fine while rebalancing is switched
off.
>>>>>>>>>>>
>>>>>>>>>>> If rebalancing switched on: when I call Ignition#stopAll,
some
>>>>> nodes
>>>>>>>>>>> go down sequentially and while one node having
gone down 
>>>>>>>>>>> another
>>>>>>> start
>>>>>>>>>>> rebalancing. When nodes started affinity function
works with a
>>>>> full
>>>>>>>>>>> set of nodes and may define a wrong partition
for a key because
>>>>> the
>>>>>>>>>>> previous state was changed at rebalancing.
>>>>>>>>>>>
>>>>>>>>>>> Maybe I'm doing something wrong. How can I avoid
rebalancing 
>>>>>>>>>>> while
>>>>>>>>>>> stopping all nodes in the cluster?
>>>>>>>>>>>
>>>>>>>>>>> Could you give me any advice, please?
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Best Regards, Vyacheslav D.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Best regards,
>>>>>>>>>> Andrey V. Mashenkov
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Best Regards, Vyacheslav D.
>>>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Best Regards, Vyacheslav D.
>>>>>>>
>>>>>
>>>>> -- 
>>>>> Best Regards, Vyacheslav D.
>>>>>
>>>
>


Mime
View raw message