ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Pavlov <dpavlov....@gmail.com>
Subject Re: Losing data during restarting cluster with persistence enabled
Date Fri, 24 Nov 2017 13:58:52 GMT
Please see the discussion on the user list. It seems that the same happened
there:

http://apache-ignite-users.70518.x6.nabble.com/Reassign-partitions-td7461.html#a7468

it contains examples when the data can diverge.

пт, 24 нояб. 2017 г. в 16:42, Dmitry Pavlov <dpavlov.spb@gmail.com>:

> If we compare native and 3rd party persistence (cache store):
>  - Updating and reading data from DBMS is slower in most scenarios.
>  - Non-clustered DBMS is a single point of failure, it is hard to scale.
>  - Ignite SQL does not extend to External (3rd party persitsence) Cache
> Store (and queries ignore DBMS changes).
>
>
> Which is why I am wondering if Native persistence is applicable in this
> case decribed by Vyacheslav.
>
> пт, 24 нояб. 2017 г. в 12:23, Evgeniy Ignatiev <
> yevgeniy.ignatyev@gmail.com>:
>
>> Sorry linked the wrong page, the latter url is not the example.
>>
>>
>> On 11/24/2017 1:12 PM, Evgeniy Ignatiev wrote:
>> > By the way I remembered that there is an annotation CacheLocalStore
>> > for marking exactly the CacheStore that is not distributed -
>> >
>> http://apache-ignite-developers.2346864.n4.nabble.com/CacheLocalStore-td734.html
>> > - here is short explanation and this -
>> >
>> https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/localstore/LocalRecoverableStoreExample.java
>> > - is example implementation.
>> >
>> >
>> > On 11/23/2017 4:42 PM, Dmitry Pavlov wrote:
>> >> Hi Evgeniy,
>> >>
>> >> Technically it is, of course, possible, but still
>> >> - it is not simple at all
>> >> - IgniteCacheOffheapManager & IgniteWriteAheadLogManager are internal
>> >> APIs,
>> >> and community can change any APIs here in any time.
>> >>
>> >> Vyacheslav,
>> >>
>> >> Why Ignite Native Persistence is not suitable for this case?
>> >>
>> >> Sincerely,
>> >> Dmitriy Pavlov
>> >>
>> >> чт, 23 нояб. 2017 г. в 11:01, Evgeniy Ignatiev
>> >> <yevgeniy.ignatyev@gmail.com
>> >>> :
>> >>> As far as I remember, last webinar I heard on Ignite Native
>> Persistence
>> >>> - it actually exposes some interfaces like IgniteWriteAheadLogManager,
>> >>> PageStore, PageStoreManager, etc., with the file-based implementation
>> >>> provided by Ignite being only one possible approach, and users can
>> >>> create their own Native Persistence variations. At least that what has
>> >>> been said by Denis Magda at that time.
>> >>>
>> >>> May be creating own implementation of Ignite Native Persistence rather
>> >>> than CacheStore based persistence is an option here?
>> >>>
>> >>> On 11/23/2017 2:23 AM, Valentin Kulichenko wrote:
>> >>>> Vyacheslav,
>> >>>>
>> >>>> There is no way to do this and I'm not sure why you want to do this.
>> >>> Ignite
>> >>>> persistence was developed to solve exactly the problems you're
>> >>> describing.
>> >>>> Just use it :)
>> >>>>
>> >>>> -Val
>> >>>>
>> >>>> On Wed, Nov 22, 2017 at 12:36 AM, Vyacheslav Daradur <
>> >>> daradurvs@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Valentin, Evgeniy thanks for your help!
>> >>>>>
>> >>>>> Valentin, unfortunately, you are right.
>> >>>>>
>> >>>>> I've tested that behavior in the following scenario:
>> >>>>> 1. Started N nodes and filled it with data
>> >>>>> 2. Shutdown one node
>> >>>>> 3. Called rebalance directly and waited to finish
>> >>>>> 4. Stopped all other (N-1) nodes
>> >>>>> 5. Started N-1 nodes and validated data
>> >>>>>
>> >>>>> Validation didn't pass - data consistency was broken. As you
say it
>> >>>>> works only on stable topology.
>> >>>>> As far as I understand Ignite doesn't manage to rebalance in
>> >>>>> underlying storage, it became clear from tests and your description
>> >>>>> that CacheStore design assumes that the underlying storage is
shared
>> >>>>> by all the
>> >>>>> nodes in the topology.
>> >>>>>
>> >>>>> I understand that PDS is the best option in case of distributing
>> >>>>> persistence.
>> >>>>> However, could you point me the best way to override default
>> >>>>> rebalance
>> >>>>> behavior?
>> >>>>> Maybe it's possible to extend it by a custom plugin?
>> >>>>>
>> >>>>> On Wed, Nov 22, 2017 at 1:35 AM, Valentin Kulichenko
>> >>>>> <valentin.kulichenko@gmail.com> wrote:
>> >>>>>> Vyacheslav,
>> >>>>>>
>> >>>>>> If you want the persistence storage to be *distributed*,
then using
>> >>>>> Ignite
>> >>>>>> persistence would be the easiest thing to do anyway, even
if you
>> >>>>>> don't
>> >>>>> need
>> >>>>>> all its features.
>> >>>>>>
>> >>>>>> CacheStore indeed can be updated from different nodes with
>> different
>> >>>>> nodes,
>> >>>>>> but the problem is in coordination. If instances of the
store are
>> >>>>>> not
>> >>>>> aware
>> >>>>>> of each other, it's really hard to handle all rebalancing
cases.
>> >>>>>> Such
>> >>>>>> solution will work only on stable topology.
>> >>>>>>
>> >>>>>> Having said that, if you can have one instance of RocksDB
(or any
>> >>>>>> other
>> >>>>> DB
>> >>>>>> for that matter) that is accessed via network by all nodes,
then
>> >>>>>> it's
>> >>>>> also
>> >>>>>> an option. But in this case storage is not distributed.
>> >>>>>>
>> >>>>>> -Val
>> >>>>>>
>> >>>>>> On Tue, Nov 21, 2017 at 4:37 AM, Vyacheslav Daradur <
>> >>> daradurvs@gmail.com
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Valentin,
>> >>>>>>>
>> >>>>>>>>> Why don't you use Ignite persistence [1]?
>> >>>>>>> I have a use case for one of the projects that need
the RAM on
>> disk
>> >>>>>>> replication only. All PDS features aren't needed.
>> >>>>>>> During the first assessment, persist to RocksDB works
faster.
>> >>>>>>>
>> >>>>>>>>> CacheStore design assumes that the underlying
storage is
>> >>>>>>>>> shared by
>> >>>>> all
>> >>>>>>> the nodes in topology.
>> >>>>>>> This is the very important note.
>> >>>>>>> I'm a bit confused because I've thought that each node
in cluster
>> >>>>>>> persists partitions for which the node is either primary
or backup
>> >>>>>>> like in PDS.
>> >>>>>>>
>> >>>>>>> My RocksDB implementation supports working with one
DB instance
>> >>>>>>> which
>> >>>>>>> shared by all the nodes in the topology, but it would
make no
>> >>>>>>> sense of
>> >>>>>>> using embedded fast storage.
>> >>>>>>>
>> >>>>>>> Is there any link to a detailed description of CacheStorage
>> >>>>>>> design or
>> >>>>>>> any other advice?
>> >>>>>>> Thanks in advance.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Fri, Nov 17, 2017 at 9:07 PM, Valentin Kulichenko
>> >>>>>>> <valentin.kulichenko@gmail.com> wrote:
>> >>>>>>>> Vyacheslav,
>> >>>>>>>>
>> >>>>>>>> CacheStore design assumes that the underlying storage
is shared
>> by
>> >>> all
>> >>>>>>> the
>> >>>>>>>> nodes in topology. Even if you delay rebalancing
on node stop
>> >>>>>>>> (which
>> >>>>> is
>> >>>>>>>> possible via CacheConfiguration#rebalanceDelay),
I doubt it will
>> >>>>> solve
>> >>>>>>> all
>> >>>>>>>> your consistency issues.
>> >>>>>>>>
>> >>>>>>>> Why don't you use Ignite persistence [1]?
>> >>>>>>>>
>> >>>>>>>> [1]
>> >>>>>>>> https://apacheignite.readme.io/docs/distributed-persistent-store
>> >>>>>>>>
>> >>>>>>>> -Val
>> >>>>>>>>
>> >>>>>>>> On Fri, Nov 17, 2017 at 4:24 AM, Vyacheslav Daradur
<
>> >>>>> daradurvs@gmail.com
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi Andrey! Thank you for answering.
>> >>>>>>>>>
>> >>>>>>>>>>> Key to partition mapping shouldn't depends
on topology, and
>> >>>>> shouldn't
>> >>>>>>>>> changed unstable topology.
>> >>>>>>>>> Key to partition mapping doesn't depend on topology
in my test
>> >>>>>>>>> affinity function. It only depends on partitions
number.
>> >>>>>>>>> But partition to node mapping depends on topology
and at cluster
>> >>>>> stop,
>> >>>>>>>>> when one node left topology, some partitions
may be moved to
>> >>>>>>>>> other
>> >>>>>>>>> nodes.
>> >>>>>>>>>
>> >>>>>>>>>>> Does all nodes share same RockDB database
or each node has
>> >>>>>>>>>>> its own
>> >>>>>>> copy?
>> >>>>>>>>> Each Ignite node has own RocksDB instance.
>> >>>>>>>>>
>> >>>>>>>>>>> Would you please share configuration?
>> >>>>>>>>> It's pretty simple:
>> >>>>>>>>>           IgniteConfiguration cfg = new IgniteConfiguration();
>> >>>>>>>>>           cfg.setIgniteInstanceName(instanceName);
>> >>>>>>>>>
>> >>>>>>>>>           CacheConfiguration<Integer, String>
cacheCfg = new
>> >>>>>>>>> CacheConfiguration<>();
>> >>>>>>>>>           cacheCfg.setName(TEST_CACHE_NAME);
>> >>>>>>>>> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
>> >>>>>>>>>           cacheCfg.setWriteSynchronizationMode(
>> >>>>>>>>> CacheWriteSynchronizationMode.PRIMARY_SYNC);
>> >>>>>>>>>           cacheCfg.setBackups(1);
>> >>>>>>>>>           cacheCfg.setAffinity(new
>> >>>>>>>>> TestAffinityFunction(partitionsNumber, backupsNumber));
>> >>>>>>>>>           cacheCfg.setWriteThrough(true);
>> >>>>>>>>>           cacheCfg.setReadThrough(true);
>> >>>>>>>>> cacheCfg.setRebalanceMode(CacheRebalanceMode.SYNC);
>> >>>>>>>>>           cacheCfg.setCacheStoreFactory(new
>> >>>>>>>>> RocksDBCacheStoreFactory<>("/test/path/to/persistence",
>> >>>>>>>>> TEST_CACHE_NAME, cfg));
>> >>>>>>>>>
>> >>>>>>>>>           cfg.setCacheConfiguration(cacheCfg);
>> >>>>>>>>>
>> >>>>>>>>> Could you give me advice on places which I need
to pay
>> attention?
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Nov 15, 2017 at 3:02 PM, Andrey Mashenkov
>> >>>>>>>>> <andrey.mashenkov@gmail.com> wrote:
>> >>>>>>>>>> Hi Vyacheslav,
>> >>>>>>>>>>
>> >>>>>>>>>> Key to partition mapping shouldn't depends
on topology, and
>> >>>>> shouldn't
>> >>>>>>>>>> changed unstable topology.
>> >>>>>>>>>> Looks like you've missed smth.
>> >>>>>>>>>>
>> >>>>>>>>>> Would you please share configuration?
>> >>>>>>>>>> Does all nodes share same RockDB database
or each node has
>> >>>>>>>>>> its own
>> >>>>>>> copy?
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Nov 15, 2017 at 12:22 AM, Vyacheslav
Daradur <
>> >>>>>>>>> daradurvs@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hi, Igniters!
>> >>>>>>>>>>>
>> >>>>>>>>>>> I’m using partitioned Ignite cache
with RocksDB as 3rd party
>> >>>>>>> persistence
>> >>>>>>>>>>> store.
>> >>>>>>>>>>> I've got an issue: if cache rebalancing
is switched on, then
>> >>>>>>>>>>> it’s
>> >>>>>>>>>>> possible to lose some data.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Basic scenario:
>> >>>>>>>>>>> 1) Start Ignite cluster and fill a cache
with RocksDB
>> >>>>>>>>>>> persistence;
>> >>>>>>>>>>> 2) Stop all nodes
>> >>>>>>>>>>> 3) Start Ignite cluster and validate
data
>> >>>>>>>>>>>
>> >>>>>>>>>>> This works fine while rebalancing is
switched off.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If rebalancing switched on: when I call
Ignition#stopAll, some
>> >>>>> nodes
>> >>>>>>>>>>> go down sequentially and while one node
having gone down
>> >>>>>>>>>>> another
>> >>>>>>> start
>> >>>>>>>>>>> rebalancing. When nodes started affinity
function works with a
>> >>>>> full
>> >>>>>>>>>>> set of nodes and may define a wrong
partition for a key
>> because
>> >>>>> the
>> >>>>>>>>>>> previous state was changed at rebalancing.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Maybe I'm doing something wrong. How
can I avoid rebalancing
>> >>>>>>>>>>> while
>> >>>>>>>>>>> stopping all nodes in the cluster?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Could you give me any advice, please?
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> Best Regards, Vyacheslav D.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Best regards,
>> >>>>>>>>>> Andrey V. Mashenkov
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> Best Regards, Vyacheslav D.
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Best Regards, Vyacheslav D.
>> >>>>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Best Regards, Vyacheslav D.
>> >>>>>
>> >>>
>> >
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message