ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Losing data during restarting cluster with persistence enabled
Date Wed, 06 Dec 2017 18:24:12 GMT
Vyacheslav,

In this case community should definitely take a look and investigate.
Please share your results when you have a chance.

-Val

On Wed, Dec 6, 2017 at 1:45 AM, Vyacheslav Daradur <daradurvs@gmail.com>
wrote:

> Evgeniy, as far as I understand PDS and rebalancing are based on
> page-memory approach instead of entry-based 3rd Party Persistence, so
> I'm not sure how to extend rebalancing behavior properly.
>
> Dmitry, the performance is the only reason of why I try to solve
> rebalancing issue.
> I've benchmarked RocksDB as 3rd party persistence and PDS via Ignite
> Yardstick with "fsync" enabled in both cases.
> The result shows that PDS is twice slower on "put" operation on the
> single node, but I had had no time to do benchmarks on all sides.
> I'll try to do that next week and will share results if the community
> is interested. Maybe there will be no reason for using RocksDB.
>
>
>
> On Fri, Nov 24, 2017 at 4:58 PM, Dmitry Pavlov <dpavlov.spb@gmail.com>
> wrote:
> > Please see the discussion on the user list. It seems that the same
> happened
> > there:
> >
> > http://apache-ignite-users.70518.x6.nabble.com/Reassign-
> partitions-td7461.html#a7468
> >
> > it contains examples when the data can diverge.
> >
> > пт, 24 нояб. 2017 г. в 16:42, Dmitry Pavlov <dpavlov.spb@gmail.com>:
> >
> >> If we compare native and 3rd party persistence (cache store):
> >>  - Updating and reading data from DBMS is slower in most scenarios.
> >>  - Non-clustered DBMS is a single point of failure, it is hard to scale.
> >>  - Ignite SQL does not extend to External (3rd party persitsence) Cache
> >> Store (and queries ignore DBMS changes).
> >>
> >>
> >> Which is why I am wondering if Native persistence is applicable in this
> >> case decribed by Vyacheslav.
> >>
> >> пт, 24 нояб. 2017 г. в 12:23, Evgeniy Ignatiev <
> >> yevgeniy.ignatyev@gmail.com>:
> >>
> >>> Sorry linked the wrong page, the latter url is not the example.
> >>>
> >>>
> >>> On 11/24/2017 1:12 PM, Evgeniy Ignatiev wrote:
> >>> > By the way I remembered that there is an annotation CacheLocalStore
> >>> > for marking exactly the CacheStore that is not distributed -
> >>> >
> >>> http://apache-ignite-developers.2346864.n4.nabble.
> com/CacheLocalStore-td734.html
> >>> > - here is short explanation and this -
> >>> >
> >>> https://github.com/gridgain/gridgain-advanced-examples/
> blob/master/src/main/java/org/gridgain/examples/localstore/
> LocalRecoverableStoreExample.java
> >>> > - is example implementation.
> >>> >
> >>> >
> >>> > On 11/23/2017 4:42 PM, Dmitry Pavlov wrote:
> >>> >> Hi Evgeniy,
> >>> >>
> >>> >> Technically it is, of course, possible, but still
> >>> >> - it is not simple at all
> >>> >> - IgniteCacheOffheapManager & IgniteWriteAheadLogManager are
> internal
> >>> >> APIs,
> >>> >> and community can change any APIs here in any time.
> >>> >>
> >>> >> Vyacheslav,
> >>> >>
> >>> >> Why Ignite Native Persistence is not suitable for this case?
> >>> >>
> >>> >> Sincerely,
> >>> >> Dmitriy Pavlov
> >>> >>
> >>> >> чт, 23 нояб. 2017 г. в 11:01, Evgeniy Ignatiev
> >>> >> <yevgeniy.ignatyev@gmail.com
> >>> >>> :
> >>> >>> As far as I remember, last webinar I heard on Ignite Native
> >>> Persistence
> >>> >>> - it actually exposes some interfaces like
> IgniteWriteAheadLogManager,
> >>> >>> PageStore, PageStoreManager, etc., with the file-based
> implementation
> >>> >>> provided by Ignite being only one possible approach, and users
can
> >>> >>> create their own Native Persistence variations. At least that
what
> has
> >>> >>> been said by Denis Magda at that time.
> >>> >>>
> >>> >>> May be creating own implementation of Ignite Native Persistence
> rather
> >>> >>> than CacheStore based persistence is an option here?
> >>> >>>
> >>> >>> On 11/23/2017 2:23 AM, Valentin Kulichenko wrote:
> >>> >>>> Vyacheslav,
> >>> >>>>
> >>> >>>> There is no way to do this and I'm not sure why you want
to do
> this.
> >>> >>> Ignite
> >>> >>>> persistence was developed to solve exactly the problems
you're
> >>> >>> describing.
> >>> >>>> Just use it :)
> >>> >>>>
> >>> >>>> -Val
> >>> >>>>
> >>> >>>> On Wed, Nov 22, 2017 at 12:36 AM, Vyacheslav Daradur <
> >>> >>> daradurvs@gmail.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> Valentin, Evgeniy thanks for your help!
> >>> >>>>>
> >>> >>>>> Valentin, unfortunately, you are right.
> >>> >>>>>
> >>> >>>>> I've tested that behavior in the following scenario:
> >>> >>>>> 1. Started N nodes and filled it with data
> >>> >>>>> 2. Shutdown one node
> >>> >>>>> 3. Called rebalance directly and waited to finish
> >>> >>>>> 4. Stopped all other (N-1) nodes
> >>> >>>>> 5. Started N-1 nodes and validated data
> >>> >>>>>
> >>> >>>>> Validation didn't pass - data consistency was broken.
As you say
> it
> >>> >>>>> works only on stable topology.
> >>> >>>>> As far as I understand Ignite doesn't manage to rebalance
in
> >>> >>>>> underlying storage, it became clear from tests and
your
> description
> >>> >>>>> that CacheStore design assumes that the underlying
storage is
> shared
> >>> >>>>> by all the
> >>> >>>>> nodes in the topology.
> >>> >>>>>
> >>> >>>>> I understand that PDS is the best option in case of
distributing
> >>> >>>>> persistence.
> >>> >>>>> However, could you point me the best way to override
default
> >>> >>>>> rebalance
> >>> >>>>> behavior?
> >>> >>>>> Maybe it's possible to extend it by a custom plugin?
> >>> >>>>>
> >>> >>>>> On Wed, Nov 22, 2017 at 1:35 AM, Valentin Kulichenko
> >>> >>>>> <valentin.kulichenko@gmail.com> wrote:
> >>> >>>>>> Vyacheslav,
> >>> >>>>>>
> >>> >>>>>> If you want the persistence storage to be *distributed*,
then
> using
> >>> >>>>> Ignite
> >>> >>>>>> persistence would be the easiest thing to do anyway,
even if you
> >>> >>>>>> don't
> >>> >>>>> need
> >>> >>>>>> all its features.
> >>> >>>>>>
> >>> >>>>>> CacheStore indeed can be updated from different
nodes with
> >>> different
> >>> >>>>> nodes,
> >>> >>>>>> but the problem is in coordination. If instances
of the store
> are
> >>> >>>>>> not
> >>> >>>>> aware
> >>> >>>>>> of each other, it's really hard to handle all rebalancing
cases.
> >>> >>>>>> Such
> >>> >>>>>> solution will work only on stable topology.
> >>> >>>>>>
> >>> >>>>>> Having said that, if you can have one instance
of RocksDB (or
> any
> >>> >>>>>> other
> >>> >>>>> DB
> >>> >>>>>> for that matter) that is accessed via network by
all nodes, then
> >>> >>>>>> it's
> >>> >>>>> also
> >>> >>>>>> an option. But in this case storage is not distributed.
> >>> >>>>>>
> >>> >>>>>> -Val
> >>> >>>>>>
> >>> >>>>>> On Tue, Nov 21, 2017 at 4:37 AM, Vyacheslav Daradur
<
> >>> >>> daradurvs@gmail.com
> >>> >>>>>> wrote:
> >>> >>>>>>
> >>> >>>>>>> Valentin,
> >>> >>>>>>>
> >>> >>>>>>>>> Why don't you use Ignite persistence
[1]?
> >>> >>>>>>> I have a use case for one of the projects that
need the RAM on
> >>> disk
> >>> >>>>>>> replication only. All PDS features aren't needed.
> >>> >>>>>>> During the first assessment, persist to RocksDB
works faster.
> >>> >>>>>>>
> >>> >>>>>>>>> CacheStore design assumes that the
underlying storage is
> >>> >>>>>>>>> shared by
> >>> >>>>> all
> >>> >>>>>>> the nodes in topology.
> >>> >>>>>>> This is the very important note.
> >>> >>>>>>> I'm a bit confused because I've thought that
each node in
> cluster
> >>> >>>>>>> persists partitions for which the node is either
primary or
> backup
> >>> >>>>>>> like in PDS.
> >>> >>>>>>>
> >>> >>>>>>> My RocksDB implementation supports working
with one DB instance
> >>> >>>>>>> which
> >>> >>>>>>> shared by all the nodes in the topology, but
it would make no
> >>> >>>>>>> sense of
> >>> >>>>>>> using embedded fast storage.
> >>> >>>>>>>
> >>> >>>>>>> Is there any link to a detailed description
of CacheStorage
> >>> >>>>>>> design or
> >>> >>>>>>> any other advice?
> >>> >>>>>>> Thanks in advance.
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> On Fri, Nov 17, 2017 at 9:07 PM, Valentin Kulichenko
> >>> >>>>>>> <valentin.kulichenko@gmail.com> wrote:
> >>> >>>>>>>> Vyacheslav,
> >>> >>>>>>>>
> >>> >>>>>>>> CacheStore design assumes that the underlying
storage is
> shared
> >>> by
> >>> >>> all
> >>> >>>>>>> the
> >>> >>>>>>>> nodes in topology. Even if you delay rebalancing
on node stop
> >>> >>>>>>>> (which
> >>> >>>>> is
> >>> >>>>>>>> possible via CacheConfiguration#rebalanceDelay),
I doubt it
> will
> >>> >>>>> solve
> >>> >>>>>>> all
> >>> >>>>>>>> your consistency issues.
> >>> >>>>>>>>
> >>> >>>>>>>> Why don't you use Ignite persistence [1]?
> >>> >>>>>>>>
> >>> >>>>>>>> [1]
> >>> >>>>>>>> https://apacheignite.readme.io/docs/distributed-
> persistent-store
> >>> >>>>>>>>
> >>> >>>>>>>> -Val
> >>> >>>>>>>>
> >>> >>>>>>>> On Fri, Nov 17, 2017 at 4:24 AM, Vyacheslav
Daradur <
> >>> >>>>> daradurvs@gmail.com
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>>> Hi Andrey! Thank you for answering.
> >>> >>>>>>>>>
> >>> >>>>>>>>>>> Key to partition mapping shouldn't
depends on topology, and
> >>> >>>>> shouldn't
> >>> >>>>>>>>> changed unstable topology.
> >>> >>>>>>>>> Key to partition mapping doesn't depend
on topology in my
> test
> >>> >>>>>>>>> affinity function. It only depends
on partitions number.
> >>> >>>>>>>>> But partition to node mapping depends
on topology and at
> cluster
> >>> >>>>> stop,
> >>> >>>>>>>>> when one node left topology, some partitions
may be moved to
> >>> >>>>>>>>> other
> >>> >>>>>>>>> nodes.
> >>> >>>>>>>>>
> >>> >>>>>>>>>>> Does all nodes share same RockDB
database or each node has
> >>> >>>>>>>>>>> its own
> >>> >>>>>>> copy?
> >>> >>>>>>>>> Each Ignite node has own RocksDB instance.
> >>> >>>>>>>>>
> >>> >>>>>>>>>>> Would you please share configuration?
> >>> >>>>>>>>> It's pretty simple:
> >>> >>>>>>>>>           IgniteConfiguration cfg =
new
> IgniteConfiguration();
> >>> >>>>>>>>>           cfg.setIgniteInstanceName(instanceName);
> >>> >>>>>>>>>
> >>> >>>>>>>>>           CacheConfiguration<Integer,
String> cacheCfg = new
> >>> >>>>>>>>> CacheConfiguration<>();
> >>> >>>>>>>>>           cacheCfg.setName(TEST_CACHE_NAME);
> >>> >>>>>>>>> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> >>> >>>>>>>>>           cacheCfg.setWriteSynchronizationMode(
> >>> >>>>>>>>> CacheWriteSynchronizationMode.PRIMARY_SYNC);
> >>> >>>>>>>>>           cacheCfg.setBackups(1);
> >>> >>>>>>>>>           cacheCfg.setAffinity(new
> >>> >>>>>>>>> TestAffinityFunction(partitionsNumber,
backupsNumber));
> >>> >>>>>>>>>           cacheCfg.setWriteThrough(true);
> >>> >>>>>>>>>           cacheCfg.setReadThrough(true);
> >>> >>>>>>>>> cacheCfg.setRebalanceMode(CacheRebalanceMode.SYNC);
> >>> >>>>>>>>>           cacheCfg.setCacheStoreFactory(new
> >>> >>>>>>>>> RocksDBCacheStoreFactory<>("/test/path/to/persistence",
> >>> >>>>>>>>> TEST_CACHE_NAME, cfg));
> >>> >>>>>>>>>
> >>> >>>>>>>>>           cfg.setCacheConfiguration(cacheCfg);
> >>> >>>>>>>>>
> >>> >>>>>>>>> Could you give me advice on places
which I need to pay
> >>> attention?
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> On Wed, Nov 15, 2017 at 3:02 PM, Andrey
Mashenkov
> >>> >>>>>>>>> <andrey.mashenkov@gmail.com>
wrote:
> >>> >>>>>>>>>> Hi Vyacheslav,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Key to partition mapping shouldn't
depends on topology, and
> >>> >>>>> shouldn't
> >>> >>>>>>>>>> changed unstable topology.
> >>> >>>>>>>>>> Looks like you've missed smth.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Would you please share configuration?
> >>> >>>>>>>>>> Does all nodes share same RockDB
database or each node has
> >>> >>>>>>>>>> its own
> >>> >>>>>>> copy?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On Wed, Nov 15, 2017 at 12:22 AM,
Vyacheslav Daradur <
> >>> >>>>>>>>> daradurvs@gmail.com>
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>> Hi, Igniters!
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> I’m using partitioned Ignite
cache with RocksDB as 3rd
> party
> >>> >>>>>>> persistence
> >>> >>>>>>>>>>> store.
> >>> >>>>>>>>>>> I've got an issue: if cache
rebalancing is switched on,
> then
> >>> >>>>>>>>>>> it’s
> >>> >>>>>>>>>>> possible to lose some data.
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> Basic scenario:
> >>> >>>>>>>>>>> 1) Start Ignite cluster and
fill a cache with RocksDB
> >>> >>>>>>>>>>> persistence;
> >>> >>>>>>>>>>> 2) Stop all nodes
> >>> >>>>>>>>>>> 3) Start Ignite cluster and
validate data
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> This works fine while rebalancing
is switched off.
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> If rebalancing switched on:
when I call Ignition#stopAll,
> some
> >>> >>>>> nodes
> >>> >>>>>>>>>>> go down sequentially and while
one node having gone down
> >>> >>>>>>>>>>> another
> >>> >>>>>>> start
> >>> >>>>>>>>>>> rebalancing. When nodes started
affinity function works
> with a
> >>> >>>>> full
> >>> >>>>>>>>>>> set of nodes and may define
a wrong partition for a key
> >>> because
> >>> >>>>> the
> >>> >>>>>>>>>>> previous state was changed
at rebalancing.
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> Maybe I'm doing something wrong.
How can I avoid
> rebalancing
> >>> >>>>>>>>>>> while
> >>> >>>>>>>>>>> stopping all nodes in the cluster?
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> Could you give me any advice,
please?
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> --
> >>> >>>>>>>>>>> Best Regards, Vyacheslav D.
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> --
> >>> >>>>>>>>>> Best regards,
> >>> >>>>>>>>>> Andrey V. Mashenkov
> >>> >>>>>>>>>
> >>> >>>>>>>>> --
> >>> >>>>>>>>> Best Regards, Vyacheslav D.
> >>> >>>>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> --
> >>> >>>>>>> Best Regards, Vyacheslav D.
> >>> >>>>>>>
> >>> >>>>>
> >>> >>>>> --
> >>> >>>>> Best Regards, Vyacheslav D.
> >>> >>>>>
> >>> >>>
> >>> >
> >>>
> >>>
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message