ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergi Vladykin <sergi.vlady...@gmail.com>
Subject Re: distributed transaction of non-single coordinator
Date Wed, 15 Mar 2017 07:28:26 GMT
Good! Basically your orchestrator just takes some predefined graph of
distributed services to be invoked, calls them by some kind of RPC and
passes the needed parameters between them, right?

Sergi

2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov.sb@gmail.com>:

> orchestrator is a custom thing. He is responsible for managing business
> scenarios flows. Many nodes are involved in scenarios. They exchange data
> and folow one another. If you acquinted with BPMN framework, so
> orchestrator is like bpmn engine.
>
> вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <sergi.vladykin@gmail.com>:
>
> > What is Orchestrator for you? Is it a thing from Microsoft or your custom
> > in-house software?
> >
> > Sergi
> >
> > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov.sb@gmail.com>:
> >
> > > Fine. Let's say we've got multiple servers which fulfills custom logic.
> > > This servers compound oriented graph (BPMN process) which controlled by
> > > Orchestrator.
> > > For instance, *server1  *creates *variable A *with value 1, persists it
> > to
> > > IGNITE cache and creates *variable B *and sends it to* server2. *The
> > > latests receives *variable B*, do some logic with it and stores to
> > IGNITE.
> > > All the work made by both servers must be fulfilled in *one*
> transaction.
> > > Because we need all information done, or nothing(rollbacked). The
> > scenario
> > > is managed by orchestrator.
> > >
> > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin <sergi.vladykin@gmail.com
> >:
> > >
> > > > Ok, it is not a business case, it is your wrong solution for it.
> > > > Lets try again, what is the business case?
> > > >
> > > > Sergi
> > > >
> > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV <
> alkuznetsov.sb@gmail.com
> > >:
> > > >
> > > > > The case is the following, One starts transaction in one node, and
> > > commit
> > > > > this transaction in another jvm node(or rollback it remotely).
> > > > >
> > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin <
> > sergi.vladykin@gmail.com
> > > >:
> > > > >
> > > > > > Because even if you make it work for some simplistic scenario,
> get
> > > > ready
> > > > > to
> > > > > > write many fault tolerance tests and make sure that you TXs
work
> > > > > gracefully
> > > > > > in all modes in case of crashes. Also make sure that we do not
> have
> > > any
> > > > > > performance drops after all your changes in existing benchmarks.
> > All
> > > in
> > > > > all
> > > > > > I don't believe these conditions will be met and your
> contribution
> > > will
> > > > > be
> > > > > > accepted.
> > > > > >
> > > > > > Better solution to what problem? Sending TX to another node?
The
> > > > problem
> > > > > > statement itself is already wrong. What business case you are
> > trying
> > > to
> > > > > > solve? I'm sure everything you need can be done in a much more
> > simple
> > > > and
> > > > > > efficient way at the application level.
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV <
> > > alkuznetsov.sb@gmail.com
> > > > >:
> > > > > >
> > > > > > > Why wrong ? You know the better solution?
> > > > > > >
> > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi Vladykin <
> > > > sergi.vladykin@gmail.com
> > > > > >:
> > > > > > >
> > > > > > > > Just serializing TX object and deserializing it on
another
> node
> > > is
> > > > > > > > meaningless, because other nodes participating in
the TX have
> > to
> > > > know
> > > > > > > about
> > > > > > > > the new coordinator. This will require protocol changes,
we
> > > > > definitely
> > > > > > > will
> > > > > > > > have fault tolerance and performance issues. IMO the
whole
> idea
> > > is
> > > > > > wrong
> > > > > > > > and it makes no sense to waste time on it.
> > > > > > > >
> > > > > > > > Sergi
> > > > > > > >
> > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > alkuznetsov.sb@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > IgniteTransactionState implememntation contains
> > IgniteTxEntry's
> > > > > which
> > > > > > > is
> > > > > > > > > supposed to be transferable
> > > > > > > > >
> > > > > > > > > пн, 13 мар. 2017 г. в 19:32, Dmitriy Setrakyan
<
> > > > > > dsetrakyan@apache.org
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > It sounds a little scary to me that we are
passing
> > > transaction
> > > > > > > objects
> > > > > > > > > > around. Such object may contain all sorts
of Ignite
> > context.
> > > If
> > > > > > some
> > > > > > > > data
> > > > > > > > > > needs to be passed across, we should create
a special
> > > transfer
> > > > > > object
> > > > > > > > in
> > > > > > > > > > this case.
> > > > > > > > > >
> > > > > > > > > > D.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 13, 2017 at 9:10 AM, ALEKSEY
KUZNETSOV <
> > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > well, there a couple of issues preventing
transaction
> > > > > proceeding.
> > > > > > > > > > > At first, After transaction serialization
and
> > > deserialization
> > > > > on
> > > > > > > the
> > > > > > > > > > remote
> > > > > > > > > > > server, there is no txState. So im
going to put it in
> > > > > > > > > > > writeExternal()\readExternal()
> > > > > > > > > > >
> > > > > > > > > > > The last one is Deserialized transaction
lacks of
> shared
> > > > cache
> > > > > > > > context
> > > > > > > > > > > field at TransactionProxyImpl. Perhaps,
it must be
> > injected
> > > > by
> > > > > > > > > > > GridResourceProcessor ?
> > > > > > > > > > >
> > > > > > > > > > > пн, 13 мар. 2017 г. в 17:27,
ALEKSEY KUZNETSOV <
> > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > while starting and continuing
transaction in
> different
> > > jvms
> > > > > in
> > > > > > > run
> > > > > > > > > into
> > > > > > > > > > > > serialization exception in writeExternalMeta
:
> > > > > > > > > > > >
> > > > > > > > > > > > @Override public void writeExternal(ObjectOutput
out)
> > > > throws
> > > > > > > > > > IOException
> > > > > > > > > > > {
> > > > > > > > > > > >     writeExternalMeta(out);
> > > > > > > > > > > >
> > > > > > > > > > > > some meta is cannot be serialized.
> > > > > > > > > > > > пт, 10 мар. 2017 г. в 17:25,
Alexey Goncharuk <
> > > > > > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > Aleksey,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I think I am starting to get what
you want, but I
> have
> > a
> > > > few
> > > > > > > > > concerns:
> > > > > > > > > > > >  - What is the API for the proposed
change? In your
> > test,
> > > > you
> > > > > > > pass
> > > > > > > > an
> > > > > > > > > > > > instance of transaction created
on ignite(0) to the
> > > ignite
> > > > > > > instance
> > > > > > > > > > > > ignite(1). This is obviously not
possible in a truly
> > > > > > distributed
> > > > > > > > > > > > (multi-jvm) environment.
> > > > > > > > > > > > - How will you synchronize cache
update actions and
> > > > > transaction
> > > > > > > > > commit?
> > > > > > > > > > > > Say, you have one node that decided
to commit, but
> > > another
> > > > > node
> > > > > > > is
> > > > > > > > > > still
> > > > > > > > > > > > writing within this transaction.
How do you make sure
> > > that
> > > > > two
> > > > > > > > nodes
> > > > > > > > > > will
> > > > > > > > > > > > not call commit() and rollback()
simultaneously?
> > > > > > > > > > > >  - How do you make sure that either
commit() or
> > > rollback()
> > > > is
> > > > > > > > called
> > > > > > > > > if
> > > > > > > > > > > an
> > > > > > > > > > > > originator failed?
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-03-10 15:38 GMT+03:00 Дмитрий
Рябов <
> > > > > > somefireone@gmail.com
> > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Alexey Goncharuk, heh, my
initial understanding was
> > > that
> > > > > > > > > transferring
> > > > > > > > > > > of
> > > > > > > > > > > > tx
> > > > > > > > > > > > > ownership from one node to
another will be happened
> > > > > > > automatically
> > > > > > > > > > when
> > > > > > > > > > > > > originating node is gone
down.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-03-10 15:36 GMT+03:00
ALEKSEY KUZNETSOV <
> > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Im aiming to span transaction
on multiple
> threads,
> > > > nodes,
> > > > > > > > > > jvms(soon).
> > > > > > > > > > > > So
> > > > > > > > > > > > > > every node is able to
rollback, or commit common
> > > > > > > transaction.It
> > > > > > > > > > > turned
> > > > > > > > > > > > > up i
> > > > > > > > > > > > > > need to transfer tx
between nodes in order to
> > commit
> > > > > > > > transaction
> > > > > > > > > in
> > > > > > > > > > > > > > different node(in the
same jvm).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > пт, 10 мар. 2017
г. в 15:20, Alexey Goncharuk <
> > > > > > > > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Aleksey,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Do you mean that
you want a concept of
> > transferring
> > > > of
> > > > > tx
> > > > > > > > > > ownership
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > one node to another?
My initial understanding
> was
> > > > that
> > > > > > you
> > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to update keys
in a transaction from multiple
> > > threads
> > > > > in
> > > > > > > > > > parallel.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --AG
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-03-10 15:01
GMT+03:00 ALEKSEY KUZNETSOV <
> > > > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Well. Consider
transaction started in one
> node,
> > > and
> > > > > > > > continued
> > > > > > > > > > in
> > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > one.
> > > > > > > > > > > > > > > > The following
test describes my idea:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ignite ignite1
= ignite(0);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IgniteTransactions
transactions =
> > > > > > ignite1.transactions();
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IgniteCache<String,
Integer> cache =
> > > > > > > > > ignite1.getOrCreateCache("
> > > > > > > > > > > > > > > > testCache");
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Transaction
tx = transactions.txStart(
> > > concurrency,
> > > > > > > > > isolation);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cache.put("key1",
1);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cache.put("key2",
2);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > tx.stop();
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IgniteInternalFuture<Boolean>
fut =
> > > > > > > > GridTestUtils.runAsync(()
> > > > > > > > > > ->
> > > > > > > > > > > {
> > > > > > > > > > > > > > > >     IgniteTransactions
ts =
> > > > ignite(1).transactions();
> > > > > > > > > > > > > > > >     Assert.assertNull(ts.tx());
> > > > > > > > > > > > > > > >     Assert.assertEquals(
> > > TransactionState.STOPPED,
> > > > > > > > > tx.state());
> > > > > > > > > > > > > > > >     ts.txStart(tx);
> > > > > > > > > > > > > > > >
> >  Assert.assertEquals(TransactionState.ACTIVE,
> > > > > > > > > tx.state());
> > > > > > > > > > > > > > > >     cache.put("key3",
3);
> > > > > > > > > > > > > > > >     Assert.assertTrue(cache.remove("key2"));
> > > > > > > > > > > > > > > >     tx.commit();
> > > > > > > > > > > > > > > >     return
true;
> > > > > > > > > > > > > > > > });
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > fut.get();
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Assert.assertEquals(
> TransactionState.COMMITTED,
> > > > > > > > tx.state());
> > > > > > > > > > > > > > > > Assert.assertEquals((long)1,
> > > > > (long)cache.get("key1"));
> > > > > > > > > > > > > > > > Assert.assertEquals((long)3,
> > > > > (long)cache.get("key3"));
> > > > > > > > > > > > > > > > Assert.assertFalse(cache.
> containsKey("key2"));
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In method
*ts.txStart(...)* we just rebind
> *tx*
> > > to
> > > > > > > current
> > > > > > > > > > > thread:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > public void
txStart(Transaction tx) {
> > > > > > > > > > > > > > > >     TransactionProxyImpl
transactionProxy =
> > > > > > > > > > > > (TransactionProxyImpl)tx;
> > > > > > > > > > > > > > > >     cctx.tm().reopenTx(
> transactionProxy.tx());
> > > > > > > > > > > > > > > >     transactionProxy.bindToCurrentThread();
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In method
*reopenTx* we alter *threadMap* so
> > that
> > > > it
> > > > > > > binds
> > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > to current
thread.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > How do u think
about it ?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > вт, 7 мар.
2017 г. в 22:38, Denis Magda <
> > > > > > > dmagda@apache.org
> > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Alexey,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please
share the rational behind this and
> the
> > > > > > thoughts,
> > > > > > > > > > design
> > > > > > > > > > > > > ideas
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > have
in mind.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > —
> > > > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
On Mar 7, 2017, at 3:19 AM, ALEKSEY
> > > KUZNETSOV <
> > > > > > > > > > > > > > > > alkuznetsov.sb@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
Hi all! Im designing distributed
> > transaction
> > > > > which
> > > > > > > can
> > > > > > > > be
> > > > > > > > > > > > started
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > >
node, and continued at other one. Has
> > anybody
> > > > > > > thoughts
> > > > > > > > on
> > > > > > > > > > it
> > > > > > > > > > > ?
> > > > > > > > > > > > > > > > > >
--
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
*Best Regards,*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
*Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Kuznetsov
Aleksey*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > >
> > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > *Best Regards,*
> > > > > > > > > > >
> > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > *Best Regards,*
> > > > > > > > >
> > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *Best Regards,*
> > > > > > >
> > > > > > > *Kuznetsov Aleksey*
> > > > > > >
> > > > > >
> > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > > >
> > > >
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message