ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>
Subject Re: distributed transaction of non-single coordinator
Date Wed, 15 Mar 2017 08:42:16 GMT
Okay, we are open for proposals on business task. I mean, we can make use
of some other thing, not distributed transaction. Not transaction yet.

ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov <vozerov@gridgain.com>:

> IMO the use case makes sense. However, as Sergi already mentioned, the
> problem is far more complex, than simply passing TX state over a wire. Most
> probably a kind of coordinator will be required still to manage all kinds
> of failures. This task should be started with clean design proposal
> explaining how we handle all these concurrent events. And only then, when
> we understand all implications, we should move to development stage.
>
> On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY KUZNETSOV <
> alkuznetsov.sb@gmail.com> wrote:
>
> > Right
> >
> > ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin <sergi.vladykin@gmail.com>:
> >
> > > Good! Basically your orchestrator just takes some predefined graph of
> > > distributed services to be invoked, calls them by some kind of RPC and
> > > passes the needed parameters between them, right?
> > >
> > > Sergi
> > >
> > > 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov.sb@gmail.com
> >:
> > >
> > > > orchestrator is a custom thing. He is responsible for managing
> business
> > > > scenarios flows. Many nodes are involved in scenarios. They exchange
> > data
> > > > and folow one another. If you acquinted with BPMN framework, so
> > > > orchestrator is like bpmn engine.
> > > >
> > > > вт, 14 Мар 2017 г., 18:56 Sergi Vladykin <sergi.vladykin@gmail.com>:
> > > >
> > > > > What is Orchestrator for you? Is it a thing from Microsoft or your
> > > custom
> > > > > in-house software?
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV <
> > alkuznetsov.sb@gmail.com
> > > >:
> > > > >
> > > > > > Fine. Let's say we've got multiple servers which fulfills custom
> > > logic.
> > > > > > This servers compound oriented graph (BPMN process) which
> > controlled
> > > by
> > > > > > Orchestrator.
> > > > > > For instance, *server1  *creates *variable A *with value 1,
> > persists
> > > it
> > > > > to
> > > > > > IGNITE cache and creates *variable B *and sends it to* server2.
> > *The
> > > > > > latests receives *variable B*, do some logic with it and stores
> to
> > > > > IGNITE.
> > > > > > All the work made by both servers must be fulfilled in *one*
> > > > transaction.
> > > > > > Because we need all information done, or nothing(rollbacked).
The
> > > > > scenario
> > > > > > is managed by orchestrator.
> > > > > >
> > > > > > вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin <
> > > sergi.vladykin@gmail.com
> > > > >:
> > > > > >
> > > > > > > Ok, it is not a business case, it is your wrong solution
for
> it.
> > > > > > > Lets try again, what is the business case?
> > > > > > >
> > > > > > > Sergi
> > > > > > >
> > > > > > > 2017-03-14 16:42 GMT+03:00 ALEKSEY KUZNETSOV <
> > > > alkuznetsov.sb@gmail.com
> > > > > >:
> > > > > > >
> > > > > > > > The case is the following, One starts transaction
in one
> node,
> > > and
> > > > > > commit
> > > > > > > > this transaction in another jvm node(or rollback it
> remotely).
> > > > > > > >
> > > > > > > > вт, 14 мар. 2017 г. в 16:30, Sergi Vladykin
<
> > > > > sergi.vladykin@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Because even if you make it work for some simplistic
> > scenario,
> > > > get
> > > > > > > ready
> > > > > > > > to
> > > > > > > > > write many fault tolerance tests and make sure
that you TXs
> > > work
> > > > > > > > gracefully
> > > > > > > > > in all modes in case of crashes. Also make sure
that we do
> > not
> > > > have
> > > > > > any
> > > > > > > > > performance drops after all your changes in existing
> > > benchmarks.
> > > > > All
> > > > > > in
> > > > > > > > all
> > > > > > > > > I don't believe these conditions will be met
and your
> > > > contribution
> > > > > > will
> > > > > > > > be
> > > > > > > > > accepted.
> > > > > > > > >
> > > > > > > > > Better solution to what problem? Sending TX to
another
> node?
> > > The
> > > > > > > problem
> > > > > > > > > statement itself is already wrong. What business
case you
> are
> > > > > trying
> > > > > > to
> > > > > > > > > solve? I'm sure everything you need can be done
in a much
> > more
> > > > > simple
> > > > > > > and
> > > > > > > > > efficient way at the application level.
> > > > > > > > >
> > > > > > > > > Sergi
> > > > > > > > >
> > > > > > > > > 2017-03-14 16:03 GMT+03:00 ALEKSEY KUZNETSOV
<
> > > > > > alkuznetsov.sb@gmail.com
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Why wrong ? You know the better solution?
> > > > > > > > > >
> > > > > > > > > > вт, 14 мар. 2017 г. в 15:46, Sergi
Vladykin <
> > > > > > > sergi.vladykin@gmail.com
> > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Just serializing TX object and deserializing
it on
> > another
> > > > node
> > > > > > is
> > > > > > > > > > > meaningless, because other nodes participating
in the
> TX
> > > have
> > > > > to
> > > > > > > know
> > > > > > > > > > about
> > > > > > > > > > > the new coordinator. This will require
protocol
> changes,
> > we
> > > > > > > > definitely
> > > > > > > > > > will
> > > > > > > > > > > have fault tolerance and performance
issues. IMO the
> > whole
> > > > idea
> > > > > > is
> > > > > > > > > wrong
> > > > > > > > > > > and it makes no sense to waste time
on it.
> > > > > > > > > > >
> > > > > > > > > > > Sergi
> > > > > > > > > > >
> > > > > > > > > > > 2017-03-14 10:57 GMT+03:00 ALEKSEY
KUZNETSOV <
> > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > IgniteTransactionState implememntation
contains
> > > > > IgniteTxEntry's
> > > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > > > supposed to be transferable
> > > > > > > > > > > >
> > > > > > > > > > > > пн, 13 мар. 2017 г. в 19:32,
Dmitriy Setrakyan <
> > > > > > > > > dsetrakyan@apache.org
> > > > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > It sounds a little scary
to me that we are passing
> > > > > > transaction
> > > > > > > > > > objects
> > > > > > > > > > > > > around. Such object may contain
all sorts of Ignite
> > > > > context.
> > > > > > If
> > > > > > > > > some
> > > > > > > > > > > data
> > > > > > > > > > > > > needs to be passed across,
we should create a
> special
> > > > > > transfer
> > > > > > > > > object
> > > > > > > > > > > in
> > > > > > > > > > > > > this case.
> > > > > > > > > > > > >
> > > > > > > > > > > > > D.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Mar 13, 2017 at 9:10
AM, ALEKSEY KUZNETSOV
> <
> > > > > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > well, there a couple
of issues preventing
> > transaction
> > > > > > > > proceeding.
> > > > > > > > > > > > > > At first, After transaction
serialization and
> > > > > > deserialization
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > > remote
> > > > > > > > > > > > > > server, there is no
txState. So im going to put
> it
> > in
> > > > > > > > > > > > > > writeExternal()\readExternal()
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The last one is Deserialized
transaction lacks of
> > > > shared
> > > > > > > cache
> > > > > > > > > > > context
> > > > > > > > > > > > > > field at TransactionProxyImpl.
Perhaps, it must
> be
> > > > > injected
> > > > > > > by
> > > > > > > > > > > > > > GridResourceProcessor
?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > пн, 13 мар. 2017
г. в 17:27, ALEKSEY KUZNETSOV <
> > > > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > while starting
and continuing transaction in
> > > > different
> > > > > > jvms
> > > > > > > > in
> > > > > > > > > > run
> > > > > > > > > > > > into
> > > > > > > > > > > > > > > serialization exception
in writeExternalMeta :
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Override public
void
> writeExternal(ObjectOutput
> > > out)
> > > > > > > throws
> > > > > > > > > > > > > IOException
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > > >     writeExternalMeta(out);
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > some meta is cannot
be serialized.
> > > > > > > > > > > > > > > пт, 10 мар.
2017 г. в 17:25, Alexey Goncharuk <
> > > > > > > > > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Aleksey,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think I am starting
to get what you want,
> but I
> > > > have
> > > > > a
> > > > > > > few
> > > > > > > > > > > > concerns:
> > > > > > > > > > > > > > >  - What is the
API for the proposed change? In
> > your
> > > > > test,
> > > > > > > you
> > > > > > > > > > pass
> > > > > > > > > > > an
> > > > > > > > > > > > > > > instance of transaction
created on ignite(0) to
> > the
> > > > > > ignite
> > > > > > > > > > instance
> > > > > > > > > > > > > > > ignite(1). This
is obviously not possible in a
> > > truly
> > > > > > > > > distributed
> > > > > > > > > > > > > > > (multi-jvm) environment.
> > > > > > > > > > > > > > > - How will you
synchronize cache update actions
> > and
> > > > > > > > transaction
> > > > > > > > > > > > commit?
> > > > > > > > > > > > > > > Say, you have one
node that decided to commit,
> > but
> > > > > > another
> > > > > > > > node
> > > > > > > > > > is
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > > writing within
this transaction. How do you
> make
> > > sure
> > > > > > that
> > > > > > > > two
> > > > > > > > > > > nodes
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not call commit()
and rollback()
> simultaneously?
> > > > > > > > > > > > > > >  - How do you make
sure that either commit() or
> > > > > > rollback()
> > > > > > > is
> > > > > > > > > > > called
> > > > > > > > > > > > if
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > originator failed?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-03-10 15:38
GMT+03:00 Дмитрий Рябов <
> > > > > > > > > somefireone@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Alexey Goncharuk,
heh, my initial
> understanding
> > > was
> > > > > > that
> > > > > > > > > > > > transferring
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > tx
> > > > > > > > > > > > > > > > ownership
from one node to another will be
> > > happened
> > > > > > > > > > automatically
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > originating
node is gone down.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2017-03-10
15:36 GMT+03:00 ALEKSEY KUZNETSOV
> <
> > > > > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Im aiming
to span transaction on multiple
> > > > threads,
> > > > > > > nodes,
> > > > > > > > > > > > > jvms(soon).
> > > > > > > > > > > > > > > So
> > > > > > > > > > > > > > > > > every
node is able to rollback, or commit
> > > common
> > > > > > > > > > transaction.It
> > > > > > > > > > > > > > turned
> > > > > > > > > > > > > > > > up i
> > > > > > > > > > > > > > > > > need
to transfer tx between nodes in order
> to
> > > > > commit
> > > > > > > > > > > transaction
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > different
node(in the same jvm).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > пт,
10 мар. 2017 г. в 15:20, Alexey
> > Goncharuk <
> > > > > > > > > > > > > > > > alexey.goncharuk@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
Aleksey,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
Do you mean that you want a concept of
> > > > > transferring
> > > > > > > of
> > > > > > > > tx
> > > > > > > > > > > > > ownership
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > >
one node to another? My initial
> > understanding
> > > > was
> > > > > > > that
> > > > > > > > > you
> > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > >
to update keys in a transaction from
> > multiple
> > > > > > threads
> > > > > > > > in
> > > > > > > > > > > > > parallel.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
--AG
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
2017-03-10 15:01 GMT+03:00 ALEKSEY
> > KUZNETSOV
> > > <
> > > > > > > > > > > > > > > alkuznetsov.sb@gmail.com
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> Well. Consider transaction started in
> one
> > > > node,
> > > > > > and
> > > > > > > > > > > continued
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > > >
> one.
> > > > > > > > > > > > > > > > > >
> The following test describes my idea:
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> Ignite ignite1 = ignite(0);
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> IgniteTransactions transactions =
> > > > > > > > > ignite1.transactions();
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> IgniteCache<String, Integer> cache =
> > > > > > > > > > > > ignite1.getOrCreateCache("
> > > > > > > > > > > > > > > > > >
> testCache");
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> Transaction tx = transactions.txStart(
> > > > > > concurrency,
> > > > > > > > > > > > isolation);
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> cache.put("key1", 1);
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> cache.put("key2", 2);
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> tx.stop();
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> IgniteInternalFuture<Boolean> fut =
> > > > > > > > > > > GridTestUtils.runAsync(()
> > > > > > > > > > > > > ->
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > >
>     IgniteTransactions ts =
> > > > > > > ignite(1).transactions();
> > > > > > > > > > > > > > > > > >
>     Assert.assertNull(ts.tx());
> > > > > > > > > > > > > > > > > >
>     Assert.assertEquals(
> > > > > > TransactionState.STOPPED,
> > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > >
>     ts.txStart(tx);
> > > > > > > > > > > > > > > > > >
>
> > > > >  Assert.assertEquals(TransactionState.ACTIVE,
> > > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > >
>     cache.put("key3", 3);
> > > > > > > > > > > > > > > > > >
>
> > >  Assert.assertTrue(cache.remove("key2"));
> > > > > > > > > > > > > > > > > >
>     tx.commit();
> > > > > > > > > > > > > > > > > >
>     return true;
> > > > > > > > > > > > > > > > > >
> });
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> fut.get();
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> Assert.assertEquals(
> > > > TransactionState.COMMITTED,
> > > > > > > > > > > tx.state());
> > > > > > > > > > > > > > > > > >
> Assert.assertEquals((long)1,
> > > > > > > > (long)cache.get("key1"));
> > > > > > > > > > > > > > > > > >
> Assert.assertEquals((long)3,
> > > > > > > > (long)cache.get("key3"));
> > > > > > > > > > > > > > > > > >
> Assert.assertFalse(cache.
> > > > containsKey("key2"));
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> In method *ts.txStart(...)* we just
> > rebind
> > > > *tx*
> > > > > > to
> > > > > > > > > > current
> > > > > > > > > > > > > > thread:
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> public void txStart(Transaction tx) {
> > > > > > > > > > > > > > > > > >
>     TransactionProxyImpl
> > transactionProxy =
> > > > > > > > > > > > > > > (TransactionProxyImpl)tx;
> > > > > > > > > > > > > > > > > >
>     cctx.tm().reopenTx(
> > > > transactionProxy.tx());
> > > > > > > > > > > > > > > > > >
>     transactionProxy.
> > bindToCurrentThread();
> > > > > > > > > > > > > > > > > >
> }
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> In method *reopenTx* we alter
> *threadMap*
> > > so
> > > > > that
> > > > > > > it
> > > > > > > > > > binds
> > > > > > > > > > > > > > > > transaction
> > > > > > > > > > > > > > > > > >
> to current thread.
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> How do u think about it ?
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> вт, 7 мар. 2017 г. в 22:38, Denis
> Magda <
> > > > > > > > > > dmagda@apache.org
> > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> > Hi Alexey,
> > > > > > > > > > > > > > > > > >
> >
> > > > > > > > > > > > > > > > > >
> > Please share the rational behind this
> > and
> > > > the
> > > > > > > > > thoughts,
> > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > ideas
> > > > > > > > > > > > > > > > > >
you
> > > > > > > > > > > > > > > > > >
> > have in mind.
> > > > > > > > > > > > > > > > > >
> >
> > > > > > > > > > > > > > > > > >
> > —
> > > > > > > > > > > > > > > > > >
> > Denis
> > > > > > > > > > > > > > > > > >
> >
> > > > > > > > > > > > > > > > > >
> > > On Mar 7, 2017, at 3:19 AM, ALEKSEY
> > > > > > KUZNETSOV <
> > > > > > > > > > > > > > > > > >
> alkuznetsov.sb@gmail.com>
> > > > > > > > > > > > > > > > > >
> > wrote:
> > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > >
> > > Hi all! Im designing distributed
> > > > > transaction
> > > > > > > > which
> > > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > > > > > started
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> one
> > > > > > > > > > > > > > > > > >
> > > node, and continued at other one.
> Has
> > > > > anybody
> > > > > > > > > > thoughts
> > > > > > > > > > > on
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > ?
> > > > > > > > > > > > > > > > > >
> > > --
> > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > >
> > > *Best Regards,*
> > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > >
> > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > >
> >
> > > > > > > > > > > > > > > > > >
> > --
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> *Best Regards,*
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> *Kuznetsov Aleksey*
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Best
Regards,*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Kuznetsov
Aleksey*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > *Best Regards,*
> > > > > > > > > > > >
> > > > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > *Best Regards,*
> > > > > > > > > >
> > > > > > > > > > *Kuznetsov Aleksey*
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > *Best Regards,*
> > > > > > > >
> > > > > > > > *Kuznetsov Aleksey*
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > >
> > > > > > *Best Regards,*
> > > > > >
> > > > > > *Kuznetsov Aleksey*
> > > > > >
> > > > >
> > > > --
> > > >
> > > > *Best Regards,*
> > > >
> > > > *Kuznetsov Aleksey*
> > > >
> > >
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message