Mailing-List: contact dev-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
User-Agent: Microsoft-MacOutlook/f.28.0.171108
Date: Fri, 17 Nov 2017 18:04:02 +0300
Subject: Re: MultiTransactionJobletEventListenerFactory
From: Murtadha Hubail <hubailmor@gmail.com>
To: <dev@asterixdb.apache.org>
Message-ID: <321A4920-A635-4F5E-A648-6B437B9CBEF2@gmail.com>
Thread-Topic: MultiTransactionJobletEventListenerFactory
References: <CAE1xAnvZZWrgu6HiQs-qXV0PRsWLr44bm3kqVfj5swha9ffUcg@mail.gmail.com>
 <F59038D4-DE9A-45DE-B952-B2BC2C982875@gmail.com>
 <CAN-bgPqg299zbr2CpDVDHkzwFGi_AqGL1oErPH01mpKH+D2X+w@mail.gmail.com>
 <7F92EF80-97A5-4B6F-BD8C-4C7514B96AD1@apache.org>
 <B43A1DC2-6A8B-44B6-A5B6-2E9810D8E757@gmail.com>
 <CAEEsEsQjD_ZaRERbPMHRkrQvBjh6G-DoBq1zeakZm3auZHccWQ@mail.gmail.com>
 <CAN-bgPoKZcLAj1ax_EomF70p6zipmRxxNopqGh-nJZu67K9bBQ@mail.gmail.com>
 <132CEC8D-7221-49F6-A355-13431921770A@gmail.com>
In-Reply-To: <132CEC8D-7221-49F6-A355-13431921770A@gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="UTF-8"
Content-transfer-encoding: quoted-printable
archived-at: Fri, 17 Nov 2017 15:04:13 -0000

A transaction context can register multiple primary indexes.
Since each entity commit log contains the dataset id, you can decrement the=
 active operations on=20
the operation tracker associated with that dataset id.

On 17/11/2017, 5:52 PM, "abdullah alamoudi" <bamousaa@gmail.com> wrote:

    Can you illustrate how a deadlock can happen? I am anxious to know.
    Moreover, the reason for the multiple transaction ids in feeds is not s=
imply because we compile them differently.
   =20
    How would a commit operator know which dataset active operation counter=
 to decrement if they share the same id for example?
   =20
    > On Nov 16, 2017, at 9:46 PM, Xikui Wang <xikuiw@uci.edu> wrote:
    >=20
    > Yes. That deadlock could happen. Currently, we have one-to-one mappin=
gs for
    > the jobs and transactions, except for the feeds.
    >=20
    > @Abdullah, after some digging into the code, I think probably we can =
use a
    > single transaction id for the job which feeds multiple datasets? See =
if I
    > can convince you. :)
    >=20
    > The reason we have multiple transaction ids in feeds is that we compi=
le
    > each connection job separately and combine them into a single feed jo=
b. A
    > new transaction id is created and assigned to each connection job, th=
us for
    > the combined job, we have to handle the different transactions as the=
y
    > are embedded in the connection job specifications. But, what if we cr=
eate a
    > single transaction id for the combined job? That transaction id will =
be
    > embedded into each connection so they can write logs freely, but the
    > transaction will be started and committed only once as there is only =
one
    > feed job. In this way, we won't need multiTransactionJobletEventListe=
ner
    > and the transaction id can be removed from the job specification easi=
ly as
    > well (for Steven's change).
    >=20
    > Best,
    > Xikui
    >=20
    >=20
    > On Thu, Nov 16, 2017 at 4:26 PM, Mike Carey <dtabass@gmail.com> wrote=
:
    >=20
    >> I worry about deadlocks.  The waits for graph may not understand tha=
t
    >> making t1 wait will also make t2 wait since they may share a thread =
-
    >> right?  Or do we have jobs and transactions separately represented t=
here
    >> now?
    >>=20
    >> On Nov 16, 2017 3:10 PM, "abdullah alamoudi" <bamousaa@gmail.com> wr=
ote:
    >>=20
    >>> We are using multiple transactions in a single job in case of feed =
and I
    >>> think that this is the correct way.
    >>> Having a single job for a feed that feeds into multiple datasets is=
 a
    >> good
    >>> thing since job resources/feed resources are consolidated.
    >>>=20
    >>> Here are some points:
    >>> - We can't use the same transaction id to feed multiple datasets. T=
he
    >> only
    >>> other option is to have multiple jobs each feeding a different data=
set.
    >>> - Having multiple jobs (in addition to the extra resources used, me=
mory
    >>> and CPU) would then forces us to either read data from external sou=
rces
    >>> multiple times, parse records multiple times, etc
    >>>  or having to have a synchronization between the different jobs and=
 the
    >>> feed source within asterixdb. IMO, this is far more complicated tha=
n
    >> having
    >>> multiple transactions within a single job and the cost far outweigh=
 the
    >>> benefits.
    >>>=20
    >>> P.S,
    >>> We are also using this for bucket connections in Couchbase Analytic=
s.
    >>>=20
    >>>> On Nov 16, 2017, at 2:57 PM, Till Westmann <tillw@apache.org> wrot=
e:
    >>>>=20
    >>>> If there are a number of issue with supporting multiple transactio=
n ids
    >>>> and no clear benefits/use-cases, I=E2=80=99d vote for simplification :)
    >>>> Also, code that=E2=80=99s not being used has a tendency to "rot" and so =
I think
    >>>> that it=E2=80=99s usefulness might be limited by the time we=E2=80=99d find a =
use for
    >>>> this functionality.
    >>>>=20
    >>>> My 2c,
    >>>> Till
    >>>>=20
    >>>> On 16 Nov 2017, at 13:57, Xikui Wang wrote:
    >>>>=20
    >>>>> I'm separating the connections into different jobs in some of my
    >>>>> experiments... but that was intended to be used for the experimen=
tal
    >>>>> settings (i.e., not for master now)...
    >>>>>=20
    >>>>> I think the interesting question here is whether we want to allow=
 one
    >>>>> Hyracks job to carry multiple transactions. I personally think th=
at
    >>> should
    >>>>> be allowed as the transaction and job are two separate concepts, =
but I
    >>>>> couldn't find such use cases other than the feeds. Does anyone ha=
ve a
    >>> good
    >>>>> example on this?
    >>>>>=20
    >>>>> Another question is, if we do allow multiple transactions in a si=
ngle
    >>>>> Hyracks job, how do we enable commit runtime to obtain the correc=
t TXN
    >>> id
    >>>>> without having that embedded as part of the job specification.
    >>>>>=20
    >>>>> Best,
    >>>>> Xikui
    >>>>>=20
    >>>>> On Thu, Nov 16, 2017 at 1:01 PM, abdullah alamoudi <
    >> bamousaa@gmail.com>
    >>>>> wrote:
    >>>>>=20
    >>>>>> I am curious as to how feed will work without this?
    >>>>>>=20
    >>>>>> ~Abdullah.
    >>>>>>> On Nov 16, 2017, at 12:43 PM, Steven Jacobs <sjaco002@ucr.edu>
    >> wrote:
    >>>>>>>=20
    >>>>>>> Hi all,
    >>>>>>> We currently have MultiTransactionJobletEventListenerFactory, w=
hich
    >>>>>> allows
    >>>>>>> for one Hyracks job to run multiple Asterix transactions togeth=
er.
    >>>>>>>=20
    >>>>>>> This class is only used by feeds, and feeds are in process of
    >>> changing to
    >>>>>>> no longer need this feature. As part of the work in pre-deployi=
ng
    >> job
    >>>>>>> specifications to be used by multiple hyracks jobs, I've been
    >> working
    >>> on
    >>>>>>> removing the transaction id from the job specifications, as we =
use a
    >>> new
    >>>>>>> transaction for each invocation of a deployed job.
    >>>>>>>=20
    >>>>>>> There is currently no clear way to remove the transaction id fr=
om
    >> the
    >>> job
    >>>>>>> spec and keep the option for MultiTransactionJobletEventLis
    >>> tenerFactory.
    >>>>>>>=20
    >>>>>>> The question for the group is, do we see a need to maintain thi=
s
    >> class
    >>>>>> that
    >>>>>>> will no longer be used by any current code? Or, an other words,=
 is
    >>> there
    >>>>>> a
    >>>>>>> strong possibility that in the future we will want multiple
    >>> transactions
    >>>>>> to
    >>>>>>> share a single Hyracks job, meaning that it is worth figuring o=
ut
    >> how
    >>> to
    >>>>>>> maintain this class?
    >>>>>>>=20
    >>>>>>> Steven
    >>>>>>=20
    >>>>>>=20
    >>>=20
    >>>=20
    >>=20
   =20
   =20