asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mingda li <limingda1...@gmail.com>
Subject Re: Let one Operator finished the job before another one begin in Hyracks
Date Tue, 11 Oct 2016 06:43:13 GMT
Yeah, that will be easier. But for example, we have N nodes and in each
node, it will generate a Bloom Filter(BF) for its own data. We need to send
these BFs to one node for constructing a complete BF and then send the BF
back to each node. I am not sure we can use multiple stage job for this,
because there should be a 1->N and a N->1 connecter among nodes. If in one
job, there may be no way to transfer data among nodes.
This is my idea. If this can be implemented by one multiple stage job, that
will decrease a lot of my work :-)

Bests,
Mingda

On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtabass@gmail.com> wrote:

> Is there a reason for wanting two jobs?  I would think that one multiple
> stage job would be preferable.
>
> On Oct 10, 2016 1:21 PM, "mingda li" <limingda1993@gmail.com> wrote:
>
> > Oh, thanks Kim~
> >
> > On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangsaeu@gmail.com> wrote:
> >
> > > Forwarded to dev.
> > >
> > > Best,
> > > Taewoo
> > >
> > > ---------- Forwarded message ----------
> > > From: mingda li <limingda1993@gmail.com>
> > > Date: Mon, Oct 10, 2016 at 11:21 AM
> > > Subject: Let one Operator finished the job before another one begin in
> > > Hyracks
> > > To: users@asterixdb.apache.org
> > >
> > >
> > > Hi,
> > >
> > > Now,I am trying to build a Bloom Filter(BF) before join. The BF is
> build
> > in
> > > each node and sent to one node to combine. I want to set a stop sign
> > there
> > > before sending the BF in each node. The stop sign means it can only
> send
> > > the BF after it is build.
> > > The class HyracksConnection.waitForCompletion may help this. But I am
> > not
> > > sure how to use it.
> > > Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
> > > jobidSendBF=hcc.startJob(); ?
> > > Has anyone ever used the HyracksConnection.waitForCompletion?
> > >
> > > Thanks,
> > > Mingda
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message