asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianfeng Jia <jianfeng....@gmail.com>
Subject Re: Let one Operator finished the job before another one begin in Hyracks
Date Tue, 11 Oct 2016 07:22:00 GMT
Based on the described example, it seems possible to implement it in one job by using MToNPartitioningConnectorDescriptor.
You can force that merge-BF-operator only runs in one partition by using PartitionConstraintHelper.addAbsoluteLocationConstraint()
function.

> On Oct 10, 2016, at 11:43 PM, mingda li <limingda1993@gmail.com> wrote:
> 
> Yeah, that will be easier. But for example, we have N nodes and in each
> node, it will generate a Bloom Filter(BF) for its own data. We need to send
> these BFs to one node for constructing a complete BF and then send the BF
> back to each node. I am not sure we can use multiple stage job for this,
> because there should be a 1->N and a N->1 connecter among nodes. If in one
> job, there may be no way to transfer data among nodes.
> This is my idea. If this can be implemented by one multiple stage job, that
> will decrease a lot of my work :-)
> 
> Bests,
> Mingda
> 
> On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtabass@gmail.com> wrote:
> 
>> Is there a reason for wanting two jobs?  I would think that one multiple
>> stage job would be preferable.
>> 
>> On Oct 10, 2016 1:21 PM, "mingda li" <limingda1993@gmail.com> wrote:
>> 
>>> Oh, thanks Kim~
>>> 
>>> On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangsaeu@gmail.com> wrote:
>>> 
>>>> Forwarded to dev.
>>>> 
>>>> Best,
>>>> Taewoo
>>>> 
>>>> ---------- Forwarded message ----------
>>>> From: mingda li <limingda1993@gmail.com>
>>>> Date: Mon, Oct 10, 2016 at 11:21 AM
>>>> Subject: Let one Operator finished the job before another one begin in
>>>> Hyracks
>>>> To: users@asterixdb.apache.org
>>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> Now,I am trying to build a Bloom Filter(BF) before join. The BF is
>> build
>>> in
>>>> each node and sent to one node to combine. I want to set a stop sign
>>> there
>>>> before sending the BF in each node. The stop sign means it can only
>> send
>>>> the BF after it is build.
>>>> The class HyracksConnection.waitForCompletion may help this. But I am
>>> not
>>>> sure how to use it.
>>>> Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
>>>> jobidSendBF=hcc.startJob(); ?
>>>> Has anyone ever used the HyracksConnection.waitForCompletion?
>>>> 
>>>> Thanks,
>>>> Mingda
>>>> 
>>> 
>> 


Mime
View raw message