Mailing-List: contact dev-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
Subject: Re: Let one Operator finished the job before another one begin in
 Hyracks
To: dev@asterixdb.apache.org
References: <CAM9x2uTf8vPubK_rXZfzj_wuJvo9UhS_xW8f1-DxPTpWW_RZnA@mail.gmail.com>
 <CABytAGo42LWPhUX9VNjZUA2d3whw74Z9gM4cdmbWUnByiuu7PQ@mail.gmail.com>
 <CAM9x2uShtaD_iveLabvKnx9_e5YX4EKAYumgpaF0UYVy5w_2kw@mail.gmail.com>
 <CAEEsEsScbpmRD35f3=gA_Po+MThYRkmLGJU9_ccVd4kC_wMr2w@mail.gmail.com>
 <CAM9x2uRV2OQHX3_+732O3W4hrzWpdK6XJpGf6Tri7Epz5TNDmA@mail.gmail.com>
 <FD392BF3-0B76-46F7-91E8-464736ED5302@gmail.com>
From: Mike Carey <dtabass@gmail.com>
Message-ID: <46e2f9df-ea12-a880-ff83-052d3c8d4656@gmail.com>
Date: Tue, 11 Oct 2016 00:51:48 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
 Gecko/20100101 Thunderbird/45.3.0
MIME-Version: 1.0
In-Reply-To: <FD392BF3-0B76-46F7-91E8-464736ED5302@gmail.com>
Content-Type: multipart/alternative;
 boundary="------------B34735148FBAA5DC9777B3CF"
archived-at: Tue, 11 Oct 2016 07:51:57 -0000

--------------B34735148FBAA5DC9777B3CF
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

And both Wenhai and Preston have examples of doing the 
fan-in-and-compute/fan-back-out pattern with blocking until the latter 
part is done - Wenhai for finding range split points for parallel 
sorting and Preston for similar things that arise in interval joins.  
Can you guys chime in when you have a chance?  (Preston may be busy from 
what I saw on Skype on Friday :-), with congrats being due!)


On 10/11/16 12:22 AM, Jianfeng Jia wrote:
> Based on the described example, it seems possible to implement it in one job by using MToNPartitioningConnectorDescriptor.
> You can force that merge-BF-operator only runs in one partition by using PartitionConstraintHelper.addAbsoluteLocationConstraint() function.
>
>> On Oct 10, 2016, at 11:43 PM, mingda li <limingda1993@gmail.com> wrote:
>>
>> Yeah, that will be easier. But for example, we have N nodes and in each
>> node, it will generate a Bloom Filter(BF) for its own data. We need to send
>> these BFs to one node for constructing a complete BF and then send the BF
>> back to each node. I am not sure we can use multiple stage job for this,
>> because there should be a 1->N and a N->1 connecter among nodes. If in one
>> job, there may be no way to transfer data among nodes.
>> This is my idea. If this can be implemented by one multiple stage job, that
>> will decrease a lot of my work :-)
>>
>> Bests,
>> Mingda
>>
>> On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtabass@gmail.com> wrote:
>>
>>> Is there a reason for wanting two jobs?  I would think that one multiple
>>> stage job would be preferable.
>>>
>>> On Oct 10, 2016 1:21 PM, "mingda li" <limingda1993@gmail.com> wrote:
>>>
>>>> Oh, thanks Kim~
>>>>
>>>> On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangsaeu@gmail.com> wrote:
>>>>
>>>>> Forwarded to dev.
>>>>>
>>>>> Best,
>>>>> Taewoo
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: mingda li <limingda1993@gmail.com>
>>>>> Date: Mon, Oct 10, 2016 at 11:21 AM
>>>>> Subject: Let one Operator finished the job before another one begin in
>>>>> Hyracks
>>>>> To: users@asterixdb.apache.org
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> Now,I am trying to build a Bloom Filter(BF) before join. The BF is
>>> build
>>>> in
>>>>> each node and sent to one node to combine. I want to set a stop sign
>>>> there
>>>>> before sending the BF in each node. The stop sign means it can only
>>> send
>>>>> the BF after it is build.
>>>>> The class HyracksConnection.waitForCompletion may help this. But I am
>>>> not
>>>>> sure how to use it.
>>>>> Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
>>>>> jobidSendBF=hcc.startJob(); ?
>>>>> Has anyone ever used the HyracksConnection.waitForCompletion?
>>>>>
>>>>> Thanks,
>>>>> Mingda
>>>>>


--------------B34735148FBAA5DC9777B3CF--