Mailing-List: contact dev-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
MIME-Version: 1.0
In-Reply-To: <635815fd-e25d-4ea5-3d05-950dfdb0f113@gmail.com>
References: <CAM9x2uTf8vPubK_rXZfzj_wuJvo9UhS_xW8f1-DxPTpWW_RZnA@mail.gmail.com>
 <CABytAGo42LWPhUX9VNjZUA2d3whw74Z9gM4cdmbWUnByiuu7PQ@mail.gmail.com>
 <CAM9x2uShtaD_iveLabvKnx9_e5YX4EKAYumgpaF0UYVy5w_2kw@mail.gmail.com>
 <CAEEsEsScbpmRD35f3=gA_Po+MThYRkmLGJU9_ccVd4kC_wMr2w@mail.gmail.com>
 <CAM9x2uRV2OQHX3_+732O3W4hrzWpdK6XJpGf6Tri7Epz5TNDmA@mail.gmail.com>
 <FD392BF3-0B76-46F7-91E8-464736ED5302@gmail.com> <46e2f9df-ea12-a880-ff83-052d3c8d4656@gmail.com>
 <7f9463d7.2a0a2.157b2eaf35a.Coremail.lwh@whu.edu.cn> <CALjEsF6gXMAs4prBV1Qop9Hz3USaPsKZzWFCApHAgSbzVZ-uaA@mail.gmail.com>
 <635815fd-e25d-4ea5-3d05-950dfdb0f113@gmail.com>
From: Yingyi Bu <buyingyi@gmail.com>
Date: Tue, 11 Oct 2016 09:35:10 -0700
Message-ID: <CALjEsF4akfMyhPBX478ODfE+Qh2bCEZUzO16yj6-2qt7p2vG6A@mail.gmail.com>
Subject: Re: Let one Operator finished the job before another one begin in Hyracks
To: dev@asterixdb.apache.org
Content-Type: multipart/alternative; boundary=94eb2c134fd6d1d1a9053e997522
archived-at: Tue, 11 Oct 2016 16:35:23 -0000

--94eb2c134fd6d1d1a9053e997522
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

+1!

Best,
Yingyi

On Tue, Oct 11, 2016 at 9:32 AM, Mike Carey <dtabass@gmail.com> wrote:

> BUT AGAIN:  I think the preferred solution in this case is to do it in on=
e
> job.  Mingda, I would suggest sync'ing up with Wenhai for a Skype meeting
> on how he/Preston have done essentially the very same thing in their use
> cases for parallel sorts and interval joins.  Hyracks has everything need=
ed
> for this, as it turns out, without a multi-job need.
>
>
>
> On 10/11/16 9:26 AM, Yingyi Bu wrote:
>
>> You can search the usage of waitForCompletion in the code base, e.g.:
>>
>> APIFramework.java:
>>
>> public void executeJobArray(IHyracksClientConnection hcc,
>> JobSpecification[] specs, PrintWriter out)
>>          throws Exception {
>>      for (JobSpecification spec : specs) {
>>          spec.setMaxReattempts(0);
>>          JobId jobId =3D hcc.startJob(spec);
>>          long startTime =3D System.currentTimeMillis();
>>          hcc.waitForCompletion(jobId);
>>          long endTime =3D System.currentTimeMillis();
>>          double duration =3D (endTime - startTime) / 1000.00;
>>          out.println("<pre>Duration: " + duration + " sec</pre>");
>>      }
>>
>> }
>>
>>
>> You start a job and get the job Id, and then you can wait on the job id.
>>
>>
>> Best,
>>
>> Yingyi
>>
>>
>> On Tue, Oct 11, 2016 at 1:45 AM, =E6=9D=8E=E6=96=87=E6=B5=B7 <lwh@whu.ed=
u.cn> wrote:
>>
>> Hi, Mingda.
>>>      What you need is quite familiar with what I and Presten have done.
>>> Actually, I think we just need a shared
>>> object accommodated by joblet or task which should be also driven by a
>>> broadcast connector inbetween its input
>>> and output operators. We can talk about this by skype if needed.
>>> Best, Wenhai
>>>
>>>
>>> -----=E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6-----
>>>> =E5=8F=91=E4=BB=B6=E4=BA=BA: "Mike Carey" <dtabass@gmail.com>
>>>> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B410=E6=9C=8811=E6=97=
=A5 =E6=98=9F=E6=9C=9F=E4=BA=8C
>>>> =E6=94=B6=E4=BB=B6=E4=BA=BA: dev@asterixdb.apache.org
>>>> =E6=8A=84=E9=80=81:
>>>> =E4=B8=BB=E9=A2=98: Re: Let one Operator finished the job before anoth=
er one begin in
>>>>
>>> Hyracks
>>>
>>>> And both Wenhai and Preston have examples of doing the
>>>> fan-in-and-compute/fan-back-out pattern with blocking until the latter
>>>> part is done - Wenhai for finding range split points for parallel
>>>> sorting and Preston for similar things that arise in interval joins.
>>>> Can you guys chime in when you have a chance?  (Preston may be busy fr=
om
>>>> what I saw on Skype on Friday :-), with congrats being due!)
>>>>
>>>>
>>>> On 10/11/16 12:22 AM, Jianfeng Jia wrote:
>>>>
>>>>> Based on the described example, it seems possible to implement it in
>>>>>
>>>> one job by using MToNPartitioningConnectorDescriptor.
>>>
>>>> You can force that merge-BF-operator only runs in one partition by
>>>>>
>>>> using PartitionConstraintHelper.addAbsoluteLocationConstraint()
>>> function.
>>>
>>>> On Oct 10, 2016, at 11:43 PM, mingda li <limingda1993@gmail.com>
>>>>>>
>>>>> wrote:
>>>
>>>> Yeah, that will be easier. But for example, we have N nodes and in
>>>>>>
>>>>> each
>>>
>>>> node, it will generate a Bloom Filter(BF) for its own data. We need
>>>>>>
>>>>> to send
>>>
>>>> these BFs to one node for constructing a complete BF and then send
>>>>>>
>>>>> the BF
>>>
>>>> back to each node. I am not sure we can use multiple stage job for
>>>>>>
>>>>> this,
>>>
>>>> because there should be a 1->N and a N->1 connecter among nodes. If
>>>>>>
>>>>> in one
>>>
>>>> job, there may be no way to transfer data among nodes.
>>>>>> This is my idea. If this can be implemented by one multiple stage
>>>>>>
>>>>> job, that
>>>
>>>> will decrease a lot of my work :-)
>>>>>>
>>>>>> Bests,
>>>>>> Mingda
>>>>>>
>>>>>> On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtabass@gmail.com>
>>>>>>
>>>>> wrote:
>>>
>>>> Is there a reason for wanting two jobs?  I would think that one
>>>>>>>
>>>>>> multiple
>>>
>>>> stage job would be preferable.
>>>>>>>
>>>>>>> On Oct 10, 2016 1:21 PM, "mingda li" <limingda1993@gmail.com> wrote=
:
>>>>>>>
>>>>>>> Oh, thanks Kim~
>>>>>>>>
>>>>>>>> On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangsaeu@gmail.com>
>>>>>>>>
>>>>>>> wrote:
>>>
>>>> Forwarded to dev.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Taewoo
>>>>>>>>>
>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>> From: mingda li <limingda1993@gmail.com>
>>>>>>>>> Date: Mon, Oct 10, 2016 at 11:21 AM
>>>>>>>>> Subject: Let one Operator finished the job before another one
>>>>>>>>>
>>>>>>>> begin in
>>>
>>>> Hyracks
>>>>>>>>> To: users@asterixdb.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Now,I am trying to build a Bloom Filter(BF) before join. The BF i=
s
>>>>>>>>>
>>>>>>>> build
>>>>>>>
>>>>>>>> in
>>>>>>>>
>>>>>>>>> each node and sent to one node to combine. I want to set a stop
>>>>>>>>>
>>>>>>>> sign
>>>
>>>> there
>>>>>>>>
>>>>>>>>> before sending the BF in each node. The stop sign means it can on=
ly
>>>>>>>>>
>>>>>>>> send
>>>>>>>
>>>>>>>> the BF after it is build.
>>>>>>>>> The class HyracksConnection.waitForCompletion may help this. But
>>>>>>>>>
>>>>>>>> I am
>>>
>>>> not
>>>>>>>>
>>>>>>>>> sure how to use it.
>>>>>>>>> Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
>>>>>>>>> jobidSendBF=3Dhcc.startJob(); ?
>>>>>>>>> Has anyone ever used the HyracksConnection.waitForCompletion?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mingda
>>>>>>>>>
>>>>>>>>>
>>>
>

--94eb2c134fd6d1d1a9053e997522--