Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
In-Reply-To: <2859816.4nhSuef7RF@nico-work>
References: <CAA6PrdQntzmAdksuBjqsi7eBBYyd+=i76kMsGYC4cwt_Lodk8A@mail.gmail.com> <8b2462a7-c427-2fcd-1ea2-1bc6f94fda56@gmail.com> <2859816.4nhSuef7RF@nico-work>
Thread-Topic: Re: Distribute crawling of a URL list using Flink
User-Agent: Android
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----2WE4R5NDHSJ5SPWY7X6D9FOTCK2TD9"
Content-Transfer-Encoding: 7bit
Subject: Re: Distribute crawling of a URL list using Flink
From: Kien Truong <duckientruong@gmail.com>
Date: Tue, 15 Aug 2017 06:16:28 +0700
To: Nico Kruber <nico@data-artisans.com>
CC: user@flink.apache.org,Eranga Heshan <eranga.h.n@gmail.com>
Message-ID: <37c57421-3ec8-4686-92d8-1fa2fc6baf3c@gmail.com>
archived-at: Mon, 14 Aug 2017 23:16:50 -0000

------2WE4R5NDHSJ5SPWY7X6D9FOTCK2TD9
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
 charset=UTF-8

Hi, 

Admittedly, I have not suggested this because I thought it was not av=
ailable for batch API=2E 

Regards, 
Kien 


On Aug 15, 2017, 00:06, at 00:=
06, Nico Kruber <nico@data-artisans=2Ecom> wrote:
>Hi Eranga and Kien,
>Fli=
nk supports asynchronous IO since version 1=2E2, see [1] for details=2E
>
>=
You basically pack your URL download into the asynchronous part and
>collec=
t 
>the resulting string for further processing in your pipeline=2E
>
>
>
>=
Nico
>
>
>[1]
>https://ci=2Eapache=2Eorg/projects/flink/flink-docs-release-=
1=2E3/dev/stream/
>asyncio=2Ehtml
>
>On Monday, 14 August 2017 17:50:47 CES=
T Kien Truong wrote:
>> Hi,
>> 
>> While this task is quite trivial to do w=
ith Flink Dataset API, using
>> readTextFile to read the input and
>> 
>> a=
 flatMap function to perform the downloading, it might not be a good
>idea=
=2E
>> 
>> The download process is I/O bound, and will block the synchronou=
s
>> flatMap function,
>> 
>> so the throughput will not be very good=2E
>>=
 
>> 
>> Until Flink supports asynchronous functions, I suggest you looks
>=
elsewhere=2E
>> 
>> An example with master-workers architecture using Akka =
can be found
>here
>> 
>> https://github=2Ecom/typesafehub/activator-akka-d=
istributed-workers
>> 
>> 
>> Regards,
>> 
>> Kien
>> 
>> On 8/14/2017 10:0=
9 AM, Eranga Heshan wrote:
>> > Hi all,
>> > 
>> > I am fairly new to Flink=
=2E I have this project where I have a list
>of
>> > URLs (In one node) whi=
ch need to be crawled distributedly=2E Then for
>> > each URL, I need the s=
erialized crawled result to be written to a
>> > single text file=2E
>> > 
=
>> > I want to know if there are similar projects which I can look into
>or=

>> > an idea on how to implement this=2E
>> > 
>> > Thanks & Regards,
>> >=
 
>> > 
>> > 
>> > 
>> > Eranga Heshan
>> > /Undergraduate/
>> > Computer S=
cience & Engineering
>> > University of Moratuwa
>> > Mobile: 	+94 71 138 2=
686 <tel:%2B94%2071%20552%202087>
>> > Email: 	eranga=2Eh=2En@gmail=2Ecom <=
mailto:eranga=2Eh=2En@gmail=2Ecom>
>> > <https://www=2Efacebook=2Ecom/erang=
aheshan>
>> > <https://twitter=2Ecom/erangaheshan>
>> > <https://www=2Elink=
edin=2Ecom/in/erangaheshan>

------2WE4R5NDHSJ5SPWY7X6D9FOTCK2TD9
Content-Type: text/html;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head></head><body><div dir=3D"auto">Hi, <br><br></div>
<div dir=3D"a=
uto">Admittedly, I have not suggested this because I thought it was not ava=
ilable for batch API=2E <br><br></div>
<div dir=3D"auto">Regards, <br></div=
>
<div dir=3D"auto">Kien <br></div>
<div class=3D"gmail_quote" >On Aug 15, =
2017, at 00:06, Nico Kruber &lt;<a href=3D"mailto:nico@data-artisans=2Ecom"=
 target=3D"_blank">nico@data-artisans=2Ecom</a>&gt; wrote:<blockquote class=
=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0=2E8ex; border-left: 1px sol=
id rgb(204, 204, 204); padding-left: 1ex;">
<pre class=3D"blue">Hi Eranga a=
nd Kien,<br>Flink supports asynchronous IO since version 1=2E2, see [1] for=
 details=2E<br><br>You basically pack your URL download into the asynchrono=
us part and collect <br>the resulting string for further processing in your=
 pipeline=2E<br><br><br><br>Nico<br><br><br>[1] <a href=3D"https://ci=2Eapa=
che=2Eorg/projects/flink/flink-docs-release-1=2E3/dev/stream">https://ci=2E=
apache=2Eorg/projects/flink/flink-docs-release-1=2E3/dev/stream</a>/<br>asy=
ncio=2Ehtml<br><br>On Monday, 14 August 2017 17:50:47 CEST Kien Truong wrot=
e:<br>
 <blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 1ex 0=2E=
8ex; border-left: 1px solid #729fcf; padding-left: 1ex;">
   Hi,
  <br> 
  =
<br> While this task is quite trivial to do with Flink Dataset API, using
 =
 <br> readTextFile to read the input and
  <br> 
  <br> a flatMap function =
to perform the downloading, it might not be a good idea=2E
  <br> 
  <br> T=
he download process is I/O bound, and will block the synchronous
  <br> fla=
tMap function,
  <br> 
  <br> so the throughput will not be very good=2E
  =
<br> 
  <br> 
  <br> Until Flink supports asynchronous functions, I suggest=
 you looks elsewhere=2E
  <br> 
  <br> An example with master-workers archi=
tecture using Akka can be found here
  <br> 
  <br> 
  <a href=3D"https://g=
ithub=2Ecom/typesafehub/activator-akka-distributed-workers">https://github=
=2Ecom/typesafehub/activator-akka-distributed-workers</a>
  <br> 
  <br> 
 =
 <br> Regards,
  <br> 
  <br> Kien
  <br> 
  <br> On 8/14/2017 10:09 AM, Er=
anga Heshan wrote:
  <br>
  <blockquote class=3D"gmail_quote" style=3D"marg=
in: 0pt 0pt 1ex 0=2E8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;=
">
    Hi all,
   <br> 
   <br> I am fairly new to Flink=2E I have this pro=
ject where I have a list of
   <br> URLs (In one node) which need to be cra=
wled distributedly=2E Then for
   <br> each URL, I need the serialized craw=
led result to be written to a
   <br> single text file=2E
   <br> 
   <br> =
I want to know if there are similar projects which I can look into or
   <b=
r> an idea on how to implement this=2E
   <br> 
   <br> Thanks &amp; Regard=
s,
   <br> 
   <br> 
   <br> 
   <br> 
   <br> Eranga Heshan
   <br> /Under=
graduate/
   <br> Computer Science &amp; Engineering
   <br> University of =
Moratuwa
   <br> Mobile: +94 71 138 2686 &lt;tel:%2B94%2071%20552%202087&gt=
;
   <br> Email: eranga=2Eh=2En@gmail=2Ecom &lt;mailto:eranga=2Eh=2En@gmail=
=2Ecom&gt;
   <br> &lt;
   <a href=3D"https://www=2Efacebook=2Ecom/erangahe=
shan">https://www=2Efacebook=2Ecom/erangaheshan</a>&gt;
   <br> &lt;
   <a =
href=3D"https://twitter=2Ecom/erangaheshan">https://twitter=2Ecom/erangahes=
han</a>&gt;
   <br> &lt;
   <a href=3D"https://www=2Elinkedin=2Ecom/in/eran=
gaheshan">https://www=2Elinkedin=2Ecom/in/erangaheshan</a>&gt;
   <br>
  </=
blockquote>
 </blockquote><br></pre></blockquote></div></body></html>
------2WE4R5NDHSJ5SPWY7X6D9FOTCK2TD9--