Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
DomainKey-Signature: s=cray1024; d=cray.com; c=nofws; q=dns;
  h=X-IronPort-AV:X-Cray-OBMMKR:Received:Received:Received:
   From:To:Subject:Thread-Topic:Thread-Index:Date:Message-ID:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:x-originating-ip:
   Content-Type:MIME-Version:Return-Path;
  b=Ew03PfNhZoYFlJ2vZZXRVb8zyyocciznzwbdYYO0uOO1HG7CZGlZ0XkX
   3AsbjDKeS70D6J9Qsw4YiL0M5dcngfDNefDxOIPxg7iQOj4E/rzn1e9z9
   N9gJgS3nz4ioGPlNPHpGIAit5t9Z1X7WyGq4NifJWk2j4F5NjhmGoeLLA
   k=;
From: Bill Sparks <jsparks@cray.com>
To: "user@flink.apache.org" <user@flink.apache.org>
Subject: RE: data conversion between flink and "other" paradigms
Thread-Topic: data conversion between flink and "other" paradigms
Thread-Index: AdC3vWnOOl/hHKitTgqazg6q+/BtlQAMYryA//+tP00=
Date: Mon, 6 Jul 2015 08:28:16 +0000
Message-ID: 
 <121F01D0B574FC4DA43817B7D9CF419E243FB07E@CFWEX01.americas.cray.com>
References: 
 <121F01D0B574FC4DA43817B7D9CF419E243FB011@CFWEX01.americas.cray.com>,<CAAdrtT0xbSs=Y23eiXkP8Xxyw6dnTbJVxESYNiz6_XaTSfjNEA@mail.gmail.com>
In-Reply-To: 
 <CAAdrtT0xbSs=Y23eiXkP8Xxyw6dnTbJVxESYNiz6_XaTSfjNEA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_121F01D0B574FC4DA43817B7D9CF419E243FB07ECFWEX01americas_"
MIME-Version: 1.0

--_000_121F01D0B574FC4DA43817B7D9CF419E243FB07ECFWEX01americas_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Fabian.


Thanks for the info and pointer to python. I'll check it out.


-Bill

________________________________
From: Fabian Hueske [fhueske@gmail.com]
Sent: Monday, July 06, 2015 3:23 AM
To: user@flink.apache.org
Subject: Re: data conversion between flink and "other" paradigms

Hi Bill,

a DataSet is just a logical concept in Flink. DataSets are often not persis=
ted and just streamed along operators. At the moment, there is no way to ac=
cess an intermediate DataSet of a Flink program directly (this might change=
 in the future).

You can process data in another function by implementing a Java user functi=
on (for example a MapPartition function) and sending the data through JNI t=
o a C function (if you need the full data set, you must set the parallelism=
 to 1). Flink's Python API follows a similar approach to ship data from Fli=
nk to an external Python process.

Best, Fabian


2015-07-06 9:30 GMT+02:00 Bill Sparks <jsparks@cray.com<mailto:jsparks@cray=
.com>>:

Just a question if there was some prior-art here. Just say someone wanted t=
o use flink for processing, but at some point they wanted to call another f=
unction via say JNI/C which doesn't understand DataSet's. How would one go =
about this ... I'm assuming the code would have to convert the data to a co=
mmon format prior to calling the function.


Regards,

   Bill.


--_000_121F01D0B574FC4DA43817B7D9CF419E243FB07ECFWEX01americas_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html dir=3D"ltr">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style id=3D"owaParaStyle">P {
	MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px
}
</style>
</head>
<body fPStyle=3D"1" ocsi=3D"0">
<div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: =
10pt;">
<p>Fabian.</p>
<p>&nbsp;</p>
<p>Thanks for the info and pointer to python. I'll check it out.</p>
<p>&nbsp;</p>
<p>-Bill</p>
<div style=3D"FONT-SIZE: 16px; FONT-FAMILY: Times New Roman; COLOR: #000000=
">
<hr tabindex=3D"-1">
<div id=3D"divRpF480768" style=3D"DIRECTION: ltr"><font color=3D"#000000" s=
ize=3D"2" face=3D"Tahoma"><b>From:</b> Fabian Hueske [fhueske@gmail.com]<br=
>
<b>Sent:</b> Monday, July 06, 2015 3:23 AM<br>
<b>To:</b> user@flink.apache.org<br>
<b>Subject:</b> Re: data conversion between flink and &quot;other&quot; par=
adigms<br>
</font><br>
</div>
<div></div>
<div>
<div dir=3D"ltr">
<div>
<div>
<div>Hi Bill,<br>
<br>
</div>
a DataSet is just a logical concept in Flink. DataSets are often not persis=
ted and just streamed along operators. At the moment, there is no way to ac=
cess an intermediate DataSet of a Flink program directly (this might change=
 in the future).
<br>
</div>
<br>
You can process data in another function by implementing a Java user functi=
on (for example a MapPartition function) and sending the data through JNI t=
o a C function (if you need the full data set, you must set the parallelism=
 to 1). Flink's Python API follows
 a similar approach to ship data from Flink to an external Python process.<=
br>
<br>
</div>
<div>Best, Fabian<br>
</div>
<br>
<br>
</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">2015-07-06 9:30 GMT&#43;02:00 Bill Sparks <span =
dir=3D"ltr">&lt;<a href=3D"mailto:jsparks@cray.com" target=3D"_blank">jspar=
ks@cray.com</a>&gt;</span>:<br>
<blockquote class=3D"gmail_quote" style=3D"PADDING-LEFT: 1ex; MARGIN: 0px 0=
px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma; COLOR: #000000; DIRECTI=
ON: ltr">
<p>Just a question if there was some prior-art here. Just say someone wante=
d to use flink for processing, but at some point they wanted to call anothe=
r function via say JNI/C which doesn't understand DataSet's. How would one =
go about this ... I'm assuming the
 code would have to convert the data to a common format prior to calling th=
e function.</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>Regards,</p>
<p>&nbsp;&nbsp; Bill.</p>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>

--_000_121F01D0B574FC4DA43817B7D9CF419E243FB07ECFWEX01americas_--