Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
From: Andrew Palumbo <ap.dev@outlook.com>
To: "dev@flink.apache.org" <dev@flink.apache.org>
Subject: Re: Iteration Intermediate Output
Thread-Topic: Iteration Intermediate Output
Thread-Index: AQHRt48Nuz87jLIbbE6qQwcEdxWLb5/R+7/6
Date: Mon, 30 May 2016 21:05:22 +0000
Message-ID: <CY1PR0601MB1228D348B41B274B52BCD3AB94450@CY1PR0601MB1228.namprd06.prod.outlook.com>
References: <CA+3U0Zn4d9+-+kd4UoMxsG8pO9T-NHxHd0rz1jtWAwVH8pumYg@mail.gmail.com>
In-Reply-To: <CA+3U0Zn4d9+-+kd4UoMxsG8pO9T-NHxHd0rz1jtWAwVH8pumYg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: SoftFail (protection.outlook.com: domain of transitioning
 outlook.com discourages use of 25.152.76.60 as permitted sender)
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [CMbZb/VIPphGREUKPZnEJvQ1z3ZT5ye1]
x-eopattributedmessage: 0
x-forefront-antispam-report: CIP:25.152.76.60;IPV:NLI;CTRY:GB;EFV:NLI;SFV:NSPM;SFS:(10019020)(98900003);DIR:OUT;SFP:1102;SCL:1;SRVR:BL2NAM02HT036;H:CY1PR0601MB1228.namprd06.prod.outlook.com;FPR:;SPF:SoftFail;MLV:ovrnspm;A:1;MX:1;LANG:en;
x-ms-office365-filtering-correlation-id: 5728f225-a659-4550-402a-08d388ce1da0
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(1601124038)(5061506196)(5061507196)(1603103041)(1603101137)(1601125047);SRVR:BL2NAM02HT036;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:BL2NAM02HT036;BCL:0;PCL:0;RULEID:;SRVR:BL2NAM02HT036;
x-forefront-prvs: 09583628E0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 May 2016 21:05:22.9803
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2NAM02HT036
X-OriginalArrivalTime: 30 May 2016 21:05:27.0474 (UTC) FILETIME=[FDC3D920:01D1BAB6]
archived-at: Mon, 30 May 2016 21:05:39 -0000

Greg,

We ran into this Issue when implementing the Mahout bindings for Flink [1].=
  It ended up being the major bottleneck for Mahout on Flink, and makes ite=
rative algorithms basically unreasonable.  While it is understook that that=
 Flink's Delta-iterations are intended for use when iterating over Flink Da=
taSet, they are not always suitable to the task.  Eg. FlinkML's  ALS.scala =
[2][3] which must flush intermediate results to the file system.

In-memory caching is a must have for any type of declarative ML DSL riding =
on top of fink.  Eg, SystemML, Mahout, etc, or an internal Flink DSL.  The =
Mahout bindings are Linear Algebra based.  In close collaboration with the =
Flink community [4] and based on a contribution from an intern hired by Dat=
a Artisans  to complete this task and other members of the community, we re=
cently finished up work on the Mahout Distributed Linear Algebra bindings.

Unfortunately the lack of an in-memory cache made the outcome of this effor=
t very sub-optimal.  After the unfinished hand-off of the bindings from the=
 Flink community to the Mahout community, we were forced to find a workarou=
nd for the caching.  We used the template of flushing and executing and flu=
shing results to the File system [5], and had to release the bindings as "E=
xperimental".

Anyone building a declarative ML interface using the Flink Dataset API as a=
 backend will run into similar issues as has been reported on the stackover=
flow thread u refer to and it would be great to have this feature.


Its great to see this being talked about.

Andy

[1] http://mahout.apache.org/users/flinkbindings/flink-internals.html
[2] https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/sr=
c/main/scala/org/apache/flink/ml/recommendation/ALS.scala#L481
[3] https://github.com/apache/flink/blob/master/flink-libraries/flink-ml/sr=
c/main/scala/org/apache/flink/ml/common/FlinkMLTools.scala#L84
[4]https://issues.apache.org/jira/browse/MAHOUT-1570
[5]https://github.com/apache/mahout/blob/master/flink/src/main/scala/org/ap=
ache/mahout/flinkbindings/drm/CheckpointedFlinkDrm.scala#L139

________________________________________
From: Greg Hogan <code@greghogan.com>
Sent: Thursday, May 26, 2016 4:41:53 PM
To: dev@flink.apache.org
Subject: Iteration Intermediate Output

Hi y'all,

I think this is an oft-requested feature [0] and there are many graph
algorithms for which intermediate output is the desired result. I'd like to
take Stephan up on his offer [1] for pointers.

I have yet to get in deep, but I see that iteration tasks are treated
specially as IterationIntermediateTask for synchronization between
supersteps. Also, when OperatorTranslation and GraphCreatingVisitor are
walking the program DAG an iteration must be first reached through the tail=
.

Greg

[0]
http://stackoverflow.com/questions/37224140/possibility-of-saving-partial-o=
utputs-from-bulk-iteration-in-flink-dataset
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Interme=
diate-output-during-delta-iterations-td436.html