Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Message-ID: <1434378723.557ee1e324324@euranova.eu>
Date: Mon, 15 Jun 2015 16:32:03 +0200
Subject: Re: Iteration stats logging
From: Nam-Luc Tran <namluc.tran@euranova.eu>
To: dev@flink.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="_=_swift_v4_1434378723557ee1e32508f_=_"

--_=_swift_v4_1434378723557ee1e32508f_=_
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Ufuk,

The kind of things we'd like to log are: time spent in=
 the iteration,
residual of the algorithm (convergence), current iter=
ation.

Best regards,

Tran Nam-Luc
=C2=A0=


At Monday, 15/06/2015 on 16:15 Ufuk Celebi wrote:

H=
ey Tran Nam-Luc,

there is currently no way to do th=
is.

The iteration sync tasks keeps track of iteration converg=
ence/max
number of iterations and signals termination to the iterati=
on head.
After this, the head flushes the produced result to the nex=
t task
(after the iteration) and the intermediate iteration tasks fi=
nish w/o
calling close again.

Because there is no "final" n=
o-op iteration happening, the iteration
tasks don't know when the las=
t iteration happened.

I'm not sure what the best way is to impl=
ement this at the moment.

What kind of stats are you recording?=


=E2=80=93 Ufuk

On 15 Jun 2015, at 15:53, Nam-Luc =
Tran  wrote:

> Hello Everyone,
>=20
> I would like to =
log certain stats during iterations in a bulk
> iterative job. The w=
ay I do this is store the things I want at each
> iteration and plan =
to flush everything to HDFS once all the
iterations
> are done. T=
o do that I would need to know when the last iteration is
> invoked =
in order to flush the data. However, the close() method in
the
>=
 RichMapFunction is executed at the end of each iteration.
>=20
> =
Is there anyway to know when I am in the last invocation? Or would
y=
ou
> have a better suggestion to achieve what I am trying to do?=

>=20
> Thank you and best regards,
>=20
> Tran Nam-Lu=
c=20
>=20
>


--_=_swift_v4_1434378723557ee1e32508f_=_--