Return-Path: <user-return-14894-archive-asf-public=cust-asf.ponee.io@flink.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 4280D166696
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 22 Aug 2017 09:51:55 +0200 (CEST)
Received: (qmail 37561 invoked by uid 500); 22 Aug 2017 07:51:54 -0000
Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@flink.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@flink.apache.org>
List-Post: <mailto:user@flink.apache.org>
List-Id: <user.flink.apache.org>
Delivered-To: mailing list user@flink.apache.org
Received: (qmail 37544 invoked by uid 99); 22 Aug 2017 07:51:54 -0000
Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Aug 2017 07:51:54 +0000
Received: from mail-qt0-f182.google.com (mail-qt0-f182.google.com [209.85.216.182])
	by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id F0E111A02CF
	for <user@flink.apache.org>; Tue, 22 Aug 2017 07:51:52 +0000 (UTC)
Received: by mail-qt0-f182.google.com with SMTP id d15so22916680qta.0
        for <user@flink.apache.org>; Tue, 22 Aug 2017 00:51:51 -0700 (PDT)
X-Gm-Message-State: AHYfb5jMQTfD8D8TRZetHnFrpo33i4xODPT3PvJZS+7dy0UtoS3pTFGg
	zIhqrlpC3lNMLcd0x1IHYh65mWMRjA==
X-Received: by 10.200.25.168 with SMTP id u37mr18291155qtj.49.1503388310374;
 Tue, 22 Aug 2017 00:51:50 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.12.168.214 with HTTP; Tue, 22 Aug 2017 00:51:09 -0700 (PDT)
In-Reply-To: <CAOT3TWp7UEEffih4fZLwnYvLeFp32fYr017m+kYrVRB=tUYNkw@mail.gmail.com>
References: <CAOT3TWrEgVSCN1KrW2mOkh=6bV0CEuP_stWYOtQ22_bJ-AGjUA@mail.gmail.com>
 <CAOT3TWp7UEEffih4fZLwnYvLeFp32fYr017m+kYrVRB=tUYNkw@mail.gmail.com>
From: Till Rohrmann <trohrmann@apache.org>
Date: Tue, 22 Aug 2017 09:51:09 +0200
X-Gmail-Original-Message-ID: <CAC27z=O0PPtdP5YWHF7CFjJrUy0ikMb3-itoAhDYrkSTwx-kmw@mail.gmail.com>
Message-ID: <CAC27z=O0PPtdP5YWHF7CFjJrUy0ikMb3-itoAhDYrkSTwx-kmw@mail.gmail.com>
Subject: Re: Deleting files in continuous processing
To: Mohit Anchlia <mohitanchlia@gmail.com>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="001a114ca06633129b055752def2"

--001a114ca06633129b055752def2
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Mohit,

Flink does not support this behaviour out of the box afaik. I think you
have to write your own source function or extend
ContinuousFileMonitoringFunction in order to do that.

Cheers,
Till
=E2=80=8B

On Mon, Aug 21, 2017 at 11:07 PM, Mohit Anchlia <mohitanchlia@gmail.com>
wrote:

> Just checking to see if there is a way to purge files after it's processe=
d.
>
> On Tue, Aug 15, 2017 at 5:11 PM, Mohit Anchlia <mohitanchlia@gmail.com>
> wrote:
>
>> Is there a way to delete a file once it has been processed?
>>
>> streamEnv
>>
>> .readFile(format, args[0], FileProcessingMode.*PROCESS_CONTINUOUSLY*,
>> 2000)
>>
>
>

--001a114ca06633129b055752def2
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"markdown-here-wrapper" style=3D""><p style=
=3D"margin:0px 0px 1.2em!important">Hi Mohit,</p>
<p style=3D"margin:0px 0px 1.2em!important">Flink does not support this beh=
aviour out of the box afaik. I think you have to write your own source func=
tion or extend <code style=3D"font-size:0.85em;font-family:Consolas,Inconso=
lata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-=
wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);bo=
rder-radius:3px;display:inline">ContinuousFileMonitoringFunction</code> in =
order to do that.</p>
<p style=3D"margin:0px 0px 1.2em!important">Cheers,<br>Till</p>
<div title=3D"MDH:SGkgTW9oaXQsPGRpdj48YnI+PC9kaXY+PGRpdj5GbGluayBkb2VzIG5vd=
CBzdXBwb3J0IHRoaXMg
YmVoYXZpb3VyIG91dCBvZiB0aGUgYm94LiBJIHRoaW5rIHlvdSBoYXZlIHRvIHdyaXRlIHlvdXI=
g
b3duIHNvdXJjZSBmdW5jdGlvbiBvciBleHRlbmQgYENvbnRpbnVvdXNGaWxlTW9uaXRvcmluZ0Z=
1
bmN0aW9uYCBpbiBvcmRlciB0byBkbyB0aGF0LjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+Q2h=
l
ZXJzLDwvZGl2PjxkaXY+VGlsbDwvZGl2Pg=3D=3D" style=3D"height:0;width:0;max-hei=
ght:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0">=E2=80=
=8B</div></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">On Mon, Aug 21, 2017 at 11:07 PM, Mohit Anchlia <span dir=3D"ltr">&lt;<=
a href=3D"mailto:mohitanchlia@gmail.com" target=3D"_blank">mohitanchlia@gma=
il.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"=
ltr">Just checking to see if there is a way to purge files after it&#39;s p=
rocessed.</div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_=
extra"><br><div class=3D"gmail_quote">On Tue, Aug 15, 2017 at 5:11 PM, Mohi=
t Anchlia <span dir=3D"ltr">&lt;<a href=3D"mailto:mohitanchlia@gmail.com" t=
arget=3D"_blank">mohitanchlia@gmail.com</a>&gt;</span> wrote:<br><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex"><div dir=3D"ltr"><div>Is there a way to delete a file =
once it has been processed?</div><div><br></div><div><font color=3D"#6a3e3e=
" size=3D"2"><font color=3D"#6a3e3e" size=3D"2"><p align=3D"LEFT">streamEnv=
</p>
</font></font><font size=3D"2"><p><font size=3D"2">				.readFile(</font><fo=
nt color=3D"#6a3e3e" size=3D"2"><font color=3D"#6a3e3e" size=3D"2">format</=
font></font><font size=3D"2">, </font><font color=3D"#6a3e3e" size=3D"2"><f=
ont color=3D"#6a3e3e" size=3D"2">args</font></font><font size=3D"2">[0], Fi=
leProcessingMode.</font><b><i><font color=3D"#0000c0" size=3D"2"><font colo=
r=3D"#0000c0" size=3D"2">PROCESS_CON<wbr>TINUOUSLY</font></font></i></b><fo=
nt size=3D"2">, 2000)<span></span></font></p></font></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a114ca06633129b055752def2--