Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E34CB200D28 for ; Mon, 23 Oct 2017 20:51:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E1C8F1609E0; Mon, 23 Oct 2017 18:51:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0C2EA1609DF for ; Mon, 23 Oct 2017 20:51:31 +0200 (CEST) Received: (qmail 34869 invoked by uid 500); 23 Oct 2017 18:51:30 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 34859 invoked by uid 99); 23 Oct 2017 18:51:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Oct 2017 18:51:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 2C8B51A124E for ; Mon, 23 Oct 2017 18:51:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.494 X-Spam-Level: **** X-Spam-Status: No, score=4.494 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FORGED_MUA_MOZILLA=1.596, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLYTO_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_SORBS_SPAM=0.5, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 1wY7ij8RcVx9 for ; Mon, 23 Oct 2017 18:51:28 +0000 (UTC) Received: from sonic304-48.consmr.mail.ne1.yahoo.com (sonic304-48.consmr.mail.ne1.yahoo.com [66.163.191.174]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 105725FB52 for ; Mon, 23 Oct 2017 18:51:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1508784681; bh=Y5IdC8aHDoAXFXNGk/QMbdTG9Ni88P21OSbr8M6dleM=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=RjZmmezjAVzA96E5EjfLMhpb76rD1OqExjrpfTDTfD6Y1cinhuoxEOSlXMaPUZSGsO0SdOtM6qre6whj+RLY0tk9s924f4x/E+T+lScQ599DGCix1IOQGfban9es2XlF6dCCTheEusH/lbHFVear4sfj+O/9BbYhkWTWJz4h0S0qSF/qR+igwQ0dh7KtuvtCnxd1sb44P4i82cUCFUGlKpAjjTFynSixpJ/4wzMxjIZS5LnJ9n8ftlR3Z6gZsokKHCgjAsKZjvNYCebLepq+gxq12euXcvmSw3mjKgKYYuYAmtksUXIT+VElZdLov02bny9zFeIuNzV9NimOMYxntw== X-YMail-OSG: MyPKt44VM1m7zpN0Zq0cmvDpo1LQgChWURp2LlOtNR3wkk7kBX.hq1nq7czfWsi o1eOEOo_dL9oPy7xgZz1uJFVmnSFWM2rsuCYfUwnhvIzx2GTYw.Xpc0e7dcGySpQs6T104EU02QG MT2afYGkbp1mOgtHc2g03xBRGF5SWymyHf2pXQEeZyQPaBinZEclgCDGWemjVXQI4CCwnpGEBeC2 SFQX.IBUn.fCMxwx3U6N8_PlgoV0DbynwcNfhgqqI7tXvV0BDecHmYVgf5OcjGL9.kW1HkQlpTcB zsf_fJX0.EFXPbQs918gNcf4XSRkoexqY7e6HKuDZFmobY_NgFkwYGLnNv5X9CfM3JyJ9ZXSmrWB IDdHGFI0P4ACN7f.wOrR_eiCk.RTreCjc5JdHs.6ENTUNNWUBGMSpWa6IGq46QnjNqHm_YduaBNz rgIeaTytMysgx93EAqnXhpBe1xQzzh6ZUZ9CfGKHMxyhcvUKf2svtK6ghVI.h._4- Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.ne1.yahoo.com with HTTP; Mon, 23 Oct 2017 18:51:21 +0000 Date: Mon, 23 Oct 2017 18:47:16 +0000 (UTC) From: Telco Phone Reply-To: Telco Phone To: "user@flink.apache.org" Message-ID: <1265178977.279590.1508784436583@mail.yahoo.com> Subject: Processing files MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_279589_685685769.1508784436580" References: <1265178977.279590.1508784436583.ref@mail.yahoo.com> X-Mailer: WebService/1.1.10668 YahooMailNeo Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38 archived-at: Mon, 23 Oct 2017 18:51:33 -0000 ------=_Part_279589_685685769.1508784436580 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable All, Im looking to process files in a directory based on files that are coming i= n via file transfer. The files are renamed once the transfer is done to a .DONE. These are binary files and I need to process billions per day. What I want to do is process the file and then create a new file called .PR= OCESSED I need to have a task thread process a file at a time (unsplitable=3Dtrue) The files are in /mnt/DATE/filename.DONE They are coming in on 4-5 servers at the moment. I can run a task manager on each host so these will be processed on each se= rver and written to the same directory. What is the best way to build a continues list of files to process and hand= that filename to tasks threads running on each host...=C2=A0 Hope this makes sense...=C2=A0 Thanks in advance. ------=_Part_279589_685685769.1508784436580 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
All,

Im looking to process = files in a directory based on files that are coming in via file transfer.

The files= are renamed once the transfer is done to a .DONE.

These are binary files and I nee= d to process billions per day.

What I want to do is process the file and then creat= e a new file called .PROCESSED

I need to have a task thread process a file at a tim= e (unsplitable=3Dtrue)

The files are in /mnt/DATE/filename.DONE

They are coming in on 4-5 s= ervers at the moment.

I can run a task manager on each host so these will be proce= ssed on each server and written to the same directory.

=
What is the best way to buil= d a continues list of files to process and hand that filename to tasks thre= ads running on each host... 

Hope this makes sense... 

Thanks in advance.

------=_Part_279589_685685769.1508784436580--