Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2872E18C94 for ; Mon, 21 Mar 2016 10:36:22 +0000 (UTC) Received: (qmail 22087 invoked by uid 500); 21 Mar 2016 10:36:21 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 22003 invoked by uid 500); 21 Mar 2016 10:36:21 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 21993 invoked by uid 99); 21 Mar 2016 10:36:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2016 10:36:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0C0D6C0223 for ; Mon, 21 Mar 2016 10:36:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.189 X-Spam-Level: * X-Spam-Status: No, score=1.189 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id zQ5yhCHDYUVz for ; Mon, 21 Mar 2016 10:36:19 +0000 (UTC) Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 921235F245 for ; Mon, 21 Mar 2016 10:36:18 +0000 (UTC) Received: by mail-wm0-f44.google.com with SMTP id l68so116026333wml.1 for ; Mon, 21 Mar 2016 03:36:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Ke3qo+qxt29+iAQUDjCGSwJ79gxK15T1R2+tuPEdX5o=; b=MVg+Sm63qdcxocQKnnKQwloTEXDBRot+qCevaUDjFDCldJuhLYVfoypvQUTy0VqQzo eQ+q3qBR8+lFlvx7STZBS6mK0UX+WrmK6C5Mmw2kLrHLdGAunpFMebuYqNdYlHekgn0a wfJUMGzSxcW1HMG6TxTGo6BWzMnp12RNlQbYIoH+21Urope29abOSMQLcjzEV4gB/xYW ZKmbhE4gqTToUJgfUnuaoEUVOqABteSco63XhAD2s2I0Cvy9rCjVwQgrJdULWNIiLiaa X8j4dCsGKWnRQ37s1AazRCsYmqiYkuwggs228ijVaFkETh0fP1WKCQo9W4/nIQVeDdT/ MvrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Ke3qo+qxt29+iAQUDjCGSwJ79gxK15T1R2+tuPEdX5o=; b=Y2X9R2CtKmEfI07ckfeygpwyfJVWCCOs32deUM5VVB618KLtb8ByWQoIzz0icXaGiO nPWz2OYS2mtL9OriNUPYtYYzMQhC2bK0Xe5g93/HEU4bOEASEKSGF74rxiWACjXubytO Dg8lqQAUTVvfC6eskYrlihUDUmBwxF8v9TCQmowWNzEwvMIBKYJCOuhHJprSdq0xo2od 0EhGOqiBrIGGrjC/8VRBcegjylThozXoZKfsg1kCLeTzSlEzuYpPSplXxn+bCROrnz2z XVRmg8dlphS34dvodpcmsO0+r6BM++W0Bfz1VXRGFdXUNPM67dIjOx0UGZ+9Bd20gl0D tc+Q== X-Gm-Message-State: AD7BkJLb1C+45BXOQQT0GldleynPogj+HFFEObskuNmdwuRi3y0ABs+qKgg2hCAIvO7ipULlzGUNP0EfBqSGIA== X-Received: by 10.194.92.107 with SMTP id cl11mr33350060wjb.21.1458556577490; Mon, 21 Mar 2016 03:36:17 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.87.21 with HTTP; Mon, 21 Mar 2016 03:35:47 -0700 (PDT) In-Reply-To: References: From: Fabian Hueske Date: Mon, 21 Mar 2016 11:35:47 +0100 Message-ID: Subject: Re: Flink 1.0.0 reading files from multiple directory with wildcards To: user@flink.apache.org Content-Type: multipart/alternative; boundary=047d7bf0d2d4afae09052e8caa2a --047d7bf0d2d4afae09052e8caa2a Content-Type: text/plain; charset=UTF-8 Hi, no, this is currently not supported. However, I agree this would be a very valuable addition to the FileInputFormat. Would you mind opening a JIRA issue with your suggestions? Until this is added to Flink, it can be implemented as a custom InputFormat based on FileInputFormat by overriding the createInputSplits() method. Best, Fabian 2016-03-21 0:11 GMT+01:00 Sourigna Phetsarath : > All, > > Do any of the Flink Data Sources support comma separated directories with > wildcards? > > For example: > > env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*, > /data/2016/01/03/*/*") > > > Thanks in advance for any help that you can provide. > -- > > > *Gna Phetsarath*System Architect // AOL Platforms // Data Services // > Applied Research Chapter > 770 Broadway, 5th Floor, New York, NY 10003 > o: 212.402.4871 // m: 917.373.7363 > vvmr: 8890237 aim: sphetsarath20 t: @sourigna > > * * > --047d7bf0d2d4afae09052e8caa2a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

no, this is currently not = supported. However, I agree this would be a very valuable addition to the F= ileInputFormat.
Would you mind opening a JIRA issue with your suggestio= ns?

Until this is added to Flink, it can be implemented as a c= ustom InputFormat based on FileInputFormat by overriding the createInputSpl= its() method.

Best, Fabian

2016-03-21 0:11 GMT+01:00 Sourigna Phetsara= th <gna.phetsarath@teamaol.com>:
All,

Do any of the Flin= k Data Sources support comma separated directories with wildcards?

For example:
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")<= /div>

Thanks in advance for any help that y= ou can provide.
--
=

Gna Phetsarath
System Architect // AOL Platforms // Da= ta Services // Applied Research Chapter
770 Broadway, 5th Floor, New Yor= k, NY 10003
o: 212.402.4871 // m: 917.373.7363
vvmr:=C2= =A08890237=C2=A0aim: = sphetsarath20 t: @sourigna

<= /a>


--047d7bf0d2d4afae09052e8caa2a--