Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
Sender: ewenstephan@gmail.com
In-Reply-To: <1DC68751-2356-4766-9347-DCDBB982A9D0@mail.polimi.it>
References: <9962C856-717B-46C1-B35B-02ABD2FD8361@mail.polimi.it>
	<CANC1h_t+XwMWD8-Qu+xTsLQPQn+OMPDjogJytFwMN+_LRf0hPw@mail.gmail.com>
	<1DC68751-2356-4766-9347-DCDBB982A9D0@mail.polimi.it>
Date: Fri, 26 Jun 2015 12:21:57 +0200
Message-ID: 
 <CANC1h_sgonf=hx6yx+_HqMcGAJbH7urC2O28Jo91WoAB9SYyoA@mail.gmail.com>
Subject: Re: open multiple file from list of uri
From: Stephan Ewen <sewen@apache.org>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a62721e6600519691c52

--bcaec548a62721e6600519691c52
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Sure, just override the "createInputSplits()" method. Call for each of your
file paths "super.createInputSplits()" and then combine the results into
one array that you return.

That should do it...

On Fri, Jun 26, 2015 at 12:19 PM, Michele Bertoni <
michele1.bertoni@mail.polimi.it> wrote:

>  Hi Stephan, thanks for answering,
> right now I am using an extension of the DelimitedInputFormat, is there a
> way to merge it with the option 2?
>
>
>
>  Il giorno 26/giu/2015, alle ore 12:17, Stephan Ewen <sewen@apache.org>
> ha scritto:
>
>  There are two ways you can realize that:
>
>  1) Create multiple sources and union them. This is easy, but probably a
> bit less efficient.
>
>  2) Override the FileInputFormat's createInputSplits method to take a
> union of the paths to create a list of all files and fils splits that wil=
l
> be read.
>
>  Stephan
>
>
> On Fri, Jun 26, 2015 at 12:12 PM, Michele Bertoni <
> michele1.bertoni@mail.polimi.it> wrote:
>
>> Hi everybody,
>> is there a way to specify a list of URI (=E2=80=9Chdfs://file1=E2=80=9D,=
=E2=80=9Dhdfs://file2=E2=80=9D,=E2=80=A6)
>> and open them as different files?
>> I know i may open the entire directory, but i want to be able to select =
a
>> subset of files in the directory
>>
>> thanks
>
>
>
>

--bcaec548a62721e6600519691c52
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Sure, just override the &quot;createInputSplits()&quot; me=
thod. Call for each of your file paths &quot;super.createInputSplits()&quot=
; and then combine the results into one array that you return.<div><br></di=
v><div>That should do it...</div></div><div class=3D"gmail_extra"><br><div =
class=3D"gmail_quote">On Fri, Jun 26, 2015 at 12:19 PM, Michele Bertoni <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:michele1.bertoni@mail.polimi.it" targe=
t=3D"_blank">michele1.bertoni@mail.polimi.it</a>&gt;</span> wrote:<br><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc=
c solid;padding-left:1ex">


<div style=3D"word-wrap:break-word">
Hi Stephan, thanks for answering,
<div>right now I am using an extension of the DelimitedInputFormat, is ther=
e a way to merge it with the option 2?
<div><div class=3D"h5"><div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<blockquote type=3D"cite">
<div>Il giorno 26/giu/2015, alle ore 12:17, Stephan Ewen &lt;<a href=3D"mai=
lto:sewen@apache.org" target=3D"_blank">sewen@apache.org</a>&gt; ha scritto=
:</div>
<br>
<div>
<div dir=3D"ltr">There are two ways you can realize that:
<div><br>
</div>
<div>1) Create multiple sources and union them. This is easy, but probably =
a bit less efficient.</div>
<div><br>
</div>
<div>2) Override the FileInputFormat&#39;s createInputSplits method to take=
 a union of the paths to create a list of all files and fils splits that wi=
ll be read.</div>
<div><br>
</div>
<div>Stephan</div>
<div><br>
</div>
</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">On Fri, Jun 26, 2015 at 12:12 PM, Michele Berton=
i <span dir=3D"ltr">
&lt;<a href=3D"mailto:michele1.bertoni@mail.polimi.it" target=3D"_blank">mi=
chele1.bertoni@mail.polimi.it</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Hi everybody,<br>
is there a way to specify a list of URI (=E2=80=9C<a>hdfs://file1=E2=80=9D,=
=E2=80=9Dhdfs://file2</a>=E2=80=9D,=E2=80=A6) and open them as different fi=
les?<br>
I know i may open the entire directory, but i want to be able to select a s=
ubset of files in the directory<br>
<br>
thanks</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div></div></div>
</div>

</blockquote></div><br></div>

--bcaec548a62721e6600519691c52--