flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adarsh Jain <eradarshj...@gmail.com>
Subject Re: Recursive Traversal of the Input Path Directory, Not working
Date Fri, 23 Jun 2017 13:08:53 GMT
Hi Stefan,

I think I found the problem, try it with a file which starts with
underscore in the name like "_part-1-0.csv".

While saving Flink appends a "_" to the file name however while reading at
folder level it does not pick those files.

Can you suggest if we can do a setting so that it does not pre appends
underscore while saving a file.

Regards,
Adarsh

On Fri, Jun 23, 2017 at 3:24 PM, Stefan Richter <s.richter@data-artisans.com
> wrote:

> No, that doesn’t make a difference and also works.
>
> Am 23.06.2017 um 11:40 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>
> I am using "val env = ExecutionEnvironment.getExecutionEnvironment", can
> this be the problem?
>
> With "import org.apache.flink.api.scala.ExecutionEnvironment"
>
> Using scala in my program.
>
> Regards,
> Adarsh
>
> On Fri, Jun 23, 2017 at 3:01 PM, Stefan Richter <
> s.richter@data-artisans.com> wrote:
>
>> I just copy pasted your code, adding the missing "val env
>> = LocalEnvironment.createLocalEnvironment()" and exchanged the string
>> with a local directory for some test files that I created. No other changes.
>>
>> Am 23.06.2017 um 11:25 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>>
>> Hi Stefan,
>>
>> Thanks for your efforts in checking the same, still doesn't work for me.
>>
>> Can you copy paste the code you used maybe I am doing some silly mistake
>> and am not able to figure out the same.
>>
>> Thanks again.
>>
>> Regards,
>> Adarsh
>>
>>
>> On Fri, Jun 23, 2017 at 2:32 PM, Stefan Richter <
>> s.richter@data-artisans.com> wrote:
>>
>>> Hi,
>>>
>>> I tried this out on the current master and the 1.3 release and both work
>>> for me everything works exactly as expected, for file names, a directory,
>>> and even nested directories.
>>>
>>> Best,
>>> Stefan
>>>
>>> Am 22.06.2017 um 21:13 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>>>
>>> Hi Stefan,
>>>
>>> Yes your understood right, when I give full path till the filename it
>>> works fine however when I give path till
>>> directory it does not read the data, doesn't print any exceptions too
>>> ... I am also not sure why it is behaving like this.
>>>
>>> Should be easily replicable, in case you can try. Will be really helpful.
>>>
>>> Regards,
>>> Adarsh
>>>
>>> On Thu, Jun 22, 2017 at 9:00 PM, Stefan Richter <
>>> s.richter@data-artisans.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am not sure I am getting the problem right: the code works if you use
>>>> a file name, but it does not work for directories? What exactly is not
>>>> working? Do you get any exceptions?
>>>>
>>>> Best,
>>>> Stefan
>>>>
>>>> Am 22.06.2017 um 17:01 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>>>>
>>>> Hi,
>>>>
>>>> I am trying to use "Recursive Traversal of the Input Path Directory" in
>>>> Flink 1.3 using scala. Snippet of my code below. If I give exact file name
>>>> it is working fine. Ref https://ci.apache.org/proj
>>>> ects/flink/flink-docs-release-1.3/dev/batch/index.html
>>>>
>>>> import org.apache.flink.api.java.utils.ParameterTool
>>>> import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}
>>>> import org.apache.flink.configuration.Configuration
>>>>
>>>> val config = new Configuration
>>>>     config.setBoolean("recursive.file.enumeration",true)
>>>>
>>>> val featuresSource: String = "file:///Users/adarsh/Document
>>>> s/testData/featurecsv/31c710ac40/2017/06/22"
>>>>
>>>> val testInput = env.readTextFile(featuresSource).withParameters(config)
>>>> testInput.print()
>>>>
>>>> Please guide how to fix this.
>>>>
>>>> Regards,
>>>> Adarsh
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message