flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adarsh Jain <eradarshj...@gmail.com>
Subject Re: Recursive Traversal of the Input Path Directory, Not working
Date Fri, 23 Jun 2017 09:40:18 GMT
I am using "val env = ExecutionEnvironment.getExecutionEnvironment", can
this be the problem?

With "import org.apache.flink.api.scala.ExecutionEnvironment"

Using scala in my program.

Regards,
Adarsh

On Fri, Jun 23, 2017 at 3:01 PM, Stefan Richter <s.richter@data-artisans.com
> wrote:

> I just copy pasted your code, adding the missing "val env
> = LocalEnvironment.createLocalEnvironment()" and exchanged the string
> with a local directory for some test files that I created. No other changes.
>
> Am 23.06.2017 um 11:25 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>
> Hi Stefan,
>
> Thanks for your efforts in checking the same, still doesn't work for me.
>
> Can you copy paste the code you used maybe I am doing some silly mistake
> and am not able to figure out the same.
>
> Thanks again.
>
> Regards,
> Adarsh
>
>
> On Fri, Jun 23, 2017 at 2:32 PM, Stefan Richter <
> s.richter@data-artisans.com> wrote:
>
>> Hi,
>>
>> I tried this out on the current master and the 1.3 release and both work
>> for me everything works exactly as expected, for file names, a directory,
>> and even nested directories.
>>
>> Best,
>> Stefan
>>
>> Am 22.06.2017 um 21:13 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>>
>> Hi Stefan,
>>
>> Yes your understood right, when I give full path till the filename it
>> works fine however when I give path till
>> directory it does not read the data, doesn't print any exceptions too ...
>> I am also not sure why it is behaving like this.
>>
>> Should be easily replicable, in case you can try. Will be really helpful.
>>
>> Regards,
>> Adarsh
>>
>> On Thu, Jun 22, 2017 at 9:00 PM, Stefan Richter <
>> s.richter@data-artisans.com> wrote:
>>
>>> Hi,
>>>
>>> I am not sure I am getting the problem right: the code works if you use
>>> a file name, but it does not work for directories? What exactly is not
>>> working? Do you get any exceptions?
>>>
>>> Best,
>>> Stefan
>>>
>>> Am 22.06.2017 um 17:01 schrieb Adarsh Jain <eradarshjain@gmail.com>:
>>>
>>> Hi,
>>>
>>> I am trying to use "Recursive Traversal of the Input Path Directory" in
>>> Flink 1.3 using scala. Snippet of my code below. If I give exact file name
>>> it is working fine. Ref https://ci.apache.org/proj
>>> ects/flink/flink-docs-release-1.3/dev/batch/index.html
>>>
>>> import org.apache.flink.api.java.utils.ParameterTool
>>> import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}
>>> import org.apache.flink.configuration.Configuration
>>>
>>> val config = new Configuration
>>>     config.setBoolean("recursive.file.enumeration",true)
>>>
>>> val featuresSource: String = "file:///Users/adarsh/Document
>>> s/testData/featurecsv/31c710ac40/2017/06/22"
>>>
>>> val testInput = env.readTextFile(featuresSource).withParameters(config)
>>> testInput.print()
>>>
>>> Please guide how to fix this.
>>>
>>> Regards,
>>> Adarsh
>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message