flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: Recursive Traversal of the Input Path Directory, Not working
Date Fri, 23 Jun 2017 09:02:34 GMT
Hi,

I tried this out on the current master and the 1.3 release and both work for me everything
works exactly as expected, for file names, a directory, and even nested directories.

Best,
Stefan

> Am 22.06.2017 um 21:13 schrieb Adarsh Jain <eradarshjain@gmail.com>:
> 
> Hi Stefan,
> 
> Yes your understood right, when I give full path till the filename it works fine however
when I give path till 
> directory it does not read the data, doesn't print any exceptions too ... I am also not
sure why it is behaving like this.
> 
> Should be easily replicable, in case you can try. Will be really helpful.
> 
> Regards,
> Adarsh
> 
> On Thu, Jun 22, 2017 at 9:00 PM, Stefan Richter <s.richter@data-artisans.com <mailto:s.richter@data-artisans.com>>
wrote:
> Hi,
> 
> I am not sure I am getting the problem right: the code works if you use a file name,
but it does not work for directories? What exactly is not working? Do you get any exceptions?
> 
> Best,
> Stefan
> 
>> Am 22.06.2017 um 17:01 schrieb Adarsh Jain <eradarshjain@gmail.com <mailto:eradarshjain@gmail.com>>:
>> 
>> Hi,
>> 
>> I am trying to use "Recursive Traversal of the Input Path Directory" in Flink 1.3
using scala. Snippet of my code below. If I give exact file name it is working fine. Ref https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html>
>> 
>> import org.apache.flink.api.java.utils.ParameterTool
>> import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}
>> import org.apache.flink.configuration.Configuration
>> 
>> val config = new Configuration
>>     config.setBoolean("recursive.file.enumeration",true)
>> 
>> val featuresSource: String = "file:///Users/adarsh/Documents/testData/featurecsv/31c710ac40/2017/06/22
<>"
>> 
>> val testInput = env.readTextFile(featuresSource).withParameters(config)
>> testInput.print()
>> 
>> Please guide how to fix this.
>> 
>> Regards,
>> Adarsh
>> 
> 
> 


Mime
View raw message