flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-4890) FileInputFormatTest#testExcludeFiles fails on Windows OS
Date Thu, 05 Jan 2017 12:12:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15801205#comment-15801205
] 

Chesnay Schepler edited comment on FLINK-4890 at 1/5/17 12:12 PM:
------------------------------------------------------------------

It does kind of work.

When matching paths the we ignore schema of the path; so given an include pattern "/*/text"
a path like "hdfs:/my/text" would be accepted. (since it is reduced to "/my/text" which matches
the pattern)

However, if the pattern would be "hdfs:/*/text" then the matching would fail; but not just
on windows but on every OS.

If we were to fix this however, and include the schema in the matching process, then it would
indeed fail by default on Windows for anything but local paths.


was (Author: zentol):
It does kind of work.

When matching paths the schema of the path is ignored; so given an include pattern "/*/text"
a path like "hdfs:/my/text" would be accepted. (since it is reduced to "/my/text" which matches
the pattern)

However, if the pattern would be "hdfs:/*/text" then the matching would fail; but not just
on windows but on every OS.

If we were to fix this however, and include the schema in the matching process, then it would
indeed fail by default on Windows for anything but local paths.

> FileInputFormatTest#testExcludeFiles fails on Windows OS
> --------------------------------------------------------
>
>                 Key: FLINK-4890
>                 URL: https://issues.apache.org/jira/browse/FLINK-4890
>             Project: Flink
>          Issue Type: Bug
>          Components: Batch Connectors and Input/Output Formats
>    Affects Versions: 1.1.3
>         Environment: Windows 10
>            Reporter: Chesnay Schepler
>
> Running the mentioned test leads to an exception:
> {code}
> Illegal char <:> at index 4: file:/C:/dev/cygwin64/tmp/junit3838395086498044255/another_file.bin
> java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:/C:/dev/cygwin64/tmp/junit3838395086498044255/anot
                                                                                         
                          her_file.bin
>         at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>         at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>         at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>         at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>         at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>         at java.nio.file.Paths.get(Paths.java:84)
>         at org.apache.flink.api.common.io.GlobFilePathFilter.filterPath(GlobFilePathFilter.java:95)
>         at org.apache.flink.api.common.io.FileInputFormat.acceptFile(FileInputFormat.java:644)
>         at org.apache.flink.api.common.io.FileInputFormat.addFilesInDir(FileInputFormat.java:600)
>         at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:476)
>         at org.apache.flink.api.common.io.FileInputFormatTest.testReadMultiplePatterns(FileInputFormatTest.java:362)
> {code}
> The problem is that we are given a flink Path, which is then converted to a String and
given to the nio FIleSystem. The passed path is thus /C:/..., which nio can't work with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message