drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hao Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1656) Multiple wildcards(*) are not supported
Date Fri, 07 Nov 2014 19:03:35 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202463#comment-14202463
] 

Hao Zhu commented on DRILL-1656:
--------------------------------

Hi Jacques,

Per globStatus Doc:
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html

It supports:
{code}
A filename pattern is composed of regular characters and special pattern matching characters,
which are:

?
Matches any single character.
*
Matches zero or more characters.
[abc]
Matches a single character from character set {a,b,c}.
[a-b]
Matches a single character from the character range {a...b}. Note that character a must be
lexicographically less than or equal to character b.
[^a]
Matches a single character that is not from character set or range {a}. Note that the ^ character
must occur immediately to the right of the opening bracket.
\c
Removes (escapes) any special meaning of character c.
{ab,cd}
Matches a string from the string set {ab, cd}
{ab,c{de,fh}}
Matches a string from the string set {ab, cde, cfh}
{code}

Per tests, many of them are supported in "hadoop fs -ls" but not supported in drill.
{code}
0: jdbc:drill:> select * from `ab*/201409*`
. . . . . . . > ;
Query failed: Failure while parsing sql. Unsupported selection path: /mapr/demo.mapr.com/data/ab*/201409*
[9a1f79a3-8485-45f5-9af4-442e4f5985da]
Node details: maprdemo:31011/31012
net.hydromatic.optiq.tools.ValidationException: java.lang.IllegalStateException: Unsupported
selection path: /mapr/demo.mapr.com/data/ab*/201409*
	at net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:175)
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:143)
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:123)
	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:132)
	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:425)
	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:219)
	at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Unsupported selection path: /mapr/demo.mapr.com/data/ab*/201409*
	at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
	at org.apache.drill.exec.store.dfs.FileSelection.create(FileSelection.java:145)
	at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:122)
	at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:54)
	at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
	at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
	at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:260)
	at net.hydromatic.optiq.jdbc.SimpleOptiqSchema.getTable(SimpleOptiqSchema.java:75)
	at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTableFrom(OptiqCatalogReader.java:87)
	at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:70)
	at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:42)
	at org.eigenbase.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:67)
	at org.eigenbase.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:75)
	at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:85)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:779)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:768)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2599)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:2807)
	at org.eigenbase.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
	at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:85)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:779)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:768)
	at org.eigenbase.sql.SqlSelect.validate(SqlSelect.java:208)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:742)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:458)
	at net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:173)
	... 9 more


Error: exception while executing query: Failure while trying to get next result batch. (state=,code=0)

[root@maprdemo]# hadoop fs -ls /mapr/demo.mapr.com/data/ab*/201409*
Found 2 items
-rw-r--r--   1 mapr mapr          6 2014-11-06 16:04 /mapr/demo.mapr.com/data/abc/20140901/a.csv
-rw-r--r--   1 mapr mapr          6 2014-11-06 16:04 /mapr/demo.mapr.com/data/abc/20140901/b.csv
Found 1 items
-rw-r--r--   1 mapr mapr          6 2014-11-06 16:04 /mapr/demo.mapr.com/data/abc/20140902/c.csv
Found 1 items
-rw-r--r--   1 root root          6 2014-11-06 18:20 /mapr/demo.mapr.com/data/abc/20140903/d.csv
{code}


Another example:
{code}
0: jdbc:drill:> select * from `ab?/201409*`; 
Query failed: Failure while parsing sql. Unsupported selection path: /mapr/demo.mapr.com/data/ab?/201409*
[960beaa3-19dd-4bd2-9dba-3f0b95f9e301]
Node details: maprdemo:31011/31012
net.hydromatic.optiq.tools.ValidationException: java.lang.IllegalStateException: Unsupported
selection path: /mapr/demo.mapr.com/data/ab?/201409*
	at net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:175)
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:143)
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:123)
	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:132)
	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:425)
	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:219)
	at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Unsupported selection path: /mapr/demo.mapr.com/data/ab?/201409*
	at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
	at org.apache.drill.exec.store.dfs.FileSelection.create(FileSelection.java:145)
	at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:122)
	at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:54)
	at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
	at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
	at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:260)
	at net.hydromatic.optiq.jdbc.SimpleOptiqSchema.getTable(SimpleOptiqSchema.java:75)
	at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTableFrom(OptiqCatalogReader.java:87)
	at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:70)
	at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:42)
	at org.eigenbase.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:67)
	at org.eigenbase.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:75)
	at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:85)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:779)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:768)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2599)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:2807)
	at org.eigenbase.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
	at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:85)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:779)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:768)
	at org.eigenbase.sql.SqlSelect.validate(SqlSelect.java:208)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:742)
	at org.eigenbase.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:458)
	at net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:173)
	... 9 more


Error: exception while executing query: Failure while trying to get next result batch. (state=,code=0)

[root@maprdemo ]# hadoop fs -ls /mapr/demo.mapr.com/data/ab?/201409*
Found 2 items
-rw-r--r--   1 mapr mapr          6 2014-11-06 16:04 /mapr/demo.mapr.com/data/abc/20140901/a.csv
-rw-r--r--   1 mapr mapr          6 2014-11-06 16:04 /mapr/demo.mapr.com/data/abc/20140901/b.csv
Found 1 items
-rw-r--r--   1 mapr mapr          6 2014-11-06 16:04 /mapr/demo.mapr.com/data/abc/20140902/c.csv
Found 1 items
-rw-r--r--   1 root root          6 2014-11-06 18:20 /mapr/demo.mapr.com/data/abc/20140903/d.csv
{code}


> Multiple wildcards(*) are not supported
> ---------------------------------------
>
>                 Key: DRILL-1656
>                 URL: https://issues.apache.org/jira/browse/DRILL-1656
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 0.6.0
>            Reporter: Hao Zhu
>
> Drill can not query the directories using multiple wild cards.
> eg:
> {code}
> 0: jdbc:drill:> select * from `abc/201409*`;
> +------------+------------+
> |  columns   |    dir0    |
> +------------+------------+
> | ["4","4","4"] | 20140903   |
> | ["3","3","3"] | 20140902   |
> | ["2","2","2"] | 20140901   |
> | ["1","1","1"] | 20140901   |
> +------------+------------+
> 4 rows selected (0.213 seconds)
> 0: jdbc:drill:> select * from `ab*/201409*`;
> Query failed: Failure while parsing sql. Unsupported selection path: /mapr/demo.mapr.com/data/ab*/201409*
[0006b50e-ef6b-475d-8488-2fc237d39517]
> Error: exception while executing query: Failure while trying to get next result batch.
(state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message