spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Okolnychyi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21850) SparkSQL cannot perform LIKE someColumn if someColumn's value contains a backslash \
Date Thu, 31 Aug 2017 09:25:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148705#comment-16148705
] 

Anton Okolnychyi commented on SPARK-21850:
------------------------------------------

Then we should not be bound to the LIKE case only. I am wondering what is expected in the
code below?

{code}
Seq((1, "\\n")).toDF("col1", "col2").write.saveAsTable("t1")
spark.sql("SELECT * FROM t1").show()
+----+----+
|col1|col2|
+----+----+
|   1|  \n|
+----+----+
spark.sql("SELECT * FROM t1 WHERE col2 = '\\n'").show() // does it give a wrong result?
+----+----+
|col1|col2|
+----+----+
+----+----+
spark.sql("SELECT * FROM t1 WHERE col2 = '\n'").show()
+----+----+
|col1|col2|
+----+----+
+----+----+
{code}

Just to mention, [PR#15398|https://github.com/apache/spark/pull/15398] was also merged in
2.1 (might not be publicly available yet). I got the same exception in the 2.1 branch using
the following code:

{code}
Seq((1, "\\n")).toDF("col1", "col2").write.saveAsTable("t1")
spark.sql("SELECT * FROM t1").show()
spark.sql("SELECT * FROM t1 WHERE col2 LIKE CONCAT(col2, '%')").show()
{code}


> SparkSQL cannot perform LIKE someColumn if someColumn's value contains a backslash \
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-21850
>                 URL: https://issues.apache.org/jira/browse/SPARK-21850
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Adrien Lavoillotte
>
> I have a test table looking like this:
> {code:none}
> spark.sql("select * from `test`.`types_basic`").show()
> {code}
> ||id||c_tinyint|| [...] || c_string||
> |  0|     -128| [...] |              string|
> |  1|        0| [...] |string 'with' "qu...|
> |  2|      127| [...] |      unicod€ strĭng|
> |  3|       42| [...] |there is a \n in ...|
> |  4|     null| [...] |                null|
> Note the line with ID 3, which has a literal \n in c_string (e.g. "some \\n string",
not a line break). I would like to join another table using a LIKE condition (to join on prefix).
If I do this:
> {code:none}
> spark.sql("select * from `test`.`types_basic` a where a.`c_string` LIKE CONCAT(a.`c_string`,
'%')").show()
> {code}
> I get the following error in spark 2.2 (but not in any earlier version):
> {noformat}
> 17/08/28 12:47:38 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 9.0 (TID 12,
cdh5.local, executor 2): org.apache.spark.sql.AnalysisException: the pattern 'there is a \n
in this line%' is invalid, the escape character is not allowed to precede 'n';
> 	at org.apache.spark.sql.catalyst.util.StringUtils$.fail$1(StringUtils.scala:42)
> 	at org.apache.spark.sql.catalyst.util.StringUtils$.escapeLikeRegex(StringUtils.scala:51)
> 	at org.apache.spark.sql.catalyst.util.StringUtils.escapeLikeRegex(StringUtils.scala)
> 	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
> 	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> 	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
> 	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
> 	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
> 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
> 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:108)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> {noformat}
> It seems to me that if LIKE requires special escaping there, then it should be provided
by SparkSQL on the value of the column.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message