spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18593) Unable to use filter function if the data length is one.
Date Sat, 26 Nov 2016 21:58:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15698567#comment-15698567
] 

Dongjoon Hyun commented on SPARK-18593:
---------------------------------------

Oh, sorry. I reproduced this in `branch-1.6`.

- When I added a testcase in JDBCWriter. There was no problem in both version. It doesn't
use PostgreSQL driver.

- However, when I tried in spark-shell with postgresql driver. Only `branch-1.6` has the same
problem.
{code}
scala> val df = sqlContext.read.option("user", "postgres").option("password", "test").jdbc("jdbc:postgresql://localhost:5432/postgres",
"t1", new java.util.Properties())
df: org.apache.spark.sql.DataFrame = [a: string]

scala> df.show
+----+
|   a|
+----+
|A   |
+----+


scala> df.filter(df("a")==="A").show
+---+
|  a|
+---+
+---+
{code}

I'll take a look at this.

> Unable to use filter function if the data length is one.
> --------------------------------------------------------
>
>                 Key: SPARK-18593
>                 URL: https://issues.apache.org/jira/browse/SPARK-18593
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.2
>            Reporter: Durga Prasad Gunturu
>            Priority: Minor
>
> when using filter using jdbc.. unable to retrieve data if the column value is having
only value of length one. 
> for example:  
> Below code works because the filter condition data is of length > 1
> val dataFromTable = sqlContext.read.jdbc("jdbc:postgresql://localhost:5433/postgres",
"stocknames", defaultProps)
>   dataFromTable.filter(dataFromTable("grouptype")==="ZP").show()
> result: 
> +------------+----------+--------------------+------+---------+---------+------------+--------------------+----------+
> |securitycode|securityid|        securityname|status|grouptype|facevalue|      isinno|
           industry|instrument|
> +------------+----------+--------------------+------+---------+---------+------------+--------------------+----------+
> |      504370|  ZARDIINV|ARDI INVESTMENT &...|Active|       ZP|     10.0|       
    |Misc.Commercial S...|    Equity|
> |      512221|     TARCF|TARRIF CINE &amp;...|Active|       ZP|     10.0|INE177E01010|Finance
(includin...|    Equity|
> |      512505|    MEENST|MEENAKSHI STEEL I...|Active|       ZP|     10.0|           
|Iron &amp; Steel ...|    Equity|
> |      514060|   EVERTEX|Evergreen Textile...|Active|       ZP|     10.0|INE229N01010|
           Textiles|    Equity|
> |      526085|   SGARRES|SAGAR TOURIST RES...|Active|       ZP|     10.0|NA         
|              Hotels|    Equity|
> +------------+----------+--------------------+------+---------+---------+------------+--------------------+----------+
> But if the same filter condition code is of length 1, no data is filtered. 
> eg: 
>  val dataFromTable = sqlContext.read.jdbc("jdbc:postgresql://localhost:5433/postgres",
"stocknames", defaultProps)
>   dataFromTable.filter(dataFromTable("grouptype")==="A").show()
> not getting any output
> +------------+----------+------------+------+---------+---------+------+--------+----------+
> |securitycode|securityid|securityname|status|grouptype|facevalue|isinno|industry|instrument|
> +------------+----------+------------+------+---------+---------+------+--------+----------+
> +------------+----------+------------+------+---------+---------+------+--------+----------+



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message