spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-9282) Filter on Spark DataFrame with multiple columns
Date Tue, 28 Jul 2015 04:59:04 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Armbrust resolved SPARK-9282.
-------------------------------------
    Resolution: Duplicate

You need to use {{&}} instead of {{and}}.  In Spark 1.4 we print an error if you try to
run this query.

> Filter on Spark DataFrame with multiple columns
> -----------------------------------------------
>
>                 Key: SPARK-9282
>                 URL: https://issues.apache.org/jira/browse/SPARK-9282
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, Spark Shell, SQL
>    Affects Versions: 1.3.0
>         Environment: CDH 5.0 on CentOS6
>            Reporter: Sandeep Pal
>
> Filter on dataframe does not work if we have more than one column inside the filter.
Nonetheless, it works on an RDD.
> Following is the example:
> df1.show()
> age coolid depid empname
> 23  7      1     sandeep
> 21  8      2     john   
> 24  9      1     cena   
> 45  12     3     bob    
> 20  7      4     tanay  
> 12  8      5     gaurav 
> df1.filter(df1.age > 21 and df1.age < 45).show(10)
> 23  7      1     sandeep
> 21  8      2     john                                   <-------------
> 24  9      1     cena   
> 20  7      4     tanay                                 <-------------
> 12  8      5     gaurav                               <--------------



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message