spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ping yan <sharon...@gmail.com>
Subject Re: weird issue with sqlContext.createDataFrame - pyspark 1.3.1
Date Fri, 09 Oct 2015 18:29:59 GMT
Thanks. It does seem like that my pandas installation is corrupted.

Thanks!


On Fri, Oct 9, 2015 at 11:04 AM, Davies Liu <davies@databricks.com> wrote:

> Is it possible that you have an very old version of pandas, that does
> not have DataFrame (or in different submodule).
>
> Could you try this:
> ```
> >>> import pandas
> >>> pandas.__version__
> '0.14.0'
> ```
>
> On Thu, Oct 8, 2015 at 10:28 PM, ping yan <sharonyan@gmail.com> wrote:
> > I really cannot figure out what this is about..
> > (tried to import pandas, in case that is a dependency, but it didn't
> help.)
> >
> >>>> from pyspark.sql import SQLContext
> >>>> sqlContext=SQLContext(sc)
> >>>> sqlContext.createDataFrame(l).collect()
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File
> >
> "/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/python/pyspark/sql/context.py",
> > line 318, in createDataFrame
> >     if has_pandas and isinstance(data, pandas.DataFrame):
> > AttributeError: 'module' object has no attribute 'DataFrame'
> >
> > Would appreciate any pointers.
> >
> > Thanks!
> > Ping
> >
> >
> >
>



-- 
Ping Yan

Mime
View raw message