spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From satish chandra j <jsatishchan...@gmail.com>
Subject Re: Re: RE: Error not found value sqlContext
Date Mon, 23 Nov 2015 10:19:18 GMT
Sorry, just to understand my issue.....if Eclipse could not understand
Scala syntax properly than it should error for the other Spark Job which is
fetching data from a RDBM and printing the output in console. I think there
is some dependencies which are missing due to which "import
sqlContext.implicits._" is not recognized during compile time

Please let me know if any further inputs needed to fix the same

Regards,
Satish Chandra

On Mon, Nov 23, 2015 at 3:29 PM, prosp4300 <prosp4300@163.com> wrote:

>
>
> So it is actually a compile time error in Eclipse, instead of jar
> generation from Eclipse, you can try to use sbt to assembly your jar, looks
> like your Eclipse does not recognize the Scala syntax properly.
>
>
>
> At 2015-11-20 21:36:55, "satish chandra j" <jsatishchandra@gmail.com>
> wrote:
>
> HI All,
> I am getting this error while generating executable Jar file itself in
> Eclipse, if the Spark Application code has "import sqlContext.implicits._"
> line in it. Spark Applicaiton code  works fine if the above mentioned line
> does not exist as I have tested by fetching data from an RDBMS by
> implementing JDBCRDD
>
> I tried couple of DataFrame related methods for which most of them errors
> stating that method has been overloaded
>
> Please let me know if any further inputs needed to analyze it
>
> Regards,
> Satish Chandra
>
> On Fri, Nov 20, 2015 at 5:46 PM, prosp4300 <prosp4300@163.com> wrote:
>
>>
>> Looks like a classpath problem, if you can provide the command you used
>> to run your application and environment variable SPARK_HOME, it will help
>> others to identify the root problem
>>
>>
>> 在2015年11月20日 18:59,Satish <jsatishchandra@gmail.com> 写道:
>>
>> Hi Michael,
>> As my current Spark version is 1.4.0 than why it error out as "error: not
>> found: value sqlContext" when I have "import sqlContext.implicits._" in my
>> Spark Job
>>
>> Regards
>> Satish Chandra
>> ------------------------------
>> From: Michael Armbrust <michael@databricks.com>
>> Sent: ‎20-‎11-‎2015 01:36
>> To: satish chandra j <jsatishchandra@gmail.com>
>> Cc: user <user@spark.apache.org>; hari krishna <harikrishna18@gmail.com>
>> Subject: Re: Error not found value sqlContext
>>
>>
>> http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13
>>
>> On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j <
>> jsatishchandra@gmail.com> wrote:
>>
>>> HI All,
>>> we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching
>>> data from an RDBMS using JDBCRDD and register it as temp table to perform
>>> SQL query
>>>
>>> Below approach is working fine in Spark 1.2.1:
>>>
>>> JDBCRDD --> apply map using Case Class --> apply createSchemaRDD -->
>>> registerTempTable --> perform SQL Query
>>>
>>> but now as createSchemaRDD is not supported in Spark 1.4.0
>>>
>>> JDBCRDD --> apply map using Case Class with* .toDF()* -->
>>> registerTempTable --> perform SQL query on temptable
>>>
>>>
>>> JDBCRDD --> apply map using Case Class --> RDD*.toDF()*.registerTempTable
>>> --> perform SQL query on temptable
>>>
>>> Only solution I get everywhere is to  use "import
>>> sqlContext.implicits._" after val SQLContext = new
>>> org.apache.spark.sql.SQLContext(sc)
>>>
>>> But it errors with the two generic errors
>>>
>>> *1. error: not found: value sqlContext*
>>>
>>> *2. value toDF is not a member of org.apache.spark.rdd.RDD*
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
>

Mime
View raw message