Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0F04A200B0F for ; Fri, 17 Jun 2016 12:41:29 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0D924160A61; Fri, 17 Jun 2016 10:41:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D61C8160A50 for ; Fri, 17 Jun 2016 12:41:27 +0200 (CEST) Received: (qmail 10377 invoked by uid 500); 17 Jun 2016 10:41:25 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 10367 invoked by uid 99); 17 Jun 2016 10:41:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2016 10:41:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 65B91C0DF1 for ; Fri, 17 Jun 2016 10:41:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id sD1XZaqA-hqm for ; Fri, 17 Jun 2016 10:41:23 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id B6F515F1E4 for ; Fri, 17 Jun 2016 10:41:22 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id j2so110386853vkg.2 for ; Fri, 17 Jun 2016 03:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=wuqnIbjEi1rBBVuvS2rkmSrKq4CFrwQK1KzBA99VwxM=; b=gxJXU0WZJ5khUcqden9yN5/YdhKQ+GUFIY/VitX9y9mgHa+A3MiieyikG5XkmLzVTI xX7uQpPSWk0ZA2rNKuOozW7GQ4srNsaPOR382Aj3wLONwyoeCCA1ddg/SzwWeuXmnCDS jegsbUAVQhNpkK63vNZX49LWwQSd/t3gN9nVJLYLm5nwIfbXzjHSNBCEuaCQuhL+VD14 eBZ6cXamC6yulbgUm8YSs4PFALyVSWACtLsQVYhWhUvY/sPHkl/WkrBS4IS62ZQfGpep p2PqKlnS1IEZXu4K8L81wigKuBitwFO7l6KeBioSn6h7g1uIBxFQnzT5kZEgQyGDxfoY mpHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=wuqnIbjEi1rBBVuvS2rkmSrKq4CFrwQK1KzBA99VwxM=; b=ODYTC57UMAGRq0BZ7Sd1GPKlBcRCB2hpM066m9KNY09PHGLjWAT/53xsQh3mdQuf5U EZcCSu6oGnjsp9nI7MuHLJfIr+g2coPDWjM7PeryLA8FGW+NJ5btBy8DNoEtKCqSoCI+ TJJbvSjKNgMLOM/s/BV/w6lkgwtVyVqy28wL4W4kpmvGXI6XwBWYe9oCgkFekkbCL2z5 1nArP6Le9stgUk4gRa86rpJcrN4DWHDnKOin/SK2AoLKhUg8e3pWbAuAYs77wIH5WAwz edMbpMtcTMVJGu5H15zg+pOHRMwvPamnKFaEk3LPHMKQu/TWIngANUEMFAYrCm+uKB2Z de0w== X-Gm-Message-State: ALyK8tJLDSt+NtEnGS26EGGqDq3RZLCUgGeRjBgmmPogyKYttRohQgKm2Tw2auAyFadQ0Juuc8dP0H42w3Tb5w== X-Received: by 10.159.32.131 with SMTP id 3mr568400uaa.75.1466160081476; Fri, 17 Jun 2016 03:41:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.1.45 with HTTP; Fri, 17 Jun 2016 03:41:20 -0700 (PDT) In-Reply-To: References: From: Marco Mistroni Date: Fri, 17 Jun 2016 11:41:20 +0100 Message-ID: Subject: Re: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.xml. Please find packages at http://spark-packages.org To: VG Cc: Siva A , User Content-Type: multipart/alternative; boundary=94eb2c0b6726d72134053576fe26 archived-at: Fri, 17 Jun 2016 10:41:29 -0000 --94eb2c0b6726d72134053576fe26 Content-Type: text/plain; charset=UTF-8 and your eclipse path is correct? i suggest, as Siva did before, to build your jar and run it via spark-submit by specifying the --packages option it's as simple as run this command spark-submit --packages com.databricks:spark-xml_: --class Indeed, if you have only these lines to run, why dont you try them in spark-shell ? hth On Fri, Jun 17, 2016 at 11:32 AM, VG wrote: > nopes. eclipse. > > > On Fri, Jun 17, 2016 at 3:58 PM, Siva A wrote: > >> If you are running from IDE, Are you using Intellij? >> >> On Fri, Jun 17, 2016 at 3:20 PM, Siva A wrote: >> >>> Can you try to package as a jar and run using spark-submit >>> >>> Siva >>> >>> On Fri, Jun 17, 2016 at 3:17 PM, VG wrote: >>> >>>> I am trying to run from IDE and everything else is working fine. >>>> I added spark-xml jar and now I ended up into this dependency >>>> >>>> 6/06/17 15:15:57 INFO BlockManagerMaster: Registered BlockManager >>>> Exception in thread "main" *java.lang.NoClassDefFoundError: >>>> scala/collection/GenTraversableOnce$class* >>>> at >>>> org.apache.spark.sql.execution.datasources.CaseInsensitiveMap.(ddl.scala:150) >>>> at >>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:154) >>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) >>>> at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19) >>>> Caused by:* java.lang.ClassNotFoundException: >>>> scala.collection.GenTraversableOnce$class* >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>> ... 5 more >>>> 16/06/17 15:15:58 INFO SparkContext: Invoking stop() from shutdown hook >>>> >>>> >>>> >>>> On Fri, Jun 17, 2016 at 2:59 PM, Marco Mistroni >>>> wrote: >>>> >>>>> So you are using spark-submit or spark-shell? >>>>> >>>>> you will need to launch either by passing --packages option (like in >>>>> the example below for spark-csv). you will need to iknow >>>>> >>>>> --packages com.databricks:spark-xml_: >>>>> >>>>> hth >>>>> >>>>> >>>>> >>>>> On Fri, Jun 17, 2016 at 10:20 AM, VG wrote: >>>>> >>>>>> Apologies for that. >>>>>> I am trying to use spark-xml to load data of a xml file. >>>>>> >>>>>> here is the exception >>>>>> >>>>>> 16/06/17 14:49:04 INFO BlockManagerMaster: Registered BlockManager >>>>>> Exception in thread "main" java.lang.ClassNotFoundException: Failed >>>>>> to find data source: org.apache.spark.xml. Please find packages at >>>>>> http://spark-packages.org >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102) >>>>>> at >>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >>>>>> at >>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) >>>>>> at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19) >>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>> org.apache.spark.xml.DefaultSource >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) >>>>>> at scala.util.Try$.apply(Try.scala:192) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) >>>>>> at scala.util.Try.orElse(Try.scala:84) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:62) >>>>>> ... 4 more >>>>>> >>>>>> Code >>>>>> SQLContext sqlContext = new SQLContext(sc); >>>>>> DataFrame df = sqlContext.read() >>>>>> .format("org.apache.spark.xml") >>>>>> .option("rowTag", "row") >>>>>> .load("A.xml"); >>>>>> >>>>>> Any suggestions please .. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 17, 2016 at 2:42 PM, Marco Mistroni >>>>>> wrote: >>>>>> >>>>>>> too little info >>>>>>> it'll help if you can post the exception and show your sbt file (if >>>>>>> you are using sbt), and provide minimal details on what you are doing >>>>>>> kr >>>>>>> >>>>>>> On Fri, Jun 17, 2016 at 10:08 AM, VG wrote: >>>>>>> >>>>>>>> Failed to find data source: com.databricks.spark.xml >>>>>>>> >>>>>>>> Any suggestions to resolve this >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --94eb2c0b6726d72134053576fe26 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
and=C2=A0 your eclipse path is correct= ?
i suggest, as Siva did before, to build your jar and run it via = spark-submit=C2=A0 by specifying the --packages option
it's as= simple as run this command

spark-submit=C2=A0=C2=A0 --packages com.= databricks:spark-xml_<scalaversion>:<packageversion>=C2=A0=C2= =A0 --class <Name of your class containing main> <path to your jar= >

Indeed, if you have only these lines to run, why dont you= try them in spark-shell ?

hth

On Fri, Jun 17, 2016 at 11:32 AM, VG <= vlinked@gmail.com> wrote:
<= div dir=3D"ltr">nopes. eclipse.=C2=A0


On Fri, Jun 17, 2016 at 3:58 PM, Siva A <siva9940261121@gmail= .com> wrote:
If you are running from IDE, Are you using Intellij?

On Fri, Jun 17, 2016 at 3:20 PM= , Siva A <siva9940261121@gmail.com> wrote:
Can you try to package as a jar an= d run using spark-submit

S= iva

On Fri, Jun 17, 2016 at 3:17 PM, VG <vlinked@gmail.c= om> wrote:
I am trying to run from IDE and everything else is working fine.=C2=A0I added spark-xml jar and now I ended up into this dependency
6/06/17 15:15:57 INFO BlockManagerMaster: Registered Bloc= kManager
Exception in thread "main" java.lang.NoClas= sDefFoundError: scala/collection/GenTraversableOnce$class
at org.apache.spark.sql.execution= .datasources.CaseInsensitiveMap.<init>(ddl.scala:150)
at org.apache.spark.sql.execution.d= atasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:154)
at org.apache.spark.sq= l.DataFrameReader.load(DataFrameReader.scala:119)
at org.apache.spark.sql.DataFrameReader.load(= DataFrameReader.scala:109)
= at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19)
Caused by: java.lang.ClassNotFoundException: scala.collect= ion.GenTraversableOnce$class
at java.net.URLClassLoader.findClass(URLClassLoader.java= :381)
at java.lang.C= lassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launch= er.java:331)
at java= .lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more
16/06/17 15:15:58= INFO SparkContext: Invoking stop() from shutdown hook

=


On Fri, Jun 17, 2016 at 2:59 PM, Marco Mistroni <mmi= stroni@gmail.com> wrote:
So you are using spark-submit=C2=A0 or spark-shell?
=
you will need to launch either by passing --packages option = (like in the example below for spark-csv). you will need to iknow

--packages com.databricks:spark-xml_<scala.version>:<pac= kage version>

hth
=C2=A0


On Fri, Jun 17, 2016 at = 10:20 AM, VG <vlinked@gmail.com> wrote:
Apologies for that.
I am trying to use s= park-xml to load data of a xml file.

here is the e= xception

16/06/17 14:49:04 INFO BlockManagerM= aster: Registered BlockManager
Exception in thread "main&quo= t; java.lang.ClassNotFoundException: Failed to find data source: org.apache= .spark.xml. Please find packages at http://spark-packages.org
at org.apache.spark.sql.execution.datasources.Re= solvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
<= span style=3D"white-space:pre-wrap"> at org.apache.spark.sql.executi= on.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102)
at org.apache.spark.sql.= DataFrameReader.load(DataFrameReader.scala:119)
at org.apache.spark.sql.DataFrameReader.load(Da= taFrameReader.scala:109)
at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19)
=
Caused by: java.lang.ClassNotFoundException: org.apache.spark.xml.Defa= ultSource
at java.ne= t.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassL= oader.java:424)
at s= un.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass= (ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfu= n$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62)
at org.apache.spark.sql.execution.datas= ources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSou= rce.scala:62)
at sca= la.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.ResolvedDataSou= rce$$anonfun$4.apply(ResolvedDataSource.scala:62)
at org.apache.spark.sql.execution.datasources= .ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62)
at scala.util.Try.orElse(Try.= scala:84)
at org.apa= che.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(Re= solvedDataSource.scala:62)
= ... 4 more

Code=C2=A0
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 SQLContext sqlContext =3D new SQLContext(sc);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 DataFrame df =3D sqlContext.read()
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .format("org.apache.s= park.xml")
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .option= ("rowTag", "row")
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 .load("A.xml");

Any= suggestions please ..




On = Fri, Jun 17, 2016 at 2:42 PM, Marco Mistroni <mmistroni@gmail.com&g= t; wrote:
too little info
it'll help if you can post the exception an= d show your sbt file (if you are using sbt), and provide minimal details on= what you are doing
kr

On Fri, Jun 17, 2016 at 10:08 AM, VG <vlinked@gmai= l.com> wrote:
Failed to find data source: com.databricks.spark.xml

=
Any suggestions to resolve this










--94eb2c0b6726d72134053576fe26--