spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor Tso-Guillen <v...@paxata.com>
Subject Re: prepending jars to the driver class path for spark-submit on YARN
Date Sun, 07 Sep 2014 00:13:56 GMT
I ran into the same issue. What I did was use maven shade plugin to shade
my version of httpcomponents libraries into another package.


On Fri, Sep 5, 2014 at 4:33 PM, Penny Espinoza <
pespino@societyconsulting.com> wrote:

>  Hey - I’m struggling with some dependency issues with
> org.apache.httpcomponents httpcore and httpclient when using spark-submit
> with YARN running Spark 1.0.2 on a Hadoop 2.2 cluster.  I’ve seen several
> posts about this issue, but no resolution.
>
>  The error message is this:
>
>
>  Caused by: java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85)
>         at
> org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
>         at
> com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
>         at
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
>         at
> com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155)
>         at
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:118)
>         at
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:102)
>         at
> com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:332)
>         at
> com.oncue.rna.realtime.streaming.config.package$.transferManager(package.scala:76)
>         at
> com.oncue.rna.realtime.streaming.models.S3SchemaRegistry.<init>(SchemaRegistry.scala:27)
>         at
> com.oncue.rna.realtime.streaming.models.S3SchemaRegistry$.schemaRegistry$lzycompute(SchemaRegistry.scala:46)
>         at
> com.oncue.rna.realtime.streaming.models.S3SchemaRegistry$.schemaRegistry(SchemaRegistry.scala:44)
>         at
> com.oncue.rna.realtime.streaming.coders.KafkaAvroDecoder.<init>(KafkaAvroDecoder.scala:20)
>         ... 17 more
>
>   The apache httpcomponents libraries include the method above as of
> version 4.2.  The Spark 1.0.2 binaries seem to include version 4.1.
>
>  I can get this to work in my driver program by adding exclusions to
> force use of 4.1, but then I get the error in tasks even when using the
> —jars option of the spark-submit command.  How can I get both the driver
> program and the individual tasks in my spark-streaming job to use the same
> version of this library so my job will run all the way through?
>
>  thanks
> p
>

Mime
View raw message