spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Penny Espinoza <pesp...@societyconsulting.com>
Subject prepending jars to the driver class path for spark-submit on YARN
Date Fri, 05 Sep 2014 23:33:32 GMT
Hey - I’m struggling with some dependency issues with org.apache.httpcomponents httpcore
and httpclient when using spark-submit with YARN running Spark 1.0.2 on a Hadoop 2.2 cluster.
 I’ve seen several posts about this issue, but no resolution.

The error message is this:


Caused by: java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
        at org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85)
        at org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93)
        at com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26)
        at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)
        at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:118)
        at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:102)
        at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:332)
        at com.oncue.rna.realtime.streaming.config.package$.transferManager(package.scala:76)
        at com.oncue.rna.realtime.streaming.models.S3SchemaRegistry.<init>(SchemaRegistry.scala:27)
        at com.oncue.rna.realtime.streaming.models.S3SchemaRegistry$.schemaRegistry$lzycompute(SchemaRegistry.scala:46)
        at com.oncue.rna.realtime.streaming.models.S3SchemaRegistry$.schemaRegistry(SchemaRegistry.scala:44)
        at com.oncue.rna.realtime.streaming.coders.KafkaAvroDecoder.<init>(KafkaAvroDecoder.scala:20)
        ... 17 more

The apache httpcomponents libraries include the method above as of version 4.2.  The Spark
1.0.2 binaries seem to include version 4.1.

I can get this to work in my driver program by adding exclusions to force use of 4.1, but
then I get the error in tasks even when using the —jars option of the spark-submit command.
 How can I get both the driver program and the individual tasks in my spark-streaming job
to use the same version of this library so my job will run all the way through?

thanks
p

Mime
View raw message