spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neelesh <neele...@gmail.com>
Subject Untangling dependency issues in spark streaming
Date Sun, 29 Mar 2015 07:10:34 GMT
Hi,
  My streaming app uses org.apache.httpcomponent:httpclient:4.3.6, but
spark uses 4.2.6 , and I believe thats what's causing the following error.
I've tried setting
spark.executor.userClassPathFirst & spark.driver.userClassPathFirst to true
in the config, but that does not solve it either. Finally I had to resort
to relocating classes using maven shade plugin while building my apps uber
jar, using

<relocations>
    <relocation>
        <pattern>org.apache.http</pattern>
        <shadedPattern>org.shaded.apache.http</shadedPattern>
    </relocation>
</relocations>


Hope this is useful to others in the same situation. It would be
really great to deal with this the right way (like tomcat or any other
servlet container - classloader hierarchy etc).


Caused by: java.lang.NoSuchFieldError: INSTANCE
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:72)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:84)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:59)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)

and ...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)

Mime
View raw message