Return-Path: 
 <user-return-30125-apmail-spark-user-archive=spark.apache.org@spark.apache.org>
X-Original-To: apmail-spark-user-archive@minotaur.apache.org
Delivered-To: apmail-spark-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C742D177DD
	for <apmail-spark-user-archive@minotaur.apache.org>;
 Sun, 29 Mar 2015 07:12:37 +0000 (UTC)
Received: (qmail 25634 invoked by uid 500); 29 Mar 2015 07:12:32 -0000
Delivered-To: apmail-spark-user-archive@spark.apache.org
Received: (qmail 25538 invoked by uid 500); 29 Mar 2015 07:12:32 -0000
Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@spark.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@spark.apache.org>
List-Post: <mailto:user@spark.apache.org>
List-Id: <user.spark.apache.org>
Delivered-To: mailing list user@spark.apache.org
Received: (qmail 25528 invoked by uid 99); 29 Mar 2015 07:12:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Mar 2015 07:12:32 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of neeleshs@gmail.com designates
 209.85.218.50 as permitted sender)
Received: from [209.85.218.50] (HELO mail-oi0-f50.google.com) (209.85.218.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Mar 2015 07:12:06 +0000
Received: by oigz129 with SMTP id z129so60623292oig.1
        for <user@spark.apache.org>; Sun, 29 Mar 2015 00:10:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=iuu4MFDR1/H3PQnm7r2oPqtrYUjvqXVot74HmI+2wnI=;
        b=X3RkCjD/O+Vds7dEthKLxmZGmOn8SdxBb3q2etn1fQhAwJb1SUVr3PlMjKU+zO1r0W
         1t9kbHk3wanOMII3N4fIaXSLHjHRqV/16x/w+l8enE4mxj9QHM8iUDY9GnqsJSK11aeJ
         GoJSCaF8HFZoFIP3QSufP8pyYYJ2f7g+4pqvhnRPh/6XjSgXNPvJEWXH6cooULXov4ah
         wmgiHl0K3TlSf3b9DLqGj98aiqCRvQHHCRsZEQ++zm0PNZ+WXiz9xJgGXvXxTftE96N+
         SSjmQO12+B30ghRp03u3PRt4QKyq4HwgnZnF1KPcrqjocnelET6kPGB3+Wj6ab/lrDFi
         r60g==
MIME-Version: 1.0
X-Received: by 10.182.28.135 with SMTP id b7mr22264071obh.61.1427613034458;
 Sun, 29 Mar 2015 00:10:34 -0700 (PDT)
Received: by 10.202.225.194 with HTTP; Sun, 29 Mar 2015 00:10:34 -0700 (PDT)
Date: Sun, 29 Mar 2015 00:10:34 -0700
Message-ID: 
 <CAJd_F2tsEH1uz3W7_LG6qXsMtit+3ncG57=vNTMASNtGsgpqXw@mail.gmail.com>
Subject: Untangling dependency issues in spark streaming
From: Neelesh <neeleshs@gmail.com>
To: "user@spark.apache.org" <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2f52ccbb47f0512680f0c
X-Virus-Checked: Checked by ClamAV on apache.org

--001a11c2f52ccbb47f0512680f0c
Content-Type: text/plain; charset=UTF-8

Hi,
  My streaming app uses org.apache.httpcomponent:httpclient:4.3.6, but
spark uses 4.2.6 , and I believe thats what's causing the following error.
I've tried setting
spark.executor.userClassPathFirst & spark.driver.userClassPathFirst to true
in the config, but that does not solve it either. Finally I had to resort
to relocating classes using maven shade plugin while building my apps uber
jar, using

<relocations>
    <relocation>
        <pattern>org.apache.http</pattern>
        <shadedPattern>org.shaded.apache.http</shadedPattern>
    </relocation>
</relocations>


Hope this is useful to others in the same situation. It would be
really great to deal with this the right way (like tomcat or any other
servlet container - classloader hierarchy etc).


Caused by: java.lang.NoSuchFieldError: INSTANCE
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:72)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:84)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:59)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)

and ...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)

--001a11c2f52ccbb47f0512680f0c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div>=C2=A0 My streaming app uses org.apache.httpcompon=
ent:httpclient:4.3.6, but spark uses 4.2.6 , and I believe thats what&#39;s=
 causing the following error. I&#39;ve tried setting=C2=A0</div><div>spark.=
executor.userClassPathFirst &amp;=C2=A0spark.driver.userClassPathFirst to t=
rue in the config, but that does not solve it either. Finally I had to reso=
rt to relocating classes using maven shade plugin while building my apps ub=
er jar, using<br></div><div><pre style=3D"color:rgb(0,0,0);font-family:Menl=
o;font-size:12pt"><span style=3D"background-color:rgb(239,239,239)">&lt;</s=
pan><span style=3D"color:rgb(0,0,128);font-weight:bold;background-color:rgb=
(239,239,239)">relocations</span><span style=3D"background-color:rgb(239,23=
9,239)">&gt;</span><br>    <span style=3D"background-color:rgb(239,239,239)=
">&lt;</span><span style=3D"color:rgb(0,0,128);font-weight:bold;background-=
color:rgb(239,239,239)">relocation</span><span style=3D"background-color:rg=
b(239,239,239)">&gt;</span><br>        <span style=3D"background-color:rgb(=
239,239,239)">&lt;</span><span style=3D"color:rgb(0,0,128);font-weight:bold=
;background-color:rgb(239,239,239)">pattern</span><span style=3D"background=
-color:rgb(239,239,239)">&gt;</span>org.apache.http<span style=3D"backgroun=
d-color:rgb(239,239,239)">&lt;/</span><span style=3D"color:rgb(0,0,128);fon=
t-weight:bold;background-color:rgb(239,239,239)">pattern</span><span style=
=3D"background-color:rgb(239,239,239)">&gt;</span><br>        <span style=
=3D"background-color:rgb(239,239,239)">&lt;</span><span style=3D"color:rgb(=
0,0,128);font-weight:bold;background-color:rgb(239,239,239)">shadedPattern<=
/span><span style=3D"background-color:rgb(239,239,239)">&gt;</span>org.shad=
ed.apache.http<span style=3D"background-color:rgb(239,239,239)">&lt;/</span=
><span style=3D"color:rgb(0,0,128);font-weight:bold;background-color:rgb(23=
9,239,239)">shadedPattern</span><span style=3D"background-color:rgb(239,239=
,239)">&gt;</span><br>    <span style=3D"background-color:rgb(239,239,239)"=
>&lt;/</span><span style=3D"color:rgb(0,0,128);font-weight:bold;background-=
color:rgb(239,239,239)">relocation</span><span style=3D"background-color:rg=
b(239,239,239)">&gt;</span><br><span style=3D"background-color:rgb(239,239,=
239)">&lt;/</span><span style=3D"color:rgb(0,0,128);font-weight:bold;backgr=
ound-color:rgb(239,239,239)">relocations</span><span style=3D"background-co=
lor:rgb(239,239,239)">&gt;</span><br></pre><pre style=3D"color:rgb(0,0,0);f=
ont-family:Menlo;font-size:12pt"><span style=3D"background-color:rgb(239,23=
9,239)"><br></span></pre><pre><font face=3D"arial, sans-serif"><span style=
=3D"white-space:normal">Hope this is useful to others in the same situation=
. It would be really great to deal with this the right way (like tomcat or =
any other servlet container - classloader=C2=A0hierarchy=C2=A0etc).</span><=
/font><font color=3D"#000000" face=3D"Menlo"><span style=3D"font-size:12pt"=
><br></span></font></pre></div><div><br></div><div><div>Caused by: java.lan=
g.NoSuchFieldError: INSTANCE</div><div><span class=3D"" style=3D"white-spac=
e:pre">	</span>at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.&=
lt;init&gt;(DefaultHttpRequestWriterFactory.java:52)</div><div><span class=
=3D"" style=3D"white-space:pre">	</span>at org.apache.http.impl.io.DefaultH=
ttpRequestWriterFactory.&lt;init&gt;(DefaultHttpRequestWriterFactory.java:5=
6)</div><div><span class=3D"" style=3D"white-space:pre">	</span>at org.apac=
he.http.impl.io.DefaultHttpRequestWriterFactory.&lt;clinit&gt;(DefaultHttpR=
equestWriterFactory.java:46)</div><div><span class=3D"" style=3D"white-spac=
e:pre">	</span>at org.apache.http.impl.conn.ManagedHttpClientConnectionFact=
ory.&lt;init&gt;(ManagedHttpClientConnectionFactory.java:72)</div><div><spa=
n class=3D"" style=3D"white-space:pre">	</span>at org.apache.http.impl.conn=
.ManagedHttpClientConnectionFactory.&lt;init&gt;(ManagedHttpClientConnectio=
nFactory.java:84)</div><div><span class=3D"" style=3D"white-space:pre">	</s=
pan>at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.&lt;cli=
nit&gt;(ManagedHttpClientConnectionFactory.java:59)</div><div><span class=
=3D"" style=3D"white-space:pre">	</span>at org.apache.http.impl.conn.Poolin=
gHttpClientConnectionManager$InternalConnectionFactory.&lt;init&gt;(Pooling=
HttpClientConnectionManager.java:494)</div><div><span class=3D"" style=3D"w=
hite-space:pre">	</span>at org.apache.http.impl.conn.PoolingHttpClientConne=
ctionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:149)</div=
><div><span class=3D"" style=3D"white-space:pre">	</span>at org.apache.http=
.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClien=
tConnectionManager.java:138)</div><div><span class=3D"" style=3D"white-spac=
e:pre">	</span>at org.apache.http.impl.conn.PoolingHttpClientConnectionMana=
ger.&lt;init&gt;(PoolingHttpClientConnectionManager.java:114)</div></div><d=
iv><br></div><div>and ...=C2=A0</div><div><div>Caused by: java.lang.NoClass=
DefFoundError: Could not initialize class org.apache.http.impl.conn.Managed=
HttpClientConnectionFactory</div><div><span class=3D"" style=3D"white-space=
:pre">	</span>at org.apache.http.impl.conn.PoolingHttpClientConnectionManag=
er$InternalConnectionFactory.&lt;init&gt;(PoolingHttpClientConnectionManage=
r.java:494)</div><div><span class=3D"" style=3D"white-space:pre">	</span>at=
 org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(=
PoolingHttpClientConnectionManager.java:149)</div><div><span class=3D"" sty=
le=3D"white-space:pre">	</span>at org.apache.http.impl.conn.PoolingHttpClie=
ntConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:13=
8)</div><div><span class=3D"" style=3D"white-space:pre">	</span>at org.apac=
he.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHt=
tpClientConnectionManager.java:114)</div></div><div><br></div></div>

--001a11c2f52ccbb47f0512680f0c--