Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of busbey@cloudera.com
 designates 209.85.216.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACA1tWLW0he+sxxgXYGi088CrBRYg_XFWkYu+Rb6G4Uvt+7_Sw@mail.gmail.com>
References: 
 <CACA1tWLW0he+sxxgXYGi088CrBRYg_XFWkYu+Rb6G4Uvt+7_Sw@mail.gmail.com>
From: Sean Busbey <busbey@cloudera.com>
Date: Mon, 16 Jun 2014 09:16:43 -0400
Message-ID: 
 <CAGHyZ6LbvdQoXvZsKHrG4pTjSGGB=+UbJjU92d3fdyzj5uN85g@mail.gmail.com>
Subject: Re: Need help. Error: java.lang.NoSuchMethodError:
 org.apache.commons.codec.binary.Base64.encodeBase64String
To: Accumulo User List <user@accumulo.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2cce8d3cf5204fbf3d7aa

--001a11c2cce8d3cf5204fbf3d7aa
Content-Type: text/plain; charset=UTF-8

Hi Jianshi!

What version of Accumulo are you using?

I believe you're running into a problem with what Accumulo presumes will be
provided by Hadoop. The immediate fault is that Accumulo wants
commons-codec. It's normally in the classpath as a part of Hadoop client.

To make sure you have the entire classpath, I believe you're going to have
to use the "Accumulo classpath" command:

${ACCUMULO_HOME}/bin/accumulo classpath

Unfortunately, the output of this command is not tailored towards passing
to another process, so you'll have to do some follow-on munging. In many of
the Accumulo set ups I've seen, there are far more jars included in this
classpath than Accumulo actually needs, so it might be best to use it as a
guide for what you include manually.


On Mon, Jun 16, 2014 at 5:19 AM, Jianshi Huang <jianshi.huang@gmail.com>
wrote:

> Hi,
>
> I'm trying to use Accumulo with Spark writing to AccumuloOutputFormat. I
> got the following errors in my spark app log:
>
> 14/06/16 02:01:44 INFO cluster.YarnClientClusterScheduler:
> YarnClientClusterScheduler.postStartHook done
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.commons.codec.binary.Base64.encodeBase64String([B)Ljava/lang/String;
>         at
> org.apache.accumulo.core.client.mapreduce.lib.impl.ConfiguratorBase.setConnectorInfo(ConfiguratorBase.java:127)
>         at
> org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat.setConnectorInfo(AccumuloOutputFormat.java:92)
>         at
> com.paypal.rtgraph.demo.MapReduceWriter$.main(MapReduceWriter.scala:44)
>         at
> com.paypal.rtgraph.demo.MapReduceWriter.main(MapReduceWriter.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Anyone knows what's wrong with my code or settings? I've added all jars to
> spark's classpath.
>
> And here's my spark-submit cmd:
>
> spark-submit --class com.paypal.rtgraph.demo.MapReduceWriter --master
> yarn-client --jars `find lib -type f | tr '\n' ','` --driver-memory 2G
> --executor-memory 20G --executor-cores 8 --num-executors 2 --verbose
> rtgraph.jar
>
> If anyone has Accumulo+Spark examples, it would be great if they could be
> shared. :)
>
>
> Cheers,
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>


-- 
Sean

--001a11c2cce8d3cf5204fbf3d7aa
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Jianshi!<div><br></div><div>What version of Accumulo ar=
e you using?</div><div><br></div><div>I believe you&#39;re running into a p=
roblem with what Accumulo presumes will be provided by Hadoop. The immediat=
e fault is that Accumulo wants commons-codec. It&#39;s normally in the clas=
spath as a part of Hadoop client.</div>

<div><br></div><div>To make sure you have the entire classpath, I believe y=
ou&#39;re going to have to use the &quot;Accumulo classpath&quot; command:<=
/div><div><br></div><div>${ACCUMULO_HOME}/bin/accumulo classpath</div>
<div>
<br></div><div>Unfortunately, the output of this command is not tailored to=
wards passing to another process, so you&#39;ll have to do some follow-on m=
unging. In many of the Accumulo set ups I&#39;ve seen, there are far more j=
ars included in this classpath than Accumulo actually needs, so it might be=
 best to use it as a guide for what you include manually.</div>

</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Mon,=
 Jun 16, 2014 at 5:19 AM, Jianshi Huang <span dir=3D"ltr">&lt;<a href=3D"ma=
ilto:jianshi.huang@gmail.com" target=3D"_blank">jianshi.huang@gmail.com</a>=
&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><span style=3D"font-family:=
arial,sans-serif;font-size:13px">Hi,</span><div style=3D"font-family:arial,=
sans-serif;font-size:13px">

<br></div><div style=3D"font-family:arial,sans-serif;font-size:13px">I&#39;=
m trying to use Accumulo with Spark writing to AccumuloOutputFormat. I got =
the following errors in my spark app log:</div>

<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px"><div>14/06/16 02:01:44=
 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postSt=
artHook done</div>


<div>Exception in thread &quot;main&quot; java.lang.NoSuchMethodError: org.=
apache.commons.codec.binary.Base64.encodeBase64String([B)Ljava/lang/String;=
</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.accumulo.core.client.m=
apreduce.lib.impl.ConfiguratorBase.setConnectorInfo(ConfiguratorBase.java:1=
27)</div>


<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.accumulo.core.client.mapredu=
ce.AccumuloOutputFormat.setConnectorInfo(AccumuloOutputFormat.java:92)</div=
><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.paypal.rtgraph.demo.MapReduceWrite=
r$.main(MapReduceWriter.scala:44)</div>


<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at com.paypal.rtgraph.demo.MapReduceWriter=
.main(MapReduceWriter.scala)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at sun.r=
eflect.NativeMethodAccessorImpl.invoke0(Native Method)</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMeth=
odAccessorImpl.java:57)</div>


<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at sun.reflect.DelegatingMethodAccessorImp=
l.invoke(DelegatingMethodAccessorImpl.java:43)</div><div>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 at java.lang.reflect.Method.invoke(Method.java:606)</div><div>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.spark.deploy.SparkSubmit$.launch(=
SparkSubmit.scala:292)</div>


<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.spark.deploy.SparkSubmit$.ma=
in(SparkSubmit.scala:55)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apach=
e.spark.deploy.SparkSubmit.main(SparkSubmit.scala)</div><div><br></div><div=
>Anyone knows what&#39;s wrong with my code or settings? I&#39;ve added all=
 jars to spark&#39;s classpath.</div>


<div><br></div><div>And here&#39;s my spark-submit cmd:</div><div><br></div=
><div>spark-submit --class com.paypal.rtgraph.demo.MapReduceWriter --master=
 yarn-client --jars `find lib -type f | tr &#39;\n&#39; &#39;,&#39;` --driv=
er-memory 2G --executor-memory 20G --executor-cores 8 --num-executors 2 --v=
erbose rtgraph.jar<br>


</div><div><br></div><div>If anyone has Accumulo+Spark examples, it would b=
e great if they could be shared. :)</div><div><br></div><div><br></div><div=
>Cheers,</div></div><span class=3D"HOEnZb"><font color=3D"#888888"><div><br=
>

</div>-- <br>Jianshi Huang<br><br>LinkedIn: jianshi<br>

Twitter: @jshuang<br>Github &amp; Blog: <a href=3D"http://huangjs.github.co=
m/" target=3D"_blank">http://huangjs.github.com/</a><br>
</font></span></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div dir=3D"=
ltr">Sean</div>
</div>

--001a11c2cce8d3cf5204fbf3d7aa--