Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of mdrob@cloudera.com designates
 209.85.214.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKthdGgFVLy=k0cb04RaovnZ4gvj=H5re1L7i_ZBEv2L3N7YAg@mail.gmail.com>
References: 
 <CAKthdGgFVLy=k0cb04RaovnZ4gvj=H5re1L7i_ZBEv2L3N7YAg@mail.gmail.com>
From: Mike Drob <madrob@cloudera.com>
Date: Wed, 29 Oct 2014 17:27:19 -0500
Message-ID: 
 <CAFAMeYK=3QtzV9qs83V14X2m-HPKPYymH-u8FoKgo7eeb7KWuQ@mail.gmail.com>
Subject: Re: Testing Map Reduce Jobs
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=089e013cc45e7e882705069745ac

--089e013cc45e7e882705069745ac
Content-Type: text/plain; charset=UTF-8

If you launch a MapReduce job in your test code, without having a cluster
present, then it will default into a local runner. This could be easy if
you invoke your job through something like ToolRunner.run(). Or just build
a job and invoke it directly. So you don't really need a mini-mr or yarn
cluster for this, 90% of the time.

As far as integrating that with a mini-accumulo... if you start a
MiniAccumuloCluster manually and keep a reference to it (which you should
anyway because you will need to stop it eventually) then you can use that
to populate your AccumuloInputFormat configuration (assuming you are using
it).

Something like...

  @Before
  public void setUp() throws Exception {
    // Start the Accumulo Cluster
    mac = new MiniAccumuloCluster(root.newFolder(), ACCUMULO_PASS);
    mac.start();

    // Get first connection to create user
    mac.getConnector(ACCUMULO_USER, ACCUMULO_PASS);
  }

  @Test
  public void setUp() throws Exception {
    AccumuloInputFormat.setZooKeeperInstance(job,
ClientConfiguration.loadDefault().withZkHosts(mac.getZooKeepers()).withInstance(mac.getInstanceName()));
    // .. and other settings

    boolean success = job.waitForCompletion(false);
    assertTrue("Job failed!", success);
  }

  @After
  public void tearDown() throws Exception {
    mac.stop();
  }

Not sure if this is helpful, but hopefully is enough to point you in the
right direction. If you have more questions, please clarify.


On Wed, Oct 29, 2014 at 4:16 PM, Ariel Valentin <ariel@arielvalentin.com>
wrote:

> I am looking for some guidance that will help me write better tests for
> our map reduce jobs. My current jobs are tested using MRUnit, which covers
> most of the "logic" but I feel like I am missing good "end-to-end"
> developer tests.
>
> I took a look at the tests for mapred classes but I am not sure that it
> achieves my goal of an end-to-end test because of the use of MockInstance.
>
> https://github.com/apache/accumulo/blob/master/mapreduce/src/test/java/org/apache/accumulo/core/client/mapred{,reduce}/
>
> For me the characteristic of a end-to-end test that I would find valuable
> is a suite that one could execute using mini-{accumulo,yarn,et.al.} but I
> don't see any examples of how one would go about making those components
> work in concert with each other.
>
> Does anyone have any guidance when it comes to writing automated developer
> end-to-end tests?
>
> What kinds of testing strategies are people out there using for MR jobs?
>
> Thanks,
> Ariel Valentin
> e-mail: ariel@arielvalentin.com
> website: http://blog.arielvalentin.com
> skype: ariel.s.valentin
> twitter: arielvalentin
> linkedin: http://www.linkedin.com/profile/view?id=8996534
> ---------------------------------------
> *simplicity *communication
> *feedback *courage *respect
>

--089e013cc45e7e882705069745ac
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>If you launch a MapReduce job in your test code,=
 without having a cluster present, then it will default into a local runner=
. This could be easy if you invoke your job through something like ToolRunn=
er.run(). Or just build a job and invoke it directly. So you don&#39;t real=
ly need a mini-mr or yarn cluster for this, 90% of the time.<br><br></div>A=
s far as integrating that with a mini-accumulo... if you start a MiniAccumu=
loCluster manually and keep a reference to it (which you should anyway beca=
use you will need to stop it eventually) then you can use that to populate =
your AccumuloInputFormat configuration (assuming you are using it).<br><br>=
</div>Something like...<br><div><br>=C2=A0 @Before<br>=C2=A0 public void se=
tUp() throws Exception {<br>=C2=A0=C2=A0=C2=A0 // Start the Accumulo Cluste=
r<br>=C2=A0=C2=A0=C2=A0 mac =3D new MiniAccumuloCluster(root.newFolder(), A=
CCUMULO_PASS);<br>=C2=A0=C2=A0=C2=A0 mac.start();<br><br>=C2=A0=C2=A0=C2=A0=
 // Get first connection to create user<br>=C2=A0=C2=A0=C2=A0 mac.getConnec=
tor(ACCUMULO_USER, ACCUMULO_PASS);<br>=C2=A0 }<br><br></div><div>=C2=A0 @Te=
st<br></div><div>=C2=A0 public void setUp() throws Exception {<br>=C2=A0=C2=
=A0=C2=A0 AccumuloInputFormat.setZooKeeperInstance(job, ClientConfiguration=
.loadDefault().withZkHosts(mac.getZooKeepers()).withInstance(mac.getInstanc=
eName()));<br></div><div>=C2=A0=C2=A0=C2=A0 // .. and other settings<br><br=
></div><div>=C2=A0=C2=A0=C2=A0 boolean success =3D job.waitForCompletion(fa=
lse);<br></div><div>=C2=A0=C2=A0=C2=A0 assertTrue(&quot;Job failed!&quot;, =
success);<br></div><div>=C2=A0 }<br><br>=C2=A0 @After<br>=C2=A0 public void=
 tearDown() throws Exception {<br>=C2=A0=C2=A0=C2=A0 mac.stop();<br>=C2=A0 =
}<br><br></div><div>Not sure if this is helpful, but hopefully is enough to=
 point you in the right direction. If you have more questions, please clari=
fy.<br></div><div><br><br></div></div><div class=3D"gmail_extra"><br><div c=
lass=3D"gmail_quote">On Wed, Oct 29, 2014 at 4:16 PM, Ariel Valentin <span =
dir=3D"ltr">&lt;<a href=3D"mailto:ariel@arielvalentin.com" target=3D"_blank=
">ariel@arielvalentin.com</a>&gt;</span> wrote:<br><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex"><div dir=3D"ltr"><div>I am looking for some guidance that will help =
me write better tests for our map reduce jobs. My current jobs are tested u=
sing MRUnit, which covers most of the &quot;logic&quot; but I feel like I a=
m missing good &quot;end-to-end&quot; developer tests.=C2=A0</div><div><br>=
</div><div>I took a look at the tests for mapred classes but I am not sure =
that it achieves my goal of an end-to-end test because of the use of MockIn=
stance.</div><div><a href=3D"https://github.com/apache/accumulo/blob/master=
/mapreduce/src/test/java/org/apache/accumulo/core/client/mapred%7B,reduce%7=
D/" target=3D"_blank">https://github.com/apache/accumulo/blob/master/mapred=
uce/src/test/java/org/apache/accumulo/core/client/mapred{,reduce}/</a><br><=
/div><div><br></div><div><div>For me the characteristic of a end-to-end tes=
t that I would find valuable is a suite that one could execute using mini-{=
accumulo,yarn,<a href=3D"http://et.al" target=3D"_blank">et.al</a>.} but I =
don&#39;t see any examples of how one would go about making those component=
s work in concert with each other.=C2=A0</div></div><div><br></div><div>Doe=
s anyone have any guidance when it comes to writing automated developer end=
-to-end tests?=C2=A0</div><div><br></div><div>What kinds of testing strateg=
ies are people out there using for MR jobs?</div><div><br></div>Thanks,<br =
clear=3D"all"><div>Ariel Valentin<br>e-mail: <a href=3D"mailto:ariel@arielv=
alentin.com" target=3D"_blank">ariel@arielvalentin.com</a><br>website: <a h=
ref=3D"http://blog.arielvalentin.com" target=3D"_blank">http://blog.arielva=
lentin.com</a><br>skype: ariel.s.valentin<br><div>twitter: arielvalentin<di=
v>linkedin:=C2=A0<a href=3D"http://www.linkedin.com/profile/view?id=3D89965=
34" target=3D"_blank">http://www.linkedin.com/profile/view?id=3D8996534</a>=
<br>---------------------------------------<br>*simplicity *communication<b=
r>*feedback *courage *respect</div></div></div>
</div>
</blockquote></div><br></div>

--089e013cc45e7e882705069745ac--