accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mario Pastorelli <mario.pastore...@teralytics.ch>
Subject Re: Testing Spark Job that uses the AccumuloInputFormat
Date Wed, 03 Aug 2016 17:34:44 GMT
Thanks for the help! I've been able to start the MiniAccumuloCluster by
forking the JVM in SBT. Now it works even if it's very slow. I use
ScalaCheck to run multiple times the same test with different input and for
each instance of the test I delete the table and recreate it in Accumulo.
This takes some seconds, which is surprising but I guess it makes sense.
That's why I usually prefer MockInstance. Do you guys know if it is
possible to use MockInstance to test Spark jobs? It's so much faster...

On Wed, Aug 3, 2016 at 6:09 PM, Keith Turner <keith@deenlo.com> wrote:

> I opened two issues :
>
> https://issues.apache.org/jira/browse/ACCUMULO-4395
> https://issues.apache.org/jira/browse/ACCUMULO-4396
>
> On Wed, Aug 3, 2016 at 6:51 AM, Mario Pastorelli
> <mario.pastorelli@teralytics.ch> wrote:
> > I'm trying to test a spark job that uses the AccumuloInputFormat but I'm
> > having many issues with both MockInstance and MiniAccumuloCluster.
> >
> > 1) MockInstance doesn't work with Spark jobs in my environment because it
> > looks like every task has a different instance of the MockInstance in
> > memory; if I add records from the driver, the executors can't find this
> > data. Is there a way to fix this?
> >
> > 2) MiniAccumuloCluster keeps giving strange errors. Two of them I can't
> > really fix:
> >   a. using sbt to run the tests throws  IllegalArgumentException Unknown
> > classloader type : sbt.classpath.NullLoader when MiniAccumuloClister is
> > instantiated. Anybody knows how to fix this? It's basically preventing me
> > from using Spark and Accumulo together.
> >   b. there is a warn that MiniDFSCluster is not found and a stub is
> used. I
> > have all the dependencies needed, included hdfs test. Is this warn ok?
> >
> > Thanks for the help,
> > Mario
> >
> > --
> > Mario Pastorelli | TERALYTICS
> >
> > software engineer
> >
> > Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> > phone: +41794381682
> > email: mario.pastorelli@teralytics.ch
> > www.teralytics.net
> >
> > Company registration number: CH-020.3.037.709-7 | Trade register Canton
> > Zurich
> > Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
> Yann
> > de Vries
> >
> > This e-mail message contains confidential information which is for the
> sole
> > attention and use of the intended recipient. Please notify us at once if
> you
> > think that it may not be intended for you and delete it immediately.
>



-- 
Mario Pastorelli | TERALYTICS

*software engineer*

Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41794381682
email: mario.pastorelli@teralytics.ch
www.teralytics.net

Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries

This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.

Mime
View raw message