accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Testing Spark Job that uses the AccumuloInputFormat
Date Wed, 03 Aug 2016 16:09:07 GMT
I opened two issues :

https://issues.apache.org/jira/browse/ACCUMULO-4395
https://issues.apache.org/jira/browse/ACCUMULO-4396

On Wed, Aug 3, 2016 at 6:51 AM, Mario Pastorelli
<mario.pastorelli@teralytics.ch> wrote:
> I'm trying to test a spark job that uses the AccumuloInputFormat but I'm
> having many issues with both MockInstance and MiniAccumuloCluster.
>
> 1) MockInstance doesn't work with Spark jobs in my environment because it
> looks like every task has a different instance of the MockInstance in
> memory; if I add records from the driver, the executors can't find this
> data. Is there a way to fix this?
>
> 2) MiniAccumuloCluster keeps giving strange errors. Two of them I can't
> really fix:
>   a. using sbt to run the tests throws  IllegalArgumentException Unknown
> classloader type : sbt.classpath.NullLoader when MiniAccumuloClister is
> instantiated. Anybody knows how to fix this? It's basically preventing me
> from using Spark and Accumulo together.
>   b. there is a warn that MiniDFSCluster is not found and a stub is used. I
> have all the dependencies needed, included hdfs test. Is this warn ok?
>
> Thanks for the help,
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastorelli@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if you
> think that it may not be intended for you and delete it immediately.

Mime
View raw message