accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mario Pastorelli <mario.pastore...@teralytics.ch>
Subject Testing Spark Job that uses the AccumuloInputFormat
Date Wed, 03 Aug 2016 10:51:26 GMT
I'm trying to test a spark job that uses the AccumuloInputFormat but I'm
having many issues with both MockInstance and MiniAccumuloCluster.

1) MockInstance doesn't work with Spark jobs in my environment because it
looks like every task has a different instance of the MockInstance in
memory; if I add records from the driver, the executors can't find this
data. Is there a way to fix this?

2) MiniAccumuloCluster keeps giving strange errors. Two of them I can't
really fix:
  a. using sbt to run the tests throws  IllegalArgumentException Unknown
classloader type : sbt.classpath.NullLoader when MiniAccumuloClister is
instantiated. Anybody knows how to fix this? It's basically preventing me
from using Spark and Accumulo together.
  b. there is a warn that MiniDFSCluster is not found and a stub is used. I
have all the dependencies needed, included hdfs test. Is this warn ok?

Thanks for the help,
Mario

-- 
Mario Pastorelli | TERALYTICS

*software engineer*

Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41794381682
email: mario.pastorelli@teralytics.ch
www.teralytics.net

Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries

This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.

Mime
View raw message