accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Winick <jaredwin...@gmail.com>
Subject Re: Accumulo and Spark
Date Wed, 15 Jan 2014 17:35:26 GMT
I tried myself a few weeks ago and saw that it "just works" too for the
very simple test I ran. I did see some error messages when running from sbt
after the job successfully completed and the SparkContext was closing. I
assume this has to do with resources within the AccumuloInputFormat? This
was with Accumulo 1.5.0. Haven't had time to look into it but I was going
contribute an Accumulo example to
https://github.com/apache/incubator-spark/tree/master/examples/src/main/scala/org/apache/spark/examplesif
I could get these messages cleared up.

...
14/01/15 10:29:13 INFO spark.SparkContext: Successfully stopped SparkContext
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
at java.lang.Thread.run(Thread.java:680)
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
at java.lang.Thread.run(Thread.java:680)
14/01/15 10:29:13 ERROR zookeeper.ClientCnxn: Event thread exiting due to
interruption
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
14/01/15 10:29:13 INFO zookeeper.ClientCnxn: EventThread shut down
[success] Total time: 3 s, completed Jan 15, 2014 10:29:13 AM
> 14/01/15 10:29:23 WARN zookeeper.ClientCnxn: Session 0x14396efc3be0022
for server vm/192.168.221.2:2181, unexpected error, closing socket
connection and attempting reconnect
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:343)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)



On Mon, Jan 13, 2014 at 10:37 AM, Matthew Molek <mmolek@clearedgeit.com>wrote:

> I just tried using AccumuloInputFormat as a data source for Spark running
> in standalone mode on a single node 'cluster'. Everything seems to work
> fine out of the box, as advertised. (Spark is supposed to work with any
> hadoop InputFormat)
>
> Just properly configure the AccumuloInputFormat, and pass it off to
> JavaSparkContext.newAPIHadoopRDD(...) to load the data into an RDD.
>
> The versions I tested with were Accumulo 1.5, Hadoop 1.2.1, and Spark
> 0.8.1.
>
> Is anyone else using Spark with Accumulo?
>
>

Mime
View raw message