ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shane Kinsella <shane.kinse...@aspect.com>
Subject Execution hangs at IgniteRDD.savePairs()
Date Mon, 18 Jan 2016 13:04:58 GMT
Hi All, I've created a StackOverflow post
(http://stackoverflow.com/questions/34815652/igniterdd-freezes-at-savepairs)
but thought I might share it here also.



I have a Spark cluster of three machines and am trying to use Apache Ignite
for caching data. On each Spark machine I have an Ignite node running and am
using the Spark REPL for testing (problem originally found using
Spark-submit so it not the REPL).

The problem is that my execution freezes at IgniteRDD.savePairs. Here is my
CacheConfig:

    <bean class="org.apache.ignite.configuration.CacheConfiguration">
        <property name="name" value="myRddCache"/>
        <property name="cacheMode" value="PARTITIONED"/>
    </bean>

I previously had this working but ran out of memory in Ignite so I
(temporarily) added some options for tiered storage:

    <!- Store cache entries on-heap. -!->
    <property name="memoryMode" value="ONHEAP_TIERED"/> 

    <!- Enable Off-Heap memory with max size of 10 Gigabytes (0 for
unlimited). -!->
    <property name="offHeapMaxMemory" value="#{10 * 1024L * 1024L *
1024L}"/>

    <!- Configure eviction policy. -!->
    <property name="evictionPolicy">
        <bean
class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy">
            <!- Evict to off-heap after cache size reaches maxSize. -!->
            <property name="maxSize" value="100000"/>
        </bean>
    </property>

These were removed to try to debug the current issue. After adding these
changes was when savePairs stopped working. I have not found anything in the
logs.

Has anyone came across this issue, any work-arounds/solutions?

I believe there could be some hidden state involved. Is there a way to
restore my cluster (delete certain directory etc.)? I have restarted the
whole cluster numerous times.

Notes: HDFS is configured as the under filesystem. When I create my
IgniteContext (in the REPL) with an Ignite Node running on the same machine
I get a warning that the IGFS/IGFS-management endpoints already in use: I
have tested this with and without an Ignite node running on driver machine.

P.S. Here is the thread trace:

Frozen threads found (potential deadlock)

It seems that the following threads have not changed their stack for more
than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

shmem-worker-#93%null% <--- Frozen for at least 21 sec
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryUtils.readSharedMemory(long,
byte[], long, long, long) IpcSharedMemoryUtils.java (native)
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemorySpace.read(byte[],
int, int, long) IpcSharedMemorySpace.java:220
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryInputStream.read(byte[],
int, int) IpcSharedMemoryInputStream.java:62
org.apache.ignite.internal.util.ipc.IpcToNioAdapter.serve()
IpcToNioAdapter.java:114
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ShmemWorker.body()
TcpCommunicationSpi.java:2943
org.apache.ignite.internal.util.worker.GridWorker.run() GridWorker.java:110
java.lang.Thread.run() Thread.java:745

I see references to Shmem; FYI the endpoint that is configured is a TCP
endpoint.

P.P.S. Forget the 21 seconds above: the thread was still hung after an hour.




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Execution-hangs-at-IgniteRDD-savePairs-tp2615.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message