ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geektimus <alexander....@gmail.com>
Subject Re: Ignite + Spark Streaming on Amazon EMR Exception
Date Thu, 27 Oct 2016 16:15:00 GMT
Hi Val,Thanks for your suggestion, the thing at this point is that this is
running inside EMR (Elastic Map Reduce in Amazon) in embedded mode and I
don't have control over the system properties. I tried to use them inside a
bootstrap action for the cluster and I found that it was deprecated for the
kind of instance that I'm using.Also, I tried to use the new json conf API
for EMR with
{        "Classification": "spark-env",        "Configurations": [           
{                "Classification": "export",                "Properties": {              
    
"JAVA_HOME": "/usr/lib/jvm/java-1.8.0",                    "JAVA_OPTS":
"-Djava.net.preferIPv4Stack=true",                   
"java.net.preferIPv4Stack": "true"                }            }        ],       
"Properties": {}    }
*Note:* This is just a fragment.None of them work in the cluster, Im seeing
IPs like "0:0:0:0:0:0:0:1%1" or "0:0:0:0:0:0:0:1%lo". Btw, I found that the
"%1" and "%lo" are the scope IDs for a IPv6.I don't know if the issue is an
incompatibility between Ignite and Spark Streaming or just an error with
Spark handling IPv6'sFYI, In the Spark code I found this  marvelous comment
<https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L948>
 
in the method that is failing! ¬¬
// This is potentially broken - when dealing with ipv6 addresses for
example, sigh ...// but then hadoop does not support ipv6 right now.// For
now, we assume that if port exists, then it is valid - not check if it is an
int > 0
I'm still trying to find a solution.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Spark-Streaming-on-Amazon-EMR-Exception-tp8410p8558.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Mime
View raw message