ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramakrishna Nalam <nramakris...@gmail.com>
Subject Getting an exception when listing partitions of IgniteDataFrame
Date Tue, 03 Jul 2018 12:20:07 GMT
Hi,

Seeking help with someone on the group who has seen this error before, or
can help me figure out what it issue is.

I'm trying to get ignite's spark integration working with Spark 2.2.1.
The ignite version I'm using is 2.5.0. I'm running Ignite co-located with
the spark workers.

I have been able to successfully load data into Ignite using the snippet:

val igniteDf = spark.read .
    format(IgniteDataFrameSettings.FORMAT_IGNITE) .
    option(IgniteDataFrameSettings.OPTION_TABLE, "tmp_table") .
    option(IgniteDataFrameSettings.OPTION_CONFIG_FILE,
"/home/hadoop/ignite-config.xml") .
load;

the command:

igniteDf.select("_c1").show()

works as expected. Other simple commands / functions like count(), max()
etc. also work well.

But when I try:

igniteDf.rdd.partitions

There is an exception thrown, with the following stacktrace:

java.lang.IllegalArgumentException: requirement failed:
partitions(0).partition == 4, but it should equal 0
  at scala.Predef$.require(Predef.scala:224)
  at
org.apache.spark.rdd.RDD$$anonfun$partitions$2$$anonfun$apply$3.apply(RDD.scala:254)
  at
org.apache.spark.rdd.RDD$$anonfun$partitions$2$$anonfun$apply$3.apply(RDD.scala:253)
  at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  ... 58 elided

It's a bit surprising, because looks like the code that is invoked is
the getPartitions
function of IgniteRDD
<https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteRDD.scala#L78>,
which should just return Array(IgnitePartition(0), IgnitePartition(1)...
and so on).

I have compiled ignite-spark module from github using the latest commit and
used that as well, but the same error persists.

Will be happy to provide any additional information that will help zero in
on the issue!


Thanks,
Rama.

Mime
View raw message