ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taras Ledkov <tled...@gridgain.com>
Subject IgniteRDD type mapping
Date Wed, 08 Jun 2016 08:23:50 GMT
Igniters,

Lets discuss the IgniteRDD type mapping features.
You can see the problem discovery at the issue: 
https://issues.apache.org/jira/browse/IGNITE-3175

* Description:
The current implementation IgniteRDD supports only types from 
conjunction of the H2 types
wrappers & spark type set.

I've created the test to illustrate the problem with type mapping. The 
test try to select the non-primitive field from the IgniteRDD
(please take a look the pull request: 
https://github.com/apache/ignite/pull/784).
The exception scala.MatchError is thrown at the 
IgniteRDDTypeMappingTest.java:111

scala.MatchError : 
org.apache.ignite.spark.IgniteRDDTypeMappingTest$StructObj@68afa21 (of 
class org.apache.ignite.spark.IgniteRDDTypeMappingTest$StructObj)

* Background:
Please take a look at the H2 type mapping & the Spark type mapping specs:
http://www.h2database.com/html/datatypes.html
https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#data-types

* Root cause:
The information about class of the TestObj.structVal field is lost in 
Ignite. IgniteRDD uses the
QueryCursor.fieldsMeta to discover type of the query fields. Ignite 
returns java.lang.Object in according with
H2 type mapping. But AFAIK Spark doesn't support plain java objects.

-- 
Taras Ledkov
Mail-To: tledkov@gridgain.com


Mime
View raw message