spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "assaf.mendelson" <assaf.mendel...@rsa.com>
Subject Converting spark types and standard scala types
Date Sun, 13 Nov 2016 11:28:45 GMT
Hi,
I am trying to write a new aggregate function (https://issues.apache.org/jira/browse/SPARK-17691)
and I wanted it to support all ordered types.
I have several  issues though:

1.       How to convert the type of the child expression to a Scala standard type (e.g. I
need an Array[Int] for IntegerType and an Array[Double] for DoubleType). The only method I
found so far is to do a match for each of the types. Is there a better way?

2.       What would be the corresponding scala type for DecimalType, TimestampType, DateType
and BinaryType? I also couldn't figure out how to do a case for DecimalType. Do I need to
do a specific case for each of its internal types?

3.       Should BinaryType be a legal type for such a function?

4.       I need to serialize the relevant array of type (i.e. turn it into an Array[Byte]
for working with TypedImperativeAggregate). Currently, I use java.io.{ByteArrayOutputStream,
ByteArrayInputStream, ObjectInputStream, ObjectOutputStream}. Is there another way which is
more standard (e.g. get a "Serialize" function which knows what to use:  java serialization,
kyro serialization etc. based on spark configuration?)
Thanks,
                Assaf.





--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Converting-spark-types-and-standard-scala-types-tp19837.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Mime
View raw message