spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "assaf.mendelson" <>
Subject Converting spark types and standard scala types
Date Sun, 13 Nov 2016 11:28:45 GMT
I am trying to write a new aggregate function (
and I wanted it to support all ordered types.
I have several  issues though:

1.       How to convert the type of the child expression to a Scala standard type (e.g. I
need an Array[Int] for IntegerType and an Array[Double] for DoubleType). The only method I
found so far is to do a match for each of the types. Is there a better way?

2.       What would be the corresponding scala type for DecimalType, TimestampType, DateType
and BinaryType? I also couldn't figure out how to do a case for DecimalType. Do I need to
do a specific case for each of its internal types?

3.       Should BinaryType be a legal type for such a function?

4.       I need to serialize the relevant array of type (i.e. turn it into an Array[Byte]
for working with TypedImperativeAggregate). Currently, I use{ByteArrayOutputStream,
ByteArrayInputStream, ObjectInputStream, ObjectOutputStream}. Is there another way which is
more standard (e.g. get a "Serialize" function which knows what to use:  java serialization,
kyro serialization etc. based on spark configuration?)

View this message in context:
Sent from the Apache Spark Developers List mailing list archive at
View raw message