flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Newport, Billy" <Billy.Newp...@gs.com>
Subject RE: Serializing NULLs
Date Wed, 04 Jan 2017 18:25:11 GMT
Map<union<null,double>> in your avro schema is what you want here if the map values
are nullable.


From: Anirudh Mallem [mailto:anirudh.mallem@247-inc.com]
Sent: Tuesday, December 20, 2016 2:26 PM
To: user@flink.apache.org
Subject: Re: Serializing NULLs

If you are using Avro generated classes then you cannot have your values null.
https://cwiki.apache.org/confluence/display/AVRO/FAQ#FAQ-Whyisn'teveryvalueinAvronullable?<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_AVRO_FAQ-23FAQ-2DWhyisn-27teveryvalueinAvronullable-3F&d=DgMGaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=rlkM70D3djmDN7dGPzzbVKG26ShcTFDMKlX5AWucE5Q&m=lDkmaFdQUGxF0wCdZ6mHjtKO8fwbxwv7ya1D-utjZ3o&s=yBn5YzwBKzOYD2wOxud0pnwWyjLa2hMDoAME91wM7JA&e=>

From: Stephan Ewen
Reply-To: "user@flink.apache.org<mailto:user@flink.apache.org>"
Date: Tuesday, December 20, 2016 at 8:17 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
Subject: Re: Serializing NULLs

Thanks for sharing the stack trace.

This seems not really Flink related, it is part of the specific Avro encoding logic.
The Avro Generic Record Type apparently does not allow the map value to be null.



On Tue, Dec 20, 2016 at 4:55 PM, Matt <dromitlabs@gmail.com<mailto:dromitlabs@gmail.com>>
wrote:
Here is the back trace: https://gist.github.com/56af4818bcf5dee6b97c248fd9233c67<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_56af4818bcf5dee6b97c248fd9233c67&d=DgMGaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=rlkM70D3djmDN7dGPzzbVKG26ShcTFDMKlX5AWucE5Q&m=lDkmaFdQUGxF0wCdZ6mHjtKO8fwbxwv7ya1D-utjZ3o&s=mUF_Xp5joSe3H2vF2CIUEom76khbr4_lcnvZHZ-H7bc&e=>

In the meanwhile I've solved the issue by creating a POJO class where null is just Long.MIN_VALUE,
that with a custom equals() made the trick. I guess it's not as fast as de/serializing Double
though.

If you need any other information let me know.

Regards,
Matt

On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen <sewen@apache.org<mailto:sewen@apache.org>>
wrote:
The "null" support in some types is not fully developed. However in that case I am wondering
why it does not work. Can you share the stack trace, so we can take a look at the serializer?



On Mon, Dec 19, 2016 at 9:56 PM, Matt <dromitlabs@gmail.com<mailto:dromitlabs@gmail.com>>
wrote:
Hello list,

I'm getting this error:

java.lang.RuntimeException: Could not forward element to next operator
...
Caused by: java.lang.NullPointerException: in com.entities.Sector in map in double null of
double of map in field properties of com.entities.Sector
...
Caused by: java.lang.NullPointerException

The field mentioned is a HashMap<String, Double>, and some keys are mapped to null values.

Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?

Regards,
Matt



Mime
View raw message