spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Tungsten in a mixed endian environment
Date Tue, 12 Jan 2016 21:34:35 GMT
(x86 is little-endian and SPARC / POWER / ARM are big-endian; I'm sure
that was just a typo)

On Tue, Jan 12, 2016 at 9:13 PM, Steve Loughran <stevel@hortonworks.com> wrote:
> It's notable that Hadoop doesn't like mixed-endianness; there is work
> (primarily from Oracle) to have consistent byteswapping —that is: work
> reliably on big-endian systems
> https://issues.apache.org/jira/browse/HADOOP-11505 ). There's no motivation
> to support mixed-endian clusters.
>
>
> The majority of clusters x86, there's only 3 cpu families that are little
> endian: Spark, Power, Arm. Adam has clearly been playing with Power + x86,
> but I'd suspect that's experimentation, not production.
>
> What is probably worth checking is mixed endian-ness between client apps
> submitting work and the servers: Java and Kryo serialization should handle
> that automatically.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message