hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@effectivemachines.com>
Subject Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk
Date Sat, 23 Jul 2016 05:08:55 GMT

> On Jul 22, 2016, at 5:47 PM, Zheng, Kai <kai.zheng@intel.com> wrote:
> For the leveldb thing, wouldn't we have an alternative option in Java for the platforms
where leveldb isn't supported yet due to whatever reasons. IMO, native library would be best
to be used for optimization and production for performance. For development and pure Java
platform, by default pure Java approach should still be provided and used. That is to say,
if no Hadoop native is used, all the functionalities should still work and not break. 

	Yes and no.  I can certainly understand some high-end features being tied to native libraries,
simply because system programming with Java is like being a touch typist with your nose. 

	That said, absolutely key functionality should definitely work. Take a look at the last Linux/ppc64le
report that was emailed to these very lists a few days ago [1]:


	Almost all of those YARN failures are due to MiniYARN trying to initiate leveldb as part
of the service startup but can't because the embedded shared library is the wrong hardware
architecture. Rather than catch the exception and do something else, the code just blows up
in a very dramatic fashion. That should translate into YARN is completely busted and unusable
without doing some very weird workarounds.

	To get us back on topic:  the class path isolation work absolutely cannot make this situation
worse.  We either need to make sure end users can replace/modify Hadoop's dependencies if
they require native lirbaries or work harder on making multiplatform stuff better supported.
 The nightly PowerPC builds should help tremendously towards this goal. [2]

1 - While I greatly appreciate the OpenPOWER Foundation getting the ASF access to these boxes
-- Mesos and Hadoop are both actively using them -- It'd be great if they were more reliable
so we could get a report every day of the week. :(

2 - At some point, I'll set up a manually triggered precommit job to test patches.  But until
both boxes are online and available on a consistent basis, it just isn't worth the effort.
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message