systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm" <>
Subject Re: SystemML Hadoop version support
Date Wed, 20 Jan 2016 21:30:04 GMT

Just to clarify because there seems to be some confusion: we are supporting
any Spark version >=1.4 and Hadoop version >=2.4. The only reason why our
build sticks to older versions is to avoid exploiting new features too
eagerly which would loose backwards compatibility if done without care.

With regard to Hadoop v1: Our code would not compile with Hadoop v1
libraries because we have several YARN features. However, the code is
structured in a way such that we could still run on Hadoop v1 because we
would never load the related classes during runtime. To be honest though,
we did not run SystemML on v1 for more than a year now.

@Deron: In order to answer your question: it would be great if you could
update MRConfigurationNames for now - it's anyway good to have all
configurations in a central place. Down the road we will systematically
remove v1 support because it would allow us to simplify a couple of things.


From:	Luciano Resende <>
Date:	01/20/2016 01:14 PM
Subject:	Re: SystemML Hadoop version support

On Wed, Jan 20, 2016 at 12:50 PM, Deron Eriksson <>

> Does SystemML still support Hadoop 1?
> I see in MRConfigurationNames a static initialization block for Hadoop
> v1/v2 properties. I'd like to remove the current deprecated property
> warnings (log messages in the console) that we get with Hadoop v2.4.1 and
> was wondering if I should add the old v1 and new v2 versions to
> MRConfigurationNames or if I can just add the new v2 versions and the v1
> versions can be removed.
> If we still support Hadoop v1, we might want to update the pom with
> profiles for Hadoop 1 and 2.
> Deron

Even Spark is moving away of supporting some old versions of Hadoop, etc...

I would really like us to be moving kind of together with other
dependencies, in the sense of supporting Hadoop 2.6 and up, and Spark 1.6
and maybe 1.5.2 as well.

Luciano Resende

  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message