systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niketan Pansare" <npan...@us.ibm.com>
Subject Re: [DISCUSS] R-Interface to SystemML
Date Fri, 22 Sep 2017 17:51:23 GMT
Hi Brendan/Alok,

Thanks for taking time to answer our questions and also for considering to
help SystemML community. I have couple more questions:

1. In case there is inconsistency, do you (as R4ML developers) feel
comfortable changing R4ML interface to be compatible with our other APIs ?
May be you can go over the below two links and imagine adding a
corresponding R tab:
- MLContext Programming guide:
http://apache.github.io/systemml/spark-mlcontext-programming-guide
- Algorithm wrappers:
http://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression

2. Other than providing R interface to SystemML as the above APIs, what
additional features/code R4ML plans to add in SystemML ? Just like we want
the R API to be functionally complete with our Python and Scala API, we
want Python and Scala APIs to be functionally complete with the R API. So a
discussion on supporting the additional features in Python and Scala APIs
is required :)

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:	Deron Eriksson <deroneriksson@gmail.com>
To:	dev@systemml.apache.org
Date:	09/22/2017 08:59 AM
Subject:	Re: [DISCUSS] R-Interface to SystemML



Hi Brendan,

Thank you for the detailed description. At a high level that sounds
feasible. Also, offering to help maintain the R codebase is extremely
helpful. Please let us know if you have any questions so that we can assist
you and Alok in your efforts, since as I said I think an R interface to
SystemML makes a lot of sense.

Deron


On Thu, Sep 21, 2017 at 4:36 PM, Brendan Dwyer <Brendan.Dwyer@ibm.com>
wrote:

> Sorry for not responding sooner. I had some issues with my email client.
>
>
>
> I will do my best to address as many as the points that have been raised
> as I can. Hopefully Alok will be able to jump in as well once he resolves
> his email issues.
>
>
>
> - I would be happy to help maintain R4ML in SystemML and I’m sure Alok
> would too.
>
> - R4ML does allow arbitrary DML script to be executed via the
> `sysml.execute()` function.
>
> - I think we would like to merge the entire R4ML github repository into
> SystemML. We could do this the same way SparkR was merged into Spark (
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e=
 )
>
> - Currently the code is not ready to be merged into SystemML because we
> are still on the old ML context. We have a PR in the works that will
update
> to the newest ML context. Once that happens we won’t need to duplicate
the
> DML scripts.
>
> - Documentation is generated automatically with the R package “roxygen”.
> We would need to discuss how to incorporate this into the SystemML
> documentation. Perhaps we could look to Spark/SparkR for ideas.
>
> - Test are done using  the R testthat package. I can work with Alan to
get
> that integrated into the systemml Jenkins  server
>
> Sent from IBM Verse
>
>
>    Matthias Boehm --- Re: [DISCUSS] R-Interface to SystemML ---
>     From:"Matthias Boehm" <mboehm7@googlemail.com>To:
> dev@systemml.apache.org, deron@apache.orgDate:Thu, Sep 21, 2017 4:13
> PMSubject:Re: [DISCUSS] R-Interface to SystemML
>
>     I pretty much agree with Niketan and Deron. In general, it would be
> usefulto provide an R API as well. However, I'm a bit concerned for two
> reasons:* Looking over the github repo, apparently R4ML is not under
> activedevelopment/maintenance anymore (last commit Jul 20). So who would
> bewilling to maintain and extend it?* Providing wrappers for our
algorithm
> scripts would be just a startbecause it hides our core value proposition
of
> custom large-scale ML.Hence, we would also need an MLContext equivalent
> that allows to executearbitrary DML scripts or R functions. Is there
> already a tentative designof such an API and if not, who would like to
take
> it over?Regards,MatthiasOn Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson <
> deroneriksson@gmail.com>wrote:> I agree with Niketan. An R interface
> definitely makes sense for SystemML.> DML itself is based on R, so it's
> surprising we have Java/Scala/Python> interfaces to SystemML but we don't
> have an R interface.>> Perhaps R4ML committers could supply a little more
> info? For instance:> 1) Would they like to merge R4ML code into the main
> SystemML project> itself? (Currently we have no modules.)> 2) What would
> they like to merge?> 3) If so, how do they propose to do so?> 4) Who will
> do the majority of the work to add R4ML code to SystemML? Or> who would
> like to volunteer to do this?> 5) Who will maintain the contributed code?
> Or who would like to volunteer> to do this?> 6) Documentation is needed
> (fit in SystemML documentation framework).> 7) Testing is needed (fit
into
> SystemML testing framework).> 8) How is this packaged?>> From a
technology
> standpoint, I think an R interface totally makes sense.> As for a minor
> criticism (which I apply to other parts of SystemML too), I> see script
> wrappers at https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_SparkTC_r4ml_tree_master_R4ML_R&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_
> gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=_qGcWSDggH-K3E_
> wTPXBBjOClp2Jub4KtvBgfeW1kbU&e= .> This tightly binds the existing DML
> scripts to R, which means DML> input/output modifications could
potentially
> require modifications to R> code.>> Deron>>>> On Thu, Sep 21, 2017
at
11:00
> AM, Niketan Pansare <npansar@us.ibm.com>> wrote:>> > Janardhan: I believe
> this is the R4ML repo:
https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=s0malh3u3v57rOz_0fpFab5WlWAJ735Cb6cGOWtiS1w&e=
 .
> com/v2/url?u=https-3A__github.com_SparkTC_&d=DwIBaQ&c=jf_
> iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1Xd
> JBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&
> s=vj-Ogu1F2fnli1JwDjU1-S-Pauc7SSmSNG0g5sjgwUk&e= > > r4ml . Arvind:
> please correct me if I am wrong.> >> > Overall, having a R interface for
> SystemML is an awesome idea. Since I am> > not an R4ML expert, may be
R4ML
> committers can comment on how they> envision> > "two code streams to work
> together".> >> > Also, comparing the features of R4ML with that of our
> Python APIs will be> > useful as it might make a stronger case for R4ML.>
> >> > As an FYI, here are different ways Python users can use SystemML:> >
-
> Using MLContext to invoke DML script (
https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=W-mdQ-T1cM2C_v9AGp-9_8RfDTP54iwaFGrXNwx-B1c&e=
 .
> proofpoint.com/v2/url?u=http-3A__apache.github.io_&d=
> DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1Xd
> JBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&
> s=7ng5kh3VaiQ045UZ-8xQtNmZS1KvD7LQ2e5gCM7aRmQ&e= > >
> systemml/beginners-guide-python#invoking-dmlpydml->
> scripts-using-mlcontext> > and
https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=s0malh3u3v57rOz_0fpFab5WlWAJ735Cb6cGOWtiS1w&e=
 .
> com/v2/url?u=http-3A__apache.github.io_systemml_spark-
> 2Dmlcontext-2D&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_
> gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=PuQtoa13gaSfRk95Bl5jvZnBeEhyI_
> Rae7ZB8-H5970&e= > > programming-guide.html)> > - Python algorithms
> wrappers (https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__apache.github.io_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_
> gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=7ng5kh3VaiQ045UZ-
> 8xQtNmZS1KvD7LQ2e5gCM7aRmQ&e= > > systemml/beginners-guide-
> python#invoke-systemmls-algorithms)> > - (not important for R4ML
> discussion): Python DSL (> >
https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=s0malh3u3v57rOz_0fpFab5WlWAJ735Cb6cGOWtiS1w&e=
 .
> com/v2/url?u=http-3A__apache.github.io_systemml_beginners-
> 2Dguide-2D&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_
> gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=7W3j9VgE_HckySvu_5-
> NDjtmgHTcdtjxB_1syxRMNRY&e= > python#matrix-operations)> >> > Thanks,>
>>
> > Niketan Pansare> > IBM Almaden Research Center> > E-mail: npansar At
> us.ibm.com> > http://researcher.watson.ibm.com/researcher/view.php?
> person=us-npansar> >> > [image: Inactive hide details for Janardhan
> ---09/21/2017 04:44:02> AM---Hi> > Arvind, This is a great idea. One
> question: the R4ML generat]Janardhan> > ---09/21/2017 04:44:02 AM---Hi
> Arvind, This is a great idea. One> question:> > the R4ML generates any
plan
> like the SystemML with `D> >> > From: Janardhan <j143@protonmail.com>>
>
> To: Arvind Surve <acs_s@yahoo.com.INVALID>, "dev@systemml.apache.org" <>
> > dev@systemml.apache.org>> > Date: 09/21/2017 04:44 AM> > Subject:
Re:
> [DISCUSS] R-Interface to SystemML> > ------------------------------> >>
> >> >> > Hi Arvind,> >> > This is a great idea. One question:
the R4ML
> generates any plan like the> > SystemML with `DML` Or with providing some
> interface we leverage this> > feature ?. Community effort in the sense of
> collaborative algorithm> > implementation.(?)> >> > Is this the Spark-R
> repo ( https://urldefense.proofpoint.com/v2/url?u=https-3A__
> urldefense.proofpoint&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_
> gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=IsTGjUhAgkp2oVtWd0RUsJcAcrSa0m
> zr9AbpU4Q86zU&e= .> > com/v2/url?u=https-3A__github.
> com_rstudio_sparklyr&d=DwIGaQ&> >
c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_>
> > 6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ->
> PSwyvQzq5gwX1GL0&s=6VRs_> > J7zXj9jZEavEP8iNvVfISAjDJeM8wFL2sBnb0g&e= 
)
> ?> >> > Thanks,> > Janardhan> >> > Sent with [ProtonMail](https://
> urldefense.proofpoint.com/v2/> > url?u=https-3A__protonmail.
> com&d=DwIGaQ&c=jf_iaSHvJObTbx-> >
siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=>
> > uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=khkGV3oXz1W5m_> >
> ueQRuKWlNMVOXXCVhV_ytNCINjJWY&e= ) Secure Email.> >> > > --------
> Original Message --------> > > Subject: [DISCUSS] R-Interface to
SystemML>
> > > Local Time: September 20, 2017 12:50 PM> > > UTC Time: September 20,
> 2017 4:50 PM> > > From: acs_s@yahoo.com.INVALID> > > To:
> dev@systemml.apache.org <dev@systemml.apache.org>> > >> > > Hi,>
> > R4ML
> is an open source project which provides a R interface to> > SystemML.Its
a
> bridge between SystemML and Spark-R.> > > Lets discuss here if and how we
> can get two code streams work together> > to benefit
development/community
> effort.> > >> > > Arvind Surve | Spark Technology Center  |>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> urldefense.proofpoint&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_
> gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=IsTGjUhAgkp2oVtWd0RUsJcAcrSa0m
> zr9AbpU4Q86zU&e= .> >
com/v2/url?u=http-3A__www.spark.tc_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=>
> > HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=uxG7P->
> 4VuICwg6yatnAEX5JBdZ-> >
PSwyvQzq5gwX1GL0&s=Yj8qfo7sjGBxX547UMGfLHRZFUxCAjZDTHWe8B7jLxI&e=>
> >> >> >>
>
>


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message