Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.apache.org
From: arijit chakraborty <akc14@hotmail.com>
To: "dev@systemml.apache.org" <dev@systemml.apache.org>
Subject: RE: Update Spark Configuration to improve SystemML performance
Thread-Topic: Update Spark Configuration to improve SystemML performance
Thread-Index: AQHTBSG5QAx3DFMUxkeplT1b4VUL5aJkQ+qwgABg4ICAACYogIAADit8
Date: Tue, 25 Jul 2017 18:14:34 +0000
Message-ID: <MAXPR01MB02202437141AF12019524575A3B80@MAXPR01MB0220.INDPRD01.PROD.OUTLOOK.COM>
References: <CAOyjQwHBG=wqim2FCjoVOLUmyos0C0DRyi9dj_ajcDWuw=zspg@mail.gmail.com>
 <MAXPR01MB022011CBEC9AE1CB1D5B9E1AA3B80@MAXPR01MB0220.INDPRD01.PROD.OUTLOOK.COM>
 <D91C5BD8-5395-4421-82B1-152876E63B83@gmail.com>,<CAObMueVb2tg0aMDPHC8CxyCu+Y2quuhzFvg=M26rQ=fUndDbcA@mail.gmail.com>
In-Reply-To: <CAObMueVb2tg0aMDPHC8CxyCu+Y2quuhzFvg=M26rQ=fUndDbcA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [vvx+/wP9Vefg56YMf7FLieGLF/bF6RGq]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1;SG2APC01HT070;7:/qGxrx71HzjHkzW3xGnsmabWqmHXgiUtz3m1bhrAOMS2PYhX1g17XbI8dA4/c8t04bCErryrcneDA1sktOEN3+6qp3oKM5CdZHUH2iTBoB4f2pazpSSXnC4XAgihIRtpQXDdW1Xso/C2KTdLar2upOl2eaNBhRuYsgEFTRpcNgt3UIYFIf2NXx6vtwkIyEfQapNiEkFEmR8UIkibwD7hK3NoyB2hAp6FoJhlUU73j+aFAsXf0uEUrjAcaEq8D5xHtVBrS2IVZOLPWS2E93bojOGFfwoI4inRvRN3y0F9I491Ufy8hCQA7ks6BGdBEJwN39SsgddLToE9jyYjfx/510x21RpaYXnZKF0WcB3L1t/y4uViHE2ZvEDy3AHyOMNQigFF44DJbNkES6foWpS9qZRT1ekaimhpqBspwb9jFw9AKEkRuYtRX6m/MCp8RWYThj+ZGxuRjFVqOA6JPnHaHVebL7t9B0dlj80HT2THyD4gJ6MmEYx/WkZb9o30MX/nPAN8ummT5Xo8ddDRXa9cRv3viTF97rhPY8f8OPddtvZJG6s1Tj6Pxz3tbs+jQCdQKg7bLsJvc2g0OWudG1gNvYGRx3xSsMP0NoDhp+Scr5JY8juAULsFs25xlXgpObYB45eT5/F02Z/ICwhLuah3YsoTVr3xJ7extfnd675FsH2a9v2I/M8Dv+2cEFpQ+MI3Ouv8AEKPd2jI9B9Ex8bUBc9dT9jrI5Ft/EOflSWog1h9dsoRj+9zPcUx4CvFb9xewlMuELr6wvH0czX7Cq5iLw==
x-incomingheadercount: 45
x-eopattributedmessage: 0
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(7070007)(98901004);DIR:OUT;SFP:1901;SCL:1;SRVR:SG2APC01HT070;H:MAXPR01MB0220.INDPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:en;
x-ms-office365-filtering-correlation-id: 364f2f74-d03f-40cd-03c3-08d4d38900b4
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(300000503095)(300135400095)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322350)(1601125374)(1603101448)(1701031045)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:SG2APC01HT070;
x-ms-traffictypediagnostic: SG2APC01HT070:
x-exchange-antispam-report-test: UriScan:(134217032509453)(158342451672863)(156600954879566)(194151415913766)(21748063052155)(5213294742642)(8415204561270);
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(444000031);SRVR:SG2APC01HT070;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:SG2APC01HT070;
x-forefront-prvs: 03793408BA
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_MAXPR01MB02202437141AF12019524575A3B80MAXPR01MB0220INDP_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jul 2017 18:14:34.1359
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SG2APC01HT070
archived-at: Tue, 25 Jul 2017 18:14:49 -0000

--_000_MAXPR01MB02202437141AF12019524575A3B80MAXPR01MB0220INDP_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

I=92ll also pick one issue and try to solve it.

And it is really a very friendly group!

Thanks!
Arijit

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window=
s 10

From: Janardhan Pulivarthi<mailto:janardhan.pulivarthi@gmail.com>
Sent: Tuesday, July 25, 2017 10:53 PM
To: himanshu.mohan78@gmail.com<mailto:himanshu.mohan78@gmail.com>; dev@syst=
emml.apache.org<mailto:dev@systemml.apache.org>
Subject: Re: Update Spark Configuration to improve SystemML performance

Hi Himanshu!

We feel great you are here. SystemML is a very friendly community.

To get started.
1. see the components of the SystemML, here in this link:
https://issues.apache.org/jira/projects/SYSTEMML?selectedItem=3Dcom.atlassi=
an.jira.jira-projects-plugin:components-page

2. Through this link you can select any component of your interest, and you
can browse through the corresponding issues and find which one(s) of them
fascinates you.

3. If you any need guidance on how to proceed on that particular issues
notify the other committers on the mailing list or in the jira itself.

Thanks a lot,

Cheers,
Janardhan

On Tue, Jul 25, 2017 at 8:36 PM, Himanshu Mohan <himanshu.mohan78@gmail.com=
>
wrote:

> I am also interested in doing some real life hands on work in SystemML
>
> Thanks and Regards
> Himanshu
>
> > On 25-Jul-2017, at 2:57 PM, arijit chakraborty <akc14@hotmail.com>
> wrote:
> >
> > Hi Matthias,
> >
> >
> > Thanks for your mail. I'm attaching again the server configurations. I'=
m
> also adding your personal email id, just to be double sure you can see th=
e
> images. Pardon me for that. I could improve the setup further so that now=
 I
> can run the code at the same speed as R (around 40 mins). But this setup
> I'm sharing is the older setup. So most probably the performance of my co=
de
> was dependent on spark configuration. So if you can help me on that.
> >
> >
> > Also, currently I'm mainly working on CNN works. And I've decent
> programming experience in python & R. But I would request you to share wi=
th
> me project which is among the least priority one. This will help me to ge=
t
> accustomed with this project setup without getting bothered about time
> lines.
> >
> >
> > Thank you!
> >
> > Arijit<cluster performance.png>
> > <cluster specs.png>
> > <cores.png>
> > From: Matthias Boehm <mboehm7@googlemail.com>
> > Sent: Tuesday, July 25, 2017 2:10:52 PM
> > To: dev@systemml.apache.org
> > Subject: Re: Update Spark Configuration to improve SystemML performance
> >
> > great to hear that - we welcome additional contributions. Just let us
> know
> > in which area you're most interested in (e.g., algorithms, APIs,
> optimizer,
> > runtime, etc) and we could identifying a couple of tasks to get you
> started.
> >
> > Regarding the performance numbers, I am not able to see the details. Al=
so
> > could you share which operation was causing the large GC overhead - may=
be
> > we can improve the runtime for the specific scenario. Thanks.
> >
> > Regards,
> > Matthias
> >
> > On Mon, Jul 24, 2017 at 12:17 PM, arijit chakraborty <akc14@hotmail.com=
>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > I tried to work on spark configuration file to improve the systemML
> > > performance. Even after much tuning R code is running in 40 mins, but
> > > SystemML is taking 2.2 hours. Please find the spark configuration
> > > screenshots. Please let me know if I'm making some mistake in tuning
> of the
> > > spark configuration. One problem we could rectify is garbage time
> error.
> > > Now, it's completely not there. That was one major bottleneck which w=
as
> > > making the code extremely slow.
> > >
> > >
> > > I'm  working in local system and created a standalone version of spar=
k,
> > > with master and workers. The following are the details:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > I also wants to know is it possible to get involved with systemML
> > > development? My project is almost on the verge of completion and I
> learned
> > > a lot from you all people. And I really liked this project. So I want
> to
> > > contribute more fruitfully in it.
> > >
> > >
> > > Thank you!
> > >
> > > Arijit
> > >
>


--_000_MAXPR01MB02202437141AF12019524575A3B80MAXPR01MB0220INDP_--