Mailing-List: contact dev-help@community.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@community.apache.org
From: muktesh mishra <mukteshkrmishra@hotmail.com>
To: "dev@community.apache.org" <dev@community.apache.org>
CC: "karanjes@usc.edu" <karanjes@usc.edu>, "pierre.smits@gmail.com"
	<pierre.smits@gmail.com>, Infra <infra@apache.org>
Subject: Re: DRAT is now scanning Apache SVN code base!
Thread-Topic: DRAT is now scanning Apache SVN code base!
Thread-Index: AQHRXiOvse2t8Az1XkWUSJBy6Z94358Z92WAgAAOfACAAAJEAIAAZf8AgAAY+JY=
Date: Wed, 3 Feb 2016 16:41:19 +0000
Message-ID: 
 <MA1PR01MB0552B24523138C9DB10B6D57A3D00@MA1PR01MB0552.INDPRD01.PROD.OUTLOOK.COM>
References: 
 <CAGaRif2ZdrhnHNhE7gDAK_UwM9NnrJaFyCPkj1w0997yw_nkxw@mail.gmail.com>
 <CADPkm85PX77dVFAtu3SnU37w=9uJuCePJmEWivO_gDpCs8_M+Q@mail.gmail.com>
 <CAF7QcwuB0xmcBzp9A_o=ojXWzvzbWPnfTM6XTKCrS5YX1C_TFg@mail.gmail.com>
 <BE877909-5D57-4D64-970F-E04676328244@pc-tony.com>,<D2D75830.74756%chris.a.mattmann@jpl.nasa.gov>
In-Reply-To: <D2D75830.74756%chris.a.mattmann@jpl.nasa.gov>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [2jF7W9ZlFXPy2bCNOMm/EDIyEMmRgu5D]
x-microsoft-exchange-diagnostics: 
 1;MA1PR01MB0552;5:PuYGokTjPZlax59m0eC3Oe0EfjAipfRc1bJhEpDE/fisbppv4v1OKnd4Gpx9R7XT1lQmH66ZnWS+DYdNd+TLYEU2tt7GMkXWbHKypvAgA2PbOQpTevRLxy6+3MU7nFkD+5Ndv3PnolgDQb50JN9VSA==;24:v5y6eu8Wscnbnx9lmJHkATTCYLe9r4RLJfmmIakFuSnRTnNZzOLvKVsKWvLpMR+bJOt1LAQQIIVeA1NO8t9iKEEeXF5lhEzxFnP1WPy23O0=
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:MA1PR01MB0552;
x-ms-office365-filtering-correlation-id: 165ee116-be08-44a9-0fb6-08d32cb8d80d
x-exchange-antispam-report-cfa-test: 
 BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:MA1PR01MB0552;BCL:0;PCL:0;RULEID:;SRVR:MA1PR01MB0552;
x-forefront-prvs: 08417837C5
x-forefront-antispam-report: 
 SFV:NSPM;SFS:(7070004)(6009001)(377454003)(13464003)(252514010)(53754006)(51914003)(24454002)(54356999)(76176999)(82202001)(50986999)(40100003)(87572001)(73972006)(93886004)(19580395003)(19580405001)(83332001)(106116001)(575784001)(122556002)(87936001)(110136002)(86362001)(2351001)(2950100001)(86152002)(2900100001)(4326007)(226693001)(16601075003)(2501003)(15975445007)(77096005)(5002640100001)(5003600100002)(1220700001)(102836003)(5008740100001)(4000100100001)(3660700001)(33656002)(10400500002)(586003)(92566002)(3280700002)(189998001)(15852004)(24704002)(42522002)(19627235001);DIR:OUT;SFP:1901;SCL:1;SRVR:MA1PR01MB0552;H:MA1PR01MB0552.INDPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;MLV:sfv;LANG:en;
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: sct-15-1-318-15-msonline-outlook-3dddf.templateTenant
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Feb 2016 16:41:19.8304
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MA1PR01MB0552
X-OriginalArrivalTime: 03 Feb 2016 16:41:24.0570 (UTC)
 FILETIME=[B853CBA0:01D15EA1]

Nice work !! Karanjeet.

It is helping in multiple ways and will improve by the time for sure.

Do we have plans to implement more custom attribute based dashboards as wel=
l??

-Muktesh

> On Feb 3, 2016, at 7:12 AM, Mattmann, Chris A (3980) <chris.a.mattmann@jp=
l.nasa.gov> wrote:
>=20
> Hey Tony,
>=20
> Sorry we should have been more clear about that. We are using
> svn-dump (we aren=92t doing this in real-time). Karanjeet can
> confirm, but from that dump file, I think we were able to
> generate this interactive app using DRAT in about a week or
> a little more. It=92s worth noting that it=92s *all* the revisions
> in that dump that we are scanning so this captures the evolution
> of the repo over time as well.
>=20
> Thanks for the pointers to Git repos too. We=92ll scan those
> with DRAT and Wrangler next.
>=20
> Also hat tip to the Solr community and so forth too - our DRAT
> stats are powered by Apache Solr (and D3).
>=20
> Cheers,
> Chris
>=20
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>=20
>=20
>=20
>=20
>=20
> -----Original Message-----
> From: Tony Stevenson <tony@pc-tony.com>
> Reply-To: "dev@community.apache.org" <dev@community.apache.org>
> Date: Wednesday, February 3, 2016 at 1:06 AM
> To: "dev@community.apache.org" <dev@community.apache.org>, Karanjeet Sing=
h
> <karanjes@usc.edu>
> Cc: "pierre.smits@gmail.com" <pierre.smits@gmail.com>, Infra
> <infra@apache.org>
> Subject: Re: DRAT is now scanning Apache SVN code base!
>=20
>> cc +=3D infra@
>>=20
>> Karanjeet,
>>=20
>> I am writing to you whilst wearing my Infrastructure hat.
>>=20
>> Please be careful if you are indeed recursing the entire ASF subversion
>> repository (http://svn.apache.org) - as you will quite likely run into
>> the aug-banning service.
>> Have you seen https://svn-dump.apache.org ?  This is an entire dump of
>> the SVN repo (at least the public one you are interested in. You can use
>> this, and it is updated monthly. If you really need fully upto date data
>> you can use the dump, and svnsync the remaining revisions.
>>=20
>> I guess this might be obvious, but I=92ll mention it just in case.  A lo=
t
>> of projects are using git repositories too. Which are mirrored here:
>> github.com/apache/
>>=20
>>=20
>>=20
>> --
>> Cheers,
>> Tony
>>=20
>> On behalf of the Apache Infrastructure Team
>>=20
>> -----------------------
>> http://www.pc-tony.com
>> GPG - 3072D/2543E323
>> -----------------------
>>=20
>>> On 3 Feb 2016, at 08:58, Karanjeet Singh <karanjes@usc.edu> wrote:
>>>=20
>>> Thanks Pierre for your feedback.
>>>=20
>>> Yes, the visualization corresponds to only 133 / 191 SVN projects (
>>> http://svn.apache.org/repos/asf/). We have successfully audited close t=
o
>>> 175 projects and hopefully by the end of this week all the remaining
>>> projects should be covered. We will update the data once done.
>>>=20
>>> Large repositories like "subversion" and "camel" having 493,420 files
>>> (size
>>> - 9,723 MB approx) and 519,584 files (size - 1,922 MB approx) taking up
>>> to
>>> 36 hours (only) to complete which is quite a good number.
>>>=20
>>> For your second question, I don't have an answer yet. Our intentions
>>> will
>>> be to update this regularly but we have some limitation at the Wrangler
>>> end
>>> that it doesn't allow us to run a job for more than 48 hours. Therefore=
,
>>> for very large repositories like openoffice, spamassassin, myfaces, etc=
,
>>> which takes more time to get audited, it will be a challenge to split
>>> the
>>> repositories every time and scan.
>>>=20
>>> Best Regards,
>>> Karanjeet Singh
>>> CS Graduate Student
>>> University of Southern California
>>> karanjes@usc.edu | +1-213-675-9583
>>>=20
>>>=20
>>> On Wed, Feb 3, 2016 at 12:06 AM, Pierre Smits <pierre.smits@gmail.com>
>>> wrote:
>>>=20
>>>> HI Karanjeet,
>>>>=20
>>>> This is surely an impressive piece of work. But I still notice that
>>>> some
>>>> projects are missing in the overview. Is this a mere PoC not intended
>>>> to be
>>>> complete? Or something that will be made available to all and be
>>>> updated
>>>> regularly?
>>>>=20
>>>> Best regards,
>>>>=20
>>>> Pierre Smits
>>>>=20
>>>> ORRTIZ.COM <
>>>>=20
>>>> https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__www.orrtiz.com&d=
=3DCwI
>>>> BaQ&c=3DclK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=3Du7neGGUaVmQKNS=
LUqJ9z
>>>> pA&m=3DI4VmXy1BbrwbVZc9758zYzQ1Vg_gsve4ety_zu60Z7o&s=3Drey8QvJVsx9VER8=
tfbyqc
>>>> WeBc3x1dze3BDFEgOry1zo&e=3D
>>>> OFBiz based solutions & services
>>>>=20
>>>> OFBiz Extensions Marketplace
>>>>=20
>>>>=20
>>>> https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__oem.ofbizci.net_=
oci-
>>>> 2D2_&d=3DCwIBaQ&c=3DclK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=3Du7=
neGGUaVm
>>>> QKNSLUqJ9zpA&m=3DI4VmXy1BbrwbVZc9758zYzQ1Vg_gsve4ety_zu60Z7o&s=3Dt-3eq=
7_jE8P
>>>> 3hTlTBYAQB9p_vFHuwoj6RqdbBBr8edI&e=3D
>>>>=20
>>>> On Wed, Feb 3, 2016 at 2:39 AM, Lewis John Mcgibbney <
>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>=20
>>>>> Hi Karanjeet,
>>>>>=20
>>>>> A good bunch of work has lready gone into this and it is looking
>>>>> really
>>>>> friggin smart indeed.
>>>>> Interesting to see some many pieces of software come together and
>>>>> result
>>>> in
>>>>> something very easy to interpret.
>>>>> Good work.
>>>>> Lewis
>>>>>=20
>>>>> On Mon, Feb 1, 2016 at 11:44 PM,
>>>>> <dev-digest-help@community.apache.org>
>>>>> wrote:
>>>>>=20
>>>>>> Hello Everyone,
>>>>>>=20
>>>>>> With great pleasure, I would like to introduce DRAT (Distributed
>>>> Release
>>>>>> Audit Tool) which is a distributed, parallelized wrapper around
>>>>>> Apache
>>>>> RAT
>>>>>> to inspect for appropriate open source licensing in software
>>>>>> projects.
>>>>>> DRAT was started by my advisor, Chris Mattmann, in an effort to get
>>>>>> RAT
>>>>>> working on a ver large code base. RAT uses Apache OODT, Apache Tika,
>>>> and
>>>>>> Apache Solr.
>>>>>>=20
>>>>>> We are now auditing the complete Apache SVN code base to check for
>>>> proper
>>>>>> licenses. Until now, we have scanned 171 / 191 repositories and
>>>>>> illustrated the statistics for 133 of them through D3 visualization
>>>>>> located at
>>>>=20
>>>> https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__drat.dyndns.org-=
3A80
>>>> 80_dratviz&d=3DCwIBaQ&c=3DclK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&=
r=3Du7ne
>>>> GGUaVmQKNSLUqJ9zpA&m=3DI4VmXy1BbrwbVZc9758zYzQ1Vg_gsve4ety_zu60Z7o&s=
=3DEiqoi
>>>> xInVvAF49_1n7AxSu4q_q7BYMJ53JbVnf7rWK4&e=3D
>>>>>>=20
>>>>>> Projects should check out the MIME analysis of the code base and
>>>>>> click
>>>>>> around. Please also note due to the sheer size of the Apache code
>>>>>> bases
>>>>>> and the fact that we scanned and included all revisions in the Apach=
e
>>>> SVN
>>>>>> repo, DRAT is not running in real time. We are running DRAT on the
>>>>>> NSF
>>>>>> Super Computer Wrangler, which has a petabyte of flash storage and
>>>>>> the
>>>>>> ability to stand up Hadoop and Spark clusters. We are also working
>>>>>> on a
>>>>>> paper describing our results.
>>>>>>=20
>>>>>> Please send feedback to myself (Karanjeet Singh <karanjes@usc.edu>),
>>>>>> Professor Mattmann <mattmann@usc.edu> and/or
>>>> irds-L@mymaillists.usc.edu.
>>>>>>=20
>>>>>> Thanks & Regards,
>>>>>> Karanjeet Singh
>>>>>> C.S. Graduate Student
>>>>>> University of Southern California
>>>>>> karanjes@usc.edu | +1-213-675-9583
>=20