incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Debo Dutta (dedutta)" <dedu...@cisco.com>
Subject Re: [VOTE] Accept Metron into Apache Incubator
Date Thu, 03 Dec 2015 18:04:33 GMT
Would like to know who in Cisco was asked actually. I am from Cisco and
can help. 

debo

On 12/3/15, 9:39 AM, "Ted Dunning" <ted.dunning@gmail.com> wrote:

>I think that there was a still pending question of why not ask Cisco for
>an
>SGA for cleanliness.
>
>The only answer that I heard was that corporations don't like to do things
>like and so Cisco hadn't been asked.
>
>That seems like an insufficient answer.
>
>
>
>On Fri, Dec 4, 2015 at 4:33 AM, Owen O'Malley <omalley@apache.org> wrote:
>
>> The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>>whether
>> Apache Incubator should accept Metron as a podling. The proposal is
>>pasted
>> below and is available on the wiki as well.
>>
>> https://wiki.apache.org/incubator/MetronProposal
>>
>> We've added a paragraph in the background section discussing how Apache
>> avoids hostile forks of projects, because we don't want to fork
>> communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>> Rhodes to the proposal.
>>
>> The vote will run until 12pm PST on Sunday.
>>
>> Thanks,
>>    Owen
>>
>> = Apache Metron Proposal =
>>
>> ----
>> /!\ '''FINAL''' /!\
>>
>> This proposal is now complete and has been submitted for a VOTE.
>> ----
>>
>> == Abstract ==
>>
>> The Metron project is an open source project dedicated to providing an
>> extensible and scalable advanced security analytics tool. It has strong
>> foundations in the Apache Hadoop ecosystem.
>>
>> == Proposal ==
>>
>> Metron integrates a variety of open source big data technologies in
>>order
>> to offer a centralized tool for security monitoring and analysis. Metron
>> provides capabilities for log aggregation, full packet capture indexing,
>> storage, advanced behavioral analytics and data enrichment, while
>>applying
>> the most current threat-intelligence information to security telemetry
>> within a single platform.
>>
>> Metron can be divided into 4 areas:
>>
>>   1. '''A mechanism to capture, store, and normalize any type of
>>security
>> telemetry at extremely high rates.''' Because security telemetry is
>> constantly being generated, it requires a method for ingesting the data
>>at
>> high speeds and pushing it to various processing units for advanced
>> computation and analytics.
>>   1. '''Real time processing and application of enrichments''' such as
>> threat intelligence, geolocation, and DNS information to telemetry being
>> collected. The immediate application of this information to incoming
>> telemetry provides the context and situational awareness, as well as the
>> ³who² and ³where² information that is critical for investigation.
>>   1. '''Efficient information storage''' based on how the information
>>will
>> be used:
>>     a. Logs and telemetry are stored such that they can be efficiently
>> mined and analyzed for concise security visibility
>>     a. The ability to extract and reconstruct full packets helps an
>>analyst
>> answer questions such as who the true attacker was, what data was
>>leaked,
>> and where that data was sent
>>     a. Long-term storage not only increases visibility over time, but
>>also
>> enables advanced analytics such as machine learning techniques to be
>>used
>> to create models on the information. Incoming data can then be scored
>> against these stored models for advanced anomaly detection.
>>   1. '''An interface that gives a security investigator a centralized
>>view
>> of data and alerts passed through the system.''' Metron¹s interface
>> presents alert summaries with threat intelligence and enrichment data
>> specific to that alert on one single page. Furthermore, advanced search
>> capabilities and full packet extraction tools are presented to the
>>analyst
>> for investigation without the need to pivot into additional tools.
>>
>> Big data is a natural fit for powerful security analytics. The Metron
>> framework integrates a number of elements from the Hadoop ecosystem to
>> provide a scalable platform for security analytics, incorporating such
>> functionality as full-packet capture, stream processing, batch
>>processing,
>> real-time search, and telemetry aggregation. With Metron, our goal is to
>> tie big data into security analytics and drive towards an extensible
>> centralized platform to effectively enable rapid detection and rapid
>> response for advanced security threats.
>>
>> == Background ==
>>
>> OpenSOC was developed by Cisco over the last two years and pushed out to
>> Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
>> development was mostly closed and has largely stopped. As evidence of
>>the
>> inactivity, users have complained that pull requests are not answered
>>for a
>> while
>> 
>>https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ.
>> Finally, no public releases of OpenSOC have been made. From an Apache
>>point
>> of view, the current community is not viable.
>>
>> However, some of the developers of the project have left Cisco and have
>> found interest from several others that would like to work together to
>>form
>> an active and open community at Apache starting from the current OpenSOC
>> code base. A message to the current support group proposing moving to
>> Apache got a single positive response.
>> https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
>>
>> In general Apache accepts only voluntary contributions and avoids
>> hostile forks. In this case, given that the community is demonstrably
>> dead, it seems fair to fork the existing code at Apache to allow a new
>> community to work on it. Once incubation starts, we will send a
>> message pointing to the new home to the OpenSOC support group.
>>
>> Because Cisco is not currently interested in being involved, the project
>> expects to change their name. The project would like to use Metron,
>> although we will perform a podling name search to check for conflicts.
>> Metron, meaning measure, is half of the greek root for the word
>> 'telemetry.'  Metron is also a DC Comics character who ³... wanders in
>> search of greater knowledge beyond his own².
>>
>>
>> == Rationale ==
>> Metron strives to move the state of the art in security analytics
>>forward.
>> We want to move away from the proprietary nature of legacy security
>>point
>> tools and develop an open platform where people can contribute and share
>> datasets, machine learning models, telemetry parsers, sources of
>>telemetry
>> enrichment, and threat intelligence feeds.  Cyber security is too large
>>of
>> a problem for a single corporation to tackle on its own and the current
>> tooling is too fragmented and proprietary for us to be able to rally
>>around
>> a single tool or vendor.
>>
>> In addition to being open and facilitating advancement in security
>> analytics, Metron has several advantages over a conventional Security
>> Information Management System (SIEM).
>>
>>   * Metron uses all open source stack under the hood and runs on
>>commodity
>> hardware.  This means Metron is much cheaper to run then the
>>competition.
>> In security cost plays a major factor because the cost of your
>> countermeasure for monitoring and reacting to a threat should not exceed
>> the cost of what is being protected.  By driving down the cost of
>>security
>> the economics works for more assets to be monitored, which means more
>> secure data centers.
>>   * Metron, being in the open, allows additional vetting and scrutiny by
>> the open source community for all of its components.  This is a better
>> model for a security-oriented tool than doing it closed source.  All the
>> problems should be flushed out and fixed in the open. The closed source
>> competition does not have this kind of rigor, is motivated by marketing
>>and
>> sales, and thus, does not inspire confidence when it comes to security.
>>   * Being Hadoop-based, Metron can process unprecedented volumes of
>> streaming data via Apache Storm.  When an organization is hit with
>>malware
>> or malicious behavior most commonly this happens as a part of a global
>> malware campaign, signatures for which are known and are available from
>> third party threat intelligence feeds.  Having the ability to take in
>>all
>> the feeds and reference them against every telemetry message processed
>>by
>> Metron in real time does not only facilitate detection of such
>>campaigns,
>> it changes the economics for the ³bad guys².  If you have to customize
>>your
>> malware for each of your targets these global attacks become a lot more
>> expensive and non viable for them.
>>   * Metron strives to shift conventional SOC workflows away from being
>> rules-driven to a more data-driven approach that incorporates machine
>> learning and a higher degree of automation and autonomous detection.
>>The
>> modern threat landscape is too dynamic to be manageable via static rules
>> alone, which is what conventional SIEMs rely on.  Rule bases tend to
>>bloat,
>> and if improperly maintained turn themselves into sources of false
>>positive
>> alerts.
>>
>> The ability to analyze and model large volumes of data at rest and then
>> being able to push up the output of that into a stream processor is
>> essential in disrupting the
>>
>> == Current Status ==
>>
>> As stated in the background section, the current community isn¹t
>>healthy,
>> which is why we are proposing moving to Apache Incubator. In this
>>section,
>> we will describe the current state of the OpenSOC project.
>>
>> === Meritocracy ===
>> The OpenSOC development is controlled by Cisco and pull requests are
>>being
>> ignored. The development list is private and requests to join are
>>rejected
>> because there is no activity on it. The goal of moving to Apache is to
>>form
>> a meritocracy where a variety of individuals, regardless of their
>>current
>> employer, come together and work together. We understand that diversity,
>> open development, and open governance are critical to being a successful
>> Apache project.
>>
>> === Community ===
>> The OpenSOC project is not responding to pull requests or making
>>releases.
>> The easiest solution would be to create a variety of forks of the
>>project
>> on github, but that would further fracture the community and prevent it
>> from reaching critical mass. Our prefered solution is to build a single
>> large diverse and open community at Apache.
>>
>> === Core Developers ===
>> The core developers of Metron are James Sirota, Charles Porter, and Mark
>> Bittmann. None of them have experience running an open source project,
>>but
>> they are eager to learn.
>>
>> === Alignment ===
>> The ASF is a natural host for Metron given that it is already the home
>>of
>> Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>> projects. Metron leverages many of Apache open-source products. We are
>>very
>> interested in a place to develop our community and integrations with the
>> other Apache big data projects.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> The current product developers are all salaried developers at a small
>> number of companies and thus there is a risk of becoming an orphaned
>> product. However, the companies view Metron as very important to their
>> product offering and plan to ramp up their work in the space. The
>>project
>> is unique in the product space and thus has strong potential to become a
>> sustainable community.
>>
>> === Inexperience with Open Source ===
>> The vast majority of the developers are inexperienced with open source
>> development and the Apache Way. One of the major hurdles to graduation
>>from
>> the Apache Incubator will be demonstrating that they have learned the
>> Apache Way and are applying it to how the project is managed. Vinod
>>Kumar
>> Vavilapalli is an Apache Member and plans on actively working as a
>> committer in the project. They also have the other mentors to help them
>> learn as they progress.
>>
>> === Homogenous Developers ===
>> The developers are employed by four diverse companies (B23, Hortonworks,
>> Mantech, and Rackspace), They are distributed across the United States.
>>We
>> hope to attract additional diversity as an Apache project.
>>
>> === Reliance on Salaried Developers ===
>> Metron is currently being developed exclusively by salaried developers,
>>but
>> the goal of coming to Apache is to form a community of users and
>>developers
>> that is much more diverse including non-salaried developers.
>>
>> === Relationships with Other Apache Products ===
>> Metron has a strong relationship and dependency with Apache Flume,
>>Hadoop,
>> HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
>> community could help with a closer collaboration among these projects
>>and
>> as well as others.
>>
>> We note that although there is a superficial resemblance to Apache
>>Eagle,
>> which does security analysis of Hadoop audit events, the projects are
>> significantly different. In particular, Metron is focused on analyzing
>> network packet traffic and thus has a very different scope and scale of
>> events than Eagle.
>>
>> === An Excessive Fascination with the Apache Brand ===
>>
>> While the Apache brand is important, we are much more interested in
>>finding
>> a home for the project that encourages open development and open
>> governance. We want to form the new community using the Apache Way with
>>its
>> strong focus on meritocracy, organizational independence, and open
>> development.
>>
>> == Documentation ==
>> The current information on the OpenSOC project is here:
>> http://opensoc.github.io/
>> A slide deck presenting background material is here:
>> http://www.slideshare.net/JamesSirota/cisco-opensoc
>>
>> == Initial Source ==
>> The initial code is on github:  http://opensoc.github.io/
>>
>> == External Dependencies ==
>> Metron has the following external dependencies:
>>   * Apache Flume
>>   * Apache Hadoop
>>   * Apache HBase
>>   * Apache Hive
>>   * Apache Kafka
>>   * Apache Spark
>>   * Apache Storm
>>   * ElasticSearch
>>   * MySQL
>>
>> The project understands that it will need to support alternatives for
>>MySQL
>> that are licensed under a ALv2 compatible license.
>>
>> == Cryptography ==
>> Metron will eventually support encryption on the wire, but this is not
>>one
>> of the initial goals, and we do not expect Metron to be a controlled
>>export
>> item due to the use of encryption. Metron supports but does not require
>>the
>> Kerberos authentication mechanism to access secured Hadoop services.
>>
>> == Required Resources ==
>>
>> === Mailing List ===
>>
>>   * metron-private for private PMC discussions
>>   * metron-dev for developers
>>   * metron-commits for all commits
>>   * metron-users for all users
>>
>> === Version Control ===
>> Git is the preferred source control system.
>>
>> === Issue Tracking ===
>>
>>   * JIRA (METRON)
>>
>> === Other Resources ===
>> The existing code already has unit tests so we will make use of existing
>> Apache continuous testing infrastructure. The resulting load should not
>>be
>> very large.
>>
>> == Initial Committers ==
>>   * Jim Baker < jim.baker at rackspace dot com >
>>   * Mark Bittmann < mark at b23 dot io >
>>   * Sheetal Dolas < sheetal at hortonworks dot com >
>>   * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>>   * P. Taylor Goetz < ptgoetz at apache dot org >
>>   * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>>   * Dave Hirko < dave at b23 dot io >
>>   * Paul Kehrer < paul.kehrer at rackspace dot com >
>>   * Brad Kolarov < brad at b23 dot io >
>>   * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>>   * Larry McCay < lmccay at appache.org >
>>   * Ryan Merriman < rmerriman at hortonworks dot com >
>>   * Michael Perez < michael.perez at hortonworks dot com>
>>   * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>>   * Phillip Rhodes < motley.crue.fan at gmail dot com >
>>   * Sean Schulte < sean.schulte at rackspace dot com >
>>   * James Sirota < jsirota at hortonworks dot com >
>>   * Casey Stella < cstella at hortonworks dot com >
>>   * Bryan Taylor < bryan.taylor at rackspace dot com >
>>   * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>>   * George Vetticaden < gvetticaden at hortonworks dot com >
>>   * Oskar Zabik < oskar.zabik at rackspace dot com >
>>
>> == Affiliations ==
>> The initial committers are employees of:
>>   * Jim Baker - Rackspace
>>   * Mark Bittmann - B23
>>   * Sheetal Dolas - Hortonworks
>>   * Discovery Gerdes - Rackspace
>>   * P. Taylor Goetz - Hortonworks
>>   * Andrew Hartnett - Rackspace
>>   * Dave Hirko - B23
>>   * Paul Kehrer - Rackspace
>>   * Brad Kolarov - B23
>>   * Kiran Komaravolu - Hortonworks
>>   * Larry McCay - Hortonworks
>>   * Ryan Merriman - Hortonworks
>>   * Michael Perez - Hortonworks
>>   * Charles Porter - Mantech
>>   * Phillip Rhodes - Fogbeam Labs
>>   * Sean Schulte - Rackspace
>>   * James Sirota - Hortonworks
>>   * Casey Stella - Hortonworks
>>   * Bryan Taylor - Rackspace
>>   * Ray Urciuoli - Mantech
>>   * Vinod Kumar Vavilapalli - Hortonworks
>>   * George Vetticaden - Hortonworks
>>   * Oskar Zabik - Rackspace
>>
>> == Sponsors ==
>>
>> === Champion ===
>>   * Owen O¹Malley - Apache IPMC member
>>
>> === Nominated Mentors ===
>>   * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>> NASA
>>   * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>> Hortonworks
>>   * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>> member, Hortonworks
>>
>> === Sponsoring Entity ===
>> We are requesting the Incubator to sponsor this project.
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message