incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: [VOTE] Accept Metron into Apache Incubator
Date Fri, 04 Dec 2015 02:41:40 GMT
I am also working on my contacts to find the right person.



On Fri, Dec 4, 2015 at 8:34 AM, P. Taylor Goetz <ptgoetz@gmail.com> wrote:

> +1 (binding)
>
> I think it would be prudent to at least try to get an SGA from Cisco, but
> it looks like Debo is trying to help out with that.
>
> -Taylor
>
> > On Dec 3, 2015, at 12:33 PM, Owen O'Malley <omalley@apache.org> wrote:
> >
> > The [DISCUSS] thread has would down, so I'd like to start a VOTE on
> whether
> > Apache Incubator should accept Metron as a podling. The proposal is
> pasted
> > below and is available on the wiki as well.
> >
> > https://wiki.apache.org/incubator/MetronProposal
> >
> > We've added a paragraph in the background section discussing how Apache
> > avoids hostile forks of projects, because we don't want to fork
> > communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
> > Rhodes to the proposal.
> >
> > The vote will run until 12pm PST on Sunday.
> >
> > Thanks,
> >   Owen
> >
> > = Apache Metron Proposal =
> >
> > ----
> > /!\ '''FINAL''' /!\
> >
> > This proposal is now complete and has been submitted for a VOTE.
> > ----
> >
> > == Abstract ==
> >
> > The Metron project is an open source project dedicated to providing an
> > extensible and scalable advanced security analytics tool. It has strong
> > foundations in the Apache Hadoop ecosystem.
> >
> > == Proposal ==
> >
> > Metron integrates a variety of open source big data technologies in order
> > to offer a centralized tool for security monitoring and analysis. Metron
> > provides capabilities for log aggregation, full packet capture indexing,
> > storage, advanced behavioral analytics and data enrichment, while
> applying
> > the most current threat-intelligence information to security telemetry
> > within a single platform.
> >
> > Metron can be divided into 4 areas:
> >
> >  1. '''A mechanism to capture, store, and normalize any type of security
> > telemetry at extremely high rates.''' Because security telemetry is
> > constantly being generated, it requires a method for ingesting the data
> at
> > high speeds and pushing it to various processing units for advanced
> > computation and analytics.
> >  1. '''Real time processing and application of enrichments''' such as
> > threat intelligence, geolocation, and DNS information to telemetry being
> > collected. The immediate application of this information to incoming
> > telemetry provides the context and situational awareness, as well as the
> > “who” and “where” information that is critical for investigation.
> >  1. '''Efficient information storage''' based on how the information will
> > be used:
> >    a. Logs and telemetry are stored such that they can be efficiently
> > mined and analyzed for concise security visibility
> >    a. The ability to extract and reconstruct full packets helps an
> analyst
> > answer questions such as who the true attacker was, what data was leaked,
> > and where that data was sent
> >    a. Long-term storage not only increases visibility over time, but also
> > enables advanced analytics such as machine learning techniques to be used
> > to create models on the information. Incoming data can then be scored
> > against these stored models for advanced anomaly detection.
> >  1. '''An interface that gives a security investigator a centralized view
> > of data and alerts passed through the system.''' Metron’s interface
> > presents alert summaries with threat intelligence and enrichment data
> > specific to that alert on one single page. Furthermore, advanced search
> > capabilities and full packet extraction tools are presented to the
> analyst
> > for investigation without the need to pivot into additional tools.
> >
> > Big data is a natural fit for powerful security analytics. The Metron
> > framework integrates a number of elements from the Hadoop ecosystem to
> > provide a scalable platform for security analytics, incorporating such
> > functionality as full-packet capture, stream processing, batch
> processing,
> > real-time search, and telemetry aggregation. With Metron, our goal is to
> > tie big data into security analytics and drive towards an extensible
> > centralized platform to effectively enable rapid detection and rapid
> > response for advanced security threats.
> >
> > == Background ==
> >
> > OpenSOC was developed by Cisco over the last two years and pushed out to
> > Github (https://github.com/OpenSOC/opensoc) under the ALv2. However, the
> > development was mostly closed and has largely stopped. As evidence of the
> > inactivity, users have complained that pull requests are not answered
> for a
> > while
> > https://groups.google.com/d/msg/opensoc-support/R2W-ZFux8Vk/Y-5tL-EmAAAJ
> .
> > Finally, no public releases of OpenSOC have been made. From an Apache
> point
> > of view, the current community is not viable.
> >
> > However, some of the developers of the project have left Cisco and have
> > found interest from several others that would like to work together to
> form
> > an active and open community at Apache starting from the current OpenSOC
> > code base. A message to the current support group proposing moving to
> > Apache got a single positive response.
> > https://groups.google.com/d/msg/opensoc-support/rFlW2uSSvmU/09PIsWL4AAAJ
> >
> > In general Apache accepts only voluntary contributions and avoids
> > hostile forks. In this case, given that the community is demonstrably
> > dead, it seems fair to fork the existing code at Apache to allow a new
> > community to work on it. Once incubation starts, we will send a
> > message pointing to the new home to the OpenSOC support group.
> >
> > Because Cisco is not currently interested in being involved, the project
> > expects to change their name. The project would like to use Metron,
> > although we will perform a podling name search to check for conflicts.
> > Metron, meaning measure, is half of the greek root for the word
> > 'telemetry.'  Metron is also a DC Comics character who “... wanders in
> > search of greater knowledge beyond his own”.
> >
> >
> > == Rationale ==
> > Metron strives to move the state of the art in security analytics
> forward.
> > We want to move away from the proprietary nature of legacy security point
> > tools and develop an open platform where people can contribute and share
> > datasets, machine learning models, telemetry parsers, sources of
> telemetry
> > enrichment, and threat intelligence feeds.  Cyber security is too large
> of
> > a problem for a single corporation to tackle on its own and the current
> > tooling is too fragmented and proprietary for us to be able to rally
> around
> > a single tool or vendor.
> >
> > In addition to being open and facilitating advancement in security
> > analytics, Metron has several advantages over a conventional Security
> > Information Management System (SIEM).
> >
> >  * Metron uses all open source stack under the hood and runs on commodity
> > hardware.  This means Metron is much cheaper to run then the competition.
> > In security cost plays a major factor because the cost of your
> > countermeasure for monitoring and reacting to a threat should not exceed
> > the cost of what is being protected.  By driving down the cost of
> security
> > the economics works for more assets to be monitored, which means more
> > secure data centers.
> >  * Metron, being in the open, allows additional vetting and scrutiny by
> > the open source community for all of its components.  This is a better
> > model for a security-oriented tool than doing it closed source.  All the
> > problems should be flushed out and fixed in the open. The closed source
> > competition does not have this kind of rigor, is motivated by marketing
> and
> > sales, and thus, does not inspire confidence when it comes to security.
> >  * Being Hadoop-based, Metron can process unprecedented volumes of
> > streaming data via Apache Storm.  When an organization is hit with
> malware
> > or malicious behavior most commonly this happens as a part of a global
> > malware campaign, signatures for which are known and are available from
> > third party threat intelligence feeds.  Having the ability to take in all
> > the feeds and reference them against every telemetry message processed by
> > Metron in real time does not only facilitate detection of such campaigns,
> > it changes the economics for the “bad guys”.  If you have to customize
> your
> > malware for each of your targets these global attacks become a lot more
> > expensive and non viable for them.
> >  * Metron strives to shift conventional SOC workflows away from being
> > rules-driven to a more data-driven approach that incorporates machine
> > learning and a higher degree of automation and autonomous detection.  The
> > modern threat landscape is too dynamic to be manageable via static rules
> > alone, which is what conventional SIEMs rely on.  Rule bases tend to
> bloat,
> > and if improperly maintained turn themselves into sources of false
> positive
> > alerts.
> >
> > The ability to analyze and model large volumes of data at rest and then
> > being able to push up the output of that into a stream processor is
> > essential in disrupting the
> >
> > == Current Status ==
> >
> > As stated in the background section, the current community isn’t healthy,
> > which is why we are proposing moving to Apache Incubator. In this
> section,
> > we will describe the current state of the OpenSOC project.
> >
> > === Meritocracy ===
> > The OpenSOC development is controlled by Cisco and pull requests are
> being
> > ignored. The development list is private and requests to join are
> rejected
> > because there is no activity on it. The goal of moving to Apache is to
> form
> > a meritocracy where a variety of individuals, regardless of their current
> > employer, come together and work together. We understand that diversity,
> > open development, and open governance are critical to being a successful
> > Apache project.
> >
> > === Community ===
> > The OpenSOC project is not responding to pull requests or making
> releases.
> > The easiest solution would be to create a variety of forks of the project
> > on github, but that would further fracture the community and prevent it
> > from reaching critical mass. Our prefered solution is to build a single
> > large diverse and open community at Apache.
> >
> > === Core Developers ===
> > The core developers of Metron are James Sirota, Charles Porter, and Mark
> > Bittmann. None of them have experience running an open source project,
> but
> > they are eager to learn.
> >
> > === Alignment ===
> > The ASF is a natural host for Metron given that it is already the home of
> > Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
> > projects. Metron leverages many of Apache open-source products. We are
> very
> > interested in a place to develop our community and integrations with the
> > other Apache big data projects.
> >
> > == Known Risks ==
> >
> > === Orphaned Products ===
> >
> > The current product developers are all salaried developers at a small
> > number of companies and thus there is a risk of becoming an orphaned
> > product. However, the companies view Metron as very important to their
> > product offering and plan to ramp up their work in the space. The project
> > is unique in the product space and thus has strong potential to become a
> > sustainable community.
> >
> > === Inexperience with Open Source ===
> > The vast majority of the developers are inexperienced with open source
> > development and the Apache Way. One of the major hurdles to graduation
> from
> > the Apache Incubator will be demonstrating that they have learned the
> > Apache Way and are applying it to how the project is managed. Vinod Kumar
> > Vavilapalli is an Apache Member and plans on actively working as a
> > committer in the project. They also have the other mentors to help them
> > learn as they progress.
> >
> > === Homogenous Developers ===
> > The developers are employed by four diverse companies (B23, Hortonworks,
> > Mantech, and Rackspace), They are distributed across the United States.
> We
> > hope to attract additional diversity as an Apache project.
> >
> > === Reliance on Salaried Developers ===
> > Metron is currently being developed exclusively by salaried developers,
> but
> > the goal of coming to Apache is to form a community of users and
> developers
> > that is much more diverse including non-salaried developers.
> >
> > === Relationships with Other Apache Products ===
> > Metron has a strong relationship and dependency with Apache Flume,
> Hadoop,
> > HBase, Hive, Kafka, Spark, and Storm. Being part of Apache’s Incubation
> > community could help with a closer collaboration among these projects and
> > as well as others.
> >
> > We note that although there is a superficial resemblance to Apache Eagle,
> > which does security analysis of Hadoop audit events, the projects are
> > significantly different. In particular, Metron is focused on analyzing
> > network packet traffic and thus has a very different scope and scale of
> > events than Eagle.
> >
> > === An Excessive Fascination with the Apache Brand ===
> >
> > While the Apache brand is important, we are much more interested in
> finding
> > a home for the project that encourages open development and open
> > governance. We want to form the new community using the Apache Way with
> its
> > strong focus on meritocracy, organizational independence, and open
> > development.
> >
> > == Documentation ==
> > The current information on the OpenSOC project is here:
> > http://opensoc.github.io/
> > A slide deck presenting background material is here:
> > http://www.slideshare.net/JamesSirota/cisco-opensoc
> >
> > == Initial Source ==
> > The initial code is on github:  http://opensoc.github.io/
> >
> > == External Dependencies ==
> > Metron has the following external dependencies:
> >  * Apache Flume
> >  * Apache Hadoop
> >  * Apache HBase
> >  * Apache Hive
> >  * Apache Kafka
> >  * Apache Spark
> >  * Apache Storm
> >  * ElasticSearch
> >  * MySQL
> >
> > The project understands that it will need to support alternatives for
> MySQL
> > that are licensed under a ALv2 compatible license.
> >
> > == Cryptography ==
> > Metron will eventually support encryption on the wire, but this is not
> one
> > of the initial goals, and we do not expect Metron to be a controlled
> export
> > item due to the use of encryption. Metron supports but does not require
> the
> > Kerberos authentication mechanism to access secured Hadoop services.
> >
> > == Required Resources ==
> >
> > === Mailing List ===
> >
> >  * metron-private for private PMC discussions
> >  * metron-dev for developers
> >  * metron-commits for all commits
> >  * metron-users for all users
> >
> > === Version Control ===
> > Git is the preferred source control system.
> >
> > === Issue Tracking ===
> >
> >  * JIRA (METRON)
> >
> > === Other Resources ===
> > The existing code already has unit tests so we will make use of existing
> > Apache continuous testing infrastructure. The resulting load should not
> be
> > very large.
> >
> > == Initial Committers ==
> >  * Jim Baker < jim.baker at rackspace dot com >
> >  * Mark Bittmann < mark at b23 dot io >
> >  * Sheetal Dolas < sheetal at hortonworks dot com >
> >  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
> >  * P. Taylor Goetz < ptgoetz at apache dot org >
> >  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
> >  * Dave Hirko < dave at b23 dot io >
> >  * Paul Kehrer < paul.kehrer at rackspace dot com >
> >  * Brad Kolarov < brad at b23 dot io >
> >  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
> >  * Larry McCay < lmccay at appache.org >
> >  * Ryan Merriman < rmerriman at hortonworks dot com >
> >  * Michael Perez < michael.perez at hortonworks dot com>
> >  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
> >  * Phillip Rhodes < motley.crue.fan at gmail dot com >
> >  * Sean Schulte < sean.schulte at rackspace dot com >
> >  * James Sirota < jsirota at hortonworks dot com >
> >  * Casey Stella < cstella at hortonworks dot com >
> >  * Bryan Taylor < bryan.taylor at rackspace dot com >
> >  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
> >  * George Vetticaden < gvetticaden at hortonworks dot com >
> >  * Oskar Zabik < oskar.zabik at rackspace dot com >
> >
> > == Affiliations ==
> > The initial committers are employees of:
> >  * Jim Baker - Rackspace
> >  * Mark Bittmann - B23
> >  * Sheetal Dolas - Hortonworks
> >  * Discovery Gerdes - Rackspace
> >  * P. Taylor Goetz - Hortonworks
> >  * Andrew Hartnett - Rackspace
> >  * Dave Hirko - B23
> >  * Paul Kehrer - Rackspace
> >  * Brad Kolarov - B23
> >  * Kiran Komaravolu - Hortonworks
> >  * Larry McCay - Hortonworks
> >  * Ryan Merriman - Hortonworks
> >  * Michael Perez - Hortonworks
> >  * Charles Porter - Mantech
> >  * Phillip Rhodes - Fogbeam Labs
> >  * Sean Schulte - Rackspace
> >  * James Sirota - Hortonworks
> >  * Casey Stella - Hortonworks
> >  * Bryan Taylor - Rackspace
> >  * Ray Urciuoli - Mantech
> >  * Vinod Kumar Vavilapalli - Hortonworks
> >  * George Vetticaden - Hortonworks
> >  * Oskar Zabik - Rackspace
> >
> > == Sponsors ==
> >
> > === Champion ===
> >  * Owen O’Malley - Apache IPMC member
> >
> > === Nominated Mentors ===
> >  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
> NASA
> >  * Owen O’Malley < omalley at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
> > Hortonworks
> >  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
> > member, Hortonworks
> >
> > === Sponsoring Entity ===
> > We are requesting the Incubator to sponsor this project.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message