incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Urs Lerch <m...@ulerch.net>
Subject Re: Real-time communication (was [VOTE] ALOIS to enter the incubator)
Date Fri, 17 Sep 2010 13:41:10 GMT
Hi

To cut a long story short: ALOIS is _not_ about a chat channel, it's a
tool for security incident and event management. Since the chat channell
in the required resources list was only a wish, I gladly dropped it off
the proposal.

Best
Urs


Am Freitag, den 17.09.2010, 07:32 -0400 schrieb Jim Jagielski:
> I still don't see how that gets around the perception, and the
> reality, that development is being done outside the list.
> 
> So I don't see that proposal as helping out at all...
> 
> On Sep 16, 2010, at 3:27 PM, Scott Deboy wrote:
> 
> > I understand the concern raised by the use of real-time communication for
> > Apache projects - that decisions may be made off-list, and that folks who
> > aren't a party to the real-time communication do not have the opportunity to
> > benefit from or impact the decisions that result from the real-time
> > communication.
> > 
> > The proposal does offer what seems to be a reasonable compromise: 'we would
> > send the logs daily to the mailing list.'
> > 
> > Daily chat logs posted to the dev list, coupled with good mentoring and
> > guidance that decisions need to be made on the mailing list, would seem to
> > minimize the risk.
> > 
> > I'm interested in what others think of their proposal for supporting
> > real-time communication, and curious what others are doing, if anything, to
> > support the growing interest in real-time communication between project
> > participants.
> > 
> > Scott
> > 
> > 
> > On Thu, Sep 16, 2010 at 10:49 AM, Craig L Russell
> > <craig.russell@oracle.com>wrote:
> > 
> >> Hi Urs,
> >> 
> >> My only concern is the request to have a chat channel. There's wide use of
> >> chat channels in Apache (the periodic board and members' meetings make use
> >> of them, and infrastructure uses channels to advantage).
> >> 
> >> But for an incubating project, I'd strongly discourage use of chat as a
> >> communication channel.
> >> 
> >> +1
> >> 
> >> Craig
> >> 
> >> 
> >> On Aug 26, 2010, at 9:09 AM, Urs Lerch wrote:
> >> 
> >> Hi,
> >>> 
> >>> I would like to call a vote for accepting "ALOIS" for incubation in
> >>> the Apache Incubator. The full proposal is available below and on the
> >>> proposal wiki page (http://wiki.apache.org/incubator/AloisProposal).  We
> >>> ask the Incubator PMC to sponsor it, with Scott Deboy volunteering as
> >>> Champion and Mentor. Additional mentors are warmly welcome.
> >>> 
> >>> Please cast your vote:
> >>> 
> >>> [ ] +1, bring ALOIS into Incubator
> >>> [ ] +0, I don't care either way,
> >>> [ ] -1, do not bring ALOIS into Incubator, because...
> >>> 
> >>> This vote will be open for 72 hours and only votes from the Incubator
> >>> PMC are binding.
> >>> 
> >>> Thanks,
> >>> Urs
> >>> 
> >>> 
> >>> --------------------------------------------
> >>> 
> >>> 
> >>> = Preface =
> >>> 
> >>> ALOIS is a log collection and correlation software with reporting and
> >>> alarming functionalities. It has been implemented by the Swiss company
> >>> IMSEC for a customer about five years ago. GPL-licenced, implemented in
> >>> Ruby and completely based on other OSS-licensed components, it was
> >>> designed for the open source community right from the start. Now that
> >>> the software has shown its functioning over several years in production
> >>> with the one customer and one IMSEC-internal installation, it seems to
> >>> be the right time to open it to a wider community.
> >>> 
> >>> 
> >>> = Abstract =
> >>> 
> >>> ALOIS stands for „Advanced Logging and Intrusion Detection System“ and
> >>> is meant to be a fully implemented open source SIEM (security
> >>> information and event management) system.
> >>> 
> >>> 
> >>> = Proposal =
> >>> 
> >>> While almost all other SIEM software, be it closed or open source,
> >>> concentrate on the technological part of security monitoring, ALOIS is
> >>> aimed to monitor the security of the content. It intends to be
> >>> pro-active in the detection of potential loss, theft, mistaken
> >>> modification or unauthorized access. ALOIS works on log messages and
> >>> thus contains all the basic functionality of a conventional SIEM, as
> >>> centralized collecting, normalizing, aggregation, analyzing and
> >>> correlating of all log messages, as well as reporting all security
> >>> related events. Therefore it can be used as any other SIEM.
> >>> 
> >>> ALOIS consists of five modules interacting to ensure a scaleable
> >>> functionality of a SIEM:
> >>> 
> >>> * Insink is the message sink, which is the receiving entry point for
> >>> all the different log messages into ALOIS. It is partly based on the
> >>> syslog-ng software. Insink listens for messages (UDP), waits for
> >>> messages (TCP), receives message collections (files, emails) and
> >>> pre-filters them to prevent from message flow overload.
> >>> 
> >>> * Pumpy is the incoming FIFO buffer, implemented as a relational
> >>> database tables. which contain the incoming original messages (in raw
> >>> format). In a complex system setup, there may be several insink
> >>> instances, e.g. for a group of hosts, for specific types of messages, or
> >>> for high-avaliablity.
> >>> 
> >>> * Prisma contains logic to split up the text of log messages into
> >>> separate fields, based on regular expressions. Actually, "prisma" is a
> >>> set of "prismi", each one prisma for one type of log message (apache,
> >>> cisco etc. Several prismi can be applied to the same message. This
> >>> allows for stacked messages, i.e. forwarded log messages contained in
> >>> compressed files contained in e-mail messages. The data retrieved form
> >>> the log messages is stored in a database called Dobby. Due to prisma
> >>> being written in Ruby, prismi can be applied interactively (when having
> >>> system access).
> >>> 
> >>> * Dobby is the central log database. It should be separated from the
> >>> Pumpy database for availability and performance reasons. The current
> >>> implementation is based on MySQL.
> >>> 
> >>> * The Analyzer contains the two sub-systems Lizard and Reptor. Lizard
> >>> is the analysis engine and user interface of ALOIS, implemented in Ruby
> >>> on Rails using AJAX. It allows for interactive browsing through the
> >>> collected data, exclusion/inclusion/selection of data, data sorting,
> >>> data filtering, creation of views, ad-hoc textual and graphical
> >>> reporting. Reptor allows for automatic activation of views and
> >>> comparison of these views' results to a predefined result (pattern
> >>> matching). In case of mismatch, Reptor sends the result to predefined
> >>> e-mail addresses.
> >>> 
> >>> Its modular design guarantees ALOIS to scale from little to large
> >>> organizations. Since there exists a Debian package, it's easy to build a
> >>> test system or even a productive system for small environments.
> >>> 
> >>> Although the software has been in productive use for a few years, there
> >>> is still a lot of desired functionality missing. The plugability of new
> >>> connected systems is given, but needs some revision. It is a given goal
> >>> of the project to allow modules in other programming language.
> >>> Furthermore, it has been discussed if parts of the existing
> >>> implementation may be replaced with other proven open source software,
> >>> e.g. the correlation engine or the web frontend. The other way round, it
> >>> has been discussed that the filter creation engine would make a good
> >>> tool for any kind of structured data, and thus could be separated from
> >>> ALOIS and standardized as a stand-alone tool.
> >>> 
> >>> 
> >>> = Background =
> >>> 
> >>> It's not simple to know what happens in a bigger network. There's a
> >>> multitude of applications, services and appliances working together.
> >>> Many of them provide some kind of events or state information. The
> >>> network administrator needs to get hands on all of them. But they come
> >>> in many different flavors and multiple canals. Therefore, it's hard to
> >>> get the big picture. Furthermore, we have learned that it's impossible
> >>> to protect a system against all malicious attacks and to keep all the
> >>> possible faulty handling away. A monitoring of the systems to guarantee
> >>> a pro-active handling is therefore needed..
> >>> 
> >>> Therefore, more and more organizations collect and analyze all logfiles
> >>> in a centralized system, called a SIEM (security information and event
> >>> management). The technology provides two major functions for security
> >>> events from networks, systems and applications: log management and
> >>> compliance reporting (SIM – security information management) and
> >>> real-time monitoring and incident management (SEM – security event
> >>> management).
> >>> 
> >>> 
> >>> = Rationale =
> >>> 
> >>> Why another security information and event management system? It's true,
> >>> there's already plenty of them. While the proprietary software is way
> >>> too expensive for smaller to mid-sized companies, we find that the open
> >>> source solutions are either too simple or not completely open. For
> >>> example, behind each of the well known systems “OSSIM” and “Prelude”,
> >>> there is a company that either closes central functionality for its own
> >>> business or has dual licensing and therefore asks the full copyright for
> >>> all contributed code.
> >>> 
> >>> ALOIS is aimed to be totally free and open for all contributions. The
> >>> openness provided for other programming languages is certainly proof of
> >>> this. The plug-ability - yet to be further developed - is meant to
> >>> guarantee that individual needs can be realized without stressing the
> >>> whole system too much. In our opinion, the Linux kernel is a good
> >>> example that this can work very well.
> >>> 
> >>> Since we are in accordance with „the Apache way“, we would be very
> >>> pleased if ALOIS could become part of the Apache community. In Addition,
> >>> the Apache Logging Services would be a perfect home for the software.
> >>> Furthermore, it's not the intention to compete with the already existing
> >>> log viewer and analyzing tool „Chainsaw“. Since Chainsaw is a relatively
> >>> easy tool, it meets a rather different need. Nevertheless, if the two
> >>> projects use synergies, both can profit.
> >>> 
> >>> 
> >>> = Initial Goals =
> >>> 
> >>> When this project started ins 2005, there was no proven SIEM open source
> >>> software and the commercial tools were way too expensive for the needed
> >>> environment. Therefore, we decided together with a customer of ours to
> >>> implement an open source SIEM tool from scratch. Now the software has
> >>> run in a production environment for several years and has proven its
> >>> functionality and reliabilty.
> >>> 
> >>> 
> >>> = Current Status =
> >>> 
> >>> == Meritocracy ==
> >>> 
> >>> As already mentioned, ALOIS is already in production use in two
> >>> organizations. All the code has been written by two persons of the same
> >>> company in a paid employment relationship. It is obvious that this is
> >>> way different from the open source approach within Apache. But
> >>> nevertheless, the two developers have always worked as a team and the
> >>> decisions were made in consensus whenever possible. But it is no secret,
> >>> that these developers have to learn to behave in an open community.
> >>> Understanding this potential problem, they already got support by a
> >>> freelance consulter, who has the corresponding experience and knowledge.
> >>> 
> >>> == Community ==
> >>> 
> >>> Until today there is no real community, because the project hasn't been
> >>> published officially, although it had been completely published on the
> >>> web site for a couple of months (until a server relaunch). Convinced by
> >>> the concept and design of the software, we are open and hope to reach
> >>> many contributors and users. We think that it is realistic, because the
> >>> SIEM issue has yet not been resolved in the OSS space.
> >>> 
> >>> == Core Developers ==
> >>> 
> >>> ALOIS was developed by Simon Hürliman and Flavio Pellanda, both employed
> >>> by the company IMSEC. Concerning Design and Architecture, Marcus
> >>> Holthaus, owner of IMSEC, gave his input as security specialist. Since
> >>> the beginning of this year, Urs Lerch, a doctorate on the subject of
> >>> commercial open source software development, supports the team with his
> >>> knowledge. Simon Hürlimann has left the company three years ago, but is
> >>> still active in the OSS environment (although not for ALOIS). Current
> >>> employee Daniel Lutz (a Debian Developer) has also contributed to the
> >>> project.
> >>> 
> >>> == Alignment ==
> >>> 
> >>> Besides that we strongly believe in the „Apache way“, we think that
> >>> although that Apache hosts the Logging Services and different security
> >>> projects, there is a gap when it comes to a superordinate security view.
> >>> We therefore think it a good idea to add our SIEM project to the Apache
> >>> repository. On the other side, Apache would become an even more complete
> >>> software repository.
> >>> 
> >>> 
> >>> = Known Risks =
> >>> 
> >>> == Orphaned products ==
> >>> 
> >>> Since the software is only maintained by employers of one company, there
> >>> is a severe risk of being orphaned. But, on the one hand, the company
> >>> has a sustained interest in keeping the project alive, because there are
> >>> plans to offer services on top of ALOIS, and IMSEC uses the software for
> >>> SIEM on their own systems. For this reason there exists a budget for the
> >>> development and support of ALOIS. On the other hand, we believe that
> >>> ALOIS is of great interest for other people and companies tied to IT
> >>> security. Therefore, our step to the Apache incubator is also a step to
> >>> a bigger community.
> >>> 
> >>> == Inexperience with Open Source ==
> >>> 
> >>> While ALOIS has always been licenced under the GPL, access to the source
> >>> code, bug tracker and version control system has been restricted to
> >>> internal users for most of the time. But the company has a strong
> >>> believe in the open source movement and therefore engages its employees
> >>> to take part in the community. Furthermore, it is also a strategic
> >>> decision to build services on top of linux.
> >>> 
> >>> We understand that the Apache Incubator is a great opportunity for us to
> >>> get assistance, when it comes to specific questions on the open source
> >>> development. Even more, the company has created a part time position for
> >>> the open source community work.
> >>> 
> >>> == Homogenous Developers ==
> >>> 
> >>> Although ALOIS has been developed by employees of only one company,
> >>> there is a thorough openness. The company is designed to stay small and
> >>> therefore works with several independent partners. Furthermore, its
> >>> employees work in geographically different parts of the country.
> >>> Therefore, it is no new experience for the developers to work in a
> >>> distributed environment and argue rather than to command. Already today
> >>> the employees are enforced to document all face-to-face communication in
> >>> the internal wiki. Sketches are photographed and stored in the project's
> >>> digital folder.
> >>> 
> >>> == Reliance on Salaried Developers ==
> >>> 
> >>> Until today all the development of ALOIS has been made in a paid
> >>> emplyoment. Therefore we know that this brings a significant danger.
> >>> Since it is our stated aim to encourage participation and recruit
> >>> commiters, we hope to eliminate this risk as soon as possible.
> >>> Furthermore, the employees of IMSEC are all open source enthusiasts and
> >>> are in one way or another active in the community. Although we have no
> >>> certainty, there is good indication that the current commiters would
> >>> continue their work on ALOIS, even if they wouldn't be paid for it.
> >>> 
> >>> == Relationships with Other Apache Products ==
> >>> 
> >>> The Apache Logging Service would be a perfect home for ALOIS as a
> >>> centralized logging collection and analyzing tool. Furthermore, we think
> >>> that we could share part of the code with the Chainsaw subproject, since
> >>> both need similar functionality in the web frontend. Since it is our
> >>> statet aim to replace our own code with proofen open source libraries,
> >>> we are open for any collaboration with other projects. For example, the
> >>> replacement of the MySQL with a NoSQL database might be useful for
> >>> performance reasons; therefore HBase is a good candidate.
> >>> 
> >>> == An Excessive Fascination with the Apache Brand ==
> >>> 
> >>> The Apache brand is in fact for its own a very good reason to join the
> >>> Incubator. But much more our desire to become part of the Apache
> >>> Incubator is our strong believe in open source software in general and
> >>> in the „Apache way“ in particular. We would love to learn from the
> >>> experience and knowledge of the foundation's members and participants,
> >>> which is an important part of the brand as well. The foundation has
> >>> shown many times, that it has the processes and people to succeed in
> >>> launching a project. We would be very proud to be part of this success
> >>> story.
> >>> 
> >>> 
> >>> = Documentation =
> >>> 
> >>> The documentation is rather weak and scattered. It has mainly been
> >>> maintained on a wiki and is open to improvement. Since we are totally
> >>> aware that this is a killer for a successfull open source project, we
> >>> have already started an internal project with its own budget to improve
> >>> this shortcomming. Once the project has been launched, writing a blog or
> >>> open a forum are other possibilities we already thought of.
> >>> 
> >>> Furthermore, as the employees are used to work in a geographycally
> >>> distributed environment, a lot of the internal communication happens in
> >>> a chat. Thus, opening a new chat channel for the community is scheduled.
> >>> (To document the discussions for all those who were off-line, we would
> >>> send the logs daily to the mailing list.)
> >>> 
> >>> 
> >>> = Initial Source =
> >>> 
> >>> Although the initial source comes from a project for a customer. it has
> >>> an open source licence since the beginning. Therefore it doesn't have
> >>> any propriatary code in it. A thorough revision before releasing it to a
> >>> public repository is recommend and is also in planning.
> >>> 
> >>> The initial source will be a snapshot of the version control system,
> >>> accompanied by a related debian package.
> >>> 
> >>> 
> >>> = Source and Intellectual Property Submission Plan =
> >>> 
> >>> ALOIS is currently under a GPL licence. Since there are only two
> >>> contributors so far, both from the same company, there is no problem to
> >>> re-licence the code and contribute it to Apache. The commitment of the
> >>> company's owner has been granted.
> >>> 
> >>> 
> >>> = External Dependencies =
> >>> 
> >>> So far, no external dependencies are known. As mentioned before, a
> >>> thorough revision of the codebase is in planning. There it can be
> >>> controlled, that no other licence is affected by the code.
> >>> 
> >>> 
> >>> = Cryptography =
> >>> 
> >>> ALOIS does not involve cryptographic code.
> >>> 
> >>> 
> >>> = Required Resources =
> >>> 
> >>> == Mailing lists ==
> >>> 
> >>> The following mailing lists will be required:
> >>> 
> >>> * alois-private
> >>> * alois-dev
> >>> * alois-commits
> >>> * alois-users
> >>> 
> >>> == Subversion Directory ==
> >>> 
> >>> https://svn.apache.org/repos/asf/incubator/alois
> >>> 
> >>> == Issue Tracking ==
> >>> 
> >>> JIRA ALOIS (ALOIS)
> >>> 
> >>> == Other Resources ==
> >>> 
> >>> We would like to open a chat channel. If this isn't possible within the
> >>> infrastructure of Apache, we would love to do this in our own already
> >>> existing infrastructure.
> >>> 
> >>> 
> >>> = Initial Commiters =
> >>> 
> >>> * NAME             EMAIL                              AFFILIATION  CLA
> >>> * Flavio Pellanda  flavio.pellanda at logintas dot ch IMSEC        no
> >>> * Urs Lerch        mail at ulerch dot net             IMSEC        no
> >>> * Daniel Lutz      daniel.lutz at logintas dot ch     IMSEC        no
> >>> * Marcus Holthaus  marcus.holthaus at imsec dot ch    IMSEC        no
> >>> 
> >>> 
> >>> = Sponsors =
> >>> 
> >>> == Champion ==
> >>> 
> >>> * Scott Deboy <sdeboy at apache dot org>
> >>> 
> >>> == Nominated Mentors ==
> >>> 
> >>> * Scott Deboy <sdeboy at apache dot org>
> >>> 
> >>> == Sponsoring Entity ==
> >>> 
> >>> The Incubator PMC (requested) b
> >>> 
> >> 
> >> Craig L Russell
> >> Architect, Oracle
> >> http://db.apache.org/jdo
> >> 408 276-5638 mailto:Craig.Russell@oracle.com
> >> P.S. A good JDO? O, Gasp!
> >> 
> >> 
> >> 
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >> 
> >> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


Mime
View raw message