Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of mail@ulerch.net designates
 78.138.113.6 as permitted sender)
Subject: Re: [PROPOSAL] ALOIS Project
From: Urs Lerch <mail@ulerch.net>
To: general@incubator.apache.org
In-Reply-To: <1281774714.1985.219.camel@X61s>
References: <1281774714.1985.219.camel@X61s>
Content-Type: multipart/signed; micalg="pgp-sha1";
 protocol="application/pgp-signature"; boundary="=-xXNMydxsbifugoiHoJ3s"
Date: Mon, 23 Aug 2010 10:06:19 +0200
Message-ID: <1282550779.1884.34.camel@X61s>
Mime-Version: 1.0

--=-xXNMydxsbifugoiHoJ3s
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi

A little more than a week ago I posted the proposal concerning the
incubation of ALOIS. Unfortunatly there wasn't much of a discussion so
far. Furthermore, we are still looking for mentors. Any feedback is
welcome!

Best
Urs

P.S.: http://wiki.apache.org/incubator/AloisProposal.


Am Samstag, den 14.08.2010, 10:31 +0200 schrieb Urs Lerch:
> Greetings All
>=20
> I would like to formally propose that the ALOIS Project be considered
> for inclusion in the ASF Incubator as a new podling. ALOIS is a log
> collection and correlation software with reporting and alarming
> functionalities (a so-called SIEM). The full details of this proposal
> are available below. I hope that the length of the text doesn't prevent
> you from reading it.
>=20
> Furthermore, we are looking for Mentors, and any additional contributors
> that we can get. And we gracefully ask the Incubator PMC for sponsoring
> this project.
>=20
> We were happy to receive your feedback about our proposal.
>=20
> Best regards
> Urs
>=20
>=20
>=20
> Here the full text of the proposal:
>=20
>=20
> =3D Preface =3D
>=20
> ALOIS is a log collection and correlation software with reporting and
> alarming functionalities. It has been implemented by the Swiss company
> IMSEC for a customer about five years ago. GPL-licenced, implemented in
> Ruby and completely based on other OSS-licensed components, it was
> designed for the open source community right from the start. Now that
> the software has shown its functioning over several years in production
> with the one customer and one IMSEC-internal installation, it seems to
> be the right time to open it to a wider community.
>=20
>=20
> =3D Abstract =3D
>=20
> ALOIS stands for =E2=80=9EAdvanced Logging and Intrusion Detection System=
=E2=80=9C and
> is meant to be a fully implemented open source SIEM (security
> information and event management) system.
>=20
>=20
> =3D Proposal =3D
>=20
> While almost all other SIEM software, be it closed or open source,
> concentrate on the technological part of security monitoring, ALOIS is
> aimed to monitor the security of the content. It intends to be
> pro-active in the detection of potential loss, theft, mistaken
> modification or unauthorized access. ALOIS works on log messages and
> thus contains all the basic functionality of a conventional SIEM, as
> centralized collecting, normalizing, aggregation, analyzing and
> correlating of all log messages, as well as reporting all security
> related events. Therefore it can be used as any other SIEM.
>=20
> ALOIS consists of five modules interacting to ensure a scaleable
> functionality of a SIEM:
>=20
>   * Insink is the message sink, which is the receiving entry point for
> all the different log messages into ALOIS. It is partly based on the
> syslog-ng software. Insink listens for messages (UDP), waits for
> messages (TCP), receives message collections (files, emails) and
> pre-filters them to prevent from message flow overload.
>        =20
>   * Pumpy is the incoming FIFO buffer, implemented as a relational
> database tables. which contain the incoming original messages (in raw
> format). In a complex system setup, there may be several insink
> instances, e.g. for a group of hosts, for specific types of messages, or
> for high-avaliablity.
>        =20
>   * Prisma contains logic to split up the text of log messages into
> separate fields, based on regular expressions. Actually, "prisma" is a
> set of "prismi", each one prisma for one type of log message (apache,
> cisco etc. Several prismi can be applied to the same message. This
> allows for stacked messages, i.e. forwarded log messages contained in
> compressed files contained in e-mail messages. The data retrieved form
> the log messages is stored in a database called Dobby. Due to prisma
> being written in Ruby, prismi can be applied interactively (when having
> system access).
>        =20
>   * Dobby is the central log database. It should be separated from the
> Pumpy database for availability and performance reasons. The current
> implementation is based on MySQL.
>        =20
>   * The Analyzer contains the two sub-systems Lizard and Reptor. Lizard
> is the analysis engine and user interface of ALOIS, implemented in Ruby
> on Rails using AJAX. It allows for interactive browsing through the
> collected data, exclusion/inclusion/selection of data, data sorting,
> data filtering, creation of views, ad-hoc textual and graphical
> reporting. Reptor allows for automatic activation of views and
> comparison of these views' results to a predefined result (pattern
> matching). In case of mismatch, Reptor sends the result to predefined
> e-mail addresses.
>=20
> Its modular design guarantees ALOIS to scale from little to large
> organizations. Since there exists a Debian package, it's easy to build a
> test system or even a productive system for small environments.
>=20
> Although the software has been in productive use for a few years, there
> is still a lot of desired functionality missing. The plugability of new
> connected systems is given, but needs some revision. It is a given goal
> of the project to allow modules in other programming language.
> Furthermore, it has been discussed if parts of the existing
> implementation may be replaced with other proven open source software,
> e.g. the correlation engine or the web frontend. The other way round, it
> has been discussed that the filter creation engine would make a good
> tool for any kind of structured data, and thus could be separated from
> ALOIS and standardized as a stand-alone tool.
>=20
>=20
> =3D Background =3D
>=20
> It's not simple to know what happens in a bigger network. There's a
> multitude of applications, services and appliances working together.
> Many of them provide some kind of events or state information. The
> network administrator needs to get hands on all of them. But they come
> in many different flavors and multiple canals. Therefore, it's hard to
> get the big picture. Furthermore, we have learned that it's impossible
> to protect a system against all malicious attacks and to keep all the
> possible faulty handling away. A monitoring of the systems to guarantee
> a pro-active handling is therefore needed..
>=20
> Therefore, more and more organizations collect and analyze all logfiles
> in a centralized system, called a SIEM (security information and event
> management). The technology provides two major functions for security
> events from networks, systems and applications: log management and
> compliance reporting (SIM =E2=80=93 security information management) and
> real-time monitoring and incident management (SEM =E2=80=93 security even=
t
> management).
>=20
>=20
> =3D Rationale =3D
>=20
> Why another security information and event management system? It's true,
> there's already plenty of them. While the proprietary software is way
> too expensive for smaller to mid-sized companies, we find that the open
> source solutions are either too simple or not completely open. For
> example, behind each of the well known systems =E2=80=9COSSIM=E2=80=9D an=
d =E2=80=9CPrelude=E2=80=9D,
> there is a company that either closes central functionality for its own
> business or has dual licensing and therefore asks the full copyright for
> all contributed code.
>=20
> ALOIS is aimed to be totally free and open for all contributions. The
> openness provided for other programming languages is certainly proof of
> this. The plug-ability - yet to be further developed - is meant to
> guarantee that individual needs can be realized without stressing the
> whole system too much. In our opinion, the Linux kernel is a good
> example that this can work very well.
>=20
> Since we are in accordance with =E2=80=9Ethe Apache way=E2=80=9C, we woul=
d be very
> pleased if ALOIS could become part of the Apache community. In Addition,
> the Apache Logging Services would be a perfect home for the software.
> Furthermore, it's not the intention to compete with the already existing
> log viewer and analyzing tool =E2=80=9EChainsaw=E2=80=9C. Since Chainsaw =
is a relatively
> easy tool, it meets a rather different need. Nevertheless, if the two
> projects use synergies, both can profit.
>=20
>=20
> =3D Initial Goals =3D
>=20
> When this project started ins 2005, there was no proven SIEM open source
> software and the commercial tools were way too expensive for the needed
> environment. Therefore, we decided together with a customer of ours to
> implement an open source SIEM tool from scratch. Now the software has
> run in a production environment for several years and has proven its
> functionality and reliabilty.
>=20
>=20
> =3D Current Status =3D
>=20
> =3D=3D Meritocracy =3D=3D
>=20
> As already mentioned, ALOIS is already in production use in two
> organizations. All the code has been written by two persons of the same
> company in a paid employment relationship. It is obvious that this is
> way different from the open source approach within Apache. But
> nevertheless, the two developers have always worked as a team and the
> decisions were made in consensus whenever possible. But it is no secret,
> that these developers have to learn to behave in an open community.
> Understanding this potential problem, they already got support by a
> freelance consulter, who has the corresponding experience and knowledge.
>=20
> =3D=3D Community =3D=3D
>=20
> Until today there is no real community, because the project hasn't been
> published officially, although it had been completely published on the
> web site for a couple of months (until a server relaunch). Convinced by
> the concept and design of the software, we are open and hope to reach
> many contributors and users. We think that it is realistic, because the
> SIEM issue has yet not been resolved in the OSS space.
>=20
> =3D=3D Core Developers =3D=3D
>=20
> ALOIS was developed by Simon H=C3=BCrliman and Flavio Pellanda, both empl=
oyed
> by the company IMSEC. Concerning Design and Architecture, Marcus
> Holthaus, owner of IMSEC, gave his input as security specialist. Since
> the beginning of this year, Urs Lerch, a doctorate on the subject of
> commercial open source software development, supports the team with his
> knowledge. Simon H=C3=BCrlimann has left the company three years ago, but=
 is
> still active in the OSS environment (although not for ALOIS). Current
> employee Daniel Lutz (a Debian Developer) has also contributed to the
> project.
>=20
> =3D=3D Alignment =3D=3D
>=20
> Besides that we strongly believe in the =E2=80=9EApache way=E2=80=9C, we =
think that
> although that Apache hosts the Logging Services and different security
> projects, there is a gap when it comes to a superordinate security view.
> We therefore think it a good idea to add our SIEM project to the Apache
> repository. On the other side, Apache would become an even more complete
> software repository.
>=20
>=20
> =3D Known Risks =3D
>=20
> =3D=3D Orphaned products =3D=3D
>=20
> Since the software is only maintained by employers of one company, there
> is a severe risk of being orphaned. But, on the one hand, the company
> has a sustained interest in keeping the project alive, because there are
> plans to offer services on top of ALOIS, and IMSEC uses the software for
> SIEM on their own systems. For this reason there exists a budget for the
> development and support of ALOIS. On the other hand, we believe that
> ALOIS is of great interest for other people and companies tied to IT
> security. Therefore, our step to the Apache incubator is also a step to
> a bigger community.
>=20
> =3D=3D Inexperience with Open Source =3D=3D
>=20
> While ALOIS has always been licenced under the GPL, access to the source
> code, bug tracker and version control system has been restricted to
> internal users for most of the time. But the company has a strong
> believe in the open source movement and therefore engages its employees
> to take part in the community. Furthermore, it is also a strategic
> decision to build services on top of linux.
>=20
> We understand that the Apache Incubator is a great opportunity for us to
> get assistance, when it comes to specific questions on the open source
> development. Even more, the company has created a part time position for
> the open source community work.
>=20
> =3D=3D Homogenous Developers =3D=3D
>=20
> Although ALOIS has been developed by employees of only one company,
> there is a thorough openness. The company is designed to stay small and
> therefore works with several independent partners. Furthermore, its
> employees work in geographically different parts of the country.
> Therefore, it is no new experience for the developers to work in a
> distributed environment and argue rather than to command. Already today
> the employees are enforced to document all face-to-face communication in
> the internal wiki. Sketches are photographed and stored in the project's
> digital folder.
>=20
> =3D=3D Reliance on Salaried Developers =3D=3D
>=20
> Until today all the development of ALOIS has been made in a paid
> emplyoment. Therefore we know that this brings a significant danger.
> Since it is our stated aim to encourage participation and recruit
> commiters, we hope to eliminate this risk as soon as possible.
> Furthermore, the employees of IMSEC are all open source enthusiasts and
> are in one way or another active in the community. Although we have no
> certainty, there is good indication that the current commiters would
> continue their work on ALOIS, even if they wouldn't be paid for it.
>=20
> =3D=3D Relationships with Other Apache Products =3D=3D
>=20
> The Apache Logging Service would be a perfect home for ALOIS as a
> centralized logging collection and analyzing tool. Furthermore, we think
> that we could share part of the code with the Chainsaw subproject, since
> both need similar functionality in the web frontend. Since it is our
> statet aim to replace our own code with proofen open source libraries,
> we are open for any collaboration with other projects. For example, the
> replacement of the MySQL with a NoSQL database might be useful for
> performance reasons; therefore HBase is a good candidate.
>=20
> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D
>=20
> The Apache brand is in fact for its own a very good reason to join the
> Incubator. But much more our desire to become part of the Apache
> Incubator is our strong believe in open source software in general and
> in the =E2=80=9EApache way=E2=80=9C in particular. We would love to learn=
 from the
> experience and knowledge of the foundation's members and participants,
> which is an important part of the brand as well. The foundation has
> shown many times, that it has the processes and people to succeed in
> launching a project. We would be very proud to be part of this success
> story.
>=20
>=20
> =3D Documentation =3D
>=20
> The documentation is rather weak and scattered. It has mainly been
> maintained on a wiki and is open to improvement. Since we are totally
> aware that this is a killer for a successfull open source project, we
> have already started an internal project with its own budget to improve
> this shortcomming. Once the project has been launched, writing a blog or
> open a forum are other possibilities we already thought of.
>=20
> Furthermore, as the employees are used to work in a geographycally
> distributed environment, a lot of the internal communication happens in
> a chat. Thus, opening a new chat channel for the community is scheduled.
> (To document the discussions for all those who were off-line, we would
> send the logs daily to the mailing list.)
>=20
>=20
> =3D Initial Source =3D
>=20
> Although the initial source comes from a project for a customer. it has
> an open source licence since the beginning. Therefore it doesn't have
> any propriatary code in it. A thorough revision before releasing it to a
> public repository is recommend and is also in planning.
>=20
> The initial source will be a snapshot of the version control system,
> accompanied by a related debian package.
>=20
>=20
> =3D Source and Intellectual Property Submission Plan =3D
>=20
> ALOIS is currently under a GPL licence. Since there are only two
> contributors so far, both from the same company, there is no problem to
> re-licence the code and contribute it to Apache. The commitment of the
> company's owner has been granted.
>=20
>=20
> =3D External Dependencies =3D
>=20
> So far, no external dependencies are known. As mentioned before, a
> thorough revision of the codebase is in planning. There it can be
> controlled, that no other licence is affected by the code.
>=20
>=20
> =3D Cryptography =3D
>=20
> ALOIS does not involve cryptographic code.
>=20
>=20
> =3D Required Resources =3D
>=20
> =3D=3D Mailing lists =3D=3D
>=20
> The following mailing lists will be required:
>=20
>   * alois-private
>   * alois-dev
>   * alois-commits
>   * alois-users
>=20
> =3D=3D Subversion Directory =3D=3D
>=20
> https://svn.apache.org/repos/asf/incubator/alois
>=20
> =3D=3D Issue Tracking =3D=3D
>=20
> JIRA ALOIS (ALOIS)
>=20
> =3D=3D Other Resources =3D=3D
>=20
> We would like to open a chat channel. If this isn't possible within the
> infrastructure of Apache, we would love to do this in our own already
> existing infrastructure.
>=20
>=20
> =3D Initial Commiters =3D
>=20
>   * NAME             EMAIL                              AFFILIATION  CLA
>   * Flavio Pellanda  flavio.pellanda at logintas dot ch IMSEC        no
>   * Urs Lerch        mail at ulerch dot net             IMSEC        no
>   * Daniel Lutz      daniel.lutz at logintas dot ch     IMSEC        no
>   * Marcus Holthaus  marcus.holthaus at imsec dot ch    IMSEC        no
>=20
>=20
> =3D Sponsors =3D
>=20
> =3D=3D Champion =3D=3D
>=20
>   * Scott Deboy <sdeboy at apache dot org>
>=20
> =3D=3D Nominated Mentors =3D=3D
>=20
>   * Scott Deboy <sdeboy at apache dot org>
>=20
> =3D=3D Sponsoring Entity =3D=3D
>=20
> The Incubator PMC (requested)
>=20


--=-xXNMydxsbifugoiHoJ3s
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Dies ist ein digital signierter Nachrichtenteil

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQEcBAABAgAGBQJMciv4AAoJEP9+V4XAg0Zpp1sIAIRBfMxfkry2CdERWh0p8FKn
VC4aM3lTKQlvpJTrKLQTIARhatnR2yHOkJmR9ERngFWCvLsWbxzJlFreia6y2qfs
BO2cv6pDHqDtSwNz0hevrFOKZ9Oj0vQzh/VkE/eL0D1xUWlLpxzxLpUQJhKrCraT
USgeKrp8cf2SwJi2iz1fTfw9mYYHy3xUWT4CqTgcC01rJVHo/Q2ySXA5uFrIMJIF
Qk5sZ/U43SNeGSQtC+ij7RP89RePSlySQRr5FxEFbYpKzEs+Du/089SFq270j+DA
j1T6iCxQpra1mIsae2EqNChAWBESb5AoOpe34ROFKEm1pghtlKDXDDDytxgc8VY=
=LO26
-----END PGP SIGNATURE-----

--=-xXNMydxsbifugoiHoJ3s--