incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John D. Ament" <johndam...@apache.org>
Subject Re: [PROPOSAL] Onyx - proposal for Apache Incubation
Date Wed, 31 Jan 2018 12:50:25 GMT
Sorry for mid-posting.

This isn't the list to determine if a project name is suitable.  There's a
JIRA project dedicated to that, and if you need a quick answer better to
email trademarks@ to get a more precise answer.

The question is really going to be, is "Apache Onyx" going to be easily
confused with something else.

John

On Sun, Jan 28, 2018 at 4:50 AM Byung-Gon Chun <bgchun@gmail.com> wrote:

> Thank you for all the information! It looks like Surf doesn't work.
>
> If possible, we'd like to keep Onyx.
> Another name we came up with is Coral.
>
> Thanks!
> -Gon
>
>
> On Sun, Jan 28, 2018 at 4:21 AM, Leif Hedstrom <zwoop@apache.org> wrote:
>
> > Did we rule out Onyx for sure? Just because some other project might use
> > it on say github doesn’t necessarily exclude us from having an Apache
> Onyx?
> >
> > FWIW, I agree that surf is too similar in pronunciation to Apache serf.
> :)
> >
> > Cheers,
> >
> > — Leif
> >
> > > On Jan 27, 2018, at 07:31, Dave Fisher <dave2wave@comcast.net> wrote:
> > >
> > > Checking “Serf Software” which sounds the same.
> > >
> > > (1) there is already Apache Serf
> > > (2) Serf is a product from Hashicorp at https://www.serf.io/. This
> > would definitely confuse as it is apparently comparable to ZooKeeper.
> > >
> > > Regards,
> > > Dave
> > >
> > > Sent from my iPhone
> > >
> > >> On Jan 27, 2018, at 3:12 AM, sebb <sebbaz@gmail.com> wrote:
> > >>
> > >> A brief search for 'Surf Software' shows quite a few hits.
> > >> I have not looked to see if they would be likely to be confused with
> > >> this project or cause problems for others.
> > >>
> > >> But it as though there might be a problem:
> > >> Surfer -  Golden Software
> > >> surf @ sourceforge
> > >> Surf Software company
> > >>
> > >>
> > >>> On 27 January 2018 at 08:03, Byung-Gon Chun <bgchun@gmail.com>
> wrote:
> > >>> Since we cannot use the name Onyx, we would like to change the
> project
> > name
> > >>> to Surf.
> > >>> I hope that this name works.
> > >>>
> > >>> -Gon
> > >>>
> > >>> ---
> > >>> Byung-Gon Chun
> > >>>
> > >>>
> > >>>> On Sat, Jan 27, 2018 at 4:57 AM, Byung-Gon Chun <bgchun@gmail.com>
> > wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Sat, Jan 27, 2018 at 4:09 AM, Davor Bonaci <davor@apache.org>
> > wrote:
> > >>>>>
> > >>>>> Great work -- I think this technology has a lot of promise,
and I'd
> > love
> > >>>>> to
> > >>>>> see its evolution inside the Foundation.
> > >>>>>
> > >>>>>
> > >>>> Thanks, Davor!
> > >>>>
> > >>>>
> > >>>>> Parts of it, like the Onyx Intermediate Representation [1],
overlap
> > with
> > >>>>> the work-in-progress inside the Apache Beam project
> ("portability").
> > We'd
> > >>>>> love to work together on this -- would you be open to such
> > collaboration?
> > >>>>> If so, it may not be necessary to start from scratch, and leverage
> > the
> > >>>>> work
> > >>>>> already done.
> > >>>>>
> > >>>>>
> > >>>> Sure. We're open to collaboration.
> > >>>>
> > >>>>
> > >>>>> Regarding the name, Onyx would likely have to be renamed, due
to a
> > >>>>> conflict
> > >>>>> with a related technology [2].
> > >>>>>
> > >>>>>
> > >>>> Thanks for pointing it out. It's difficult to come up with a good
> > short
> > >>>> name. :)
> > >>>> Do you have any suggestion?
> > >>>>
> > >>>> Thanks!
> > >>>> -Gon
> > >>>>
> > >>>> ---
> > >>>> Byung-Gon Chun
> > >>>>
> > >>>>
> > >>>>
> > >>>>> Davor
> > >>>>>
> > >>>>> [1] https://snuspl.github.io/onyx/docs/ir/
> > >>>>> [2] http://www.onyxplatform.org/
> > >>>>>
> > >>>>>> On Thu, Jan 25, 2018 at 3:28 PM, Byung-Gon Chun <bgchun@gmail.com
> >
> > wrote:
> > >>>>>>
> > >>>>>> Dear Apache Incubator Community,
> > >>>>>>
> > >>>>>> Please accept the following proposal for presentation and
> > discussion:
> > >>>>>> https://wiki.apache.org/incubator/OnyxProposal
> > >>>>>>
> > >>>>>> Onyx is a data processing system that aims to flexibly
control the
> > >>>>> runtime
> > >>>>>> behaviors of a job to adapt to varying deployment characteristics
> > (e.g.,
> > >>>>>> harnessing transient resources in datacenters, cross-datacenter
> > >>>>> deployment,
> > >>>>>> changing runtime based on job characteristics, etc.). Onyx
> provides
> > >>>>> ways to
> > >>>>>> extend the system’s capabilities and incorporate the
extensions to
> > the
> > >>>>>> flexible job execution.
> > >>>>>> Onyx translates a user program (e.g., Apache Beam, Apache
Spark)
> > into an
> > >>>>>> Intermediate Representation (IR) DAG, which Onyx optimizes
and
> > deploys
> > >>>>>> based on a deployment policy.
> > >>>>>>
> > >>>>>> I've attached the proposal below.
> > >>>>>>
> > >>>>>> Best regards,
> > >>>>>> Byung-Gon Chun
> > >>>>>>
> > >>>>>> = OnyxProposal =
> > >>>>>>
> > >>>>>> == Abstract ==
> > >>>>>> Onyx is a data processing system for flexible employment
with
> > >>>>>> different execution scenarios for various deployment
> characteristics
> > >>>>>> on clusters.
> > >>>>>>
> > >>>>>> == Proposal ==
> > >>>>>> Today, there is a wide variety of data processing systems
with
> > >>>>>> different designs for better performance and datacenter
> efficiency.
> > >>>>>> They include processing data on specific resource environments
and
> > >>>>>> running jobs with specific attributes. Although each system
> > >>>>>> successfully solves the problems it targets, most systems
are
> > designed
> > >>>>>> in the way that runtime behaviors are built tightly inside
the
> > system
> > >>>>>> core to hide the complexity of distributed computing. This
makes
> it
> > >>>>>> hard for a single system to support different deployment
> > >>>>>> characteristics with different runtime behaviors without
> substantial
> > >>>>>> effort.
> > >>>>>>
> > >>>>>> Onyx is a data processing system that aims to flexibly
control the
> > >>>>>> runtime behaviors of a job to adapt to varying deployment
> > >>>>>> characteristics. Moreover, it provides a means of extending
the
> > >>>>>> system’s capabilities and incorporating the extensions
to the
> > flexible
> > >>>>>> job execution.
> > >>>>>>
> > >>>>>> In order to be able to easily modify runtime behaviors
to adapt to
> > >>>>>> varying deployment characteristics, Onyx exposes runtime
behaviors
> > to
> > >>>>>> be flexibly configured and modified at both compile-time
and
> runtime
> > >>>>>> through a set of high-level graph pass interfaces.
> > >>>>>>
> > >>>>>> We hope to contribute to the big data processing community
by
> > enabling
> > >>>>>> more flexibility and extensibility in job executions. Furthermore,
> > we
> > >>>>>> can benefit more together as a community when we work together
as
> a
> > >>>>>> community to mature the system with more use cases and
> understanding
> > >>>>>> of diverse deployment characteristics. The Apache Software
> > Foundation
> > >>>>>> is the perfect place to achieve these aspirations.
> > >>>>>>
> > >>>>>> == Background ==
> > >>>>>> Many data processing systems have distinctive runtime behaviors
> > >>>>>> optimized and configured for specific deployment characteristics
> > like
> > >>>>>> different resource environments and for handling special
job
> > >>>>>> attributes.
> > >>>>>>
> > >>>>>> For example, much research have been conducted to overcome
the
> > >>>>>> challenge of running data processing jobs on cheap, unreliable
> > >>>>>> transient resources. Likewise, techniques for disaggregating
> > different
> > >>>>>> types of resources, like memory, CPU and GPU, are being
actively
> > >>>>>> developed to use datacenter resources more efficiently.
Many
> > >>>>>> researchers are also working to run data processing jobs
in even
> > more
> > >>>>>> diverse environments, such as across distant datacenters.
> Similarly,
> > >>>>>> for special job attributes, many works take different approaches,
> > such
> > >>>>>> as runtime optimization, to solve problems like data skew,
and to
> > >>>>>> optimize systems for data processing jobs with small-scale
input
> > data.
> > >>>>>>
> > >>>>>> Although each of the systems performs well with the jobs
and in
> the
> > >>>>>> environments they target, they perform poorly with unconsidered
> > cases,
> > >>>>>> and do not consider supporting multiple deployment characteristics
> > on
> > >>>>>> a single system in their designs.
> > >>>>>>
> > >>>>>> For an application writer to optimize an application to
perform
> well
> > >>>>>> on a certain system engraved with its underlying behaviors,
it
> > >>>>>> requires a deep understanding of the system itself, which
is an
> > >>>>>> overhead that often requires a lot of time and effort.
Moreover,
> > for a
> > >>>>>> developer to modify such system behaviors, it requires
> modifications
> > >>>>>> of the system core, which requires an even deeper understanding
of
> > the
> > >>>>>> system itself.
> > >>>>>>
> > >>>>>> With this background, Onyx is designed to represent all
of its
> jobs
> > as
> > >>>>>> an Intermediate Representation (IR) DAG. In the Onyx compiler,
> user
> > >>>>>> applications from various programming models (ex. Apache
Beam) are
> > >>>>>> submitted, transformed to an IR DAG, and optimized/customized
for
> > the
> > >>>>>> deployment characteristics. In the IR DAG optimization
phase, the
> > DAG
> > >>>>>> is modified through a series of compiler “passes” which
reshape or
> > >>>>>> annotate the DAG with an expression of the underlying runtime
> > >>>>>> behaviors. The IR DAG is then submitted as an execution
plan for
> the
> > >>>>>> Onyx runtime. The runtime includes the unmodified parts
of data
> > >>>>>> processing in the backbone which is transparently integrated
with
> > >>>>>> configurable components exposed for further extension.
> > >>>>>>
> > >>>>>> == Rationale ==
> > >>>>>> Onyx’s vision lies in providing means for flexibly supporting
a
> wide
> > >>>>>> variety of job execution scenarios for users while facilitating
> > system
> > >>>>>> developers to extend the execution framework with various
> > >>>>>> functionalities at the same time. The capabilities of the
system
> can
> > >>>>>> be extended as it grows to meet a more variety of execution
> > scenarios.
> > >>>>>> We require inputs from users and developers from diverse
domains
> in
> > >>>>>> order to make it a more thriving and useful project. The
Apache
> > >>>>>> Software Foundation provides the best tools and community
to
> support
> > >>>>>> this vision.
> > >>>>>>
> > >>>>>> == Initial Goals ==
> > >>>>>> Initial goals will be to move the existing codebase to
Apache and
> > >>>>>> integrate with the Apache development process. We further
plan to
> > >>>>>> develop our system to meet the needs for more execution
scenarios
> > for
> > >>>>>> a more variety of deployment characteristics.
> > >>>>>>
> > >>>>>> == Current Status ==
> > >>>>>> Onyx codebase is currently hosted in a repository at github.com.
> > The
> > >>>>>> current version has been developed by system developers
at Seoul
> > >>>>>> National University, Viva Republica, Samsung, and LG.
> > >>>>>>
> > >>>>>> == Meritocracy ==
> > >>>>>> We plan to strongly support meritocracy. We will discuss
the
> > >>>>>> requirements in an open forum, and those that continuously
> > contribute
> > >>>>>> to Onyx with the passion to strengthen the system will
be invited
> as
> > >>>>>> committers. Contributors that enrich Onyx by providing
various use
> > >>>>>> cases, various implementations of the configurable components
> > >>>>>> including ideas for optimization techniques will be especially
> > >>>>>> welcome. Committers with a deep understanding of the system’s
> > >>>>>> technical aspects as a whole and its philosophy will definitely
be
> > >>>>>> voted as the PMC. We will monitor community participation
so that
> > >>>>>> privileges can be extended to those that contribute.
> > >>>>>>
> > >>>>>> == Community ==
> > >>>>>> We hope to expand our contribution community by becoming
an Apache
> > >>>>>> incubator project. The contributions will come from both
users and
> > >>>>>> system developers interested in flexibility and extensibility
of
> job
> > >>>>>> executions that Onyx can support. We expect users to mainly
> > contribute
> > >>>>>> to diversify the use cases and deployment characteristics,
and
> > >>>>>> developers to  contribute to implement them.
> > >>>>>>
> > >>>>>> == Alignment ==
> > >>>>>> Apache Spark is one of many popular data processing frameworks.
> The
> > >>>>>> system is designed towards optimizing jobs using RDDs in
memory
> and
> > >>>>>> many other optimizations built tightly within the framework.
In
> > >>>>>> contrast to Spark, Onyx aims to provide more flexibility
for job
> > >>>>>> execution in an easy manner.
> > >>>>>>
> > >>>>>> Apache Tez enables developers to build complex task DAGs
with
> > control
> > >>>>>> over the control plane of job execution. In Onyx, a high-level
> > >>>>>> programming layer (ex. Apache Beam) is automatically converted
to
> a
> > >>>>>> basic IR DAG and can be converted to any IR DAG through
a series
> of
> > >>>>>> easy user writable passes, that can both reshape and modify
the
> > >>>>>> annotation (of execution properties) of the DAG. Moreover,
Onyx
> > leaves
> > >>>>>> more parts of the job execution configurable, such as the
> scheduler
> > >>>>>> and the data plane. As opposed to providing a set of properties
> for
> > >>>>>> solid optimization, Onyx’s configurable parts can be
easily
> extended
> > >>>>>> and explored by implementing the pre-defined interfaces.
For
> > example,
> > >>>>>> an arbitrary intermediate data store can be added.
> > >>>>>>
> > >>>>>> Onyx currently supports Apache Beam programs and we are
working on
> > >>>>>> supporting Apache Spark programs as well. Onyx also utilizes
> Apache
> > >>>>>> REEF for container management, which allows Onyx to run
in Apache
> > YARN
> > >>>>>> and Apache Mesos clusters. If necessary, we plan to contribute
to
> > and
> > >>>>>> collaborate with these other Apache projects for the benefit
of
> all.
> > >>>>>> We plan to extend such integrations with more Apache softwares.
> > Apache
> > >>>>>> software foundation already hosts many major big-data systems,
and
> > we
> > >>>>>> expect to help further growth of the big-data community
by having
> > Onyx
> > >>>>>> within the Apache foundation.
> > >>>>>>
> > >>>>>> == Known Risks ==
> > >>>>>> === Orphaned Products ===
> > >>>>>> The risk of the Onyx project being orphaned is minimal.
There is
> > >>>>>> already plenty of work that arduously support different
deployment
> > >>>>>> characteristics, and we propose a general way to implement
them
> with
> > >>>>>> flexible and extensible configuration knobs. The domain
of data
> > >>>>>> processing is already of high interest, and this domain
is
> expected
> > to
> > >>>>>> evolve continuously with various other purposes, such as
resource
> > >>>>>> disaggregation and using transient resources for better
datacenter
> > >>>>>> resource utilization.
> > >>>>>>
> > >>>>>> === Inexperience with Open Source ===
> > >>>>>> The initial committers include PMC members and committers
of other
> > >>>>>> Apache projects. They have experience with open source
projects,
> > >>>>>> starting from their incubation to the top-level. They have
been
> > >>>>>> involved in the open source development process, and are
familiar
> > with
> > >>>>>> releasing code under an open source license.
> > >>>>>>
> > >>>>>> === Homogeneous Developers ===
> > >>>>>> The initial set of committers is from a limited set of
> > organizations,
> > >>>>>> but we expect to attract new contributors from diverse
> organizations
> > >>>>>> and will thus grow organically once approved for incubation.
Our
> > prior
> > >>>>>> experience with other open source projects will help various
> > >>>>>> contributors to actively participate in our project.
> > >>>>>>
> > >>>>>> === Reliance on Salaried Developers ===
> > >>>>>> Many developers are from Seoul National University. This
is not
> > >>>>> applicable.
> > >>>>>>
> > >>>>>> === Relationships with Other Apache Products ===
> > >>>>>> Onyx positions itself among multiple Apache products. It
runs on
> > >>>>>> Apache REEF for container management. It also utilizes
many useful
> > >>>>>> development tools including Apache Maven, Apache Log4J,
and
> multiple
> > >>>>>> Apache Commons components. Onyx supports the Apache Beam
> programming
> > >>>>>> model for user applications. We are currently working on
> supporting
> > >>>>>> the Apache Spark programming APIs as well.
> > >>>>>>
> > >>>>>> === An Excessive Fascination with the Apache Brand ===
> > >>>>>> We hope to make Onyx a powerful system for data processing,
> meeting
> > >>>>>> various needs for different deployment characteristics,
under a
> more
> > >>>>>> variety of environments. We see the limitations of simply
putting
> > code
> > >>>>>> on GitHub, and we believe the Apache community will help
the
> growth
> > of
> > >>>>>> Onyx for the project to become a positively impactful and
> innovative
> > >>>>>> open source software. We believe Onyx is a great fit for
the
> Apache
> > >>>>>> Software Foundation due to the collaboration it aims to
achieve
> from
> > >>>>>> the big data processing community.
> > >>>>>>
> > >>>>>> == Documentation ==
> > >>>>>> The current documentation for Onyx is at
> > https://snuspl.github.io/onyx/
> > >>>>> .
> > >>>>>>
> > >>>>>> == Initial Source ==
> > >>>>>> The Onyx codebase is currently hosted at
> > https://github.com/snuspl/onyx
> > >>>>> .
> > >>>>>>
> > >>>>>> == External Dependencies ==
> > >>>>>> To the best of our knowledge, all Onyx dependencies are
> distributed
> > >>>>>> under Apache compatible licenses. Upon acceptance to the
> incubator,
> > we
> > >>>>>> would begin a thorough analysis of all transitive dependencies
to
> > >>>>>> verify this fact and further introduce license checking
into the
> > build
> > >>>>>> and release process.
> > >>>>>>
> > >>>>>> == Cryptography ==
> > >>>>>> Not applicable.
> > >>>>>>
> > >>>>>> == Required Resources ==
> > >>>>>> === Mailing Lists ===
> > >>>>>> We will operate two mailing lists as follows:
> > >>>>>>  * Onyx PMC discussions: private@onyx.incubator.apache.org
> > >>>>>>  * Onyx developers: dev@onyx.incubator.apache.org
> > >>>>>>
> > >>>>>> === Git Repositories ===
> > >>>>>> Upon incubation: https://github.com/apache/incubator-onyx.
> > >>>>>> After the incubation, we would like to move the existing
repo
> > >>>>>> https://github.com/snuspl/onyx to the Apache infrastructure
> > >>>>>>
> > >>>>>> === Issue Tracking ===
> > >>>>>> Onyx currently tracks its issues using the Github issue
tracker:
> > >>>>>> https://github.com/snuspl/onyx/issues. We plan to migrate
to
> Apache
> > >>>>>> JIRA.
> > >>>>>>
> > >>>>>> == Initial Committers ==
> > >>>>>> * Byung-Gon Chun
> > >>>>>> * Jeongyoon Eo
> > >>>>>> * Geon-Woo Kim
> > >>>>>> * Joo Yeon Kim
> > >>>>>> * Gyewon Lee
> > >>>>>> * Jung-Gil Lee
> > >>>>>> * Sanha Lee
> > >>>>>> * Wooyeon Lee
> > >>>>>> * Yunseong Lee
> > >>>>>> * JangHo Seo
> > >>>>>> * Won Wook Song
> > >>>>>> * Taegeon Um
> > >>>>>> * Youngseok Yang
> > >>>>>>
> > >>>>>> == Affiliations ==
> > >>>>>> * SNU (Seoul National University)
> > >>>>>>   * Byung-Gon Chun
> > >>>>>>   * Jeongyoon Eo
> > >>>>>>   * Geon-Woo Kim
> > >>>>>>   * Gyewon Lee
> > >>>>>>   * Sanha Lee
> > >>>>>>   * Wooyeon Lee
> > >>>>>>   * Yunseong Lee
> > >>>>>>   * JangHo Seo
> > >>>>>>   * Won Wook Song
> > >>>>>>   * Taegeon Um
> > >>>>>>   * Youngseok Yang
> > >>>>>>
> > >>>>>> * LG
> > >>>>>>   * Jung-Gil Lee
> > >>>>>>
> > >>>>>> * Samsung
> > >>>>>>   * Joo Yeon Kim
> > >>>>>>
> > >>>>>> * Viva Republica
> > >>>>>>   * Geon-Woo Kim
> > >>>>>>
> > >>>>>> == Sponsors ==
> > >>>>>> === Champions ===
> > >>>>>> Byung-Gon Chun
> > >>>>>>
> > >>>>>> === Mentors ===
> > >>>>>> * Hyunsik Choi
> > >>>>>> * Byung-Gon Chun
> > >>>>>> * Markus Weimer
> > >>>>>> * Reynold Xin
> > >>>>>>
> > >>>>>> === Sponsoring Entity ===
> > >>>>>> The Apache Incubator
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Byung-Gon Chun
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Byung-Gon Chun
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Byung-Gon Chun
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: general-help@incubator.apache.org
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
> --
> Byung-Gon Chun
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message