incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leif Hedstrom <zw...@apache.org>
Subject Re: [PROPOSAL] Onyx - proposal for Apache Incubation
Date Sat, 27 Jan 2018 19:21:11 GMT
Did we rule out Onyx for sure? Just because some other project might use it on say github doesn’t
necessarily exclude us from having an Apache Onyx?

FWIW, I agree that surf is too similar in pronunciation to Apache serf. :)

Cheers,

— Leif 

> On Jan 27, 2018, at 07:31, Dave Fisher <dave2wave@comcast.net> wrote:
> 
> Checking “Serf Software” which sounds the same.
> 
> (1) there is already Apache Serf
> (2) Serf is a product from Hashicorp at https://www.serf.io/. This would definitely confuse
as it is apparently comparable to ZooKeeper.
> 
> Regards,
> Dave
> 
> Sent from my iPhone
> 
>> On Jan 27, 2018, at 3:12 AM, sebb <sebbaz@gmail.com> wrote:
>> 
>> A brief search for 'Surf Software' shows quite a few hits.
>> I have not looked to see if they would be likely to be confused with
>> this project or cause problems for others.
>> 
>> But it as though there might be a problem:
>> Surfer -  Golden Software
>> surf @ sourceforge
>> Surf Software company
>> 
>> 
>>> On 27 January 2018 at 08:03, Byung-Gon Chun <bgchun@gmail.com> wrote:
>>> Since we cannot use the name Onyx, we would like to change the project name
>>> to Surf.
>>> I hope that this name works.
>>> 
>>> -Gon
>>> 
>>> ---
>>> Byung-Gon Chun
>>> 
>>> 
>>>> On Sat, Jan 27, 2018 at 4:57 AM, Byung-Gon Chun <bgchun@gmail.com>
wrote:
>>>> 
>>>> 
>>>> 
>>>>> On Sat, Jan 27, 2018 at 4:09 AM, Davor Bonaci <davor@apache.org>
wrote:
>>>>> 
>>>>> Great work -- I think this technology has a lot of promise, and I'd love
>>>>> to
>>>>> see its evolution inside the Foundation.
>>>>> 
>>>>> 
>>>> Thanks, Davor!
>>>> 
>>>> 
>>>>> Parts of it, like the Onyx Intermediate Representation [1], overlap with
>>>>> the work-in-progress inside the Apache Beam project ("portability").
We'd
>>>>> love to work together on this -- would you be open to such collaboration?
>>>>> If so, it may not be necessary to start from scratch, and leverage the
>>>>> work
>>>>> already done.
>>>>> 
>>>>> 
>>>> Sure. We're open to collaboration.
>>>> 
>>>> 
>>>>> Regarding the name, Onyx would likely have to be renamed, due to a
>>>>> conflict
>>>>> with a related technology [2].
>>>>> 
>>>>> 
>>>> Thanks for pointing it out. It's difficult to come up with a good short
>>>> name. :)
>>>> Do you have any suggestion?
>>>> 
>>>> Thanks!
>>>> -Gon
>>>> 
>>>> ---
>>>> Byung-Gon Chun
>>>> 
>>>> 
>>>> 
>>>>> Davor
>>>>> 
>>>>> [1] https://snuspl.github.io/onyx/docs/ir/
>>>>> [2] http://www.onyxplatform.org/
>>>>> 
>>>>>> On Thu, Jan 25, 2018 at 3:28 PM, Byung-Gon Chun <bgchun@gmail.com>
wrote:
>>>>>> 
>>>>>> Dear Apache Incubator Community,
>>>>>> 
>>>>>> Please accept the following proposal for presentation and discussion:
>>>>>> https://wiki.apache.org/incubator/OnyxProposal
>>>>>> 
>>>>>> Onyx is a data processing system that aims to flexibly control the
>>>>> runtime
>>>>>> behaviors of a job to adapt to varying deployment characteristics
(e.g.,
>>>>>> harnessing transient resources in datacenters, cross-datacenter
>>>>> deployment,
>>>>>> changing runtime based on job characteristics, etc.). Onyx provides
>>>>> ways to
>>>>>> extend the system’s capabilities and incorporate the extensions
to the
>>>>>> flexible job execution.
>>>>>> Onyx translates a user program (e.g., Apache Beam, Apache Spark)
into an
>>>>>> Intermediate Representation (IR) DAG, which Onyx optimizes and deploys
>>>>>> based on a deployment policy.
>>>>>> 
>>>>>> I've attached the proposal below.
>>>>>> 
>>>>>> Best regards,
>>>>>> Byung-Gon Chun
>>>>>> 
>>>>>> = OnyxProposal =
>>>>>> 
>>>>>> == Abstract ==
>>>>>> Onyx is a data processing system for flexible employment with
>>>>>> different execution scenarios for various deployment characteristics
>>>>>> on clusters.
>>>>>> 
>>>>>> == Proposal ==
>>>>>> Today, there is a wide variety of data processing systems with
>>>>>> different designs for better performance and datacenter efficiency.
>>>>>> They include processing data on specific resource environments and
>>>>>> running jobs with specific attributes. Although each system
>>>>>> successfully solves the problems it targets, most systems are designed
>>>>>> in the way that runtime behaviors are built tightly inside the system
>>>>>> core to hide the complexity of distributed computing. This makes
it
>>>>>> hard for a single system to support different deployment
>>>>>> characteristics with different runtime behaviors without substantial
>>>>>> effort.
>>>>>> 
>>>>>> Onyx is a data processing system that aims to flexibly control the
>>>>>> runtime behaviors of a job to adapt to varying deployment
>>>>>> characteristics. Moreover, it provides a means of extending the
>>>>>> system’s capabilities and incorporating the extensions to the flexible
>>>>>> job execution.
>>>>>> 
>>>>>> In order to be able to easily modify runtime behaviors to adapt to
>>>>>> varying deployment characteristics, Onyx exposes runtime behaviors
to
>>>>>> be flexibly configured and modified at both compile-time and runtime
>>>>>> through a set of high-level graph pass interfaces.
>>>>>> 
>>>>>> We hope to contribute to the big data processing community by enabling
>>>>>> more flexibility and extensibility in job executions. Furthermore,
we
>>>>>> can benefit more together as a community when we work together as
a
>>>>>> community to mature the system with more use cases and understanding
>>>>>> of diverse deployment characteristics. The Apache Software Foundation
>>>>>> is the perfect place to achieve these aspirations.
>>>>>> 
>>>>>> == Background ==
>>>>>> Many data processing systems have distinctive runtime behaviors
>>>>>> optimized and configured for specific deployment characteristics
like
>>>>>> different resource environments and for handling special job
>>>>>> attributes.
>>>>>> 
>>>>>> For example, much research have been conducted to overcome the
>>>>>> challenge of running data processing jobs on cheap, unreliable
>>>>>> transient resources. Likewise, techniques for disaggregating different
>>>>>> types of resources, like memory, CPU and GPU, are being actively
>>>>>> developed to use datacenter resources more efficiently. Many
>>>>>> researchers are also working to run data processing jobs in even
more
>>>>>> diverse environments, such as across distant datacenters. Similarly,
>>>>>> for special job attributes, many works take different approaches,
such
>>>>>> as runtime optimization, to solve problems like data skew, and to
>>>>>> optimize systems for data processing jobs with small-scale input
data.
>>>>>> 
>>>>>> Although each of the systems performs well with the jobs and in the
>>>>>> environments they target, they perform poorly with unconsidered cases,
>>>>>> and do not consider supporting multiple deployment characteristics
on
>>>>>> a single system in their designs.
>>>>>> 
>>>>>> For an application writer to optimize an application to perform well
>>>>>> on a certain system engraved with its underlying behaviors, it
>>>>>> requires a deep understanding of the system itself, which is an
>>>>>> overhead that often requires a lot of time and effort. Moreover,
for a
>>>>>> developer to modify such system behaviors, it requires modifications
>>>>>> of the system core, which requires an even deeper understanding of
the
>>>>>> system itself.
>>>>>> 
>>>>>> With this background, Onyx is designed to represent all of its jobs
as
>>>>>> an Intermediate Representation (IR) DAG. In the Onyx compiler, user
>>>>>> applications from various programming models (ex. Apache Beam) are
>>>>>> submitted, transformed to an IR DAG, and optimized/customized for
the
>>>>>> deployment characteristics. In the IR DAG optimization phase, the
DAG
>>>>>> is modified through a series of compiler “passes” which reshape
or
>>>>>> annotate the DAG with an expression of the underlying runtime
>>>>>> behaviors. The IR DAG is then submitted as an execution plan for
the
>>>>>> Onyx runtime. The runtime includes the unmodified parts of data
>>>>>> processing in the backbone which is transparently integrated with
>>>>>> configurable components exposed for further extension.
>>>>>> 
>>>>>> == Rationale ==
>>>>>> Onyx’s vision lies in providing means for flexibly supporting a
wide
>>>>>> variety of job execution scenarios for users while facilitating system
>>>>>> developers to extend the execution framework with various
>>>>>> functionalities at the same time. The capabilities of the system
can
>>>>>> be extended as it grows to meet a more variety of execution scenarios.
>>>>>> We require inputs from users and developers from diverse domains
in
>>>>>> order to make it a more thriving and useful project. The Apache
>>>>>> Software Foundation provides the best tools and community to support
>>>>>> this vision.
>>>>>> 
>>>>>> == Initial Goals ==
>>>>>> Initial goals will be to move the existing codebase to Apache and
>>>>>> integrate with the Apache development process. We further plan to
>>>>>> develop our system to meet the needs for more execution scenarios
for
>>>>>> a more variety of deployment characteristics.
>>>>>> 
>>>>>> == Current Status ==
>>>>>> Onyx codebase is currently hosted in a repository at github.com.
The
>>>>>> current version has been developed by system developers at Seoul
>>>>>> National University, Viva Republica, Samsung, and LG.
>>>>>> 
>>>>>> == Meritocracy ==
>>>>>> We plan to strongly support meritocracy. We will discuss the
>>>>>> requirements in an open forum, and those that continuously contribute
>>>>>> to Onyx with the passion to strengthen the system will be invited
as
>>>>>> committers. Contributors that enrich Onyx by providing various use
>>>>>> cases, various implementations of the configurable components
>>>>>> including ideas for optimization techniques will be especially
>>>>>> welcome. Committers with a deep understanding of the system’s
>>>>>> technical aspects as a whole and its philosophy will definitely be
>>>>>> voted as the PMC. We will monitor community participation so that
>>>>>> privileges can be extended to those that contribute.
>>>>>> 
>>>>>> == Community ==
>>>>>> We hope to expand our contribution community by becoming an Apache
>>>>>> incubator project. The contributions will come from both users and
>>>>>> system developers interested in flexibility and extensibility of
job
>>>>>> executions that Onyx can support. We expect users to mainly contribute
>>>>>> to diversify the use cases and deployment characteristics, and
>>>>>> developers to  contribute to implement them.
>>>>>> 
>>>>>> == Alignment ==
>>>>>> Apache Spark is one of many popular data processing frameworks. The
>>>>>> system is designed towards optimizing jobs using RDDs in memory and
>>>>>> many other optimizations built tightly within the framework. In
>>>>>> contrast to Spark, Onyx aims to provide more flexibility for job
>>>>>> execution in an easy manner.
>>>>>> 
>>>>>> Apache Tez enables developers to build complex task DAGs with control
>>>>>> over the control plane of job execution. In Onyx, a high-level
>>>>>> programming layer (ex. Apache Beam) is automatically converted to
a
>>>>>> basic IR DAG and can be converted to any IR DAG through a series
of
>>>>>> easy user writable passes, that can both reshape and modify the
>>>>>> annotation (of execution properties) of the DAG. Moreover, Onyx leaves
>>>>>> more parts of the job execution configurable, such as the scheduler
>>>>>> and the data plane. As opposed to providing a set of properties for
>>>>>> solid optimization, Onyx’s configurable parts can be easily extended
>>>>>> and explored by implementing the pre-defined interfaces. For example,
>>>>>> an arbitrary intermediate data store can be added.
>>>>>> 
>>>>>> Onyx currently supports Apache Beam programs and we are working on
>>>>>> supporting Apache Spark programs as well. Onyx also utilizes Apache
>>>>>> REEF for container management, which allows Onyx to run in Apache
YARN
>>>>>> and Apache Mesos clusters. If necessary, we plan to contribute to
and
>>>>>> collaborate with these other Apache projects for the benefit of all.
>>>>>> We plan to extend such integrations with more Apache softwares. Apache
>>>>>> software foundation already hosts many major big-data systems, and
we
>>>>>> expect to help further growth of the big-data community by having
Onyx
>>>>>> within the Apache foundation.
>>>>>> 
>>>>>> == Known Risks ==
>>>>>> === Orphaned Products ===
>>>>>> The risk of the Onyx project being orphaned is minimal. There is
>>>>>> already plenty of work that arduously support different deployment
>>>>>> characteristics, and we propose a general way to implement them with
>>>>>> flexible and extensible configuration knobs. The domain of data
>>>>>> processing is already of high interest, and this domain is expected
to
>>>>>> evolve continuously with various other purposes, such as resource
>>>>>> disaggregation and using transient resources for better datacenter
>>>>>> resource utilization.
>>>>>> 
>>>>>> === Inexperience with Open Source ===
>>>>>> The initial committers include PMC members and committers of other
>>>>>> Apache projects. They have experience with open source projects,
>>>>>> starting from their incubation to the top-level. They have been
>>>>>> involved in the open source development process, and are familiar
with
>>>>>> releasing code under an open source license.
>>>>>> 
>>>>>> === Homogeneous Developers ===
>>>>>> The initial set of committers is from a limited set of organizations,
>>>>>> but we expect to attract new contributors from diverse organizations
>>>>>> and will thus grow organically once approved for incubation. Our
prior
>>>>>> experience with other open source projects will help various
>>>>>> contributors to actively participate in our project.
>>>>>> 
>>>>>> === Reliance on Salaried Developers ===
>>>>>> Many developers are from Seoul National University. This is not
>>>>> applicable.
>>>>>> 
>>>>>> === Relationships with Other Apache Products ===
>>>>>> Onyx positions itself among multiple Apache products. It runs on
>>>>>> Apache REEF for container management. It also utilizes many useful
>>>>>> development tools including Apache Maven, Apache Log4J, and multiple
>>>>>> Apache Commons components. Onyx supports the Apache Beam programming
>>>>>> model for user applications. We are currently working on supporting
>>>>>> the Apache Spark programming APIs as well.
>>>>>> 
>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>> We hope to make Onyx a powerful system for data processing, meeting
>>>>>> various needs for different deployment characteristics, under a more
>>>>>> variety of environments. We see the limitations of simply putting
code
>>>>>> on GitHub, and we believe the Apache community will help the growth
of
>>>>>> Onyx for the project to become a positively impactful and innovative
>>>>>> open source software. We believe Onyx is a great fit for the Apache
>>>>>> Software Foundation due to the collaboration it aims to achieve from
>>>>>> the big data processing community.
>>>>>> 
>>>>>> == Documentation ==
>>>>>> The current documentation for Onyx is at https://snuspl.github.io/onyx/
>>>>> .
>>>>>> 
>>>>>> == Initial Source ==
>>>>>> The Onyx codebase is currently hosted at https://github.com/snuspl/onyx
>>>>> .
>>>>>> 
>>>>>> == External Dependencies ==
>>>>>> To the best of our knowledge, all Onyx dependencies are distributed
>>>>>> under Apache compatible licenses. Upon acceptance to the incubator,
we
>>>>>> would begin a thorough analysis of all transitive dependencies to
>>>>>> verify this fact and further introduce license checking into the
build
>>>>>> and release process.
>>>>>> 
>>>>>> == Cryptography ==
>>>>>> Not applicable.
>>>>>> 
>>>>>> == Required Resources ==
>>>>>> === Mailing Lists ===
>>>>>> We will operate two mailing lists as follows:
>>>>>>  * Onyx PMC discussions: private@onyx.incubator.apache.org
>>>>>>  * Onyx developers: dev@onyx.incubator.apache.org
>>>>>> 
>>>>>> === Git Repositories ===
>>>>>> Upon incubation: https://github.com/apache/incubator-onyx.
>>>>>> After the incubation, we would like to move the existing repo
>>>>>> https://github.com/snuspl/onyx to the Apache infrastructure
>>>>>> 
>>>>>> === Issue Tracking ===
>>>>>> Onyx currently tracks its issues using the Github issue tracker:
>>>>>> https://github.com/snuspl/onyx/issues. We plan to migrate to Apache
>>>>>> JIRA.
>>>>>> 
>>>>>> == Initial Committers ==
>>>>>> * Byung-Gon Chun
>>>>>> * Jeongyoon Eo
>>>>>> * Geon-Woo Kim
>>>>>> * Joo Yeon Kim
>>>>>> * Gyewon Lee
>>>>>> * Jung-Gil Lee
>>>>>> * Sanha Lee
>>>>>> * Wooyeon Lee
>>>>>> * Yunseong Lee
>>>>>> * JangHo Seo
>>>>>> * Won Wook Song
>>>>>> * Taegeon Um
>>>>>> * Youngseok Yang
>>>>>> 
>>>>>> == Affiliations ==
>>>>>> * SNU (Seoul National University)
>>>>>>   * Byung-Gon Chun
>>>>>>   * Jeongyoon Eo
>>>>>>   * Geon-Woo Kim
>>>>>>   * Gyewon Lee
>>>>>>   * Sanha Lee
>>>>>>   * Wooyeon Lee
>>>>>>   * Yunseong Lee
>>>>>>   * JangHo Seo
>>>>>>   * Won Wook Song
>>>>>>   * Taegeon Um
>>>>>>   * Youngseok Yang
>>>>>> 
>>>>>> * LG
>>>>>>   * Jung-Gil Lee
>>>>>> 
>>>>>> * Samsung
>>>>>>   * Joo Yeon Kim
>>>>>> 
>>>>>> * Viva Republica
>>>>>>   * Geon-Woo Kim
>>>>>> 
>>>>>> == Sponsors ==
>>>>>> === Champions ===
>>>>>> Byung-Gon Chun
>>>>>> 
>>>>>> === Mentors ===
>>>>>> * Hyunsik Choi
>>>>>> * Byung-Gon Chun
>>>>>> * Markus Weimer
>>>>>> * Reynold Xin
>>>>>> 
>>>>>> === Sponsoring Entity ===
>>>>>> The Apache Incubator
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Byung-Gon Chun
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Byung-Gon Chun
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Byung-Gon Chun
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message