Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2CE0D200BDA for ; Tue, 29 Nov 2016 07:15:29 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 2B6AB160B22; Tue, 29 Nov 2016 06:15:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 750B7160B0D for ; Tue, 29 Nov 2016 07:15:28 +0100 (CET) Received: (qmail 44519 invoked by uid 500); 29 Nov 2016 06:15:27 -0000 Mailing-List: contact dev-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@apex.apache.org Delivered-To: mailing list dev@apex.apache.org Received: (qmail 44508 invoked by uid 99); 29 Nov 2016 06:15:27 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Nov 2016 06:15:27 +0000 Received: from mail-oi0-f54.google.com (mail-oi0-f54.google.com [209.85.218.54]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 364F61A0309 for ; Tue, 29 Nov 2016 06:15:27 +0000 (UTC) Received: by mail-oi0-f54.google.com with SMTP id w63so179259591oiw.0 for ; Mon, 28 Nov 2016 22:15:26 -0800 (PST) X-Gm-Message-State: AKaTC02mRvzfZnHTPeDhgj/lT9rZ0W+aA47w63wuTyJT7oMFYT4oKUPRfn3koemvMrPmU55zfao+VkMPyNjjZWbe X-Received: by 10.157.47.240 with SMTP id b45mr15777874otd.242.1480400126380; Mon, 28 Nov 2016 22:15:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.87.41 with HTTP; Mon, 28 Nov 2016 22:15:25 -0800 (PST) In-Reply-To: References: From: Chinmay Kolhatkar Date: Tue, 29 Nov 2016 11:45:25 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Apex internal documentation. To: dev@apex.apache.org Content-Type: multipart/alternative; boundary=001a113edf5ea8ae9105426a83b7 archived-at: Tue, 29 Nov 2016 06:15:29 -0000 --001a113edf5ea8ae9105426a83b7 Content-Type: text/plain; charset=UTF-8 +1... Good to have. On Tue, Nov 29, 2016 at 11:42 AM, Devendra Tagare wrote: > +1 > > ~Dev > > On Mon, Nov 28, 2016 at 10:10 PM, Priyanka Gugale > wrote: > > > +1 for documenting this. > > > > I had covered the basic cycle of launch in couple of meetups but there is > > no documentation. Also many things like bufferserver management, > heartbeat > > mechanism, command execution, stats etc are not documented. > > > > -Priyanka > > > > On Tue, Nov 29, 2016 at 11:23 AM, Tushar Gosavi > wrote: > > > > > Do we have documents explaining internals of Apex? Would it be a good > > > idea to add document explaining following on apex.apache.org to help > > > newcomers understand overall execution flow of application and apex > > > capabilities, this might also help them to add new features. > > > > > > - Startup of application > > > - logical plan formation and properties injection. > > > - copying of jars/resources > > > - handling of old application state. > > > - Master initialization > > > - converting logical plan to physical plan > > > - container assignments with locality and affinity > > > - initial checkpoints > > > - deployment > > > - restarting from killed application (-originalId) > > > - StramChild functionality > > > - Operator deploy requests and operator lifecycle management. > > > - handling of checkpoints > > > - WindowGenerator > > > - Sending hearbeats > > > - handling of various commands from Stram. > > > - Operator request response model. > > > - Bufferserver management. (purge/reset/disk spooling/back-pressure > > > handling) > > > - Run-time activities at master. > > > - heartbeat monitoring > > > - stats listeners > > > - recovery from operator failure > > > - dynamical plan change > > > > > > - Tushar. > > > > > > --001a113edf5ea8ae9105426a83b7--