stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirmal Fernando <nirmal070...@gmail.com>
Subject Re: FW: Cartridge Agent Extension Points
Date Thu, 05 Jun 2014 13:09:26 GMT
Hi Shaheedur,
AFAIK waitUntilPortsActive is blocking till the ports get active.


On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <
shahhaqu@cisco.com> wrote:

>  (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> ·        All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute the
> extension.
>
> a.      In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b.      This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> ·        The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to the
> left and also down? The down path follows the artifact update path:
>
>
>
> 1.      Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2.      Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could even
> argue that just because an artifact update happens, the main state of the
> agent should not be affected. (I know there is another discussion about
> having artifact updates come much earlier in the lifecycle, and that
> further illustrates this point). So, I tried a simple change to address
> item 1:
>
>
>
> ·        Always do “CA publishes Instance Activated event” after port
> activation
>
> ·        Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member
> activation (i.e. item 2). If you agree that item 2 should be fixed, then a
> slightly simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would you
> prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1]  Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
>      private boolean terminated;
>
>
>
> +    // We have an asynchronous activity running to respond to ADC
> updates. We want to ensure
>
> +    // that no publishInstanceActivatedEvent() call is made *before* the
> port activation test
>
> +    // has succeeded. This flag controls that.
>
> +    private boolean portsActivated;
>
> +
>
>      @Override
>
>      public void run() {
>
>          if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
>          validateRequiredSystemProperties();
>
>
>
>          // Start instance notifier listener thread
>
> +        portsActivated = false;
>
>          subscribeToTopicsAndRegisterListeners();
>
>
>
>          // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
>          // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>
>                  CartridgeAgentConfiguration.getInstance().getPorts());
>
> +        portsActivated = true;
>
> +
>
> +        // Publish instance activated event
>
> +        CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
>          // Check repo url
>
>          String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
>              if(log.isInfoEnabled()) {
>
>                  log.info("No artifact repository found");
>
>              }
>
> -
>
> -            // Publish instance activated event
>
> -            CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>          } else {
>
>              //Start periodical file checker task
>
>                 if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
>              ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> -            if(!cloneExists){
>
> +            if (!cloneExists && portsActivated) {
>
>                  // Executed git clone, publish instance activated event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>              }
>
> =========================
>
>
>
>
>
>
>
> *From:* Lakmal Warusawithana [mailto:lakmal@wso2.com]
> *Sent:* 05 June 2014 09:22
> *To:* Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> *Cc:* Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1.      The key requirements are:
>
> a.      start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b.      clean.sh must block the termination of the VM.
>
> 2.      The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3.      I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a.      Can you advise if making these two operations will adversely
> affect the cartridge agent in other ways?
>
> b.      For exmaple, I guess there is some other thread which is looping
> to deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Shaheedur Haque (shahhaqu)
> *Sent:* 31 May 2014 20:36
> *To:* dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel (meppel)
> *Cc:* Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> *Subject:* Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> ·                  The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> ·                  Worse, it makes the grouping/dependency work useless.
> The whole point there is to start up VMs in some given order.
>
> ·                  "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread
> here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
> > Yes, We should have python based agent in 4.1.0 release.
>
> >
>
> >
>
> > On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> >
>
> > <ravihansa@wso2.com>wrote:
>
> > > Hi all,
>
> > >
>
> > > AFAIK, the Stratos community has decided to put effort in writing a
>
> > > new cartridge-agent using Python. In that case, Java based agent will
>
> > > become deprecated and all DevOps level extensions will have to be done
>
> > > using Python. But until then we can use this shell script based model.
>
> > >
>
> > > I'll update the thread once I've finished writing documentation for
>
> > > agent extension points.
>
> > >
>
> > > Thanks.
>
> > >
>
> > > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
> > >
>
> > > <ravihansa@wso2.com> wrote:
>
> > > > Hi Shaheed,
>
> > > >
>
> > > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
> > > >
>
> > > > <shahhaqu@cisco.com> wrote:
>
> > > >> Is there any documentation on how this works form the scripter's
> point
>
> > >
>
> > > of view? Specifically:
>
> > > > I'm working on writing documentation content for cartridge-agent
>
> > > > extension points. Also planning to publish some articles on cartridge
>
> > > > deployment use cases (for eg - clustering MongoDB) in which these
>
> > > > extensions are being used.
>
> > > >
>
> > > >> - For the old events (i.e. the non-toplogy events hooked via the
>
> > >
>
> > > /extensions directory), is the new code backwards compatible with
> scripts
>
> > > using the /extensions mechanism?
>
> > >
>
> > > > There will be no changes done for previous non-topology events. The
>
> > > > new code will use the same /extensions directory mechanism. Only the
>
> > > > shell scripts' file names for each extension point will be passed as
>
> > > > parameters to the cartridge-agent.
>
> > > >
>
> > > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
> > >
>
> > > you confirm if environment variables set before the JVM is invoked will
>
> > > still be visible to the shell scripts?
>
> > >
>
> > > > I've tested and confirmed that environment variables set before the
>
> > > > JVM is invoked are visible to the shell scripts. On the side note,
> the
>
> > > > JavaProcessBuilder will only append our custom parameters to the
> shell
>
> > > > script process environment block. This is same as writing to
>
> > > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
> note
>
> > > > that these env variables set by the cartridge-agent are *not* shared
>
> > > > globally across other processes.
>
> > > >
>
> > > >> Thanks, Shaheed
>
> > > >>
>
> > > >>
>
> > > >> -----Original Message-----
>
> > > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
> > > >> Sent: 07 May 2014 05:24
>
> > > >> To: dev@stratos.incubator.apache.org
>
> > > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
> > > >> Subject: Re: Cartridge Agent Extension Points
>
> > > >>
>
> > > >> Pl note that patch is against the master branch (as of yesterday).
>
> > > >>
>
> > > >> Following are the list of event listeners that it will subscribe to
> 1.
>
> > >
>
> > > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
> > > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
> > > SubscriptionDomainsAddedEventListener
>
> > >
>
> > > >> 6. SubscriptionDomainsRemovedEventListener
>
> > > >> 7. CompleteTenantEvent
>
> > > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
> > >
>
> > > InstanceCleanupMemberEventListener (only locally) 10.
>
> > > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
> > > Event (only locally) 12. Start Servers event (only locally) 13.
> Instance
>
> > > Activated Event (only locally)
>
> > >
>
> > > >> These event handlers are called through an interface
>
> > >
>
> > > (ExtensionHandler). Therefore someone else can easily implement this
>
> > > interface and customize extensions from code level. The main purpose is
>
> > > however to extend the cartridge agent through shell scripts without
>
> > > changing the code.
>
> > >
>
> > > >> PS - Is there an on-going development effort to develop the
> cartridge
>
> > >
>
> > > agent in Python or Google GO? If so I would like to get involved :)
>
> > >
>
> > > >> Thanks.
>
> > > >>
>
> > > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
> > >
>
> > > ravihansa@wso2.com> wrote:
>
> > > >>> Hi,
>
> > > >>>
>
> > > >>> I've developed cartridge agent extension points feature for Stratos
>
> > > >>> cartridge agent. This will work by listening to event messages
>
> > > >>> published by CC, ADC and agents running on instances.
>
> > > >>>
>
> > > >>> Once an event is received it will execute a shell script (system
>
> > > >>> command) with parameters passed to it as environment variables
> through
>
> > > >>> Java ProcessBuilder class. These environment variables are visible
>
> > > >>> only to that process so there is no risk of dirty read/write.
>
> > > >>>
>
> > > >>> These script names can be set from stratos.sh and these are not
>
> > > >>> hard-coded. Puppet templates can be used to set script names for
> each
>
> > > >>> event type. This can be useful when dealing with different
> platforms
>
> > > >>> (For eg - Linux, Windows, OS X etc)
>
> > > >>>
>
> > > >>> I had to make some changes to the core messaging component to
get
>
> > > >>> LBClusterID from Member Activated event. This is because it is
>
> > > >>> necessary for LBs to identify which members belong to their LB
> cluster
>
> > > >>> when they receive a member activated event. If someone can tell
a
>
> > > >>> workaround, please do share.
>
> > > >>>
>
> > > >>> I have attached the patch in [1] as a JIRA issue.
>
> > > >>>
>
> > > >>> Thanks.
>
> > > >>>
>
> > > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
> > > >>>
>
> > > >>>
>
> > > >>> --
>
> > > >>> Akila Ravihansa Perera
>
> > > >>> Software Engineer
>
> > > >>> WSO2 Inc.
>
> > > >>> http://wso2.com
>
> > > >>>
>
> > > >>> Phone: +94 77 64 154 38
>
> > > >>> Blog: http://ravihansa3000.blogspot.com
>
> > > >>
>
> > > >> --
>
> > > >> Akila Ravihansa Perera
>
> > > >> Software Engineer
>
> > > >> WSO2 Inc.
>
> > > >> http://wso2.com
>
> > > >>
>
> > > >> Phone: +94 77 64 154 38
>
> > > >> Blog: http://ravihansa3000.blogspot.com
>
> > > >
>
> > > > --
>
> > > > Akila Ravihansa Perera
>
> > > > Software Engineer
>
> > > > WSO2 Inc.
>
> > > > http://wso2.com
>
> > > >
>
> > > > Phone: +94 77 64 154 38
>
> > > > Blog: http://ravihansa3000.blogspot.com
>
> > >
>
> > > --
>
> > > Akila Ravihansa Perera
>
> > > Software Engineer
>
> > > WSO2 Inc.
>
> > > http://wso2.com
>
> > >
>
> > > Phone: +94 77 64 154 38
>
> > > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>



-- 
Best Regards,
Nirmal

Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.

Blog: http://nirmalfdo.blogspot.com/

Mime
View raw message