Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
From: davidyan74 <git@git.apache.org>
To: dev@apex.incubator.apache.org
Reply-To: dev@apex.incubator.apache.org
References: <git-pr-294-incubator-apex-core@git.apache.org>
In-Reply-To: <git-pr-294-incubator-apex-core@git.apache.org>
Subject: [GitHub] incubator-apex-core pull request: APEXCORE-411 support
 restarting ...
Content-Type: text/plain
Message-Id: <20160405185410.99649E0211@git1-us-west.apache.org>
Date: Tue,  5 Apr 2016 18:54:10 +0000 (UTC)

Github user davidyan74 commented on a diff in the pull request:

    https://github.com/apache/incubator-apex-core/pull/294#discussion_r58595340
  
    --- Diff: engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java ---
    @@ -210,14 +239,67 @@ public StramAppLauncher(FileSystem fs, Path path, Configuration conf) throws Exc
       public StramAppLauncher(String name, Configuration conf) throws Exception
       {
         this.propertiesBuilder = new LogicalPlanConfiguration(conf);
    +    this.conf = conf;
         init(name);
       }
     
    +  public StramAppLauncher(FileSystem fs, Configuration conf) throws Exception
    +  {
    +    this.propertiesBuilder = new LogicalPlanConfiguration(conf);
    +    this.fs = fs;
    +    this.conf = conf;
    +    init();
    +  }
    +
       public String getMvnBuildClasspathOutput()
       {
         return mvnBuildClasspathOutput.toString();
       }
     
    +  /**
    +   * This is for recovering an app without specifying apa or appjar file
    +   * @throws Exception
    +   */
    +  private void init() throws Exception
    +  {
    +    String originalAppId = propertiesBuilder.conf.get(ORIGINAL_APP_ID);
    +    if (originalAppId == null) {
    +      throw new AssertionError("Need original app id if launching without apa or appjar");
    +    }
    +    Path appsBasePath = new Path(StramClientUtils.getDTDFSRootDir(fs, conf), StramClientUtils.SUBDIR_APPS);
    +    Path origAppPath = new Path(appsBasePath, originalAppId);
    +    StringWriter writer = new StringWriter();
    +    try (FSDataInputStream in = fs.open(new Path(origAppPath, "meta.json"))) {
    +      IOUtils.copy(in, writer);
    +    }
    +    JSONObject metaJson = new JSONObject(writer.toString());
    +    String originalLibJars = null;
    +    try {
    +      JSONObject attributes = metaJson.getJSONObject("attributes");
    +      originalLibJars = attributes.getString(LogicalPlan.LIBRARY_JARS.getSimpleName());
    +      recoveryAppName = attributes.getString(Context.DAGContext.APPLICATION_NAME.getSimpleName());
    +    } catch (JSONException ex) {
    +      // ignore
    +    }
    +
    +    LinkedHashSet<URL> clUrls = new LinkedHashSet<>();
    --- End diff --
    
    That was what I thought too. But StramClient.copyInitialState itself needs all the operator classes to be loaded before we can even launch. Here's the exception stack trace if you remove all that code:
    {code}
    com.esotericsoftware.kryo.KryoException: Unable to find class: com.datatorrent.lib.testbench.RandomEventGenerator
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
    	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
    	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
    	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:193)
    	at com.datatorrent.stram.plan.logical.LogicalPlan$OperatorMeta.readObject(LogicalPlan.java:857)
    ...
    ...
    	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    	at com.datatorrent.stram.FSRecoveryHandler.restore(FSRecoveryHandler.java:247)
    	at com.datatorrent.stram.StramClient.copyInitialState(StramClient.java:275)
    	at com.datatorrent.stram.StramClient.startApplication(StramClient.java:513)
    	at com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:616)
    	at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2067)
    	at com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1454)
    {code}


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---