apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From davidyan74 <...@git.apache.org>
Subject [GitHub] incubator-apex-core pull request: APEXCORE-411 support restarting ...
Date Tue, 05 Apr 2016 18:54:10 GMT
Github user davidyan74 commented on a diff in the pull request:

    https://github.com/apache/incubator-apex-core/pull/294#discussion_r58595340
  
    --- Diff: engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java ---
    @@ -210,14 +239,67 @@ public StramAppLauncher(FileSystem fs, Path path, Configuration
conf) throws Exc
       public StramAppLauncher(String name, Configuration conf) throws Exception
       {
         this.propertiesBuilder = new LogicalPlanConfiguration(conf);
    +    this.conf = conf;
         init(name);
       }
     
    +  public StramAppLauncher(FileSystem fs, Configuration conf) throws Exception
    +  {
    +    this.propertiesBuilder = new LogicalPlanConfiguration(conf);
    +    this.fs = fs;
    +    this.conf = conf;
    +    init();
    +  }
    +
       public String getMvnBuildClasspathOutput()
       {
         return mvnBuildClasspathOutput.toString();
       }
     
    +  /**
    +   * This is for recovering an app without specifying apa or appjar file
    +   * @throws Exception
    +   */
    +  private void init() throws Exception
    +  {
    +    String originalAppId = propertiesBuilder.conf.get(ORIGINAL_APP_ID);
    +    if (originalAppId == null) {
    +      throw new AssertionError("Need original app id if launching without apa or appjar");
    +    }
    +    Path appsBasePath = new Path(StramClientUtils.getDTDFSRootDir(fs, conf), StramClientUtils.SUBDIR_APPS);
    +    Path origAppPath = new Path(appsBasePath, originalAppId);
    +    StringWriter writer = new StringWriter();
    +    try (FSDataInputStream in = fs.open(new Path(origAppPath, "meta.json"))) {
    +      IOUtils.copy(in, writer);
    +    }
    +    JSONObject metaJson = new JSONObject(writer.toString());
    +    String originalLibJars = null;
    +    try {
    +      JSONObject attributes = metaJson.getJSONObject("attributes");
    +      originalLibJars = attributes.getString(LogicalPlan.LIBRARY_JARS.getSimpleName());
    +      recoveryAppName = attributes.getString(Context.DAGContext.APPLICATION_NAME.getSimpleName());
    +    } catch (JSONException ex) {
    +      // ignore
    +    }
    +
    +    LinkedHashSet<URL> clUrls = new LinkedHashSet<>();
    --- End diff --
    
    That was what I thought too. But StramClient.copyInitialState itself needs all the operator
classes to be loaded before we can even launch. Here's the exception stack trace if you remove
all that code:
    {code}
    com.esotericsoftware.kryo.KryoException: Unable to find class: com.datatorrent.lib.testbench.RandomEventGenerator
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
    	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
    	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
    	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
    	at com.datatorrent.common.util.FSStorageAgent.retrieve(FSStorageAgent.java:193)
    	at com.datatorrent.stram.plan.logical.LogicalPlan$OperatorMeta.readObject(LogicalPlan.java:857)
    ...
    ...
    	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    	at com.datatorrent.stram.FSRecoveryHandler.restore(FSRecoveryHandler.java:247)
    	at com.datatorrent.stram.StramClient.copyInitialState(StramClient.java:275)
    	at com.datatorrent.stram.StramClient.startApplication(StramClient.java:513)
    	at com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:616)
    	at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2067)
    	at com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1454)
    {code}


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message