apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay M Pujare (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-586) Enhancement to persist logical and physical plan snapshots in HDFS
Date Tue, 06 Dec 2016 00:43:59 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723879#comment-15723879
] 

Sanjay M Pujare commented on APEXCORE-586:
------------------------------------------

The following is the implementation plan. We will soon start the implementation so pls send
your feedback as soon as possible.

We will refactor code to move the logic in

com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)

and

com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan()

to the class

com.datatorrent.stram.StreamingContainerManager

So these 2 methods will return JSONObject representation of the logical or physical plan.

2 new methods will be added to StreamingContainerManager:

writePhysicalPlan()
writeLogicalPlan()

The first method will create an HDFS file with the name physicalPlan_NNN_MMM.json under the
current app's app directory (where NNN is the current app attempt id and MMM is	a running
count (maintained in the StreamingContainerManager object) that	is incremented on every call
to the method. The method will call getPhysicalPlan() to get the JSON object and write it
out to the HDFS file. writeLogicalPlan() will be similarly written to write the logical plan
to the file logicalPlan_NNN_MMM.json. There is a separate count for the MMM for logical plans.

The method com.datatorrent.stram.plan.physical.PhysicalPlan.onStatusUpdate(PTOperator) will
be modified to accept a parameter for the StreamingContainerManager object. Inside this method
after redoPartition() is	called, writePhysicalPlan() will be called on the StreamingContainerManager
object which will write a new physicalPlan_NNN_MMM.json file.

Similarly in com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call()
at the end writeLogicalPlan() will be called which will write a new logicalPlan_NNN_MMM.json
file.

Unit tests will be written to cover existing functionality and new functionality to minimize
the possibility of regressions and to cover the new code.


> Enhancement to persist logical and physical plan snapshots in HDFS
> ------------------------------------------------------------------
>
>                 Key: APEXCORE-586
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-586
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Sanjay M Pujare
>            Assignee: Sanjay M Pujare
>
> Pls refer to the discussion on dev@apex 
> http://apache-apex-developers-list.78491.x6.nabble.com/Proposing-a-new-feature-to-persist-logical-and-physical-plan-snapshots-in-HDFS-td11592.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message