falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: [jira] [Commented] (FALCON-2051) Falcon post-processing services are not getting invoked
Date Wed, 29 Jun 2016 21:31:16 GMT
Ok, let us reenable post processing sending the jms message and take up the fix in master.
  This means we will revert FACON-1926 only in 0.10 and apply to master and continue to make
the fixes in master – right?


On 6/29/16, 12:48 PM, "Venky Ramachandran" <vramachandran@hortonworks.com> wrote:

>(Apache JIRA is down so continuing with email)
>[~pallavi.rao], [~peeyushb], [~bvellanki], [~vrangan]
>As it makes sense, I took a look at constructing the counters.txt path mentioned below
in the JMSMessageConsumer class.
>6-06-29 02:46:11,574 INFO [main] org.apache.falcon.workflow.WorkflowExecutionContext:
counterFile - hdfs://c6401.ambari.apache.org:8020/tmp/fs-backup/falcon/workflows/feed/sampleReplFeed/logs/job-2016-06-23-08-36/primaryCluster/counter.txt
>From the OOZIE JMS message, I can get the entity name and entity type (decoding the WF
name), but the cluster names are not there -- in this case, the replication WF runs on the
destination cluster (backupCluster and so the /tmp/fs-backup staging dir), but there is also
primaryCluster in the path. 
>So, I think we need to rethink about where the counters file gets stored -- possibly in
the STATS dir as pallavi points out or some other place that can be derived independently.

>Given that this is blocking 0.10 release, I would prefer to go ahead with Post Processing
sending the JMS message so that we can have this functionality work and address the counters
path in a separate JIRA. 
>What do people think? 

View raw message