falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (FALCON-2163) Feed replication based on partition does not produce expected output
Date Tue, 25 Oct 2016 11:02:58 GMT

     [ https://issues.apache.org/jira/browse/FALCON-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pallavi Rao resolved FALCON-2163.
---------------------------------
       Resolution: Fixed
         Assignee: Pallavi Rao
    Fix Version/s: trunk

> Feed replication based on partition does not produce expected output
> --------------------------------------------------------------------
>
>                 Key: FALCON-2163
>                 URL: https://issues.apache.org/jira/browse/FALCON-2163
>             Project: Falcon
>          Issue Type: Bug
>          Components: feed, prism
>    Affects Versions: trunk
>            Reporter: Pragya Mittal
>            Assignee: Pallavi Rao
>             Fix For: trunk
>
>
> Feed :
> {noformat}
> <?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="PrismFeedReplicationPartitionExpTest-InputFeed-021eefe6"
description="Input File">
>     <partitions>
>         <partition name="colo"/>
>         <partition name="eventTime"/>
>         <partition name="impressionHour"/>
>         <partition name="pricingModel"/>
>     </partitions>
>     <frequency>minutes(5)</frequency>
>     <late-arrival cut-off="days(100000)"/>
>     <clusters>
>         <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-5698a59c"
type="source" partition="${cluster.colo}">
>             <validity start="2012-10-01T12:00Z" end="2099-10-01T12:10Z"/>
>             <retention limit="days(1000000)" action="delete"/>
>             <locations>
>                 <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>             </locations>
>         </cluster>
>         <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-7a96289d"
type="target">
>             <validity start="2012-10-01T12:00Z" end="2099-10-01T12:25Z"/>
>             <retention limit="days(1000000)" action="delete"/>
>             <locations>
>                 <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>             </locations>
>         </cluster>
>         <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-d4fbf2d9"
type="source" partition="${cluster.colo}">
>             <validity start="2012-10-01T12:00Z" end="2099-01-01T00:00Z"/>
>             <retention limit="days(1000000)" action="delete"/>
>             <locations>
>                 <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/localDC/rc/billing/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>             </locations>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/data/regression/fetlrc/billing/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>         <location type="stats" path="/data/regression/fetlrc/billing/stats"/>
>         <location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
>     </locations>
>     <ACL owner="pragya" group="dataqa" permission="*"/>
>     <schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
>     <properties>
>         <property name="field1" value="value1"/>
>         <property name="field2" value="value2"/>
>     </properties>
> </feed>
> {noformat}
> Cluster 1 data :
> {noformat}
> dataqa@falcon1001:~$ hlr /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/*/*
> 16/09/28 10:53:02 INFO util.Shell: ===Loading custom Shell class JIRA:GRIDOPS-4518 ==
> lsr: DEPRECATED: Please use 'ls -R' instead.
> -rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua1/dataFile.xml
> -rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua2/dataFile.xml
> -rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua3/dataFile.xml
> -rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua1/id.pig
> -rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua2/id.pig
> -rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua3/id.pig
> {noformat}
> Cluster 3 data :
> {noformat}
> dataqa@hydra:~$ hlr /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/*/*
> lsr: DEPRECATED: Please use 'ls -R' instead.
> -rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua1/dataFile.xml
> -rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua2/dataFile.xml
> -rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua3/dataFile.xml
> -rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua1/id.pig
> -rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua2/id.pig
> -rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua3/id.pig
> -rw-r--r--   3 pragya hdfs       1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua1/cluster-0.1.xml
> -rw-r--r--   3 pragya hdfs       1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua2/cluster-0.1.xml
> -rw-r--r--   3 pragya hdfs       1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua3/cluster-0.1.xml
> -rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua1/dataFile.properties
> -rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua2/dataFile.properties
> -rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua3/dataFile.properties
> -rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua1/dataFile.properties
> -rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua2/dataFile.properties
> -rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua3/dataFile.properties
> {noformat}
> Target data :
> {noformat}
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/dataFile.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/id.pig
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/cluster-0.1.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/dataFile.properties
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/dataFile.properties
> {noformat}
> Expected output :
> {noformat}
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/dataFile.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/id.pig
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua3/dataFile.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua3/id.pig
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/cluster-0.1.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/dataFile.properties
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/dataFile.properties
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message