falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-129) Disable Late data handling for hive tables
Date Thu, 17 Oct 2013 08:40:41 GMT

    [ https://issues.apache.org/jira/browse/FALCON-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797715#comment-13797715
] 

Srikanth Sundarrajan commented on FALCON-129:
---------------------------------------------

Patch doesn't apply. Can you please fix this?

{code}
sriksun:falcon-for-commit sriksun$ curl "https://issues.apache.org/jira/secure/attachment/12608821/FALCON-129-r2-cumulative.patch"
| git apply --check -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 65944  100 65944    0     0  24206      0  0:00:02  0:00:02 --:--:-- 27170
Checking patch client/src/main/java/org/apache/falcon/client/FalconClient.java...
Checking patch common/src/main/java/org/apache/falcon/catalog/CatalogPartition.java...
error: common/src/main/java/org/apache/falcon/catalog/CatalogPartition.java: No such file
or directory
Checking patch common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java...
error: while searching for:

    protected void delete(Cluster cluster, Entity entity, long retention, FileStatus[] logs)
        throws FalconException {

        long now = System.currentTimeMillis();


error: patch failed: common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java:106
error: common/src/main/java/org/apache/falcon/cleanup/AbstractCleanupHandler.java: patch does
not apply
Checking patch common/src/main/java/org/apache/falcon/entity/EntityUtil.java...
Checking patch common/src/main/java/org/apache/falcon/entity/FeedHelper.java...
Hunk #1 succeeded at 145 (offset -10 lines).
Checking patch common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java...
Hunk #1 succeeded at 74 (offset 2 lines).
Checking patch common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java...
Checking patch common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java...
Hunk #1 succeeded at 172 (offset -3 lines).
error: while searching for:
            }
        }

        if (process.getLateProcess() != null) {
            for (LateInput lp : process.getLateProcess().getLateInputs()) {
                if (!feeds.keySet().contains(lp.getInput())) {
                    throw new ValidationException("Late Input: " + lp.getInput() + " is not
specified in the inputs");
                }
                try {
                    Feed feed = ConfigurationStore.get().get(EntityType.FEED, feeds.get(lp.getInput()));
                    if (feed.getLateArrival() == null) {
                        throw new ValidationException(
                                "Late Input feed: " + lp.getInput() + " is not configured
with late arrival cut-off");
                    }
                } catch (FalconException e) {
                    throw new ValidationException(e);
                }
            }
        }
    }

    private void validateOptionalInputsForTableStorage(Feed feed, Input input) throws FalconException
{
        if (input.isOptional() && FeedHelper.createStorage(feed).getType() == Storage.TYPE.TABLE)
{
            throw new ValidationException("Optional Input is not supported for feeds with
table storage! "
                    + input.getName());
        }

error: patch failed: common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java:182
error: common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java: patch
does not apply
Checking patch common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java...
Checking patch common/src/test/resources/config/feed/hive-table-feed.xml...
Checking patch feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java...
error: while searching for:
                } else if (sourceStorage.getType() == Storage.TYPE.TABLE) {
                    instancePaths = "${coord:dataIn('input')}";

                    props.put("shouldRecord", "false"); // todo - override until late data
is handled
                    final CatalogStorage sourceTableStorage = (CatalogStorage) sourceStorage;
                    propagateTableStorageProperties(srcCluster, sourceTableStorage, props,
"falconSource");
                    final CatalogStorage targetTableStorage = (CatalogStorage) targetStorage;

error: patch failed: feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java:416
error: feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java: patch does not
apply
Checking patch feed/src/main/resources/config/workflow/replication-workflow.xml...
error: while searching for:
            <arg>${falconInPaths}</arg>
            <arg>-falconInputFeeds</arg>
            <arg>${falconInputFeeds}</arg>
            <capture-output/>
        </java>
        <ok to="replication-decision"/>

error: patch failed: feed/src/main/resources/config/workflow/replication-workflow.xml:46
error: feed/src/main/resources/config/workflow/replication-workflow.xml: patch does not apply
Checking patch feed/src/main/resources/config/workflow/retention-workflow.xml...
Hunk #1 succeeded at 96 (offset -11 lines).
Checking patch messaging/src/main/java/org/apache/falcon/messaging/EntityInstanceMessage.java...
Checking patch messaging/src/main/java/org/apache/falcon/messaging/MessageProducer.java...
Checking patch messaging/src/test/java/org/apache/falcon/messaging/FalconTopicProducerTest.java...
Checking patch messaging/src/test/java/org/apache/falcon/messaging/FeedProducerTest.java...
Checking patch messaging/src/test/java/org/apache/falcon/messaging/ProcessProducerTest.java...
Checking patch oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java...
Checking patch oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java...
Checking patch oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java...
Checking patch prism/src/main/java/org/apache/falcon/service/FalconTopicSubscriber.java...
Checking patch process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java...
error: while searching for:
import org.apache.falcon.entity.FeedHelper;
import org.apache.falcon.entity.ProcessHelper;
import org.apache.falcon.entity.Storage;
import org.apache.falcon.entity.v0.EntityType;
import org.apache.falcon.entity.v0.Frequency;
import org.apache.falcon.entity.v0.SchemaHelper;

error: patch failed: process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java:27
error: process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java: patch does
not apply
Checking patch process/src/main/resources/config/workflow/process-parent-workflow.xml...
Hunk #2 succeeded at 140 (offset -9 lines).
Hunk #3 succeeded at 209 (offset -9 lines).
Checking patch process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java...
Hunk #1 succeeded at 40 (offset -5 lines).
Hunk #2 succeeded at 155 (offset -9 lines).
error: while searching for:
            props.put(prefix + "_partition_filter_java", "${coord:dataInPartitionFilter('input',
'java')}");
        } else if (prefix.equals("output")) {
            props.put(prefix + "_dataout_partitions", "${coord:dataOutPartitions('output')}");
            props.put("shouldRecord", "false"); // todo - override until late data is handled
        }
    }


error: patch failed: process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java:285
error: process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java: patch
does not apply
Checking patch process/src/test/resources/config/process/process-0.1.xml...
Checking patch rerun/src/main/java/org/apache/falcon/latedata/LateDataHandler.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/event/LaterunEvent.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/event/RerunEvent.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/event/RerunEventFactory.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/event/RetryEvent.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/handler/AbstractRerunHandler.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunConsumer.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java...
Checking patch rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java...
Checking patch rerun/src/test/java/org/apache/falcon/rerun/queue/ActiveMQTest.java...
Checking patch webapp/src/test/java/org/apache/falcon/resource/FeedEntityValidationIT.java...
Checking patch webapp/src/test/resources/hive-table-feed.xml...
{code}

> Disable Late data handling for hive tables
> ------------------------------------------
>
>                 Key: FALCON-129
>                 URL: https://issues.apache.org/jira/browse/FALCON-129
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>         Attachments: FALCON-129.patch, FALCON-129-r1.patch, FALCON-129-r2-cumulative.patch
>
>
> HCat nor Hive APIs expose internal stats about a given partition. The only way to get
the partition size is to get the location of the partition on HDFS and then use globStatus
and contentSummary APIs.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message