Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E97A8100DC for ; Tue, 25 Mar 2014 09:09:13 +0000 (UTC) Received: (qmail 58966 invoked by uid 500); 25 Mar 2014 09:09:13 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 58938 invoked by uid 500); 25 Mar 2014 09:09:12 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 58926 invoked by uid 99); 25 Mar 2014 09:09:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2014 09:09:07 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 25 Mar 2014 09:09:06 +0000 Received: (qmail 58602 invoked by uid 99); 25 Mar 2014 09:08:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2014 09:08:46 +0000 Date: Tue, 25 Mar 2014 09:08:45 +0000 (UTC) From: "Shwetha G S (JIRA)" To: dev@falcon.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FALCON-365) Remove the checked in oozie xsds MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/FALCON-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946319#comment-13946319 ] Shwetha G S commented on FALCON-365: ------------------------------------ I haven't changed any java code that prepares the hive workflow(not even imports) and its the java code that sets params for hive workflow. Since the compilation is fine, I don't see any issue why hive workflow should fail. Moreover there was no difference between hive xsd that was there in falcon and the one in oozie-client. Anyway, I decided to test a process with hive workflow and it doesn't work. I think the issue is with the oozie EL extension that we use. Here are the details of hql: {noformat} Script [wordcount.hql] content: ------------------------ INSERT OVERWRITE TABLE $falcon_output_table PARTITION($falcon_output_dataout_partitions) SELECT word, SUM(cnt) as cnt FROM $falcon_input_table WHERE $falcon_input_filter GROUP BY word; ------------------------ Parameters: ------------------------ falcon_input_table=in_table falcon_input_database=default falcon_input_storage_type=TABLE falcon_input_catalog_url=thrift://localhost:12000 falcon_input_filter=(ds='2013-11-15-00-04') OR (ds='2013-11-15-00-03') OR (ds='2013-11-15-00-02') OR (ds='2013-11-15-00-01') OR (ds='2013-11-15-00-00') falcon_output_catalog_url=thrift://localhost:12000 falcon_output_dataout_partitions='ds=2013-11-15-00-05' falcon_output_dated_partition_value=2013-11-15-00-05 falcon_output_storage_type=TABLE falcon_output_database=default falcon_output_table=out_table ------------------------ Hive command arguments : --hivevar falcon_input_table=in_table --hivevar falcon_input_database=default --hivevar falcon_input_storage_type=TABLE --hivevar falcon_input_catalog_url=thrift://localhost:12000 --hivevar falcon_input_filter=(ds='2013-11-15-00-04') OR (ds='2013-11-15-00-03') OR (ds='2013-11-15-00-02') OR (ds='2013-11-15-00-01') OR (ds='2013-11-15-00-00') --hivevar falcon_output_catalog_url=thrift://localhost:12000 --hivevar falcon_output_dataout_partitions='ds=2013-11-15-00-05' --hivevar falcon_output_dated_partition_value=2013-11-15-00-05 --hivevar falcon_output_storage_type=TABLE --hivevar falcon_output_database=default --hivevar falcon_output_table=out_table -f wordcount.hql {noformat} The issue is, this generates hql as {noformat} INSERT OVERWRITE TABLE out_table PARTITION(ds=2013-11-15-00-05) SELECT word, SUM(cnt) as cnt FROM in_table WHERE (ds='2013-11-15-00-04') OR (ds='2013-11-15-00-03') OR (ds='2013-11-15-00-02') OR (ds='2013-11-15-00-01') OR (ds='2013-11-15-00-00') GROUP BY word; {noformat} it should be (quote around output partition value) {noformat} INSERT OVERWRITE TABLE out_table PARTITION(ds='2013-11-15-00-05') SELECT word, SUM(cnt) as cnt FROM in_table WHERE (ds='2013-11-15-00-04') OR (ds='2013-11-15-00-03') OR (ds='2013-11-15-00-02') OR (ds='2013-11-15-00-01') OR (ds='2013-11-15-00-00') GROUP BY word; {noformat} > Remove the checked in oozie xsds > -------------------------------- > > Key: FALCON-365 > URL: https://issues.apache.org/jira/browse/FALCON-365 > Project: Falcon > Issue Type: Bug > Reporter: Shwetha G S > Assignee: Shwetha G S > Fix For: 0.5 > > Attachments: FALCON-365-v2.patch, FALCON-365.patch > > > Oozie xsds for workflow, coordinator, bundle etc are part of oozie-client. We should use the xsds from the client jar -- This message was sent by Atlassian JIRA (v6.2#6252)