Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5E98F1007C for ; Thu, 17 Apr 2014 10:06:58 +0000 (UTC) Received: (qmail 58944 invoked by uid 500); 17 Apr 2014 10:06:57 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 58844 invoked by uid 500); 17 Apr 2014 10:06:57 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 58835 invoked by uid 99); 17 Apr 2014 10:06:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 10:06:56 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shwetha.gs@inmobi.com designates 209.85.215.54 as permitted sender) Received: from [209.85.215.54] (HELO mail-la0-f54.google.com) (209.85.215.54) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 10:06:52 +0000 Received: by mail-la0-f54.google.com with SMTP id mc6so186960lab.41 for ; Thu, 17 Apr 2014 03:06:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inmobi.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=AA1N5OJyBjo2pzEbqsUrINvz/7TVas1yW2N5BbTJQd8=; b=knzrSarMxlM/w08x0ZvQqvrMy2BfsTtw2Tiy5HxY9R17q7Grljm1yFpA/oSPVtVmTB dDWyjqoj9CQftEYvVTtctXiFvjmP0j0FDdztsEKx4LLY3mfX58fukPxE4SkCfKLL2/5i HqGUYffUdb9hjIO9IgsYAG0BYx875VgHbcDiw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=AA1N5OJyBjo2pzEbqsUrINvz/7TVas1yW2N5BbTJQd8=; b=gBRF7aF+ZCyXI5MpMSnsSrFvZQQAPkHB2x4zOZIv7lxlVJw8+Sib12Ui1QoUy3ncgv D+IYtjruPNh80GA94FjsfE7AmMvKcNVexEhdTOX/aMe5tyMHTL3NAkg0WFxAsZA0o29J +NCInLhc2h6CKc2Yn2tA5gnUAAYwJSIeXwnNSCooONgbAP/NyMtnQgxARfftqwQXu0Xa rIJd9TcaDwBYnz4JgeYm5uKsTO2yXgF/7gfNCGU+DfLKVu5BMNG96RwGQW1he2ZbKmlE 6JB5Pzz/HgZfoTKPR/EgslgpaenVYMGDQv/hJcQ0xcrDnqh5bEsId97vvK0DvgvGfQX/ 6/nw== X-Gm-Message-State: ALoCoQlcHsMzge3GRV0VBz+qpLg9mnIMKpaHCCUuM3+gscaB66jnMkrJNNOjOeedYTMYMPHbCGgcopRDC73RmzVfeAgTcn3hBzWI6lInvFyOV42OGB8NVX4= MIME-Version: 1.0 X-Received: by 10.112.61.199 with SMTP id s7mr6578513lbr.25.1397729190349; Thu, 17 Apr 2014 03:06:30 -0700 (PDT) Received: by 10.114.243.229 with HTTP; Thu, 17 Apr 2014 03:06:30 -0700 (PDT) In-Reply-To: <056a80beacb24b008f0b06e7d2de365c@MBX1.impetus.co.in> References: <34d3ca1efb094318a8f4305c1fb3b802@MBX1.impetus.co.in> <5e06ac4731bd42d49bad830638430376@MBX1.impetus.co.in> <2a8301f60f3b415b92b425b34843d6e7@MBX1.impetus.co.in> <66b20a2f51cc4326ab4690cf4a1c8f5b@MBX1.impetus.co.in> <056a80beacb24b008f0b06e7d2de365c@MBX1.impetus.co.in> Date: Thu, 17 Apr 2014 15:36:30 +0530 Message-ID: Subject: Re: Hive Tables as Falcon Feed Input Error From: Shwetha GS To: "dev@falcon.incubator.apache.org" Content-Type: multipart/alternative; boundary=001a1133d17ae240eb04f73a2f8c X-Virus-Checked: Checked by ClamAV on apache.org --001a1133d17ae240eb04f73a2f8c Content-Type: text/plain; charset=UTF-8 In hcat cli, check what partitions are there for reservation table On Wed, Apr 16, 2014 at 7:35 PM, Sunil N Kumar wrote: > strange part is that it remain in waiting state in Oozie and there is no > error in any of the log of Hadoop, oozie of falcon. > > > -----Original Message----- > From: Sunil N Kumar > Sent: Wednesday, April 16, 2014 7:33 PM > To: dev@falcon.incubator.apache.org > Subject: RE: Hive Tables as Falcon Feed Input Error > > Hi Shwetha, > I have tried that as well but getting same issue. > HQL: > > set hive.exec.dynamic.partition.mode=nonstrict; > INSERT OVERWRITE TABLE reservationmale partition(gender) select > passenger_id ,first_name ,last_name ,dob ,phone ,email ,address ,city > ,zipcode ,state ,passport_num ,passport_issue_country > ,passport_expiration_date ,ticket_id ,reservation_id ,reservation_status > ,reservation_date ,itinerary_id ,agent_id ,agent_name ,ticket_type > ,ticket_class ,special_request_code ,meal_preference ,payment_id > ,payment_amount ,payment_date ,payment_status, gender from reservationext > where gender='Male'; > > > -----Original Message----- > From: Shwetha GS [mailto:shwetha.gs@inmobi.com] > Sent: Wednesday, April 16, 2014 6:56 PM > To: dev@falcon.incubator.apache.org > Subject: Re: Hive Tables as Falcon Feed Input Error > > Sunil, > > In the input feed: > > > The part after # denotes the partition columns in the hcat table. In this > case, gender is defined twice and not sure how oozie handles this. You can > replace this with
> > Whats the hql that you are using (script.hql)? In hcat, can you check what > partitions are there for reservation table? > > > > On Wed, Apr 16, 2014 at 5:55 PM, Sunil N Kumar > wrote: > > > Hi Shwetha, > > I have created oozie workflow for the Hive using Hive action it is > > working fine. Oozie Workflow sample: > > > > > > > > > > > > ${jobTracker} > > ${nameNode} > > > > ${nameNode}/falcon/sample/hive-site.xml > > > > > > > > > > > > > > > > > > Hive failed, error > > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > > > > > > > Not sure why it is in waiting state in ooze when I am executing as > > oozie process. Find below Falcon entities details: > > Cluster Entity: > > > > > name="demoClsHive" xmlns="uri:falcon:cluster:0.1"> > > > > > > > > > > > > > version="1.2.1"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Input Feed: > > > > > xmlns="uri:falcon:feed:0.1"> > > sample,ac > > hours(1) > > UTC > > > > > > > > > > > > > > > >
uri="catalog:default:reservation#gender='Female';gender='Male'" > > /> > > > > > > > > > > Output Feed: > > > xmlns="uri:falcon:feed:0.1"> > > sample,ac > > hours(1) > > UTC > > > > > > > end="2014-05-21T00:00Z"/> > > > > > > > > > >
> > > > > > > > > > > > Process Definition: > > > xmlns="uri:falcon:process:0.1"> > > > > > > > > > > > > > > 1 > > FIFO > > days(1) > > UTC > > > > > > > name="input"/> > > > > > > > > name="output"/> > > > > > > > > > value="/falcon/sample/hive-site.xml"/> > > > value="/falcon/oozie/share/lib/hive/"/> > > > > > > > > > > > > > > > > > > > > > > Please let me know if any thing suspicious there? > > > > > > > > > > Thanks and Regard. > > Sunil Kumar > > > > > > > > -----Original Message----- > > From: Shwetha GS [mailto:shwetha.gs@inmobi.com] > > Sent: Wednesday, April 16, 2014 2:56 PM > > To: dev@falcon.incubator.apache.org > > Subject: Re: Hive Tables as Falcon Feed Input Error > > > > warning about config in /falcon/staging/falcon/ > > workflows/process/hiveprocess/c3257ea288706fe84d843f6067799d8d/DEFAULT > > / is not an issue. Its just a warning from oozie. > > > > Check that the partition(hcat:// > > Impetus-942.impetus.co.in:9083/default/reservation/gender='Female< > > http://impetus-942.impetus.co.in:9083/default/reservation/gender='Fema > > le> > > ') exists in HCat or if there is an issue with oozie connecting to > > Hcat(exception stacktace in oozie logs). > > > > There are some examples in > > https://issues.apache.org/jira/browse/FALCON-392. > > You can apply the patch and use them > > > > > > On Tue, Apr 15, 2014 at 7:14 PM, Sunil N Kumar > > wrote: > > > > > Hi Srikanth, > > > > > > There is no coord-config-default.xml in the hdfs path > > > /falcon/staging/falcon/workflows/process/hiveprocess/c3257ea288706fe > > > 84 > > > d843f6067799d8d/DEFAULT/ > > > . There is > > > /falcon/staging/falcon/workflows/process/hiveprocess/coordinator.xml > > > file in the path. How does coord-config-default.xml file get > > > created and why workflow is referring to this file. > > > > > > Bundle Job LOG Error: > > > > > > 2014-04-16 00:04:25,018 INFO CoordSubmitXCommand:539 - USER[-] > > > GROUP[-] TOKEN[-] APP[-] JOB[0000000-140416000225157-oozie-clou-B] > > > ACTION[-] configDefault Doesn't exist hdfs:// > > > Impetus-942.impetus.co.in:9000/falcon/staging/falcon/workflows/proce > > > ss > > > /hiveprocess/c3257ea288706fe84d843f6067799d8d/DEFAULT/coord-config-d > > > ef > > > ault.xml > > > > > > > > > > > > -----Original Message----- > > > From: Sunil N Kumar > > > Sent: Tuesday, April 15, 2014 6:23 PM > > > To: dev@falcon.incubator.apache.org > > > Subject: RE: Hive Tables as Falcon Feed Input Error > > > > > > There is no error following are the warning message coming in > > > oozie.log > > > > > > 2014-04-15 23:35:48,554 INFO CoordPushDependencyCheckXCommand:539 - > > > USER[-] GROUP[-] TOKEN[-] APP[-] > > > JOB[0000001-140415003049552-oozie-clou-C] > > > ACTION[0000001-140415003049552-oozie-clou-C@1] First Push missing > > > dependency is [hcat:// > > > Impetus-942.impetus.co.in:9083/default/reservation/gender='Female'] > > > 2014-04-15 23:35:48,573 INFO HCatURIHandler:539 - USER[-] GROUP[-] > > > TOKEN[-] APP[-] JOB[0000001-140415003049552-oozie-clou-C] > > > ACTION[0000001-140415003049552-oozie-clou-C@1] Creating HCatClient > > > for user [cloud] login_user [cloud] and server [thrift:// > > > Impetus-942.impetus.co.in:9083] > > > 2014-04-15 23:35:48,597 WARN HiveConf:1003 - DEPRECATED: > > > Configuration property hive.metastore.local no longer has any effect. > > > Make sure to provide a valid value for hive.metastore.uris if you > > > are connecting to a remote metastore. > > > 2014-04-15 23:35:51,410 INFO CoordPushDependencyCheckXCommand:539 - > > > USER[-] GROUP[-] TOKEN[-] APP[-] > > > JOB[0000003-140415003049552-oozie-clou-C] > > > ACTION[0000003-140415003049552-oozie-clou-C@12] First Push missing > > > dependency is [hcat:// > > > Impetus-942.impetus.co.in:9083/default/reservation/gender='Female'] > > > > > > > > > > > > -----Original Message----- > > > From: Srikanth Sundarrajan [mailto:sriksun@hotmail.com] > > > Sent: Tuesday, April 15, 2014 6:05 PM > > > To: dev@falcon.incubator.apache.org > > > Subject: RE: Hive Tables as Falcon Feed Input Error > > > > > > I would suspect that Oozie has issues connecting to the hive > > > metastore while determining availability of instance. Some logs > > > might help nail down the issue. > > > RegardsSrikanth Sundarrajan > > > > > > > From: suniln.kumar@impetus.co.in > > > > To: dev@falcon.incubator.apache.org > > > > Subject: RE: Hive Tables as Falcon Feed Input Error > > > > Date: Tue, 15 Apr 2014 12:29:29 +0000 > > > > > > > > Hi Srikanth, > > > > It is still in waiting state at oozie. > > > > > > > > > > > > Thanks and Regard. > > > > Sunil Kumar > > > > > > > > > > > > -----Original Message----- > > > > From: Srikanth Sundarrajan [mailto:sriksun@hotmail.com] > > > > Sent: Tuesday, April 15, 2014 5:53 PM > > > > To: dev@falcon.incubator.apache.org > > > > Subject: RE: Hive Tables as Falcon Feed Input Error > > > > > > > > Hi Sunil, Can you confirm if the process has kicked off in hadoop > or > > > is it still in PREP / WAITING state in oozie ? > > > > RegardsSrikanth Sundarrajan > > > > > > > > > From: suniln.kumar@impetus.co.in > > > > > To: dev@falcon.incubator.apache.org > > > > > Subject: RE: Hive Tables as Falcon Feed Input Error > > > > > Date: Tue, 15 Apr 2014 06:37:01 +0000 > > > > > > > > > > Hi, > > > > > I am able to resolve Feed definition issue for the Hive tables. > > > > > It is > > > mandatory to have Hive partition for you hive tables and it also > > > needs to build Oozie with the Hive and Hcatalog jars. Please refer > > > to below blog for building oozie with Hive Dependencies: > > > > > http://blog.nanthrax.net/2014/03/hadoop-cdc-and-processes-notifi > > > > > ca ti on -with-apache-falcon-apache-activemq-and-apache-camel/ > > > > > > > > > > > > > > > But now I am facing issue with the Process execution of falcon > > > > > for the > > > Hive tables. It remain running state from last 18 hours . Not sure > > > what is the issue . can somebody help me to resolve this. > > > > > > > > > > Following is the process definition of the Hive process I have > > > created : > > > > > > > > > xmlns="uri:falcon:process:0.1"> > > > > > > > > > > > > > > > > > start="2014-04-01T00:00Z"/> > > > > > > > > > > > > > > > > > > > > 1 > > > > > FIFO > > > > > days(1) > > > > > UTC > > > > > > > > > > > > > > > > feed="reservation2" > > > name="input"/> > > > > > > > > > > > > > > > > > > > > > > name="output"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > workflow-path="/falcon/hive_run/"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Sunil N Kumar > > > > > Sent: Monday, April 14, 2014 12:58 PM > > > > > To: dev@falcon.incubator.apache.org > > > > > Subject: Hive Tables as Falcon Feed Input Error > > > > > > > > > > Hi, > > > > > > > > > > I am trying to build sample around executing Hive Script as part > > > > > of > > > Falcon Oozie Process. I have created two tables in the Hive and want > > > to join data from these tables and put data into third Hive table > > > using falcon Process. But I am not able to submit Input feed into > > > Falcon Server. I am using Falcon 0.4 release. > > > > > > > > > > > > > > > > > > > > Input Feed XML: > > > > > > > > > > > > > > > > > > > > > > > > xmlns="uri:falcon:feed:0.1"> > > > > > > > > > > hours(1) > > > > > > > > > > UTC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > end="2012-04-21T00:00Z"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Error: > > > > > > > > > > [cloud@Impetus-942 falcon-distributed-0.4-incubating]$ > > > > > bin/falcon entity -submit -type feed -file > > > > > examples/entity/in-feed.xml > > > > > > > > > > Error: java.net.URISyntaxException: URI path is not in expected > > > format: database:table: catalog:default:reservationnew:gender='Male' > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > > > proprietary, privileged or otherwise protected by law. The message > > > is intended solely for the named addressee. If received in error, > > > please destroy and notify the sender. Any use of this email is > > > prohibited when received in error. Impetus does not represent, > > > warrant and/or guarantee, that the integrity of this communication > > > has been maintained nor that the communication is free of errors, > > > virus, > > interception or interference. > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > > > proprietary, privileged or otherwise protected by law. The message > > > is intended solely for the named addressee. If received in error, > > > please destroy and notify the sender. Any use of this email is > > > prohibited when received in error. Impetus does not represent, > > > warrant and/or guarantee, that the integrity of this communication > > > has been maintained nor that the communication is free of errors, > > > virus, > > interception or interference. > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > > > proprietary, privileged or otherwise protected by law. The message > > > is intended solely for the named addressee. If received in error, > > > please destroy and notify the sender. Any use of this email is > > > prohibited when received in error. Impetus does not represent, > > > warrant and/or guarantee, that the integrity of this communication > > > has been maintained nor that the communication is free of errors, > > > virus, > > interception or interference. > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > > > proprietary, privileged or otherwise protected by law. The message > > > is intended solely for the named addressee. If received in error, > > > please destroy and notify the sender. Any use of this email is > > > prohibited when received in error. Impetus does not represent, > > > warrant and/or guarantee, that the integrity of this communication > > > has been maintained nor that the communication is free of errors, > > > virus, > > interception or interference. > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > > > proprietary, privileged or otherwise protected by law. The message > > > is intended solely for the named addressee. If received in error, > > > please destroy and notify the sender. Any use of this email is > > > prohibited when received in error. Impetus does not represent, > > > warrant and/or guarantee, that the integrity of this communication > > > has been maintained nor that the communication is free of errors, > > > virus, > > interception or interference. > > > > > > > -- > > _____________________________________________________________ > > The information contained in this communication is intended solely for > > the use of the individual or entity to whom it is addressed and others > > authorized to receive it. It may contain confidential or legally > > privileged information. If you are not the intended recipient you are > > hereby notified that any disclosure, copying, distribution or taking > > any action in reliance on the contents of this information is strictly > > prohibited and may be unlawful. If you have received this > > communication in error, please notify us immediately by responding to > > this email and then delete it from your system. The firm is neither > > liable for the proper and complete transmission of the information > > contained in this communication nor for any delay in its receipt. > > > > ________________________________ > > > > > > > > > > > > > > NOTE: This message may contain information that is confidential, > > proprietary, privileged or otherwise protected by law. The message is > > intended solely for the named addressee. If received in error, please > > destroy and notify the sender. Any use of this email is prohibited > > when received in error. Impetus does not represent, warrant and/or > > guarantee, that the integrity of this communication has been > > maintained nor that the communication is free of errors, virus, > interception or interference. > > > > -- > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. > > ________________________________ > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. --001a1133d17ae240eb04f73a2f8c--