nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Percivall <joeperciv...@yahoo.com>
Subject Re: Nifi Kite-dataset URI issue
Date Tue, 22 Dec 2015 13:13:43 GMT
Are you saying the Hive dataset is not yet supported in Kite? Joe- - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
joepercivall@yahoo.com
 

    On Tuesday, December 22, 2015 7:41 AM, panfei <cnweike@gmail.com> wrote:
 

 hive dataset is not supported  yet.


2015年12月22日星期二,Chandu Koripella <ckoripel@starbucks.com> 写道:
> Hi Joe,
>
>  
>
> I am still facing this issue. any help is appreciated.
>
> I am not sure why dataset:hive:namespace/dataset_name didn’t get validated in NIFI.
Same does work when I use kite-dataset CLI.
>
>  
>
> Thanks,
>
> Chandra
>
>  
>
>  
>
> From: Joe Percivall [mailto:joepercivall@yahoo.com]
> Sent: Monday, December 21, 2015 6:38 AM
> To: users@nifi.apache.org
> Subject: Re: Nifi Kite-dataset URI issue
>
>  
>
> Hello Chandra,
>
>  
>
> Are you still running into issues?
>
>  
>
> If so, can someone with more experience with Kite and Hadoop chime in?
>
>  
>
> Joe
>
> - - - - - - 
>
> Joseph Percivall
>
> linkedin.com/in/Percivall
>
> e: joepercivall@yahoo.com
>
>  
>
>  
>
> On Wednesday, December 16, 2015 2:08 PM, Chandu Koripella <ckoripel@starbucks.com>
wrote:
>
>  
>
> HI,
>
>  
>
> I am running into another issue. I really appreciate your help. My apology if I am asking
very basic question. I am first time user for both Nifi, Kite dataset and Hadoop
>
>  
>
> My Nifi is service is running as HDFS user.
>
>  
>
> ·        Create a dataset as HDFS and I could able to import just fine as HDFS
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.1&disp=emb&zw&atsh=1>
>
>  
>
> ·        Permissions:
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.2&disp=emb&zw&atsh=1>
>
>  
>
>  
>
> ·        URI Validates just fine in NIfi
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.3&disp=emb&zw&atsh=1>
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.4&disp=emb&zw&atsh=1>
>
>  
>
> ·        Hadoop Configuration files: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml
>
>  
>
> Any help is appreciated.
>
>  
>
> Thanks,
>
> Chandra
>
>  
>
> From: Chandu Koripella [mailto:ckoripel@starbucks.com]
> Sent: Monday, December 14, 2015 11:49 AM
> To: users@nifi.apache.org
> Subject: RE: Nifi Kite-dataset URI issue
>
>  
>
> You are right. It works just fine by changing the keyword to hdfs. Kit-dataset accepting
the both hdfs and hive
>
>  
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.5&disp=emb&zw&atsh=1>
>
>  
>
> Thanks,
>
> Chandu
>
>  
>
>  
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Monday, December 14, 2015 11:37 AM
> To: users@nifi.apache.org
> Subject: Re: Nifi Kite-dataset URI issue
>
>  
>
> the property to focus on is 'Target Dataset URI'
>
>  
>
> I'm no expert on Kite but a quick check of their docs makes me think the current URI
being used really is not valid.
>
>  
>
> Try dataset:hdfs:/default/ctest1
>
>  
>
>  
>
> On Mon, Dec 14, 2015 at 2:33 PM, Chandu Koripella <ckoripel@starbucks.com> wrote:
>
> HI Alan,
>
>  
>
> Thanks for responding. Hive dataset has 777 permissions. I am not sure why it still unable
to validate it. I can load the data just fine if execute kite-dataset command on the server.
can you please take a look?
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.6&disp=emb&zw&atsh=1>
>
>  
>
> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-log4j.properties
>
>  
>
> Here is last few lines from nifi logs.
>
>  
>
> 2015-12-14 11:07:26,671 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 27 milliseconds
>
> 2015-12-14 11:09:16,643 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false
>
> 2015-12-14 11:09:26,672 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:09:26,700 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 28 milliseconds (Stop-the-world time = 12 milliseconds, Clear Edit Logs time = 13 millis),
max Transaction ID -1
>
> 2015-12-14 11:09:26,700 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 28 milliseconds
>
> 2015-12-14 11:09:42,649 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false
>
> 2015-12-14 11:10:33,658 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false
>
> 2015-12-14 11:11:26,700 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:11:26,740 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 39 milliseconds (Stop-the-world time = 12 milliseconds, Clear Edit Logs time = 24 millis),
max Transaction ID -1
>
> 2015-12-14 11:11:26,740 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 39 milliseconds
>
> 2015-12-14 11:13:26,740 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:13:26,769 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 28 milliseconds (Stop-the-world time = 12 milliseconds, Clear Edit Logs time = 13 millis),
max Transaction ID -1
>
> 2015-12-14 11:13:26,769 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 28 milliseconds
>
> 2015-12-14 11:15:21,314 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false
>
> 2015-12-14 11:15:26,769 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:15:26,796 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 27 milliseconds (Stop-the-world time = 11 milliseconds, Clear Edit Logs time = 12 millis),
max Transaction ID -1
>
> 2015-12-14 11:15:26,796 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 27 milliseconds
>
> 2015-12-14 11:15:47,820 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false
>
> 2015-12-14 11:16:28,328 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService
Saved flow controller org.apache.nifi.controller.FlowController@66f2383d // Another save pending
= false
>
> 2015-12-14 11:17:26,797 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:17:26,824 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 27 milliseconds (Stop-the-world time = 10 milliseconds, Clear Edit Logs time = 13 millis),
max Transaction ID -1
>
> 2015-12-14 11:17:26,825 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 27 milliseconds
>
> 2015-12-14 11:19:26,825 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:19:26,900 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 74 milliseconds (Stop-the-world time = 59 milliseconds, Clear Edit Logs time = 13 millis),
max Transaction ID -1
>
> 2015-12-14 11:19:26,900 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 records in 75 milliseconds
>
> 2015-12-14 11:21:26,900 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Initiating checkpoint of FlowFile Repository
>
> 2015-12-14 11:21:26,928 INFO [pool-18-thread-1] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records and 0 Swap Files
in 27 milliseconds (Stop-the-world time = 12 milliseconds, Clear Edit Logs time = 13 millis),
max Transaction ID -1
>
> 2015-12-14 11:21:26,928 INFO [pool-18-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository
Successfully checkpointed FlowFile Repository with 0 record
>
>  
>
>  
>
>  
>
>  
>
> From: Alan Jackoway [mailto:alanj@cloudera.com]
> Sent: Monday, December 14, 2015 11:05 AM
>
> To: users@nifi.apache.org
> Subject: Re: Nifi Kite-dataset URI issue
>
>  
>
> Try adding the path to hive-site.xml.
>
>  
>
> Did it give you any error message beyond just "cannot validate url"? Is there anything
interesting in the nifi-app.log?
>
>  
>
> On Mon, Dec 14, 2015 at 12:54 PM, Chandu Koripella <ckoripel@starbucks.com> wrote:
>
> It would be great help if someone can take a look at this.
>
>  
>
> Thanks,
>
> Chandu
>
>  
>
> From: Chandu Koripella
> Sent: Friday, December 11, 2015 11:44 AM
> To: 'users@nifi.apache.org'
> Subject: RE: Nifi Kite-dataset URI issue
>
>  
>
> Hi,
>
>  
>
> This is final step in implementing first NIfi job. I really appreciate any quick tips
to resolve this this.
>
>  
>
> Thanks,
>
> Chandu
>
>  
>
>  
>
> From: Chandu Koripella
> Sent: Friday, December 11, 2015 7:03 AM
> To: users@nifi.apache.org
> Subject: RE: Nifi Kite-dataset URI issue
>
>  
>
> Hi Allen,
>
>  
>
> Thanks for helping me with this issue. Here is the full value Hadoop config property.
>
>  
>
> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.7&disp=emb&zw&atsh=1>
>
>  
>
> This works just fine with putHDFS processor
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.8&disp=emb&zw&atsh=1>
>
>  
>
> From: Alan Jackoway [mailto:alanj@cloudera.com]
> Sent: Friday, December 11, 2015 5:42 AM
> To: users@nifi.apache.org
> Subject: Re: Nifi Kite-dataset URI issue
>
>  
>
> Can you give the full value of that hadoop configuration files property?
>
> On Dec 10, 2015 11:23 PM, "Chandu Koripella" <ckoripel@starbucks.com> wrote:
>
> Hi,
>
>  
>
> I am trying to load data into a hive external table. Nifi  is unable to validate the
URI. It is working just fine when I use kite-dataset.
>
> Can you please advice if am missing any parameters here?
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.9&disp=emb&zw&atsh=1>
>
>  
>
> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.10&disp=emb&zw&atsh=1>
>
>  
>
> Thanks,
>
> Chandu
>
>  
>
>  
>
>  

-- 
不学习,不知道



  
Mime
View raw message