nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzhu <js.tianlu...@gmail.com>
Subject Re: How to count the number of occurrences of a certain string in file
Date Wed, 08 Nov 2017 21:46:25 GMT
Hi Mark,

I am confused about the whole process. I have the following questions:

1. From what I read, I can use TailFile to read the log file. However, it
would only read the file once (as the input file does not change). Is there
a way that I can read the file every time it gets started?

2. As you suggested, I am writing a personal Python script to handle the
count. Most of the examples online are in Jython.(My NiFi version is 1.3.0.
I suppose it's similar to Python, correct?) 
I find 
https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html
<https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html>
 
as a useful guide, but I don't understand what to choose from. What's the
input and output for the script? I want to read the file line by line and
count the string occurrences. I'm currently using  key,value =
flowFile.getAttributes().iteritems() to get the file content, but it shows
"too many values to unpack". (The original file is 41.14MB)
For the output, the common way seems to be using a callback. Is it necessary
in my case? Or can I just add these attributes to the output file and
extract the attributes later?

3. To write the columns into the SQL table, the common way seems to use
"ReplaceText" and "PutSQL". I also noticed there's a processor called
PutDatabaseRecord that might combine the function of ExecuteScript and
PutSQL together. Since my Python script doesn't work so far I can't really
test the result. But is this an easier approach?

Any help is appreciated...

Thanks,
Tina



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Mime
View raw message