nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tripathi, Shiv Deepak" <>
Subject RE: Use Case...Please help
Date Mon, 16 May 2016 12:44:23 GMT
Thanks a lot Mark.

Looking forward to try it out.

If I understood correctly than I can drop log copying script and staging machine and can directly
pull the logs from repository.

Please confirm.


From: Mark Payne []
Sent: Monday, May 16, 2016 5:06 PM
Subject: Re: Use Case...Please help


Thanks for providing such a detailed description of your use case. I think NiFi would be an
tool to help you out here!

As I mentioned before, you would typically use ListFile -> FetchFile to pull the data in.
Clearly, here,
though, you want to be more selective about what you pull in. You can accomplish this by using
RouteOnAttribute processor. So you'd have something like: ListFile -> RouteOnAttribute
-> FetchFile.
The RouteOnAttribute processor is very powerful and allows you to configure how to route each
of data based on whatever attributes are available. The ListFile Processor adds the following
to each piece of data that it pulls in:

filename (name of file)
path (relative path of file)
absolute.path (absolute directory of file)
fs.owner (owner of the file) (group that the file belongs to)
fs.lastModified (last modified date)
fs.length (file length)
fs.permissions (file permissions, such as rw-rw-r--)

From these, you can make all sorts of routing decisions, based on name, timestamp, etc. You
can choose
to terminate data that does not meet your criteria.

When you use FetchFile, you have the option of deleting the source file, moving it elsewhere,
or leaving
it as-is. So you wouldn't need to delete it if you don't want to. This is possible because
ListFile keeps track
of what has been 'listed'. So it won't ingest duplicate data, but it will pick up new files
(if any existing
file is modified, it will pick up the new version of the file.)

You can then use UnpackContent if you want to unzip the data, or you can leave it zipped.
After the FetchFile,
you can also use a RouteOnAttribute processor to separate out the XML from the log files and
put those to
different directories in HDFS.

Does this sound like it will provide you all that you need?


On May 16, 2016, at 3:06 AM, Tripathi, Shiv Deepak <<>>

Hi Mark,

I am very happy to see the detailed reply. I am very thankful to you. So explaining more about
my use case below.

1-      Screenshot Name--> “Stagingdirectory_copiedfiles”

Log copying script will copy the log in the  staging directory which is in my case “D:\PaiValidation”
and will maintain multiple folders. These folders are nothing but device serial no. Every
serial no will have multiple log files and xml files as on  Each day one new log files used
to come in this directory, as you can see.

In log copy script we defined that how many days logs we want. So lets say we passed 360 ,
so it will copy logs from last 360 days and as it is continuously running so after 10 days
of when you passed configuration very first time it will have logs 360(last 360 days from
the time when you passed this parameter to script) +10 days+ growing day by day= 370++++++

And after pushing the files to cluster we are renaming or creating dummy files with 0 byte
as you can see in screenshot

Also we are passing one more parameter which specifies the device serial no of which we want
logs not from all devices.

2-      The source repository
Screenshot Name --> “Repository files”

This is actual repository from where we are taking the logs and copying it to staging directory.
These are incoming logs from the device and every serial no is having multiple types of files
as you can see in screenshot.  We need only  log files with log************.zip pattern and
xml file and rest of them we will not pick up. Also these logs in repository we can’t delete.

3-      HDFS directory

From staging directory flume is  moving files to our on HDFS premise cluster
Screenshot Name--> HDFS1

You can see two highlighted folder in this screenshot one will have only log files other will
have xml files.
If you go back and see the “Stagingdirectory_copiedfiles” you will find xml and log files
under same device serial no which we are storing separately in cluster

Screenshot name --> hdfs2

Logs will be stored under same directory structure as it was in staging. For both xml files
and log files.

So if I want to accomplish above goals nifi will be best solution?

If I use nifi directly to the repository to pull the logs whether I can be able to do these
few things:

1-      It should not copy duplicate logs as from destination we will be deleting logs.
2-      It should only copy the logs of last 20 days or last 50 days like any days and if
the new logs comes in directory each day it should pull up that too.
3-      It should not delete any logs from the source repository.
4-      It should copy specified logs in one directory and xml in other directory in HDFS.

In such a case we can remove the concept of script.

Hoping for best.


From: Mark Payne []
Sent: Monday, May 16, 2016 1:25 AM
Subject: Re: Use Case...Please help

Hi Deepak,

Certainly, this is something that you could use NiFi for. We often see people using NiFi to
sync data from
a directory on local disk to a directory in HDFS. This is typically accomplished by using
a flow like:

ListFile -> FetchFile -> PutHDFS

You can then create a file in the source directory with the same name by using ReplaceText
to set the content
to nothing and then PutFile to write out the 0-byte content. So the flow would look like:

ListFile -> FetchFile -> PutHDFS -> ReplaceText -> PutFile

PutHDFS has a "Directory" property. If you set this value to "${path}" it will use the same
directory structure that
ListFile found the file to be in when it performed the listing. I.e., if you set ListFile
to pull from /data/mydir
and "Recurse Subdirectories" to true, then any file found in /data/mydir will have a 'path'
of './' and anything found in
/data/mydir/subdir1 will have a path of './subdir1'. If you would rather have the fully qualified
path (/data/mydir/subdir1)
you would use "${absolute.path}" instead of "${path}"

One thing that I find curious about your scenario though is the concept of a 'log copy script'
and then putting back
a 0-byte file so that the script does not pick up the data again. Why not just use NiFi to
pull directly from the source
and avoid using a script all together? The ListFile processor will keep track of what has
been pulled in already,
so it won't copy the data multiple times. But I may not be clear on this point. Is the "Log
repository" that you mention
just a directory that NiFi could pull from, or is it some other sort of repository?


On May 15, 2016, at 3:23 PM, Tripathi, Shiv Deepak <<>>


Currently I am using flume for data ingestion and my use case as follows

Log repository--------log copy Script-------> Staging directory for  copied logs

Staging directory for  copied logs folder structure----Machine1log----a.log

Flume will copy these logs and replicate same structure in HDFS cluster. Beginning with which
is :

And creates 0 byte dummy file with same name so that Script wont copy the same log again as
it find 0 byte file already existing in source directory.

Can we do same things with apache nifi?

Keeping in mind two goals- same folder structure in HDFS and after moving file to HDFS it
should crete 0 byte dummy file in source directory.

Please help


With Best Regards,
Deepak Tripathi
Philips Innovation campus

The information contained in this message may be confidential and legally protected under
applicable law. The message is intended solely for the addressee(s). If you are not the intended
recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction
of this message is strictly prohibited and may be unlawful. If you are not the intended recipient,
please contact the sender by return e-mail and destroy all copies of the original message.

<repository files.jpg><Stagingdirectory_copiedfiles.jpg><HDFS1.jpg><hdfs2.jpg>

View raw message