Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
MIME-Version: 1.0
In-Reply-To: <CALo_M19TNjAYb=3EFCX3NTb1bW1dZ9OLcqd8SJ5UWnDA-oOsGQ@mail.gmail.com>
References: <9D4F093F9B7E72499E2ED901D1A60B4831710050@hou150w8xmbx04.hou150.chevrontexaco.net>
 <CAEV8zdXoPXBNhZ+cqh1qMXeC1qtL-MdbLDw3YCSu+3yQhYF6nQ@mail.gmail.com> <CALo_M19TNjAYb=3EFCX3NTb1bW1dZ9OLcqd8SJ5UWnDA-oOsGQ@mail.gmail.com>
From: Matt Burgess <mattyb149@apache.org>
Date: Tue, 17 Jan 2017 18:57:00 -0500
Message-ID: <CAEV8zdUKpT9DCUVMcZiXBLF=78RN=+cyavgg=592n7+b3S_-0g@mail.gmail.com>
Subject: Re: Regular Expression, or Similar Processor
To: dev@nifi.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Tue, 17 Jan 2017 23:57:04 -0000

Good call Bryan, thanks!  That template and another
(SyslogExample.xml) are available at [1].

Regards,
Matt

https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

On Tue, Jan 17, 2017 at 6:45 PM, Bryan Bende <bbende@gmail.com> wrote:
> One more resource to add to Matt's list:
>
> https://blogs.apache.org/nifi/entry/storing_syslog_events_in_hbase
>
> Thanks,
>
> Bryan
>
> On Tue, Jan 17, 2017 at 6:35 PM, Matt Burgess <mattyb149@apache.org> wrote:
>
>> Oziel,
>>
>> The ListenSyslog processor [1] will apply the regular expression and
>> extract each of the fields into flow file attributes. From there you
>> could use AttributesToJSON [2] with "Include Core Attributes" set to
>> false, that should give you fields named "syslog.hostname" for
>> example. You could use JoltTransformJSON [3] if you need to
>> rename/reorganize the fields, or if you need more complex logic and
>> are familiar with a scripting language such as Groovy, Javascript,
>> Python, Ruby, or Lua, you could use ExecuteScript [4] to build a
>> custom JSON output, I have examples of custom JSON transformations
>> using Groovy [5], Javascript [6], and Jython [7].
>>
>> If you are looking for the regular expression(s) to do the parsing
>> yourself (perhaps with ExtractText [8]), you can find them here [9].
>>
>> Regards,
>> Matt
>>
>> [1] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.ListenSyslog/index.html
>> [2] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.AttributesToJSON/index.html
>> [3] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.JoltTransformJSON/index.html
>> [4] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.script.ExecuteScript/index.html
>> [5] http://funnifi.blogspot.com/2016/02/executescript-json-to-
>> json-conversion.html
>> [6] http://funnifi.blogspot.com/2016/03/executescript-json-to-
>> json-revisited.html
>> [7] http://funnifi.blogspot.com/2016/03/executescript-json-to-
>> json-revisited_14.html
>> [8] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.ExtractText/index.html
>> [9] https://github.com/apache/nifi/blob/master/nifi-nar-
>> bundles/nifi-standard-bundle/nifi-standard-processors/src/
>> main/java/org/apache/nifi/processors/standard/syslog/SyslogParser.java#L36
>>
>> On Tue, Jan 17, 2017 at 5:28 PM, Hinojosa, Ozzie
>> <Oziel.Hinojosa@chevron.com> wrote:
>> > Hi,
>> >
>> > I am in the process of reading the Apache NiFi documentation and setting
>> up a single instance to obtain basic working knowledge of the platform. But
>> in the meantime I need an answer regarding Syslog messages and guidance on
>> which educational resources would be best to leverage in order to
>> understand NiFi in depth and in the shortest time possible.
>> >
>> > My questions regarding Syslog. How can I parse the FlowFile contents
>> produce by a ListenSyslog processor. By parse, I mean apply regular
>> expressions and transform the entire contents to JSON. I know extensibility
>> is an integral part of the platform, but again I am new to NiFi and I am
>> looking to confirm the feasibility of parsing Syslog messages; to add
>> additional complexity Netflows would have the same requirement as the
>> Syslog messages. What is available out of the box for Netflows and JFlows?
>> >
>> > Thanks, much appreciated.
>> >
>> > Oziel Hinojosa
>> > Security Technologies Analyst
>> > Strategy, Service Delivery & Programs (SSDP)
>> > oziel.hinojosa@chevron.com
>> >
>> > CITC - IRSM
>> > 1600 Smith St.  HOU160-23042B
>> > Houston, TX 77002
>> >
>> > Tel +1 713 754 4749
>> >
>> > Use http://csocbehavior.chevron.com/ to give me or others CSOC
>> Behavioral Feedback, my CAI is OZNH.
>> >
>>