nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" <chris.mcderm...@hpe.com>
Subject Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Date Tue, 22 Mar 2016 14:55:18 GMT
I’ve entered a Jira issue, NIFI-1660, to enhance the expression language to be able to evaluate
a Json path.  I plan on submitting a pull request on it in the next week or so.  




On 3/22/16, 2:53 AM, "Conrad Crampton" <conrad.crampton@SecData.com> wrote:

>My 2p. 
>If the kaka.key value (very simple json), you could use UpdateAttribute and use some expression
language - specifically the string manipulation functions to extract the part you want.
>I like the power or ExecuteProcessor by the way.
>
>And I agree, this community is phenomenally responsive and helpful.
>
>Regards
>Conrad
>
>
>
>
>On 21/03/2016, 18:38, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" <chris.mcdermott@hpe.com>
wrote:
>
>>Thanks everyone.  While I’m naturally disappointed that this doesn’t exist, I
am hyper charged about the responsiveness and enthusiasm of the NiFi community!
>>
>>From: Matt Burgess <mattyb149@gmail.com<mailto:mattyb149@gmail.com>>
>>Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
>>Date: Monday, March 21, 2016 at 1:58 PM
>>To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
>>Subject: Re: Help on creating that flow that requires processing attributes in a flow
content but need to preserve the original flow content
>>
>>One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which gives you full
control over the session and flowfile(s).  For example if you had JSON in your "kafka.key"
attribute such as "{"data": {"myKey": "myValue"}}" , you could use the following Groovy script
to parse out the value of the 'data.myKey' field:
>>
>>def flowfile = session.get()
>>if(!flowfile) return
>>def json = new groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
>>flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
>>session.transfer(flowfile, REL_SUCCESS)
>>
>>
>>I put an example of this up as a Gist (https://gist.github.com/mattyb149/478864017ec70d76f74f)
>>
>>A possible improvement could be to add a "jsonPath" function to Expression Language,
which could take any value (including an attribute) along with a JSONPath expression to evaluate
against it...
>>
>>Regards,
>>Matt
>>
>>On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)
<chris.mcdermott@hpe.com<mailto:chris.mcdermott@hpe.com>> wrote:
>>Joe,
>>
>>Thanks for the reply.  I think I was not clear.
>>
>>The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I need to
be able to evaluate without modifying the original FlowFile content (which was read from the
Kafka topic).  What I can’t figure out is how to squirrel away the flowfile content so that
I can write the value of the kafka.key attribute to the FlowFile content, so that I can process
it with EvaluateJsonPath, and then read content I squirreled away back into the FlowFile content.
I considered using the the DistributedMapCache, but there would be no guarantee what I added
to the cache would still be there when I needed it back.
>>
>>
>>
>>
>>On 3/21/16, 1:37 PM, "Joe Witt" <joe.witt@gmail.com<mailto:joe.witt@gmail.com>>
wrote:
>>
>>>Chris,
>>>
>>>Sounds like you have the right flow in mind already.  EvaluateJSONPath
>>>does not write content.  It merely evaluates the given jsonpath
>>>expression against the content of the flowfile and if appropriate
>>>creates a flowfile attribute of what it finds.
>>>
>>>For example if you have JSON from Twitter you can use EvaluateJsonPath
>>>and add a property with a name
>>>'twitter.user' and a value of '$.user.name<http://user.name>'
>>>
>>>Once you run the tweets through each flow file will have an attribute
>>>called 'twitter.user' with the name found in the message.  No
>>>manipulation of content at all.  Just promotes things it finds to flow
>>>file attributes.
>>>
>>>Thanks
>>>Joe
>>>
>>>On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU -
>>>STaTS/StorefrontRemote) <chris.mcdermott@hpe.com<mailto:chris.mcdermott@hpe.com>>
wrote:
>>>> What I need to do is read a file from Kafka.  The Kafka key contains a JSON
string which I need to turn in FlowFile attributes while preserving the original FlowFile
content.  Obviously I can use EvaluteJsonPath but that necessitates replacing the FlowFile
content with the kaka.key attribute, thus loosing the original FlowFile content.  I feel like
I’m missing something fundamental.
>>
>>
>>
>> ***This email originated outside SecureData***
>>
>>Click https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==  to report this email
as spam.
>
>
>SecureData, combating cyber threats
>______________________________________________________________________ 
>The information contained in this message or any of its attachments may be privileged
and confidential and intended for the exclusive use of the intended recipient. If you are
not the intended recipient any disclosure, reproduction, distribution or other dissemination
or use of this communications is strictly prohibited. The views expressed in this email are
those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are
only valid if followed up by a formal written quote.
>
>SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address:
SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
Mime
View raw message