nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: GetDynamoDB Processor returns only item attribute from DynamoDB but not all attributes.
Date Tue, 05 Jul 2016 13:15:53 GMT
Hi Mike,

Personally, I'm not familiar with DynamoDB either, but a quick look at the source & configuration
do indeed
show that it expects the user to fetch only a single 'attribute' from a Dynamo Item. Looking
at the AWS javadocs,
it appears that this is the typical usage, as they don't seem to provide a method for obtaining
the entire item
as JSON (unless I'm missing something).

One approach, though, would be to allow the 'Json Document attribute' property to be optional
instead of required
and if not specified, simply iterate over all of the keys in the Dynamo DB Item and create
a JSON object ourselves
that we write to the content of the FlowFile.

Often what happens, though, is that a member of the NiFi community develops a processor to
provide the functionality
that they need and sometimes makes the processor a little more 'generic' to handle additional
use cases, if it makes sense.
As the community begins to use the processor, it begins to evolved over time to meet more
and more use cases. This is
a fairly new processor and so likely will evolve over time to meet more & more use cases.
I think the use case of pulling back
an entire DynamoDB item is very reasonable and likely will be needed by others.

Is there any chance that you're interested in working on building out that capability and
contributing it to the community? 
We would be more than happy to help shepherd you through the process of contributing it back
to make NiFi's dynamo
support better!

Thanks
-Mark


> On Jul 5, 2016, at 8:51 AM, Mike Connolly <mike.connolly@oracle.com> wrote:
> 
> James,
>    Hmmm. I'm not too familiar with DynamoDB or NiFi, but having a processor that can
only return one attribute at a time doesn't seem very useful. I would think the processor
would retrieve all the attributes and package them into one variable and send it back as one
unit so when the user configures the processor they could do something like set the json attribute
value to "ALL" or something like that. I tried modifying the processor source code to call
item.getJSON() to populate the return value with everything, but something isn't liking {
brace in the string value that gets sent back. Its complaining about unacceptable characters
in the string. I might need to swich to MongoDB as my datasource and see if that works.
> 
> Thanks,
> 
> Mike
> 
> On 07/01/2016 02:41 AM, James Wing wrote:
>> Mike,
>> 
>> I believe it was an intentional design choice to separate the hash/range keys from
the flowfile content for validation and handling.  There is currently no option in PutDynamoDB
or GetDynamoDB  to match the entire flowfile content to the entire DynamoDB item, but I believe
that is certainly a possible extension.
>> 
>> Thanks,
>> 
>> James
>> 
>> On Thu, Jun 30, 2016 at 9:57 AM, Mike Connolly <mike.connolly@oracle.com <mailto:mike.connolly@oracle.com>>
wrote:
>> I checked out the master branch of Nifi and noticed the GetDynamoDB processor which
was something I really needed for a workflow. I tried using it to return the json representation
of a DynamoDB item.  It looks like it will only return a json representation of a a single
attribute of a DynamoDB item. I'm wondering it this was the intention or if there is a specific
value you need to set in the processor configuration that will return the whole item.  Anybody
ever use this?
>> 
>> Thanks,
>> 
>> Mike Connolly
>> 
> 


Mime
View raw message