nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brajendra Mishra <brajendra_mis...@persistent.com>
Subject RE: GetMongoDB : How to pass parameters as input to GetMongoDB processor
Date Fri, 04 May 2018 07:26:35 GMT
Hi Mike, 

Thanks a lot for your valuable inputs. 
We tried GetMongoDB Template which you shared in previous mail with new version of Apache
NiFi (NiFi-1.6) , Its working fine and we are able to get mongoDB data in desired format.

One query: Is such input flow functionality available with 'GetSplunk' processor too with
NiFI 1.6.0? if yes, please share sample template or if not, When can expect such implementation
with 'GetSplunk' processor too?

Brajendra Mishra
Persistent Systems Ltd.

-----Original Message-----
From: Mike Thomsen <mikerthomsen@gmail.com> 
Sent: Thursday, May 03, 2018 6:56 PM
To: dev@nifi.apache.org
Subject: Re: GetMongoDB : How to pass parameters as input to GetMongoDB processor

Brajendra,

I would recommend an update to 1.6.0. It'll make your life a lot easier on this. I did that
patch to GetMongo because I had a client that had an explosion of GetMongos due to that inflexibility.
With that said, *be aware of this bug* in 1.6.0 w/ PutMongo if you use it and upgrade. It
is fixed in
1.7.0 (still in development):

Migrating from 1.5.0 to 1.6.0

   - PutMongo can fail in insert mode. Will be fixed in next release. In
   the mean time you can set query keys for insert even though they'll be
   ignored it should workaround the validation bug.


What it means is there is a validator function that is broken in PutMongo when one is using
the "insert mode" instead of "update mode." You can do a work around by putting a dummy value
in the "query key" field to make it happy.

On Thu, May 3, 2018 at 8:26 AM Pierre Villard <pierre.villard.fr@gmail.com>
wrote:

> Hi,
>
> As Mike said: incoming relationship has been added for NiFi 1.6.0.
> https://issues.apache.org/jira/browse/NIFI-4827
>
> Pierre
>
> 2018-05-03 14:09 GMT+02:00 Brajendra Mishra < 
> brajendra_mishra@persistent.com
> >:
>
> > Hi Mike,
> >
> > I did attach the same in my previous mail. Well reattaching it again.
> > Well error is at GetMongoDB Processor and error text is : "Upstream 
> > Connections is invalid because Processor does not allow upstream 
> > connections but currently has 1"
> >
> > Brajendra Mishra
> > Persistent Systems Ltd.
> >
> > -----Original Message-----
> > From: Mike Thomsen <mikerthomsen@gmail.com>
> > Sent: Thursday, May 03, 2018 5:20 PM
> > To: dev@nifi.apache.org
> > Subject: Re: GetMongoDB : How to pass parameters as input to 
> > GetMongoDB processor
> >
> > Brajendra,
> >
> > Looks like the image didn't make it.
> >
> > On Wed, May 2, 2018 at 11:36 PM Brajendra Mishra < 
> > brajendra_mishra@persistent.com> wrote:
> >
> > > Hi Mike,
> > >
> > > Thanks for responding.
> > >
> > > Here, I have attached missing image attachment.
> > >
> > >
> > >
> > > Brajendra Mishra
> > >
> > > Persistent Systems Ltd.
> > >
> > >
> > >
> > > *From:* Mike Thomsen <mikerthomsen@gmail.com>
> > > *Sent:* Wednesday, May 02, 2018 6:24 PM
> > >
> > >
> > > *To:* dev@nifi.apache.org
> > > *Subject:* Re: GetMongoDB : How to pass parameters as input to 
> > > GetMongoDB processor
> > >
> > >
> > >
> > > That might require 1.6.0. Also, your image didn't come through in 
> > > your response to Sivaprasanna so resend that too.
> > >
> > > On Wed, May 2, 2018 at 8:37 AM Brajendra Mishra < 
> > > brajendra_mishra@persistent.com> wrote:
> > >
> > > Hi Mike,
> > >
> > > Thanks a lot for responding.
> > >
> > > On your statement
> > > "That is its new default behavior if you leave the query field 
> > > blank and have an incoming connection from another processor. That 
> > > would be a good way to integrate the flow with another application"
> > >
> > > Could you please share a sample template for the same?
> > >
> > >
> > > Brajendra Mishra
> > > Persistent Systems Ltd.
> > >
> > > -----Original Message-----
> > > From: Mike Thomsen <mikerthomsen@gmail.com>
> > > Sent: Wednesday, May 02, 2018 5:58 PM
> > > To: dev@nifi.apache.org
> > > Subject: Re: GetMongoDB : How to pass parameters as input to 
> > > GetMongoDB processor
> > >
> > > GetMongo can also use the body of a flowfile for the query. That 
> > > is its new default behavior if you leave the query field blank and 
> > > have an incoming connection from another processor. That would be 
> > > a good way to integrate the flow with another application. For 
> > > example, you could add FetchKafka to the flow and have your 
> > > applications post messages to Kafka with the queries they want it 
> > > to run and FetchKafka would send that JSON to GetMongo as it comes 
> > > in. Or you could build a REST service that writes the JSON to disk and use
GetFile to load it.
> > Lots of ways to do this.
> > >
> > > On Wed, May 2, 2018 at 6:42 AM Sivaprasanna 
> > > <sivaprasanna246@gmail.com>
> > > wrote:
> > >
> > > > Since I'm not so sure about your exact use case, I have just 
> > > > created a rough template based on the simple example flow that I 
> > > > had posted earlier which is GenerateFlowfile -> UpdateAttribute 
> > > > -> GetMongo. I have attached the template here.
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Wed, May 2, 2018 at 2:55 PM, Brajendra Mishra < 
> > > > brajendra_mishra@persistent.com> wrote:
> > > >
> > > >> Hi Sivaprasanna,
> > > >>
> > > >> Could you please provide me the sample template for the same, 
> > > >> where I can pass parameters (and get those parameters' value to 
> > > >> process
> > > >> further) to GetMongoDB processor?
> > > >> It would be a great help for us.
> > > >>
> > > >> Brajendra Mishra
> > > >> Persistent Systems Ltd.
> > > >>
> > > >> -----Original Message-----
> > > >> From: Sivaprasanna <sivaprasanna246@gmail.com>
> > > >> Sent: Wednesday, May 02, 2018 2:28 PM
> > > >> To: dev@nifi.apache.org
> > > >> Subject: Re: GetMongoDB : How to pass parameters as input to 
> > > >> GetMongoDB processor
> > > >>
> > > >> Hi.
> > > >>
> > > >> GetMongo can take input. So technically you can use a processor 
> > > >> before and then connect it  to GetMongo.
> > > >>
> > > >> A simple example :
> > > >> GenerateFlowfile -> UpdateAttribute -> GetMongo
> > > >>
> > > >> In the UpdateAttribute, you can add attributes for the database 
> > > >> and collection and then use them in GetMong using NiFi 
> > > >> Expression
> > Language.
> > > >>
> > > >> Let me know, if that doesn’t help.
> > > >>
> > > >> -
> > > >> Sivaprasanna
> > > >>
> > > >> On Wed, 2 May 2018 at 1:26 PM, Brajendra Mishra < 
> > > >> brajendra_mishra@persistent.com> wrote:
> > > >>
> > > >> > Hi Team,
> > > >> > We have found there is only 'GetMongoDB' processor to connect

> > > >> > and query to MongoDB in Apache NiFi.
> > > >> > Hence, we this processor does not take any type or input.
> > > >> >
> > > >> > Do we have another type to Apache NiFi processor which can 
> > > >> > take parameters as input (details of MongoDB, query, instance

> > > >> > etc.) from
> > > >> other processor?
> > > >> > If not then please suggest when such type of processor can be

> > > >> > expected in upcoming release?
> > > >> >
> > > >> > Brajendra Mishra
> > > >> > Persistent Systems Ltd.
> > > >> >
> > > >> > DISCLAIMER
> > > >> > ==========
> > > >> > This e-mail may contain privileged and confidential 
> > > >> > information which is the property of Persistent Systems Ltd.

> > > >> > It is intended only for the use of the individual or entity 
> > > >> > to which it is addressed. If you are not the intended 
> > > >> > recipient, you are not authorized to read, retain, copy, 
> > > >> > print, distribute or use this message. If you have received 
> > > >> > this communication in error, please notify the sender and 
> > > >> > delete all copies
> > > >> of this message.
> > > >> > Persistent Systems Ltd. does not accept any liability for 
> > > >> > virus infected mails.
> > > >> >
> > > >> DISCLAIMER
> > > >> ==========
> > > >> This e-mail may contain privileged and confidential information 
> > > >> which is the property of Persistent Systems Ltd. It is intended 
> > > >> only for the use of the individual or entity to which it is 
> > > >> addressed. If you are not the intended recipient, you are not 
> > > >> authorized to read, retain, copy, print, distribute or use this 
> > > >> message. If you have received this communication in error, 
> > > >> please notify the sender and
> > > delete all copies of this message.
> > > >> Persistent Systems Ltd. does not accept any liability for virus 
> > > >> infected mails.
> > > >>
> > > >
> > > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information 
> > > which is the property of Persistent Systems Ltd. It is intended 
> > > only for the use of the individual or entity to which it is 
> > > addressed. If you are not the intended recipient, you are not 
> > > authorized to read, retain, copy, print, distribute or use this 
> > > message. If you have received this communication in error, please 
> > > notify the sender and delete all copies
> > of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus 
> > > infected mails.
> > >
> > >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information 
> > which is the property of Persistent Systems Ltd. It is intended only 
> > for the use
> of
> > the individual or entity to which it is addressed. If you are not 
> > the intended recipient, you are not authorized to read, retain, 
> > copy, print, distribute or use this message. If you have received 
> > this communication
> in
> > error, please notify the sender and delete all copies of this message.
> > Persistent Systems Ltd. does not accept any liability for virus 
> > infected mails.
> >
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed.
If you are not the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems Ltd. does not accept
any liability for virus infected mails.
Mime
View raw message