nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giovanni Lanzani <giovannilanz...@godatadriven.com>
Subject RE: Is it possible to reference python requests module in ExecuteScript?
Date Sat, 06 May 2017 08:29:55 GMT
Please do not remove the Python scripting facilities. 

I believe most people experience it as very slow because (ok, Python is slow) they only get
a flow file at a time.

I think Matt pointed out in this ML once that you can use 

flowfiles = session.get(1000)
for flowfile in filter(None, flowfiles):
    # do things

In that case Jython will be kept alive much longer. Or am I missing something?

Giovanni

> -----Original Message-----
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Saturday, May 6, 2017 1:12 AM
> To: users@nifi.apache.org
> Subject: Re: Is it possible to reference python requests module in
> ExecuteScript?
> 
> It is worth discussing whether there is sufficient value to warrant keeping
> jython/python support in the processors or whether we should pull it.  It is
> certainly something we can document as being highly limited but we don't
> really know how limited.  Frankly given the performance I've seen with it I'd be
> ok removing it entirely.  One is better off calling the script via a system call.
> Groovy is one that I've seen perform very well and be fully featured.
> 
> On Fri, May 5, 2017 at 6:38 PM, Russell Bateman <russ@windofkeltia.com>
> wrote:
> > We really want to use ExecuteScript because our end users are Pythonistas.
> > They tend to punctuate their flows with the equivalent of PutFile and
> > GetFile with Python scripts doing stuff on flowfiles that pass out of
> > NiFi before returning into NiFi.
> >
> > However, we find it nearly impossible to replace even the tamest of
> > undertakings. If there were a good set of NiFi/Python shims that, from
> > PyCharm, etc., gave us the ability to prototype, test and debug before
> > copying and pasting into ExecuteScript, that would be wonderful. It
> > hasn't worked out that way. Most of our experience is copying, pasting
> > into the processor property, only to find something wrong, sometimes
> > syntax, sometimes something runtime.
> >
> > On their behalf, I played with this processor a few hours a while back.
> > Another colleague too. Googling this underused tool hasn't been
> > helpful, so the overall experience is negative so far. I can get most
> > of the examples out there to work, but as soon as I try to do "real"
> > work from my point of view, my plans sort of cave in.
> >
> > Likely the Groovy and/or Ruby options are better? But, we're not
> > Groovy or Ruby guys here. I understand the problems with this tool and
> > so I understand what the obstacles are to it growing stronger. The
> > problems won't yield to a few hours one Saturday afternoon. Better
> > problem-logging underneath and
> > better- and more lenient Python support on top. The second one is
> > tough, though.
> >
> > My approach is to minimize those black holes these guys put into their
> > flows by creating custom processors for what I can't solve using
> > standard processors.
> >
> > Trying not to be too negative here...
> >
> >
> > On 05/05/2017 04:09 PM, Andre wrote:
> >
> > Mike,
> >
> > I believe it is possible to use requests under jython, however the
> > process isn't very intuitive.
> >
> > I know one folk that if I recall correctly has used it. Happy to try
> > to find out how it is done.
> >
> > Cheers
> >
> > On Sat, May 6, 2017 at 4:57 AM, Mike Harding <mikeyharding@gmail.com>
> wrote:
> >>
> >> Hi All, I'm now looking at using ExecuteScript and python engine to
> >> execute HTTP requests using the requests module. I've tried
> >> referencing requests the module but when I try to import requests I
> >> get a module reference error.
> >> I downloaded the module from here >
> >> https://pypi.python.org/pypi/requests
> >> Not sure why it isnt picking it up. Ive tried referencing the
> >> directory and the .py directly with no success.
> >> Any ideas where im going wrong?
> >> Cheers,
> >> Mike
Mime
View raw message