nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: send http request to nifi input port
Date Tue, 28 Feb 2017 17:09:24 GMT
Mohammed,

NiFi Input Ports are definitely not the answer you are looking for here. As stated, they are
specifically for NiFi to NiFi communication using Site to Site protocol, either over raw sockets
or HTTPS. They do not accept arbitrary HTTP requests from other entities.

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 28, 2017, at 12:04 PM, mohammed shambakey <shambakey1@gmail.com> wrote:
> 
> Thanks Koji
> 
> On Tue, Feb 28, 2017 at 8:44 AM, Koji Kawamura <ijokarumawak@gmail.com <mailto:ijokarumawak@gmail.com>>
wrote:
> Mohammed,
> 
> NiFi REST API is for managing NiFi itself and data flow running on it,
> as Andy described earlier, it's not for data transfer. It does have
> endpoints to receive POST requests but those are for adding Processors
> or Templates ... etc. Please refer NiFi REST API docs for details.
> https://nifi.apache.org/docs/nifi-docs/rest-api/index.html <https://nifi.apache.org/docs/nifi-docs/rest-api/index.html>
> 
> Thanks,
> Koji
> 
> On Tue, Feb 28, 2017 at 9:25 PM, mohammed shambakey
> <shambakey1@gmail.com <mailto:shambakey1@gmail.com>> wrote:
> > Thank you all for your help
> >
> > I think Koji's thread at JIRA will be the best solution when it is
> > implemented
> >
> > Just a quick question, Andy mentioned REST APIs for NIFI. I didn't find much
> > examples about using NIFI REST APIs. Is it possible to POST the WEB UI
> > (including all inputs from multipart form and uploaded files in a POST
> > request to a NIFI input port of a remote NIFI instance? because, I think,
> > this will avoid using "handlehttprequest" processor and allow users to
> > communicate with the remote NIFI instance both from WEB UI and/or NIFI
> > instance (e.g., site-to-site).
> >
> > Regards
> >
> > On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ijokarumawak@gmail.com <mailto:ijokarumawak@gmail.com>>
> > wrote:
> >>
> >> If the multipart contains only text data, it might be possible to use
> >> those processors Andy suggested. However, if we support multipart
> >> natively, it can be done much easier and safer. I created a JIRA for
> >> this improvement before, but haven't been able to find a time to
> >> implement it myself.
> >> If anyone interested in tackling this enhancement, it'd be greatly
> >> appreciated.
> >> https://issues.apache.org/jira/browse/NIFI-3469 <https://issues.apache.org/jira/browse/NIFI-3469>
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Feb 28, 2017 at 4:59 AM, Andy LoPresto <alopresto@apache.org <mailto:alopresto@apache.org>>
> >> wrote:
> >> > Absolutely. Use some combination of SplitText, SplitContent, SplitJSON,
> >> > and
> >> > ExtractText to split the content into multiple flowfiles on regular
> >> > expressions, new lines, and/or byte sequences and then RouteOnContent to
> >> > send the flowfiles which reference a file to a processor to
> >> > retrieve/parse
> >> > the file contents.
> >> >
> >> > Andy LoPresto
> >> > alopresto@apache.org <mailto:alopresto@apache.org>
> >> > alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com>
> >> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >> >
> >> > On Feb 27, 2017, at 2:08 PM, mohammed shambakey <shambakey1@gmail.com
<mailto:shambakey1@gmail.com>>
> >> > wrote:
> >> >
> >> > This is an example of the output file from "handlehttprequest":
> >> >
> >> > "
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="userid"^M
> >> > ^M
> >> > jpl_user^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="a.py"^M
> >> > Content-Type: text/x-python^M
> >> > ^M
> >> > ^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="b.py"^M
> >> > Content-Type: text/x-python^M
> >> > ^M
> >> > ^M
> >> > -----------------------------831805444208556434014280960^M
> >> > Content-Disposition: form-data; name="scripts"; filename="conf.json"^M
> >> > Content-Type: application/json^M
> >> > ^M
> >> > {
> >> > "userid":"jpl_user_1",
> >> > "scripts":["a.py","b.py"]
> >> > }
> >> > ^M
> >> > -----------------------------831805444208556434014280960--^M
> >> > "
> >> >
> >> > Can I extract different components of this file without script at server
> >> > side?
> >> >
> >> > Regards
> >> >
> >> > On Mon, Feb 27, 2017 at 10:52 AM, Andy LoPresto <alopresto@apache.org
<mailto:alopresto@apache.org>>
> >> > wrote:
> >> >>
> >> >> Mohammed,
> >> >>
> >> >> So from what I understand you have a separate web application which
> >> >> accepts a variety of data (a combination of form fields and file
> >> >> uploads)
> >> >> and submits that (I’m assuming via POST) to NiFi. I’m not sure
what
> >> >> problem
> >> >> you had extracting these “components” from the output flowfile
from the
> >> >> HandleHTTPRequest processor. Can you change your web interface to send
> >> >> separate POST requests to NiFi and logically separate these components
> >> >> from
> >> >> the beginning? Can you modify the format of the request (i.e. make
it
> >> >> JSON,
> >> >> etc.) so it is properly structured for NiFi to parse and operate on
> >> >> this
> >> >> data?
> >> >>
> >> >> NiFi treats the content of the flowfile the same once it is in the
> >> >> system
> >> >> regardless of the source processor, so you should be able to split,
> >> >> extract,
> >> >> or manipulate the contents of a single flowfile containing multiple
> >> >> form
> >> >> elements and file bodies.
> >> >>
> >> >> If you are referencing NiFi’s Input Port components as a destination
> >> >> for
> >> >> an arbitrary HTTP request, this will not work. The input ports are
> >> >> designed
> >> >> to work with NiFi’s Site to Site protocol (either over raw sockets
or
> >> >> HTTP(S)).
> >> >>
> >> >>
> >> >> Andy LoPresto
> >> >> alopresto@apache.org <mailto:alopresto@apache.org>
> >> >> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com>
> >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >> >>
> >> >> On Feb 27, 2017, at 11:09 AM, mohammed shambakey <shambakey1@gmail.com
<mailto:shambakey1@gmail.com>>
> >> >> wrote:
> >> >>
> >> >> Thanks Andy
> >> >>
> >> >> I have a user web interface that send some form information (e.g.,
user
> >> >> id), as well as upload multiple files, to a remote NIFI instance. When
> >> >> using
> >> >> "handlhttprequest" processor, it results in a single file that contains
> >> >> "form inputs" and bodies of uploaded files.
> >> >>
> >> >> I want to extract individual components of this file (i.e., download
> >> >> all
> >> >> uploaded files and extract other form information (e.g., userid) in
a
> >> >> JSON
> >> >> or other text file. I had a similar problem before and I learnt that
> >> >> "httphandlerequest" has a problem to extract individual compoents from,
> >> >> so I
> >> >> have to write my own scripts to extract these components (form inputs
> >> >> and
> >> >> uploaded files).
> >> >>
> >> >> As NIFI input ports does not face this problem of "handlehttprequest",
> >> >> I
> >> >> though if I can send form inputs and upload files directly to "NIFI
> >> >> input
> >> >> port" at remote site, then I don't have to write my own scripts to
> >> >> extract
> >> >> individual components.
> >> >>
> >> >> Regards
> >> >>
> >> >> On Mon, Feb 27, 2017 at 10:02 AM, Andy LoPresto <alopresto@apache.org
<mailto:alopresto@apache.org>>
> >> >> wrote:
> >> >>>
> >> >>> Mohammed,
> >> >>>
> >> >>> I’m not sure I understand your question. NiFi has a REST API
which
> >> >>> supports the UI and allows scripts/other applications to make requests
> >> >>> of
> >> >>> the platform. This runs on the input port specified in
> >> >>> nifi.properties.
> >> >>>
> >> >>> If you want to submit form data and uploaded files as data to be
> >> >>> ingested
> >> >>> by NiFi, HandleHTTPRequest or (less likely) ListenHTTP are the
> >> >>> processors
> >> >>> you would want to use.
> >> >>>
> >> >>> If you can clarify your use case, we should be able to offer more
> >> >>> assistance.
> >> >>>
> >> >>> Andy LoPresto
> >> >>> alopresto@apache.org <mailto:alopresto@apache.org>
> >> >>> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com>
> >> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D
EF69
> >> >>>
> >> >>> On Feb 27, 2017, at 9:15 AM, mohammed shambakey <shambakey1@gmail.com
<mailto:shambakey1@gmail.com>>
> >> >>> wrote:
> >> >>>
> >> >>> Hi
> >> >>>
> >> >>> Is it possible to send http request (including form data and uploaded
> >> >>> files) to NIFI input port instead of "HandleHTTPRequest" processor?
> >> >>>
> >> >>> Regards
> >> >>>
> >> >>> --
> >> >>> Mohammed
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Mohammed
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Mohammed
> >> >
> >> >
> >
> >
> >
> >
> > --
> > Mohammed
> 
> 
> 
> --
> Mohammed


Mime
View raw message