Return-Path: Extract attachments from a mime formatted email file, splitting them into individual flowfiles. split, email This component has no required or optional properties. Using the flowfile content as source of data, extract header from an RFC compliant email file adding the relevant attributes to the flowfile. This processor does not perform extensive RFC validation but still requires a bare minimum compliance with RFC 2822 split, email In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values. Extract attachments from a mime formatted email file, splitting them into individual flowfiles. split, email This component has no required or optional properties. This processor implements a lightweight SMTP server to an arbitrary port, allowing nifi to listen for incoming email. Note this server does not perform any email validation. If direct exposure to the internet is sought, it may be a better idea to use the combination of NiFi and an industrial scale MTA (e.g. Postfix). Threading for this processor is managed by the underlying smtp server used so the processor need not support more than one thread. listen, email, smtp In the list below, the names of required properti
es appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values. Looks up geolocation information for an IP address and adds the geo information to FlowFile attributes. The geo data is provided as a MaxMind database. The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the the attributes added by enrichment will take the form X.geo.<fieldName> geo, enrich, ip, maxmind In the list below, the names of required properties appear in bold. Any other propert
ies (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. Looks up ISP information for an IP address and adds the information to FlowFile attributes. The ISP data is provided as a MaxMind ISP database (Note that this is NOT the same as the GeoLite database utilizedby some geo enrichment tools). The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the the attributes added by enrichment will take the form X.isp.<fieldName> ISP, enrich, ip, maxmind In the list belo
w, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. A powerful DNS query processor primary designed to enrich DataFlows with DNS based APIs (e.g. RBLs, ShadowServer's ASN lookup) but that can be also used to perform regular DNS lookups. dns, enrich, ip In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. A powerful whois query processor primary designed to enrich DataFlows with whois based APIs (e.g. ShadowServer's ASN lookup) but that can be also used to perform regular whois lookups. whois, enrich, ip In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
This processor is used to parse Windows event logs in the binary evtx format. The input flow files' content should be evtx files. The processor has 4 outputs:
+ ExtractEmailAttachments
Description:
Tags:
Properties:
Relationships:
Name Description failure Flowfiles that could not be parsed original The original file attachments Each individual attachment will be routed to the attachments relationship Reads Attributes:
None specified.Writes Attributes:
Name Description filename The filename of the attachment email.attachment.parent.filename The filename of the parent FlowFile email.attachment.parent.uuid The UUID of the original FlowFile. mime.type The mime type of the attachment. State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1 @@
+ExtractEmailHeaders
Description:
Tags:
Properties:
Name<
/th> Default Value Allowable Values Description Additional Header List x-mailer COLON separated list of additional headers to be extracted from the flowfile content.NOTE the header key is case insensitive and will be matched as lower-case. Values will respect email contents. Email Address Parsing true If "strict", strict address format parsing rules are applied to mailbox and mailbox list fields, such as "to" and "from" headers, and FlowFiles with poorly formed addresses will be routed to the failure relationship, similar to messages that fail RFC compliant format validation. If "non-strict", the processor will extract the contents of mailbox list headers as comma-separated values without attempting to parse each value as well-formed Internet mailbox addresses. This is optional and defaults to Strict Address Parsing Relationships:
Name Description success Extraction was successful failure Flowfiles that could not be parsed as a RF
C-2822 compliant message Reads Attributes:
None specified.Writes Attributes:
Name Description email.headers.bcc.* Each individual BCC recipient (if available) email.headers.cc.* Each individual CC recipient (if available) email.headers.from.* Each individual mailbox contained in the From of the Email (array as per RFC-2822) email.headers.message-id The value of the Message-ID header (if available) email.headers.received_date The Received-Date of the message (if available) email.headers.sent_date Date the message was sent email.headers.subject Subject of the message (if available) email.headers.to.* Each individual TO recipient (if available) email.attachment_count Number of attachm
ents of the message State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1 @@
+ExtractTNEFAttachments
Description:
Tags:
Properties:
Relationships:
Name Description failure Each individual flowfile that could not be parsed will be routed to the failure relationship original Each original flowfile (i.e. before extraction) will be routed to the original relati
onship attachments Each individual attachment will be routed to the attachments relationship Reads Attributes:
None specified.Writes Attributes:
Name Description filename The filename of the attachment email.tnef.attachment.parent.filename The filename of the parent FlowFile email.tnef.attachment.parent.uuid The UUID of the original FlowFile. State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1 @@
+ListenSMTP
Description:
Tags:
Properties:
Name Default Value Allowable Values Description Listening Port The TCP port the ListenSMTP processor will bind to.NOTE that on Unix derivative operating systems this port must be higher than 1024 unless NiFi is running as with root user permissions. Maximum number of SMTP connection 1 The maximum number of simultaneous SMTP connections. SMTP connection timeout 60 seconds The maximum time to wait for an action of
SMTP client. SMTP Maximum Message Size 20 MB The maximum number of bytes the server will accept. SSL Context Service Controller Service API:
RestrictedSSLContextService
Implementation: StandardRestrictedSSLContextServiceThe Controller Service to use in order to obtain an SSL Context. If this property is set, messages will be received over a secure connection. Client Auth The client authentication policy to use for the SSL Context. Only
used if an SSL Context Service is provided. SMTP hostname The hostname to be embedded into the banner displayed when an SMTP client connects to the processor TCP port . Relationships:
Name Description success All new messages will be routed as FlowFiles to this relationship Reads Attributes:
None specified.Writes Attributes:
Name Description smtp.helo The value used during HELO smtp.certificates.*.serial The serial numbers for each of the certificates used by an TLS peer smtp.certificates.*.principal The principal for each of the certificates used by an TLS peer smtp.src The source IP and port of the SMTP
connection smtp.from The value used during MAIL FROM (i.e. envelope) smtp.recipient.* The values used during RCPT TO (i.e. envelope) mime.type Mime type of the message State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component does not allow an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1 @@
+GeoEnrichIP
Description:
Tags:
Properties:
Name Default Value Allowable Values Description MaxMind Database File Path to Maxmind IP Enrichment Database File IP Address Attribute The name of an attribute whose value is a dotted decimal IP address for which enrichment should occur
Supports Expression Language: trueRelationships:
Name Description not found Where to route flow files after unsucces
sfully enriching attributes because no data was found found Where to route flow files after successfully enriching attributes with data provided by database Reads Attributes:
None specified.Writes Attributes:
Name Description X.geo.lookup.micros The number of microseconds that the geo lookup took X.geo.city The city identified for the IP address X.geo.accuracy The accuracy radius if provided by the database (in Kilometers) X.geo.latitude The latitude identified for this IP address X.geo.longitude The longitude identified for this IP address X.geo.subdivision.N Each subdivision that is identified for this IP address is added with a one-up number appended to the attribute name, starting with 0 X.geo.subdivision.isoco
de.N The ISO code for the subdivision that is identified by X.geo.subdivision.N X.geo.country The country identified for this IP address X.geo.country.isocode The ISO Code for the country identified X.geo.postalcode The postal code for the country identified State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1 @@
+ISPEnrichIP
Description:
Tags:
Properties:
Name Default Value Allowable Values Description MaxMind Database File Path to Maxmind IP Enrichment Database File IP Address Attribute The name of an attribute whose value is a dotted decimal IP address for which enrichment should occur
Supports Expression Language: trueRelationships:
Name D
escription not found Where to route flow files after unsuccessfully enriching attributes because no data was found found Where to route flow files after successfully enriching attributes with data provided by database Reads Attributes:
None specified.Writes Attributes:
Name Description X.isp.lookup.micros The number of microseconds that the geo lookup took X.isp.asn The Autonomous System Number (ASN) identified for the IP address X.isp.asn.organization The Organization Associated with the ASN identified X.isp.name The name of the ISP associated with the IP address provided X.isp.organization The Organization associated with the IP address provided State management:
This component does not store state.R
estricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1,2 @@
+QueryDNS
Description:
Tags:
Properties:
Name Default Value Allowable Values Description Lookup value The value that should be used to populate the query
Supports Expression Language: trueResults Parser None The method used to slice the results into attribute groups Parser RegEx Choice between a splitter and regex matcher used to parse the results of the query into attribute groups.
+NOTE: This is a multiline regular expression, therefore, the DFM should decide how to handle trailing new line characters. DNS Query Retries 1 The number of attempts before giving up and moving on DNS Query Timeout 1500 ms The amount of time to wait until considering a query as failed DNS Servers A comma separated list of DNS servers to be used. (Defaults to system wide if none is used) DNS Query Type TXT The DNS query type to be used by the processor (e.g. TXT, A) Relationships
:
Name Description not found Where to route flow files if data enrichment query rendered no results found Where to route flow files after successfully enriching attributes with data Reads Attributes:
None specified.Writes Attributes:
Name Description enrich.dns.record*.group* The captured fields of the DNS query response for each of the records received State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1,2 @@
+QueryWhois
Description:
Tags:
Properties:
Name Default Value Allowable Values Description Lookup value The value that should be used to populate the query
Supports Expression Language: trueWhois Query Type The Whois query type to be used by the processor (if used) Whois Server The Whois server to be used Whois Server Port 43 The TCP port of the remote Whois server Whois Query Timeout 1500 m
s The amount of time to wait until considering a query as failed Batch Size 25 The number of incoming FlowFiles to process in a single execution of this processor. Bulk Protocol None The protocol used to perform the bulk query. Results Parser None The method used to slice the results into attribute groups Parser RegEx Choice between a splitte
r and regex matcher used to parse the results of the query into attribute groups.
+NOTE: This is a multiline regular expression, therefore, the DFM should decide how to handle trailing new line characters. Key lookup group (multiline / batch) When performing a batched lookup, the following RegEx numbered capture group or Column number will be used to match the whois server response with the lookup field Relationships:
Name Description not found Where to route flow files if data enrichment query rendered no results found Where to route flow files after successfully enriching attributes with data Reads Attributes:
None specified.Writes Attributes:
Name Description enrich.dns.record*.group* The captured fields of the Whois query response
for each of the records received State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html?rev=1811008&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html Tue Oct 3 13:30:16 2017
@@ -0,0 +1,91 @@
+
+
+
+
+
+ Description:
+
+
+
+
+<?xml version="1.0"?> +<Events> + <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> + <System> + <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" Ev +entSourceName="Service Control Manager"/> + <EventID Qualifiers="16384">7036</EventID> + <Version>0</Version> + <Level>4</Level> + <Task>0</Task> + <Opcode>0</Opcode> + <Keywords>0x8080000000000000</Keywords> + <TimeCreated SystemTime="2016-01-08 16:49:47.518"/> + <EventRecordID>780</EventRecordID> + <Correlation ActivityID="" RelatedActivityID=""/> + <Execution ProcessID="480" ThreadID="596"/> + <Channel>System</Channel> + <Computer>win7-pro-vm</Computer> + <Security UserID=""/> + </System> + <EventData> + <Data Name="param1">Workstation</Data> + <Data Name="param2">running</Data> + <Binary>TABhAG4AbQBhAG4AVwBvAHIAawBzAHQAYQB0AGkAbwBuAC8ANAAAAA==</Binary> + </EventData> + </Event> + <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> + <System> + <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager"/> + <EventID Qualifiers="16384">7036</EventID> + <Version>0</Version> + <Level>4</Level> + <Task>0</Task> + <Opcode>0</Opcode> + <Keywords>0x8080000000000000</Keywords> + <TimeCreated SystemTime="2016-01-08 16:49:47.535"/> + <EventRecordID>781</EventRecordID> + <Correlation ActivityID="" RelatedActivityID=""/> + <Execution ProcessID="480" ThreadID="576"/> + <Channel>System</Channel> + <Computer>win7-pro-vm</Computer> + <Security UserID=""/> + </System> + <EventData> + <Data Name="param1">Cryptographic Services</Data> + <Data Name="param2">running</Data> + <Binary>QwByAHkAcAB0AFMAdgBjAC8ANAAAAA==</Binary> + </EventData> + </Event> +</Events> ++ + + Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +
Parses the contents of a Windows Event Log file (evtx) and writes the resulting XML to the FlowFile
logs, windows, event, evtx, message, file
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
Granularity | Chunk |
| Output flow file for each Record, Chunk, or File encountered in the event log |
Name | Description |
---|---|
success | Any FlowFile that was successfully converted from evtx to XML |
failure | Any FlowFile that encountered an exception during conversion will be transferred to this relationship with as much parsing as possible done |
original | The unmodified input FlowFile will be transferred to this relationship |
bad chunk | Any bad chunks of records will be transferred to this relationship in their original binary form |
Name | Description |
---|---|
filename | The filename of the evtx file |
Name | Description |
---|---|
filename | The output filename |
mime.type | The output filetype (application/xml for success and failure relationships, original value for bad chunk and original relationships) |
+ This processor executes an Apache Flume sink. FlowFiles are wrapped in Flume's + Event interface. The content of the FlowFile becomes the body of the Event and + the attributes of the FlowFile become Event headers. The following special + headers are also set: +
+Flume Event Header | +FlowFile Attribute | +
---|---|
nifi.entry.date | +FlowFile#getEntryDate() | +
nifi.id | +FlowFile#getId() | +
nifi.last.queue.date | +FlowFile#getLastQueueDate() | +
nifi.lineage.start.date | +FlowFile#getLineageStartDate() | +
nifi.size | +FlowFile#getSize() | +
+ In NiFi, the contents of a FlowFile are accessed via a stream, but in Flume it is + stored in a byte array. This means the full content will be loaded into memory when + a FlowFile is processed by the ExecuteFlumeSink processor. You should consider the + typical size of the FlowFiles you'll process and the batch size, if any, your sink + is configured with when setting NiFi's heap size. +
++ This processor is designed to execute arbitrary Flume sinks. Most of the details + of configuring the sink is deferred to Flume's built-in configuration system. + For details on the available settings for each sink type, refer to the Flume + User Guide. + Configuring the Flume sink is a four step process: +
+tier1
+ tier1
, then the Sink Name
+ is the value of the tier1.sinks
property. Example: sink-1
+ tier1.sinks.sink-1
.
+ Do not copy the tier1.sinks.sink-1.type
or
+ tier1.sinks.sink-1.channel
properties.
+ + Assuming you had the following existing Flume configuration file: +
++a1.sources = r1 +a1.sinks = k1 +a1.channels = c1 + +a1.sources.r1.type = netcat +a1.sources.r1.bind = localhost +a1.sources.r1.port = 44444 + +a1.sinks.k1.type = hdfs +a1.sinks.k1.channel = c1 +a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S +a1.sinks.k1.hdfs.filePrefix = events- +a1.sinks.k1.hdfs.round = true +a1.sinks.k1.hdfs.roundValue = 10 +a1.sinks.k1.hdfs.roundUnit = minute + +a1.channels.c1.type = memory +a1.channels.c1.capacity = 1000 +a1.channels.c1.transactionCapacity = 100 + +a1.sources.r1.channels = c1 +a1.sinks.k1.channel = c1+
+ Then you'd configure the ExecuteFlumeSink as follows: +
+Property | +Value | +
---|---|
Sink Type | +hdfs | +
Agent Name | +a1 | +
Sink Name | +k1 | +
Flume Configuration | +
+
+ a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+ |
+
Execute a Flume sink. Each input FlowFile is converted into a Flume Event for processing by the sink.
flume, hadoop, put, sink, restricted
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
Sink Type | The component type name for the sink. For some sinks, this is a short, symbolic name (e.g. hdfs). For others, it's the fully-qualified name of the Sink class. See the Flume User Guide for details. | ||
Agent Name | tier1 | The name of the agent used in the Flume sink configuration | |
Sink Name | sink-1 | The name of the sink used in the Flume sink configuration | |
Flume Configuration | The Flume configuration for the sink copied from the flume.properties file |
Name | Description |
---|---|
success | |
failure |
+ This processor executes an Apache Flume source. Each Flume Event is turned into + a FlowFile. The content of the FlowFile is set to the body of the Event and + the Event headers become FlowFile attributes. +
++ This processor is designed to execute arbitrary Flume sources. Most of the details + of configuring the source is deferred to Flume's built-in configuration system. + For details on the available settings for each source type, refer to the Flume + User Guide. + Configuring the Flume source is a four step process: +
+tier1
+ tier1
, then the Source Name
+ is the value of the tier1.sources
property. Example: src-1
+ tier1.sources.src-1
.
+ Do not copy the tier1.sources.src-1.type
or
+ tier1.sources.src-1.channel
properties.
+ + Assuming you had the following existing Flume configuration file: +
++a1.sources = r1 +a1.sinks = k1 +a1.channels = c1 + +a1.sources.r1.type = multiport_syslogtcp +a1.sources.r1.channels = c1 +a1.sources.r1.host = 0.0.0.0 +a1.sources.r1.ports = 10001 10002 10003 +a1.sources.r1.portHeader = port + +a1.sinks.k1.type = logger + +a1.channels.c1.type = memory +a1.channels.c1.capacity = 1000 +a1.channels.c1.transactionCapacity = 100 + +a1.sources.r1.channels = c1 +a1.sinks.k1.channel = c1+
+ Then you'd configure the ExecuteFlumeSource as follows: +
+Property | +Value | +
---|---|
Source Type | +multiport_syslogtcp | +
Agent Name | +a1 | +
Source Name | +r1 | +
Flume Configuration | +
+
+ a1.sources.r1.host = 0.0.0.0
+ |
+
Execute a Flume source. Each Flume Event is sent to the success relationship as a FlowFile
flume, hadoop, get, source, restricted
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
Source Type | The component type name for the source. For some sources, this is a short, symbolic name (e.g. spooldir). For others, it's the fully-qualified name of the Source class. See the Flume User Guide for details. | ||
Agent Name | tier1 | The name of the agent used in the Flume source configuration | |
Source Name | src-1 | The name of the source used in the Flume source configuration | |
Flume Configuration | The Flume configuration for the source copied from the flume.properties file |
Name | Description |
---|---|
success |
Defines credentials for Google Cloud Platform processors. Uses Application Default credentials without configuration. Application Default credentials support environmental variable (GOOGLE_APPLICATION_CREDENTIALS) pointing to a credential file, the config generated by `gcloud auth application-default login`, AppEngine/Compute Engine service accounts, etc.
gcp, credentials, provider
In the list below, the names of required properties appear in bold. Any oth er properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
Use Application Default Credentials | false |
| If true, uses Google Application Default Credentials, which checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filepath to a service account JSON key, the config generated by the gcloud sdk, the A pp Engine service account, and the Compute Engine service account. |
Use Compute Engine Credentials | false |
| If true, uses Google Compute Engine Credentials of the Compute Engine VM Instance which NiFi is running on. |
Service Account JSON File | Path to a file containing a Service Account key file in JSON format. | ||
Service Account JSON | The raw JSON containing a Service Account keyfile. Sensitive Property: true Supports Expression Language: true |
Deletes objects from a Google Cloud Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.
google cloud, gcs, google, storage, delete
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
GCP Credentials Provider Service | Controller Service API: GCPCredentialsService Implementation: GCPCredentialsControllerService | The Controller Service used to obtain Google Cloud Platform credentials. | |
Project ID | Google Cloud Project ID | ||
Number of retries | 6 | How many retry attempts should be made before routing to the failure relationship. | |
Buck et | ${gcs.bucket} | Bucket of the object. Supports Expression Language: true | |
Key | ${filename} | Name of the object. Supports Expression Language: true | |
Generation | The generation of the object to be deleted. If null, will use latest version of the object. Supports Expression Language: true |
Name | Description |
---|---|
success | FlowFiles are routed to this relationship after a successful Google Cloud Storage operation. |
failure | FlowFiles are routed to this relationship if the Google Cloud Storage operation fails. |
PutGCSObject, FetchGCSObject, ListGCSBucket
\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.
google cloud, google, storage, gcs, fetch
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
GCP Credentials Provider Service | Controller Service API: GCPCredentialsService Implementation: GCPCredentialsControllerService | The Controller Service used to obtain Google Cloud Platform credentials. | |
Project ID | Google Cloud Project ID | ||
Number of retries | 6 | How many retry attempts should be made before routing to the failure relationship. | |
Bucket | ${gcs.bucket} | Bucket of the object. Supports Expression Language: true | |
Key | ${filename} | Name of the object. Supports Expression Language: true | |
Object Generation | The generation of the Object to download. If null, will download latest generation. Supports Expression Language: true | ||
Server Side Encryption Key | An AES256 Key (encoded in base64) which the object has been encrypted in. Sensitive Property: true Supports Expression Language: true |
Name | Description |
---|---|
success | FlowFiles are routed to this relationship after a successful Google Cloud Storage operation. |
failure | FlowFiles are routed to this relationship if the Google Cloud Storage operation fails. |
Name | Description |
---|---|
filename | The name of the file, parsed if possible from the Content-Disposition response header |
gcs.bucket | Bucket of the object. |
gcs.key | Name of the object. |
gcs.size | Size of the objec t. |
gcs.cache.control | Data cache control of the object. |
gcs.component.count | The number of components which make up the object. |
gcs.content.disposition | The data content disposition of the object. |
gcs.content.encoding | The content encoding of the object. |
gcs.content.language | The content language of the object. |
mime.type | The MIME/Content-Type of the object |
gcs.crc32c | The CRC32C checksum of object's data, encoded in base64 in big-endian order. |
gcs.create.time | The creation time of the object (milliseconds) |
gcs.update.time | The last modification time of the object (milliseconds) |
gcs.encryption.algorithm | The algorithm used to encrypt the object. |
gcs.encryption.sha256 | The SHA256 hash of the key used to encrypt the object |
gcs.etag | The HTTP 1.1 Entity tag for the object. |
gcs.generated.id | The service-generated for the object |
gcs.generation | The data generation of the object. |
gcs.md5 | The MD5 hash of the object's data encoded in base64. |
gcs.media.link | The media download link to the object. |
gcs.metageneration | The metageneration of the object. |
gcs.owner | The owner (uploader) of the object. |
gcs.owner.type | The ACL entity type of the uploader of the object. |
gcs.uri | The URI of the object as a string. |
ListGCSBucket, PutGCSObject, DeleteGCSObject
\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +Retrieves a listing of objects from an GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
google cloud, google, storage, gcs, list
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.
Name | Default Value | Allowable Values | Description |
---|---|---|---|
GCP Credentials Provider Service | Controller Service API: GCPCredentialsService Implementation: GCPCredentialsControllerService | The Controller Service used to obtain Google Cloud Platform credentials. | |
Project ID | Google Cloud Project ID | ||
Number of retries | 6 | How many retry attempts should be made before routing to the failure relationship. | |
Bucket | Bucket of the object. | ||
Prefix | The prefix used to filter the object list. In most cases, it should end with a forward slash ('/'). | ||
Use Generations | false |
| Specifies whether to use GCS Generations, if applicable. If false, only the latest version of each object will be returned. |
Name | Description |
---|---|
success | FlowFiles are routed to this relationship after a successful Google Cloud Storage ope ration. |
Name | Description |
---|---|
filename | The name of the file |
gcs.bucket | Bucket of the object. |
gcs.key | Name of the object. |
gcs.size | Size of the object. |
gcs.cache.control | Data cache control of the object. |
gcs.component.count | The number of components which make up the object. |
gcs.content.disposition | The data content disposition of the object. |
gcs.content.encoding | The content encoding of the object. |
gcs.content.language | The content language of the object. |
mime.type | The MIME/Content-Type of the object |
gcs.crc32c | The CRC32C checksum of object's data, encoded in base64 in big-endian order. | < /tr>
gcs.create.time | The creation time of the object (milliseconds) |
gcs.update.time | The last modification time of the object (milliseconds) |
gcs.encryption.algorithm | The algorithm used to encrypt the object. |
gcs.encryption.sha256 | The SHA256 hash of the key used to encrypt the object |
gcs.etag | The HTTP 1.1 Entity tag for the object. |
gcs.generated.id | The service-generated for the object |
gcs.generation | The data generation of the object. |
gcs.md5 | The MD5 hash of the object's data encoded in base64. |
gcs.media.link | The media download link to the object. |
gcs.metageneration | The metageneration of the object. |
gcs.owner | The owner (uploader) of the object. |
gcs.owner.type | The ACL entity type of the uploader of the object. |
gcs.uri | The URI of the object as a string. |
Scope | Description |
---|---|
CLUSTER | After performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data. |