ParseCEF

From commits-return-42390-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Mon Feb 3 21:53:52 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id DFA69180647 for ; Mon, 3 Feb 2020 22:53:51 +0100 (CET) Received: (qmail 79114 invoked by uid 500); 3 Feb 2020 21:53:42 -0000 Mailing-List: contact commits-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list commits@nifi.apache.org Received: (qmail 78471 invoked by uid 99); 3 Feb 2020 21:53:40 -0000 Received: from Unknown (HELO svn01-us-east.apache.org) (13.90.137.153) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Feb 2020 21:53:40 +0000 Received: from svn01-us-east.apache.org (svn01-us-east.apache.org [127.0.0.1]) by svn01-us-east.apache.org (ASF Mail Server at svn01-us-east.apache.org) with ESMTP id 4473D17A159 for ; Mon, 3 Feb 2020 21:53:40 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1873556 [41/48] - in /nifi/site/trunk/docs/nifi-docs: ./ components/org.apache.nifi/nifi-ambari-nar/1.11.1/ components/org.apache.nifi/nifi-ambari-nar/1.11.1/org.apache.nifi.reporting.ambari.AmbariReportingTask/ components/org.apache.nifi/... Date: Mon, 03 Feb 2020 21:53:33 -0000 To: commits@nifi.apache.org From: joewitt@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20200203215340.4473D17A159@svn01-us-east.apache.org> Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.ParseCEF/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.ParseCEF/index.html?rev=1873556&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.ParseCEF/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.ParseCEF/index.html Mon Feb 3 21:53:27 2020 @@ -0,0 +1,2 @@ +ParseCEF

Description:

Parses the contents of a CEF formatted message and adds attributes to the FlowFile for headers and extensions of the parts of the CEF message. +Note: This Processor expects CEF messages WITHOUT the syslog headers (i.e. starting at "CEF:0"

Tags:

logs, cef, attributes, system, event, message

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name	Default Value	Allowable Values	Description
Parsed fields destination	flowfile-content	flowfile-content flowfile-attribute	Indicates whether the results of the CEF parser are written to the FlowFile content or a FlowFile attribute; if using flowfile-attributeattribute, fields will be populated as attributes. If set to flowfile-content, the CEF extension field will be converted into a flat JSON object.
Append raw message to JSON	true		When using flowfile-content (i.e. JSON output), add the original CEF message to the resulting JSON object. The original message is added as a string to _raw.
Timezone	Local Timezone (system Default)	UTC Local Timezone (system Default)	Timezone to be used when representing date fields. UTC will convert all dates to UTC, while Local Timezone will convert them to the timezone used by NiFi.
DateTime Locale	en-US		The IETF BCP 47 representation of the Locale to be used when parsing date fields with long or short month names (e.g. may <en-US> vs. mai. & lt;fr-FR>. The defaultvalue is generally safe. Only change if having issues parsing CEF messages

Relationships:

Name	Description
success	Any FlowFile that is successfully parsed as a CEF message will be transferred to this Relationship.
failure	Any FlowFile that could not be parsed as a CEF message will be transferred to this Relationship without any attributes being added

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
cef.header.version	The version of the CEF message.
cef.header.deviceVendor	The Device Vendor of the CEF message.
cef.header.deviceProduct	The Device Product of the CEF message.
cef.header.deviceVersion	The Device Version of the CEF message.
cef.header.deviceEventClassId	The Device Event Class ID of the CEF message.
cef.header.name	The name of the CEF message.
cef.header.severity	The severity of the CEF message.
cef.extension.*	The key and value generated by the parsing of the message.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

Description:

Attempts to parses the contents of a Syslog message in accordance to RFC5424 and RFC3164 formats and adds attributes to the FlowFile for each of the parts of the Syslog message.Note: Be mindfull that RFC3164 is informational and a wide range of different implementations are present in the wild. If messages fail parsing, considering using RFC5424 or using a generic parsing processors such as ExtractGrok.

Tags:

logs, syslog, attributes, system, event, message

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name	Default Value	Allowable Values	Description
Character Set	UTF-8		Specifies which character set of the Syslog messages

Relationships:

Name	Description
success	Any FlowFile that is successfully parsed as a Syslog message will be to this Relationship.
failure	Any FlowFile that could not be parsed as a Syslog message will be transferred to this Relationship without any attributes being added

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
sysl og.priority	The priority of the Syslog message.
syslog.severity	The severity of the Syslog message derived from the priority.
syslog.facility	The facility of the Syslog message derived from the priority.
syslog.version	The optional version from the Syslog message.
syslog.timestamp	The timestamp of the Syslog message.
syslog.hostname	The hostname or IP address of the Syslog message.
syslog.sender	The hostname of the Syslog server that sent the message.
syslog.body	The body of the Syslog message, everything after the hostname.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

Description:

Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data.Structured Data will be written to attributes as one attribute per item id + parameter see https://tools.ietf.org/html/rfc5424.Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs will fail wit h this parser, where they would not fail with ParseSyslog.

Tags:

logs, syslog, syslog5424, attributes, system, event, message

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name	Default Value	Allowable Values	Description
Character Set	UTF-8		Specifies which character set of the Syslog messages
NIL Policy	NULL	OMIT NULL DASH	Defines how NIL values are handled for header fields.
Include Message Body in Attributes	true	true false	If true, then the Syslog Message body will be included in the attributes.

Relationships:

Name	Description
success	Any FlowFile that is successfully parsed as a Syslog message will be to this Relationship.
failure	Any FlowFile that could not be parsed as a Syslog message will be transferred to this Relationship without any attributes being added

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
syslog.priority	The priority of the Syslog message.
syslog.severity	The severity of the Syslog message derived from the priority.
syslog.facility	The facility of the Syslog message derived from the priority.
syslog.version	The optional version from the Syslog message.
syslog.timestamp	The timestamp of the Syslog message.
syslog.hostname	The hostname or IP address of the Syslog message.
syslog.appname	The appname of the Syslog message.
syslog.procid	The procid of the Syslog message.
syslog.me ssageid	The messageid the Syslog message.
syslog.structuredData	Multiple entries per structuredData of the Syslog message.
syslog.sender	The hostname of the Syslog server that sent the message.
syslog.body	The body of the Syslog message, everything after the hostname.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

Examples

+ +

+ To better understand how this Processor works, we will lay out a few examples. For the sake of these examples, let's assume that our input + data is JSON formatted and looks like this: +

+ +


++[ {
+  "name": "John Doe",
+  "dob": "11/30/1976",
+  "favorites": [ "spaghetti", "basketball", "blue" ],
+  "locations": {
+  	"home": {
+  		"number": 123,
+  		"street": "My Street",
+  		"city": "New York",
+  		"state": "NY",
+  		"country": "US"
+  	},
+  	"work": {
+  		"number": 321,
+  		"street": "Your Street",
+  		"city": "New York",
+  		"state": "NY",
+  		"country": "US"
+  	}
+  }
+}, {
+  "name": "Jane Doe",
+  "dob": "10/04/1979",
+  "favorites": [ "spaghetti", "football", "red" ],
+  "locations": {
+  	"home": {
+  		"number": 123,
+  		"street": "My Street",
+  		"city": "New York",
+  		"state": "NY",
+  		"country": "US"
+  	},
+  	"work": {
+  		"number": 456,
+  		"street": "Our Street",
+  		"city": "New York",
+  		"state": "NY",
+  		"country": "US"
+  	}
+  }
+}, {
+  "name": "Jacob Doe",
+  "dob": "04/02/2012",
+  "favorites": [ "chocolate", "running", "yellow" ],
+  "locations": {
+  	"home": {
+  		"number": 123,
+  		"street": "My Street",
+  		"city": "New York",
+  		"state": "NY",
+  		"country": "US"
+  	},
+  	"work": null
+  }
+}, {
+  "name": "Janet Doe",
+  "dob": "02/14/2007",
+  "favorites": [ "spaghetti", "reading", "white" ],
+  "locations": {
+  	"home": {
+  		"number": 1111,
+  		"street": "Far Away",
+  		"city": "San Francisco",
+  		"state": "CA",
+  		"country": "US"
+  	},
+  	"work": null
+  }
+}]
+
+

+ + +

Example 1 - Partition By Simple Field

+ +

+ For a simple case, let's partition all of the records based on the state that they live in. + We can add a property named state with a value of /locations/home/state. + The result will be that we will have two outbound FlowFiles. The first will contain an attribute with the name + state and a value of NY. This FlowFile will consist of 3 records: John Doe, Jane Doe, and Jacob Doe. + The second FlowFile will consist of a single record for Janet Doe and will contain an attribute named state that + has a value of CA. +

+ + +

Example 2 - Partition By Nullable Value

+ +

+ In the above example, there are three different values for the work location. If we use a RecordPath of /locations/work/state + with a property name of state, then we will end up with two different FlowFiles. The first will contain records for John Doe and Jane Doe + because they have the same value for the given RecordPath. This FlowFile will have an attribute named state with a value of NY. +

+ The second FlowFile will contain the two records for Jacob Doe and Janet Doe, because the RecordPath will evaluate + to null for both of them. This FlowFile will have no state attribute (unless such an attribute existed on the incoming FlowFile, + in which case its value will be unaltered). +

+ + +

Example 3 - Partition By Multiple Values

+ +

+ Now let's say that we want to partition records based on multiple different fields. We now add two properties to the PartitionRecord processor. + The first property is named home and has a value of /locations/home. The second property is named favorite.food + and has a value of /favorites[0] to reference the first element in the "favorites" array. +

+ +

+ This will result in three different FlowFiles being created. The first FlowFile will contain records for John Doe and Jane Doe. If will contain an attribute + named "favorite.food" with a value of "spaghetti." However, because the second RecordPath pointed to a Record field, no "home" attribute will be added. + In this case, both of these records have the same value for both the first element of the "favorites" array + and the same value for the home address. Janet Doe has the same value for the first element in the "favorites" array but has a different home address. Similarly, + Jacob Doe has the same home address but a different value for the favorite food. +

+ +

+ The second FlowFile will consist of a single record: Jacob Doe. This FlowFile will have an attribute named "favorite.food" with a value of "chocolate." + The third FlowFile will consist of a single record: Janet Doe. This FlowFile will have an attribute named "favorite.food" with a value of "spaghetti." +

+ + + \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PartitionRecord/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PartitionRecord/index.html?rev=1873556&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PartitionRecord/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PartitionRecord/index.html Mon Feb 3 21:53:27 2020 @@ -0,0 +1 @@ +PartitionRecord

Description:

Receives Record-oriented data (i.e., data that can be read by the configured Record Reader) and evaluates one or more RecordPaths against the each record in the incoming FlowFile. Each record is then grouped with other "like records" and a FlowFile is created for each group of "like records." What it means for two records to be "like records" is determined by user-defined properties. The user is required to enter at least one user-defined property whose value is a RecordPath. Two records are considered alike if they have the same value for all configu red RecordPaths. Because we know that all records in a given output FlowFile have the same value for the fields that are specified by the RecordPath, an attribute is added for each field. See Additional Details on the Usage page for more information and examples.

Additional Details...

Tags:

record, partition, recordpath, rpath, segment, split, group, bin, organize

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name	Default Value	Allowable Values	Description
Record Reader		Controller Service API: RecordReaderFactory Implementations: ScriptedReader JsonTreeReader ParquetReader JsonPathReader GrokReader Syslog5424Reader XMLReader AvroReader CSVReader SyslogReader	Specifies the Controller Service to use for reading incoming data
Record Writer		Controller Service API: RecordSetWriterFactory Implementations: AvroRecordSetWriter ScriptedRecordSetWriter ParquetRecordSetWriter XMLRecordSetWriter JsonRecordSetWriter FreeFormTextRecordSetWriter CSVRecordSetWriter	Specifies the Controller Service to use for writing out the records

Dynamic Properties:

Dynamic Properties allow the user to specify both the name and value of a property.
Name Value Description
The name given to the dynamic property is the name of the attribute that will be used to de note the value of the associted RecordPath. A RecordPath that points to a field in the Record. Each dynamic property represents a RecordPath that will be evaluated against each record in an incoming FlowFile. When the value of the RecordPath is determined for a Record, an attribute is added to the outgoing FlowFile. The name of the attribute is the same as the name of this property. The value of the attribute is the same as the value of the field in the Record that the RecordPath points to. Note that no attribute will be added if the value returned for the RecordPath is null or is not a scalar value (i.e., the value is an Array, Map, or Record).
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Name	Value	Description
The name given to the dynamic property is the name of the attribute that will be used to de note the value of the associted RecordPath.	A RecordPath that points to a field in the Record.	Each dynamic property represents a RecordPath that will be evaluated against each record in an incoming FlowFile. When the value of the RecordPath is determined for a Record, an attribute is added to the outgoing FlowFile. The name of the attribute is the same as the name of this property. The value of the attribute is the same as the value of the field in the Record that the RecordPath points to. Note that no attribute will be added if the value returned for the RecordPath is null or is not a scalar value (i.e., the value is an Array, Map, or Record). Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

Name	Description
success	FlowFiles that are successfully partitioned will be ro uted to this relationship
failure	If a FlowFile cannot be partitioned from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship
original	Once all records in an incoming FlowFile have been partitioned, the original FlowFile is routed to this relationship.

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
record.count	The number of records in an outgoing FlowFile
mime.type	The MIME Type that the configured Record Writer indicates is appropriate
fragment.identifier	All partitioned FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute
fragment.index	A one-up number that indicates the ordering of the partitioned Fl owFiles that were created from a single parent FlowFile
fragment.count	The number of partitioned FlowFiles generated from the parent FlowFile
segment.original.filename	The filename of the parent FlowFile
<dynamic property name>	For each dynamic property that is added, an attribute may be added to the FlowFile. See the description for Dynamic Properties for more information.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

Deprecation notice:

This processor is deprecated and may be removed in future releases.

Please consider using one the following alternatives: InvokeHTTP

Description:

Please be aware this processor is deprecated and may be removed in the near future. Use InvokeHTTP instead. Performs an HTTP Post with the content of the FlowFile. Uses a connection pool with max number of connections equal to the number of possible endpoints multiplied by the Concurrent Tasks configuration.

Tags:

http, https, remote, copy, archive

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

Name	Default Value	Allowable Values	Description
URL			The URL to POST to. The URL may be defined using the Attribute Expression Language. A separate connection p ool will be created for each unique host:port combination. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Max Batch Size	100 MB		If the Send as FlowFile property is true, specifies the max data size for a batch of FlowFiles to send in a single HTTP POST. If not specified, each FlowFile will be sent separately. If the Send as FlowFile property is false, this property is ignored
Max Data to Post per Second			The maximum amount of data to send per second; this allows the bandwidth to be throttled to a specified data rate; if not specified, the data rate is not throttled
SSL Context Service		Controller Ser vice API: SSLContextService Implementations: StandardSSLContextService StandardRestrictedSSLContextService	The Controller Service to use in order to obtain an SSL Context
Username			Username required to access the URL
Password			Password required to access the URL Sensitive Property: true
Send as FlowFile	false	true false	If true, will package the FlowFile's contents and attributes together and send the FlowFile Package; otherwise, will send only the FlowFile's content
Use Chunked Encoding		true false	Specifies whether or not to use Chunked Encoding to send the data. This property is ignored in the event the contents are compressed or sent as FlowFiles.
Compression Level	0		Determines the GZIP Compression Level to use when sending the file; the value must be in the range of 0-9. A value of 0 indicates that the file will not be GZIP'ed
Connection Timeout	30 sec		How long to wait when attempting to connect to the remote server before giving up
Data Timeout	30 sec		How long to wait between receiving segments of data from the remote server before giving up and discarding the partial file
Attributes to Send as HTTP Headers (Regex)			Specifies the Regular Expression that determines the names of FlowFile attributes that should be sent as HTTP Headers
User Agent	Apache-HttpClient/4.5.5 (Java/1.8.0_232)		What to report as the User Agent when we connect to the remote server
Proxy Configuration Service		Controller Service API: ProxyConfiguratio nService Implementation: StandardProxyConfigurationService	Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN
Proxy Host			The fully qualified hostname or IP address of the proxy server
Proxy Port			The port of the proxy server
Content-Type	${mime.type}		The Content-Type to specify for the content of the FlowFile being POSTed if Send as FlowFile is false. In the case of an empty value after evaluating an expression language expression, Content-Type defaults to application/octet-stream Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

Name	Description
success	Files that are successfully send will be transferred to success
failure	Files that fail to send will transferred to failure

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html?rev=1873556&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html Mon Feb 3 21:53:27 2020 @@ -0,0 +1 @@ +PutDatabaseRecord

Description:

The PutDatabaseRecord processor uses a specified RecordReader to input (possibly multiple) records from an incoming flow file. These records are translated to SQL statements and executed as a single batch. If any errors occur, the flow file is routed to failure or retry, and if the records are transmitted successfully, the incoming flow file is routed to success. The type of statement executed by the processor is specified via the Statement Type property, which accepts some hard-coded values such as INSERT, UPDATE, and DELETE, as well as 'Use sta tement.type Attribute', which causes the processor to get the statement type from a flow file attribute. IMPORTANT: If the Statement Type is UPDATE, then the incoming records must not alter the value(s) of the primary keys (or user-specified Update Keys). If such records are encountered, the UPDATE statement issued to the database may do nothing (if no existing records with the new primary key values are found), or could inadvertently corrupt the existing data (by changing records for which the new values of the primary keys exist).

Tags:

sql, record, jdbc, put, database, update, insert, delete

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

N ame	Default Value	Allowable Values	Description
Record Reader		Controller Service API: RecordReaderFactory Implementations: ScriptedReader JsonTreeReader ParquetReader JsonPathReader GrokReader Syslog5424Reader XMLReader AvroReader CSVReader SyslogReader	Specifies the Controller Service to use for parsing incoming data and determining the data's schema.
Statement Type		UPDATE INSERT DELETE Use statement.type Attribute	Specifies the type of SQL Statement to generate. If 'Use statem ent.type Attribute' is chosen, then the value is taken from the statement.type attribute in the FlowFile. The 'Use statement.type Attribute' option is the only one that allows the 'SQL' statement type. If 'SQL' is specified, the value of the field specified by the 'Field Containing SQL' property is expected to be a valid SQL statement on the target database, and will be executed as-is.
Database Connection Pooling Service		Controller Service API: DBCPService Implementations: DBCPConnectionPoolLookup DBCPConnectionPool HiveConnection Pool	The Controller Service that is used to obtain a connection to the database for sending records.
Catalog Name			The name of the catalog that the statement should update. This may not apply for the database that you are updating. In this case, leave the field empty Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Schema Name			The name of the schema that the table belongs to. This may not apply for the database that you are updating. In this case, leave the field empty Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Table Name			The name of the table that the statement should affect. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Translate Field Names	true	true false	If true, the Processor will attempt to translate field names into the appropriate column names for the table specified. If false, the field names must match the column names exactly, or the column will not be updated
Unmatched Field Behavior	Ignore Unmatched Fields	Ignore Unmatched Fields Fail on Unmatched Fields	If an incoming record has a field that does not map to any of the database table's columns, this property specifies how to handle the situation
Unmatched Column Behavior	Fail on Unmatched Columns	Ignore Unmatched Columns Warn on Unmatched Columns Fail on Unmatched Columns	If an incoming record does not have a field mapping for all of the database table's columns, this property specifies how to handle the situation
Update Keys			A comma-separated list of column names that uniquely identifies a row in the database for UPDATE statements. If the Statement Type is UPDATE and this property is not set, the table's Primary Keys are used. In this case, if no Primary Key exists, the conversion to SQL will fail if Unmatched Column Behaviour is set to FAIL. This property is ignored if the Statement Type is INSERT Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Field Containing SQL			If the Statement Type is 'SQL' (as set in the statement.type attribute), this field indicates which field in the record(s) contains the SQL statement to execute. The value of the field must be a single SQL statement . If the Statement Type is not 'SQL', this field is ignored. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Allow Multiple SQL Statements	false	true false	If the Statement Type is 'SQL' (as set in the statement.type attribute), this field indicates whether to split the field value by a semicolon and execute each statement separately. If any statement causes an error, the entire set of statements will be rolled back. If the Statement Type is not 'SQL', this field is ignored.
Quote Column Identifiers	false	true false	Enabling this option will cause all column names to be quoted, allowing you to use reserved words as column nam es in your tables.
Quote Table Identifiers	false	true false	Enabling this option will cause the table name to be quoted to support the use of special characters in the table name.
Max Wait Time	0 seconds		The maximum amount of time allowed for a running SQL statement , zero means there is no limit. Max time less than 1 second will be equal to zero. Supports Expression Language: true (will be evaluated using variable registry only)
Rollback On Failure	false	true false	Specify how to handle error. By default (false), if an error occurs while processing a Flow File, the FlowFile will be routed to 'failure' or 'retry' relationship based on error type, and processor can continue with next FlowFile. Instead, you may want to rollback currently processed FlowFiles and stop further processing immediately. In that case, you can do so by enabling this 'Rollback On Failure' property. If enabled, failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. It is important to set adequate 'Yield Duration' to avoid retrying too frequently.
Table Schema Cache Size	100		Specifies how many Table Schemas should be cached
Maximum Batch Size	0		Specifies maximum batch size for INSERT and UPDATE statements. This parame ter has no effect for other statements specified in 'Statement Type'. Zero means the batch size is not limited. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

Name	Description
retry	A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed
success	Successfully created FlowFile from SQL query result set.
failure	A FlowFile is routed to this relationship if the database cannot be updated and retrying the operation will also fail, such as an invalid query or an integrity constraint violation

Reads Attributes:

Name	Description
statement.type	If 'Use statement.type Attribute' is selected for the St atement Type property, the value of this attribute will be used to determine the type of statement (INSERT, UPDATE, DELETE, SQL, etc.) to generate and execute.

Writes Attributes:

Name	Description
putdatabaserecord.error	If an error occurs during processing, the flow file will be routed to failure or retry, and this attribute will be populated with the cause of the error.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html?rev=1873556&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html Mon Feb 3 21:53:27 2020 @@ -0,0 +1 @@ +PutDistributedMapCache

Description:

Gets the content of a FlowFile and puts it to a distributed map cache, using a cache key computed from FlowFile attributes. If the cache already contains the entry and the cache update strategy is 'keep original' the entry is not replaced.'

Tags:

map, cache, put, distributed

Properties:

Name	Default Value	Allowable Values	Description
Cache Entry Identifier			A FlowFile attribute, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the cache key Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Distributed Cache Service		Controller Service API: DistributedMapCacheClient Implementations: DistributedMapCacheClientService CouchbaseMapCacheClient HBase_1_1_2_ClientMapCacheService RedisDistributedMapCacheClientService HBase_2_ClientMapCacheService	The Controller Service that is used to cache flow files
Cache update strategy	replace	Replace if present Keep original	Determines how the cache is updated if the cache already contains the entry
Max cache entry size	1 MB		The maximum amount of data to put into cache

Relationships:

Name	Description
success	Any FlowFile that is successfully inserted into cache will be routed to this relationship
failure	Any FlowFile that cannot be inserted into the cac he will be routed to this relationship

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
cached	All FlowFiles will have an attribute 'cached'. The value of this attribute is true, is the FlowFile is cached, otherwise false.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

Description:

Sends an e-mail to configured recipients for each incoming FlowFile

Tags:

email, put, notify, smtp

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value i n a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

Name	Default Value	Allowable Values	Description
SMTP Hostname			The hostname of the SMTP host Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP Port	25		The Port used for SMTP communications Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP Username			Username for th e SMTP account Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP Password			Password for the SMTP account Sensitive Property: true Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP Auth	true		Flag indicating whether authentication should be used Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP TLS	false		Flag indicating whether TLS should be enabled Supports E xpression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP Socket Factory	javax.net.ssl.SSLSocketFactory		Socket Factory to use for SMTP Connection Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SMTP X-Mailer Header	NiFi		X-Mailer used in the header of the outgoing email Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Attributes to Send as Headers (Regex)			A Regular Expression that is matched against all FlowFile attribute name s. Any attribute whose name matches the regex will be added to the Email messages as a Header. If not specified, no FlowFile attributes will be added as headers.
Content Type	text/plain		Mime Type used to interpret the contents of the email, such as text/plain or text/html Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
From			Specifies the Email address to use as the sender. Comma separated sequence of addresses following RFC822 syntax. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
To			The recipients to include in the To-Line of the email. Comma separated sequence of addresses following RFC822 syntax. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
CC			The recipients to include in the CC-Line of the email. Comma separated sequence of addresses following RFC822 syntax. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
BCC			The recipients to include in the BCC-Line of the email. Comma separated sequence of addresses following RFC822 syntax. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Subject	Message from NiFi		The email subject Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Message			The body of the email message Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Flow file content as message	false		Specifies whether or not the FlowFile content should be the message of the email. If true, the 'Message' property is ignored. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Atta ch File	false	true false	Specifies whether or not the FlowFile content should be attached to the email
Include All Attributes In Message	false	true false	Specifies whether or not all FlowFile attributes should be recorded in the body of the email message

Relationships:

Name	Description
success	FlowFiles that are successfully sent will be routed to this relationship
failure	FlowFiles that fail to send will be routed to this relationship

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

R estricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

Resource	Description
MEMORY	The entirety of the FlowFile's content (as a String object) will be read into memory in case the property to use the flow file content as the email body is set to true.

\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutFTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutFTP/index.html?rev=1873556&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutFTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.PutFTP/index.html Mon Feb 3 21:53:27 2020 @@ -0,0 +1 @@ +PutFTP

Description:

Sends FlowFiles to an FTP Server

Tags:

remote, copy, egress, put, ftp, archive, files

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive pro perty, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

Name	Default Value	Allowable Values	Description
Hostname			The fully qualified hostname or IP address of the remote system Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Port	21		The port that the remote system is listening on for file transfers Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Username			Username Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Password			Password for the user account Sensitive Property: true Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Remote Path			The path on the remote system from which to pull or push files Supports Expression Language: true (will be evaluated using variable registry only)
Create Directory	false	true false	Specifies whether or not the rem ote directory should be created if it does not exist.
Batch Size	500		The maximum number of FlowFiles to send in a single connection
Connection Timeout	30 sec		Amount of time to wait before timing out while creating a connection
Data Timeout	30 sec		When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems
Conflict Resolution	NONE	REPLACE IGNORE RENAME REJECT FAIL < li>NONE	Determines how to handle the problem of filename collisions
Dot Rename	true	true false	If true, then the filename of the sent file is prepended with a "." and then renamed back to the original once the file is completely sent. Otherwise, there is no rename. This property is ignored if the Temporary Filename property is set.
Temporary Filename			If set, the filename of the sent file will be equal to the value specified during the transfer and after successful completion will be renamed to the original filename. If this value is set, the Dot Rename property is ignored. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Transfer Mode	Binary	Binary ASCII	The FTP Transfer Mode
Connection Mode	Passive	Active Passive	The FTP Connection Mode
Reject Zero-Byte Files	true	true false	Determines whether or not Zero-byte files should be rejected without attempting to transfer
Last Modified Time			The lastModifiedTime to assign to the file after transferring it. If not set, the lastModifiedTime will not be changed. Format must be yyyy-MM-dd'T'HH:mm:ssZ. You may also use expression language such as ${file.lastModified Time}. If the value is invalid, the processor will not be invalid but will fail to change lastModifiedTime of the file. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Permissions			The permissions to assign to the file after transferring it. Format must be either UNIX rwxrwxrwx with a - in place of denied permissions (e.g. rw-r--r--) or an octal number (e.g. 644). If not set, the permissions will not be changed. You may also use expression language such as ${file.permissions}. If the value is invalid, the processor will not be invalid but will fail to change permissions of the file. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Use Compression	false	true false	Indicates whether or not ZLIB compression should be used when transferring files
Proxy Configuration Service		Controller Service API: ProxyConfigurationService Implementation: StandardProxyConfigurationService	Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS
Proxy Type	DIRECT	DIRECT HTTP SOCKS	Proxy type used for file transfers
Proxy Host			The fully qualified hostname or IP address of the proxy server Supports Expression Language: true (will be evaluated using variable registry only)
Proxy Port			The port of the proxy server Supports Expression Language: true (will be evaluated using variable registry only)
Http Proxy Username			Http Proxy Username Supports Expression Language: true (will be evaluated using variable registry only)
Http Proxy Password			Http Proxy Password Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only)
Internal Buffer Size	16KB		Set the internal buffer size for buffered data streams
Use UTF-8 Encoding	false	true false	Tells the client to use UTF-8 encoding when processing files and filenames. If set to true, the server must also support UTF-8 encoding.

Dynamic Properties:

Dynamic Properties allow the user to specify both the name and value of a property.
Name Value Description
pre.cmd._____ Not used The command specified in the key will be executed before doing a put. You may add these optional properties to send any commands to the FTP server before the file is actually transferred (before the put command). This option is only available for the PutFTP processor, as only FTP has this functionality. This is essentially the same as sending quote commands to an FTP server from the command line. While this is the same as sending a quote command, it is very important that you leave off the .
Supports Expression Language: false
post.cmd._____ Not used The command specified in the key will be executed after doing a put. You may add these optional properties to send any commands to the FTP server before the file is actually transferred (before the put command). This option is only available for the PutFTP processor, as only FTP has this functionality. This is essentially the same as sending quote commands to an FTP server from the command line. While this is the same as sending a quote command, it is very importan t that you leave off the .
Supports Expression Language: false

Name	Value	Description
pre.cmd._____	Not used	The command specified in the key will be executed before doing a put. You may add these optional properties to send any commands to the FTP server before the file is actually transferred (before the put command). This option is only available for the PutFTP processor, as only FTP has this functionality. This is essentially the same as sending quote commands to an FTP server from the command line. While this is the same as sending a quote command, it is very important that you leave off the . Supports Expression Language: false
post.cmd._____	Not used	The command specified in the key will be executed after doing a put. You may add these optional properties to send any commands to the FTP server before the file is actually transferred (before the put command). This option is only available for the PutFTP processor, as only FTP has this functionality. This is essentially the same as sending quote commands to an FTP server from the command line. While this is the same as sending a quote command, it is very importan t that you leave off the . Supports Expression Language: false

Relationships:

Name	Description
success	FlowFiles that are successfully sent will be routed to success
reject	FlowFiles that were rejected by the destination system
failure	FlowFiles that failed to send to the remote system; failure is usually looped back to this processor

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

ParseCEF

Description:

Tags:

Properties:

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

System Resource Considerations:

See Also:

ParseSyslog

Description:

Tags:

Properties:

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

System Resource Considerations:

See Also:

ParseSyslog5424

Description:

Tags:

Properties:

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

System Resource Considerations:

See Also:

Examples

Example 1 - Partition By Simple Field

Example 2 - Partition By Nullable Value

Example 3 - Partition By Multiple Values

PartitionRecord

Description:

Tags:

Properties:

Dynamic Properties:

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

System Resource Considerations:

See Also:

PostHTTP

Deprecation notice:

Description:

Tags:

Properties:

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

System Resource Considerations:

PutDatabaseRecord

Description:

Tags:

Properties:

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

System Resource Considerations:

PutDistributedMapCache

Description:

Tags:

Properties: