Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.HashContent/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.HashContent/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.HashContent/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.HashContent/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +HashContent

HashContent

Description:

Calculates a hash value for the Content of a FlowFile and puts that hash value on the FlowFile as an attribute whose name is determined by the <Hash Attribute Name> property

Tags:

hash, content, MD5, SHA-1, SHA-256

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
Hash Attribute Namehash.valueThe name of the FlowFile Attribute into which the Hash Value should be written. If the value already exists, it will be overwritten
Hash AlgorithmMD5
  • SHA-1
  • SHA-384
  • SKEIN-512-256
  • SKEIN-1024-384
  • BLAKE2B-160
  • SHA
  • KECCAK-288
  • WHIRLPOOL
  • SKEIN-512-384
  • SHA-224
  • SM3
  • BLAKE2B-512
  • GOST3411-2012-512
  • SKEIN-512-128
  • KECCAK-256
  • BLAKE2B-384
  • OID.2.16.840.1.101.3.4.2.10
  • SKEIN-256-160
  • SHA-256
  • SHA3-256
  • KECCAK-384
  • SKEIN-256-128
  • KECCAK-224
  • SKEIN-512-512
  • SKEIN-1024-512
  • SKEIN-512-160
  • GOST3411
  • BLAKE2B-256
  • SKEIN-1024-1024
  • SHA3- 384
  • OID.2.16.840.1.101.3.4.2.9
  • SHA-512/224
  • OID.2.16.840.1.101.3.4.2.8
  • OID.2.16.840.1.101.3.4.2.7
  • TIGER
  • RIPEMD256
  • SKEIN-256-256
  • SHA3-224
  • SHA3-512
  • RIPEMD320
  • 2.16.840.1.101.3.4.2.9
  • RIPEMD160
  • GOST3411-2012-256
  • KECCAK-512
  • SKEIN-512-224
  • 2.16.840.1.101.3.4.2.8
  • 2.16.840.1.101.3.4.2.7
  • 2.16.840.1.101.3.4.2.10
  • SHA-512/256
  • MD2
  • RIPEMD128
  • MD4
  • SHA-512
  • SKEIN-256-224
  • MD5
Determines what hashing algorithm should be used to perform the hashing function

Relationships:

NameDescription
successFlowFiles that are process successfully will be sent to this relationship
failureAny FlowFile that cannot be processed successfully will be sent to this relationship without any attribute being added

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
<Hash Attribute Name>This Processor adds an attribute whose value is the result of Hashing the existing FlowFile content. The name of this attribute is specified by the <Hash Attribute Name> property

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/additionalDetails.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/additionalDetails.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,64 @@ + + + + + + IdentifyMimeType + + + + + +

The following is a non-exhaustive list of MIME Types detected: +

+ +

For a complete list, please refer to + + Apache Tika's source code +

+ + + Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.IdentifyMimeType/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +IdentifyMimeType

IdentifyMimeType

Description:

Attempts to identify the MIME Type used for a FlowFile. If the MIME Type can be identified, an attribute with the name 'mime.type' is added with the value being the MIME Type. If the MIME Type cannot be determined, the value will be set to 'application/octet-stream'. In addition, the attribute mime.extension will be set if a common file extension for the MIME Type is known.

Additional Details...

Tags:

compression, gzip, bzip2, zip, MIME, mime.type, file, identify

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
Use Filename In Detectiontrue
  • true
  • false
If true will pass the filename to Tika to aid in detection.

Relationships:

NameDescription
successAll FlowFiles are routed to success

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
mime.typeThis Processor sets the FlowFile's mime.type attribute to the detected MIME Type. If unable to detect the MIME Type, the attribute's value will be set to application/octet-stream

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +InvokeHTTP

InvokeHTTP

Description:

An HTTP client processor which can interact with a configurable HTTP Endpoint. The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).

Tags:

http, https, rest, client

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
HTTP MethodGETHTTP request method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS). Arbitrary methods are also supported. Methods other than POST, PUT and PATCH will be sent without a message body.
Supports Expression Language: true
Remote URLRemote URL which will be connected to, including scheme, host, port, path.
Supports Expression Language: true
SSL Context ServiceController Service API:
SSLContextService
Implementations: StandardSSLContextService
StandardRestrictedSSLContextService
The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.
Connection Timeout5 secsMax wait time for connection to remote service.
Read Timeout15 secsMax wait time for response from remote service.
Include Date HeaderTrue
  • True
  • False
Include an RFC-2616 Date header in the request.
Follow RedirectsTrue
  • True
  • False
Follow HTTP redirects issued by remote server.
Attributes to SendRegular expression that defines which attributes to send as HTTP headers in the request. If not defined, no attributes are sent as headers. Also any dynamic properties set will be sent as headers. The dynamic propert y key will be the header key and the dynamic property value will be interpreted as expression language will be the header value.
Basic Authentication UsernameThe username to be used by the client to authenticate against the Remote URL. Cannot include control characters (0-31), ':', or DEL (127).
Basic Authentication PasswordThe password to be used by the client to authenticate against the Remote URL.
Sensitive Property: true
Proxy HostThe fully qualified hostname or IP address of the proxy server
Proxy PortThe port of the proxy server
Proxy UsernameUsername to set when authenticating against proxy
Proxy PasswordPassword to set when authenticating against proxy
Sensitive Property: true
Put Response Body In AttributeIf set, the response body received back will be put into an attribute of the original FlowFile instead of a separate FlowFile. The attribute key to put to is determined by evaluating value of this property.
Supports Expression Language: true
Max Length To Put In Attribute256If routing the response body to an attribute of the original (by setting the "Put response body in attribute" property or by receiving an error status code), the number of characters put to the attribute value will be at most this amount. This is important because attributes are held in memory and large attributes will quickly cause out of memory issues. If the output goes longer than this value, it will be truncated to fit. Consider making this smaller if able.
Use Digest Authenticationfalse
  • true
  • false
Whether to communicate with the website using Digest Authentication. 'Basic Authentication Username' and 'Basic Authentication Password' are used for authentication.
Always Output Responsefalse
  • true
  • false
Will force a response FlowFile to be generate d and routed to the 'Response' relationship regardless of what the server status code received is or if the processor is configured to put the server response body in the request attribute. In the later configuration a request FlowFile with the response body in the attribute and a typical response FlowFile will be emitted to their respective relationships.
Trusted HostnameBypass the normal truststore hostname verifier to allow the specified remote hostname as trusted. Enabling this property has MITM security implications, use wisely. Will still accept other connections based on the normal truststore hostname verifier. Only valid with SSL (HTTPS) connections.
Add Response Headers to Requestfalse
  • true
  • false
Enabling this property saves all t he response headers to the original request. This may be when the response headers are needed but a response is not generated due to the status code received.
Content-Type${mime.type}The Content-Type to specify for when content is being transmitted through a PUT, POST or PATCH. In the case of an empty value after evaluating an expression language expression, Content-Type defaults to application/octet-stream
Supports Expression Language: true
Send Message Bodytrue
  • true
  • false
If true, sends the HTTP message body on POST/PUT/PATCH requests (default). If false, suppresses the message body and content-type header for these requests.
Use Chunked Encodingfalse
  • true
  • false
When POST'ing, PUT'ing or PATCH'ing content set this property to true in order to not pass the 'Content-length' header and instead send 'Transfer-Encoding' with a value of 'chunked'. This will enable the data transfer mechanism which was introduced in HTTP 1.1 to pass data of unknown lengths in chunks.
Penalize on "No Retry"false
  • true
  • false
Enabling this property will penalize FlowFiles that are routed to the "No Retry" relationship.

Dynamic Properties:

Dynamic Properties allow the user to specify both the name and value of a property.
NameValueDescription
Header NameAttribute Expression LanguageS end request header with a key matching the Dynamic Property Key and a value created by evaluating the Attribute Expression Language set in the value of the Dynamic Property.
Supports Expression Language: true

Relationships:

NameDescription
OriginalThe original FlowFile will be routed upon success (2xx status codes). It will have new attributes detailing the success of the request.
FailureThe original FlowFile will be routed on any type of connection failure, timeout or general exception. It will have new attributes detailing the request.
RetryThe original FlowFile will be routed on any status code that can be retried (5xx status codes). It will have new attributes detailing the request.
No RetryThe original FlowFile will be routed on any status code that should NOT be retried (1xx, 3xx, 4 xx status codes). It will have new attributes detailing the request.
ResponseA Response FlowFile will be routed upon success (2xx status codes). If the 'Output Response Regardless' property is true then the response will be sent to this relationship regardless of the status code received.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
invokehttp.status.codeThe status code that is returned
invokehttp.status.messageThe status message that is returned
invokehttp.response.bodyIn the instance where the status code received is not a success (2xx) then the response body will be put to the 'invokehttp.response.body' attribute of the request FlowFile.
invokehttp.request.urlThe request URL
invokehttp.tx.idThe transaction ID that is retur ned after reading the response
invokehttp.remote.dnThe DN of the remote server
invokehttp.java.exception.classThe Java exception class raised when the processor fails
invokehttp.java.exception.messageThe Java exception message raised when the processor fails
user-definedIf the 'Put Response Body In Attribute' property is set then whatever it is set to will become the attribute key and the value would be the body of the HTTP response.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component allows an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/additionalDetails.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/additionalDetails.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,40 @@ + + + + + + JoltTransformJSON + + + + + +

Usage Information

+ +

+ The Jolt utilities processing JSON are not not stream based therefore large JSON document + transformation may consume large amounts of memory. Currently UTF-8 FlowFile content and Jolt specifications are supported. + A specification can be defined using Expression Language where attributes can be referred either on the left or right hand side within the specification syntax. + + Custom Jolt Transformations (that implement the Transform interface) are supported. Modules containing custom libraries which do not + existing on the current class path can be included via the custom module directory property. + + Note: When configuring a processor if user selects of the Default transformation yet provides a + Chain specification the system does not alert that the specification is invalid and and will produce failed flow files. + This is a known issue identified within the Jolt library. +

+ + Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +JoltTransformJSON

JoltTransformJSON

Description:

Applies a list of Jolt specifications to the flowfile JSON payload. A new FlowFile is created with transformed content and is routed to the 'success' relationship. If the JSON transform fails, the original FlowFile is routed to the 'failure' relationship.

Additional Details...

Tags:

json, jolt, transform, shiftr, chainr, defaultr, removr, cardinality, sort

Properties:

In the list below, the names of required properties appear in bold. Any other properties (no t in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Jolt Transformation DSLjolt-transform-chain
  • Cardinality Change the cardinality of input elements to create the output JSON.
  • Chain Execute list of Jolt transformations.
  • Default  Apply default values to the output JSON.
  • Modify - Default Writes when key is missing or value is null
  • Modify - Define Writes when key is missing
  • Modify - Overwrite  Always overwrite value
  • Remove  Remove values from input data to create the output JSON.
  • Shift Shift input JSON/data to create the output JSON.
  • Sort Sort input json key values alphabe
 tically. Any specification set is ignored.
  • Custom Custom Transformation. Requires Custom Transformation Class Name
Specifies the Jolt Transformation that should be used with the provided specification.
Custom Transformation Class NameFully Qualified Class Name for Custom Transformation
Custom Module DirectoryComma-separated list of paths to files and/or directories which contain modules containing custom transformations (that are not included on NiFi's classpath).
Jolt SpecificationJolt Specification for transform of JSON data. This value is ignored if the Jolt Sort Transformation is selected.
Supports Expression Language: true
Transform Cache Size1Compiling a Jolt Transform can be fairly expensive. Ideally, this will be done only once. However, if the Expression Language is used in the transform, we may need a new Transform for each FlowFile. This value controls how many of those Transforms we cache in memory in order to avoid having to compile the Transform each time.

Relationships:

NameDescription
successThe FlowFile with transformed content will be routed to this relationship
failureIf a FlowF ile fails processing for any reason (for example, the FlowFile is not valid JSON), it will be routed to this relationship

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
mime.typeAlways set to application/json

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListDatabaseTables/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListDatabaseTables/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListDatabaseTables/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListDatabaseTables/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListDatabaseTables

ListDatabaseTables

Description:

Generates a set of flow files, each containing attributes corresponding to metadata about a table from a database connection. Once metadata about a table has been fetched, it will not be fetched again until the Refresh Interval (if set) has elapsed, or until state has been manually cleared.

Tags:

sql, list, jdbc, table, database

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any defa ult values.

NameDefault ValueAllowable ValuesDescription
Database Connection Pooling ServiceController Service API:
DBCPService
Implementations: DBCPConnectionPool
HiveConnectionPool
The Controller Service that is used to obtain connection to database
CatalogThe name of a catalog from which to list database tables. The name must match the catalog name as it is stored in the database. If the property is not set, the catalog name will not be used to narrow the search for tables. If the property is set to an empty string, tables without a catalog will be listed.
Schema PatternA pattern for matching schemas in the database. Within a pattern, "%" means match any substring of 0 or more characters, and "_" means match any one character. The pattern must match the schema name as it is stored in the database. If the property is not set, the schema name will not be used to narrow the search for tables. If the property is set to an empty string, tables without a schema will be listed.
Table Name PatternA pattern for matching tables in the database. Within a pattern, "%" means match any substring of 0 or more characters, and "_" means match any one character. The pattern must match the table name as it is stored i n the database. If the property is not set, all tables will be retrieved.
Table TypesTABLEA comma-separated list of table types to include. For example, some databases support TABLE and VIEW types. If the property is not set, tables of all types will be returned.
Include Countfalse
  • true
  • false
Whether to include the table's row count as a flow file attribute. This affects performance as a database query will be generated for each table in the retrieved list.
Refresh Interval0 secThe amount of time to elapse before resetting the processor state, thereby causing all current tables to be listed. Duri ng this interval, the processor may continue to run, but tables that have already been listed will not be re-listed. However new/added tables will be listed as the processor runs. A value of zero means the state will never be automatically reset, the user must Clear State manually.

Relationships:

NameDescription
successAll FlowFiles that are received are routed to success

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
db.table.nameContains the name of a database table from the connection
db.table.catalogContains the name of the catalog to which the table belongs (may be null)
db.table.schemaContains the name of the schema to which the table belongs (may be null)
db.table.fullnameContain s the fully-qualifed table name (possibly including catalog, schema, etc.)
db.table.typeContains the type of the database table from the connection. Typical types are "TABLE", "VIEW", "SYSTEM TABLE", "GLOBAL TEMPORARY", "LOCAL TEMPORARY", "ALIAS", "SYNONYM"
db.table.remarksContains the name of a database table from the connection
db.table.countContains the number of rows in the table

State management:

ScopeDescription
CLUSTERAfter performing a listing of tables, the timestamp of the query is stored. This allows the Processor to not re-list tables the next time that the Processor is run. Specifying the refresh interval in the processor properties will indicate that when the processor detects the interval has elapsed, the state will be reset and tables will be re-listed as a result. This processor is meant to be run on the primary node only.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFTP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFTP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListFTP

ListFTP

Description:

Performs a listing of the files residing on an FTP server. For each file that is found on the remote server, a new FlowFile will be created with the filename attribute set to the name of the file on the remote server. This can then be used in conjunction with FetchFTP in order to fetch those files.

Tags:

list, ftp, remote, ingest, source, input, files

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any def ault values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
HostnameThe fully qualified hostname or IP address of the remote system
Supports Expression Language: true
Port21The port to connect to on the remote host to fetch the data from
Supports Expressio n Language: true
UsernameUsername
Supports Expression Language: true
PasswordPassword for the user account
Sensitive Property: true
Supports Expression Language: true
Remote Path.The path on the remote system from which to pull or push files
Supports Expression Language: true
Distributed Cache ServiceController Service API:
DistributedMapCacheClient
Implementations: HBase_1_1_2_ClientMapCacheService
RedisDistributedMapCacheClientService
DistributedMapCacheClientService
Specifies the Controller Service that should be used to maintain state about what has been pulled from the remote server so that if a new node begins pulling data, it won't duplicate all of the work that has been done. If not specified, the information will not be shared across the cluster. This property does not need to be set for standalone instances of NiFi but should be configured if NiFi is run within a cluster.
Search Recursivelyfalse
  • true
  • false
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories
File Filter RegexProvides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched
Path Filter RegexWhen Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned
Ignore Dotted Filestrue
  • true
  • false
If true, files whose names begin with a dot (".") will be ignore d
Remote Poll Batch Size5000The value specifies how many file paths to find in a given directory on the remote system when doing a file listing. This value in general should not need to be modified but when polling against a remote system with a tremendous number of files this value can be critical. Setting this value too high can result very poor performance and setting it too low can cause the flow to be slower than normal.
Connection Timeout30 secAmount of time to wait before timing out while creating a connection
Data Timeout30 secWhen transferring a file between the local and remote system, this v alue specifies how long is allowed to elapse without any data being transferred between systems
Connection ModePassive
  • Active
  • Passive
The FTP Connection Mode
Transfer ModeBinary
  • Binary
  • ASCII
The FTP Transfer Mode
Proxy TypeDIRECT
  • DIRECT
  • HTTP
  • SOCKS
Proxy type used for file transfers
Proxy HostThe fully qualified hostname or IP address of the proxy server
Proxy PortThe port of the proxy server
Http Proxy UsernameHttp Proxy Username
Http Proxy PasswordHttp Proxy Password
Sensitive Property: true
Internal Buffer Size16KBSet the internal buffer size for buffered data streams
Target System Timestamp Precisionauto-detect
  • Auto Detect Automatically detect time unit deterministically based on candidate entries timestamp. Please note that this option may take longer to list entities unnecessarily, if none of entries has a precise 
 precision timestamp. E.g. even if a target system supports millis, if all entries only have timestamps without millis, such as '2017-06-16 09:06:34.000', then its precision is determined as 'seconds'.
  • Milliseconds This option provides the minimum latency for an entry from being available to being listed if target system supports millis, if not, use other options.
  • Seconds For a target system that does not have millis precision, but has in seconds.
  • Minutes For a target system that only supports precision in minutes.
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision.

Relationships:

NameDescription
successAll FlowFiles that are received are routed to success

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
ftp.remote.hostThe hostname of the FTP Server
ftp.remote.portThe port that was connected to on the FTP Server
ftp.listing.userThe username of the user that performed the FTP Listing
file.ownerThe numeric owner id of the source file
file.groupThe numeric group id of the source file
file.permissionsThe read/write/execute permissions of the source file
file.sizeThe number of bytes in the source file
file.lastModifiedTimeThe timestamp of when the file in the filesystem waslast modified as 'yyyy-MM-dd'T'HH:mm:ssZ'
filenameThe name of the file on the SFTP Server
pathThe fully qualified name of the directory on the SFTP Server from which the file was pulled

State management:

ScopeDescription
CLUSTERAfter performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node will not duplicate the data that was listed by the previous Primary Node.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

See Also:

FetchFTP, GetFTP, PutFTP

\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFile/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFile/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFile/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListFile/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListFile

ListFile

Description:

Retrieves a listing of files from the local filesystem. For each file that is listed, creates a FlowFile that represents the file so that it can be fetched in conjunction with FetchFile. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data. Unlike GetFile, this Processor does not delete any data from the local filesystem.

Tags:

file, get, list, ingest, source, filesystem

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Input DirectoryThe input directory from which files to pull files
Supports Expression Language: true
Recurse Subdirectoriestrue
  • true
  • false
Indicates whether to list files from subdirectories of the directory
Input Directory LocationLocal
  • Local Input Directory is located on a local disk. State will be stored locally on each node in the cluster.
  • Remote Input Directory is located on a remote system. State will be stored across the cluster so that the listing can be performed on Primary Node Only and another node can pick up where the last node left off, if the Primary Node changes
Specifies where the Input Directory is located. This is use d to determine whether state should be stored locally or across the cluster.
File Filter[^\.].*Only files whose names match the given regular expression will be picked up
Path FilterWhen Recurse Subdirectories is true, then only subdirectories whose path matches the given regular expression will be scanned
Minimum File Age0 secThe minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored
Maximum File AgeThe maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored
Minimum File Size0 BThe minimum size that a file must be in order to be pulled
Maximum File SizeThe maximum size that a file can be in order to be pulled
Ignore Hidden Filestrue
  • true
  • false
Indicates whether or not hidden files should be ignored
Target System Timestamp Precisionauto-detect
  • Auto Detect
  • Milliseconds This option provides the minimum latency for an entry from being available to being listed if target syst
 em supports millis, if not, use other options.
  • Seconds For a target system that does not have millis precision, but has in seconds.
  • Minutes For a target system that only supports precision in minutes.
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision.

Relationships:

NameDescription
successAll FlowFiles that are received are routed to success

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
filenameThe name of the file that was read from filesystem.
pathThe path is set to the relative path of the file's directory on filesystem compared to the Input Directory property. For example, if Input Directory is set to /tmp, then files picked up from /tmp will have the path attribute set to "/". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "abc/1/2/3/".
absolute.pathThe absolute.path is set to the absolute path of the file's directory on filesystem. For example, if the Input Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to "/tmp/". If t he Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "/tmp/abc/1/2/3/".
file.ownerThe user that owns the file in filesystem
file.groupThe group that owns the file in filesystem
file.sizeThe number of bytes in the file in filesystem
file.permissionsThe permissions for the file in filesystem. This is formatted as 3 characters for the owner, 3 for the group, and 3 for other users. For example rw-rw-r--
file.lastModifiedTimeThe timestamp of when the file in filesystem was last modified as 'yyyy-MM-dd'T'HH:mm:ssZ'
file.lastAccessTimeThe timestamp of when the file in filesystem was last accessed as 'yyyy-MM-dd'T'HH:mm:ssZ'
file.creationTimeThe timestamp of when the file in filesystem was created as 'yyyy-MM-dd'T'HH:mm:ssZ'

State management:

ScopeDescription
LOCAL, CLUSTERAfter performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. Whether the state is stored with a Local or Cluster scope depends on the value of the <Input Directory Location> property.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

See Also:

GetFile, PutFile, FetchFile

\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListSFTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListSFTP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListSFTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListSFTP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListSFTP

ListSFTP

Description:

Performs a listing of the files residing on an SFTP server. For each file that is found on the remote server, a new FlowFile will be created with the filename attribute set to the name of the file on the remote server. This can then be used in conjunction with FetchSFTP in order to fetch those files.

Tags:

list, sftp, remote, ingest, source, input, files

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates an y default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

< td id="name">Remote Path
NameDefault ValueAllowable ValuesDescription
HostnameThe fully qualified hostname or IP address of the remote system
Supports Expression Language: true
Port22The port to connect to on the remote host to fetch the data from
Supports Expr ession Language: true
UsernameUsername
Supports Expression Language: true
PasswordPassword for the user account
Sensitive Property: true
Supports Expression Language: true
Private Key PathThe fully qualified path to the Private Key file
Supports Expression Language: true
Private Key PassphrasePassword for the private key
Sensitive Property: true
Supports Expression Language: true
.The path on the remote system from which to pull or push files
Supports Expression Language: true
Distributed Cache ServiceController Service API:
DistributedMapCacheClient
Implementations: HBase_1_1_2_ClientMapCacheService
RedisDistributedMapCacheClientService
DistributedMapCacheClientService
Spec ifies the Controller Service that should be used to maintain state about what has been pulled from the remote server so that if a new node begins pulling data, it won't duplicate all of the work that has been done. If not specified, the information will not be shared across the cluster. This property does not need to be set for standalone instances of NiFi but should be configured if NiFi is run within a cluster.
Search Recursivelyfalse
  • true
  • false
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories
File Filter RegexProvides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched< /td>
Path Filter RegexWhen Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned
Ignore Dotted Filestrue
  • true
  • false
If true, files whose names begin with a dot (".") will be ignored
Strict Host Key Checkingfalse
  • true
  • false
Indicates whether or not strict enforcement of hosts keys should be applied
Host Key FileIf supplied, the given file will be used as the Host Key; otherwise, no use host key file will be used
Connection Timeout30 secAmount of time to wait before timing out while creating a connection
Data Timeout30 secWhen transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems
Send Keep Alive On Timeouttrue
  • true
  • false
Indicates whether or not to send a single Keep Alive message when SSH socket times out
Target System Timestamp Precisionauto-detect
  • Auto Detect Automatically detect time unit deterministically based on candidate entries timestamp. Please note that this option may take longer to list entities unnecessarily, if none of entries has a precise precision timestamp. E.g. even if a target system supports millis, if all entries only have timestamps without millis, such as '2017-06-16 09:06:34.000', then its precision is determined as 'seconds'.
  • Milliseconds This option provides the minimum latency for an entry from be
 ing available to being listed if target system supports millis, if not, use other options.
  • Seconds For a target system that does not have millis precision, but has in seconds.
  • Minutes For a target system that only supports precision in minutes.
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision.

Relationships:

NameDescription
successAll FlowFiles that are received are routed to success

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
sftp.remote.hostThe hostname of the SFTP Server
sftp.remote.portThe port that was connected to on the SFTP Server
sftp.listing.userThe username of the user that performed the SFTP Listing
file.ownerThe numeric owner id of the source file
file.groupThe numeric group id of the source file
file.permissionsThe read/write/execute permissions of the source file
file.sizeThe number of bytes in the source file
file.lastModifiedTimeThe timestamp of when the file in the filesystem waslast modified as 'yyyy-MM-dd'T'HH:mm:ssZ '
filenameThe name of the file on the SFTP Server
pathThe fully qualified name of the directory on the SFTP Server from which the file was pulled

State management:

ScopeDescription
CLUSTERAfter performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node will not duplicate the data that was listed by the previous Primary Node.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

See Also:

FetchSFTP, GetSFTP, PutSFTP

\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenHTTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenHTTP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenHTTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenHTTP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListenHTTP

ListenHTTP

Description:

Starts an HTTP Server and listens on a given base path to transform incoming requests into FlowFiles. The default URI of the Service will be http://{hostname}:{port}/contentListener. Only HEAD and POST requests are supported. GET, PUT, and DELETE will result in an error and the HTTP response status code 405.

Tags:

ingest, http, https, rest, listen

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any d efault values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Base PathcontentListenerBase path for incoming connections
Supports Expression Language: true
Listening PortThe Port to listen on for incoming connections
Supports Expression Language: true
Max Data to Receive per SecondThe maximum amount of data to receive per second; this allows the bandwidth to be throttled to a specified data rate; if not specified, the data rate is not throttled
SSL Context ServiceController Service API:
RestrictedSSLContextService
Implementation: StandardRestrictedSSLContextService
The Controller Service to use in order to obtain an SSL Context
Authorized DN Pattern.*A Regular Expression to apply against the Distinguished Name of incoming connections. If the Pattern does not match the DN, the connection will be refused.
Max Unconfirmed Flowfile Time60 secsThe maxi mum amount of time to wait for a FlowFile to be confirmed before it is removed from the cache
HTTP Headers to receive as Attributes (Regex)Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes

Relationships:

NameDescription
successRelationship for successfully received FlowFiles

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenRELP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenRELP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenRELP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.ListenRELP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListenRELP

ListenRELP

Description:

Listens for RELP messages being sent to a given port over TCP. Each message will be acknowledged after successfully writing the message to a FlowFile. Each FlowFile will contain data portion of one or more RELP frames. In the case where the RELP frames contain syslog messages, the output of this processor can be sent to a ParseSyslog processor for further processing.

Tags:

listen, relp, tcp, logs

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are c onsidered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

< tr>
NameDefault ValueAllowable ValuesDescription
Local Network InterfaceThe name of a local network interface to be used to restrict listening to a specific LAN.
Supports Expression Language: true
PortThe port to listen on for communication.
Receive Buffer Size65507 BThe size of each buffer used to receive messages. Adjust this value appropriately based o n the expected size of the incoming messages.
Max Size of Message Queue10000The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor.
Max Size of Socket Buffer1 MBThe maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped.
Character SetUTF-8Specifies the character set of the received data.
Max Batch Size1The maximum number of messages to add to a single FlowFile. If multiple messages are available, they will be concatenated along with the <Message Delimiter> up to this configured maximum number of messages
Batching Message Delimiter\nSpecifies the delimiter to place between messages when multiple messages are bundled together (see <Max Batch Size> property).
Max Number of TCP Connections2The maximum number of concurrent TCP connections to accept.
SSL Context ServiceController Service API:
RestrictedSSLContextService
Implementation: StandardRestrictedSSLContextService
The Controller Service to use in order to obtain an SSL Context. If this property is set, messages will be received over a secure connection.
Client AuthREQUIRED
  • WANT
  • REQUIRED
  • NONE
The client authentication policy to use for the SSL Context. Only used if an SSL Context Service is provided.

Relationships:

NameDescription
successMessages received successfully will b e sent out this relationship.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
relp.commandThe command of the RELP frames.
relp.senderThe sending host of the messages.
relp.portThe sending port the messages were received over.
relp.txnrThe transaction number of the message. Only included if <Batch Size> is 1.
mime.typeThe mime.type of the content which is text/plain

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

See Also:

ParseSyslog

\ No newline at end of file