Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
From: Arturo Michel <Arturo.Michel@leotech.com.sg>
To: "dev@nifi.apache.org" <dev@nifi.apache.org>
Subject: Hadoop Sequence File Processor changes the key.
Thread-Topic: Hadoop Sequence File Processor changes the key.
Thread-Index: AQHRrPPygc7JY1+H4kus93ZFmPQFig==
Date: Fri, 13 May 2016 08:49:35 +0000
Message-ID: <2a44f2befa2947439d2bd69b46c92e0f@leotech.com.sg>
Accept-Language: en-GB, en-US
Content-Language: en-GB
Content-Type: multipart/alternative;
	boundary="_000_2a44f2befa2947439d2bd69b46c92e0fleotechcomsg_"
MIME-Version: 1.0
archived-at: Fri, 13 May 2016 08:49:51 -0000

--_000_2a44f2befa2947439d2bd69b46c92e0fleotechcomsg_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I am using the createHadoopSequenceFile processor to create a sequence file=
 from incoming data to time stamp the data, using the current time as the k=
ey and the data as the value of the sequence file.


I change the file name attribute (momentarily) to ${now()} as to get a sequ=
ence file where the key is the time and the content is the data. However th=
e processor adds the .sf suffix which makes it all the way to the key.


I end up with the following structure [40668712567.sf | [data bytes]]


I understand that the file is written as filename.sf but shouldn't the key =
omit the .sf suffix and only be the file name?


Looking at Processor code in


<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bu=
ndle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/C=
reateHadoopSequenceFile.java>https://github.com/apache/nifi/blob/master/nif=
i-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apa=
che/nifi/processors/hadoop/CreateHadoopSequenceFile.java


155     final String fileName =3D flowFile.getAttribute(CoreAttributes.FILE=
NAME.key()) + ".sf";
156     flowFile =3D session.putAttribute(flowFile, CoreAttributes.FILENAME=
.key(), fileName);
157        try {
158            flowFile =3D sequenceFileWriter.writeSequenceFile(flowFile, =
session, getConfiguration(), compressionType);
159            session.transfer(flowFile, RELATIONSHIP_SUCCESS);
160            getLogger().info("Transferred flowfile {} to {}", new Object=
[]{flowFile, RELATIONSHIP_SUCCESS});
161        } catch (ProcessException e) {
162            getLogger().error("Failed to create Sequence File. Transferr=
ing {} to 'failure'", new Object[]{flowFile}, e);
163            session.transfer(flowFile, RELATIONSHIP_FAILURE);
164        }


The file name is changed before passing the flow file to the writer. The de=
fault sequence writer (and I think also the others) use the file name as re=
ceived to write the key.


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bun=
dle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/Se=
quenceFileWriterImpl.java


117 String key =3D flowFile.getAttribute(CoreAttributes.FILENAME.key());

118 writer.append(new Text(key), inStreamWritable);


If there is a better way of accomplishing this?


Best Regards.


This email is intended only for the individual or entity to which it is add=
ressed and may contain information that is private, restricted, confidentia=
l or secret and exempt from disclosure under applicable law.
If the reader of this disclaimer is not the intended recipient, you are her=
eby notified that any dissemination, distribution or copying of this docume=
nt is strictly prohibited. If you received this in error, please notify the=
 sender and delete it immediately after reading this disclaimer.
Thank you.


--_000_2a44f2befa2947439d2bd69b46c92e0fleotechcomsg_--