orc-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <omal...@apache.org>
Subject Re: append to the existing orc file
Date Wed, 27 Jul 2016 20:26:55 GMT
You can use org.apache.orc.Writer.writeIntermediateFooter(). That will
write a temporary footer to the file and flush it to hdfs.
writeIntermediateFooter will return the length of the file after the flush,
which should be passed in to OrcFile.ReaderOptions.maxLength(). In Hive, we
write the lengths in a side file so that if the writer dies suddenly, it
can always get the last footer. Look at OrcAcidUtils.getLastFlushLength().

.. Owen

On Wed, Jul 27, 2016 at 11:27 AM, praveen reddy <
praveen.onlinecourse@gmail.com> wrote:

> Hi,
> i am creating orc file based on particular action(kind of action user does
> from UI) and that action may happen in future as well. so ,one ORC file
> will be created for particular action and data will keep appending to that
> file when that action happens again in future.
> is there a way i can flush the data to HDFS without closing the Writer
> object. i will close the Writer object when the file on HDFS reaches
> particular length. only when i create a new file, writer on old file should
> be closed.
> currently i can see only close , flushes the data to HDFS. i don't want to
> close the file, keep writing to the file, flush it and close write only
> when file is of particular size.
> is there a way i can achieve it?
> Thanks,
> Praveen

View raw message