apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogi Devendra <devendra.vyavah...@gmail.com>
Subject Re: Proposal for concrete operator for writing to HDFS file
Date Sun, 06 Mar 2016 03:50:31 GMT

Please see my replies inline:

On 5 March 2016 at 22:42, Ashwin Chandra Putta <ashwinchandrap@gmail.com>

> I think the concrete implementation should contain the following to allow
> for the most common use cases.
> 1. Take any java object as input and get the bytes of the string returned
> from toString method on the object.

Yes. It would allow any java object and byte[] will be derived from the
toString(). If input is byte[]; then it would be passed on without any

> 2. The separator should be configurable. Null separator should also be
> valid.

Implementation will allow any String separator. Default would be newline.
Even empty string will be supported.
Are you referring to no-separator case by Null separator? How about using
empty string for no-separator instead of Null to avoid any special handling?

> 3. Should have one time configurable file path and name.

Yes. Filepath and name will be configurable as a property.

> 4. Should have configurable time based and size based rotation policy.

Do you mean rotate based on whichever happens first?

Size based rotation policy will be inherited from

For time based rotation, are you referring to write one file for X windows?
OR rotate if there is no new data for X windows?

In either case, can we say that set appropriate value X for
OR should we expose another property rotationWindowCount for this?

> Regards,
> Ashwin.

~ Yogi

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message