apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogi Devendra <devendra.vyavah...@gmail.com>
Subject Re: Proposal for concrete operator for writing to HDFS file
Date Sun, 06 Mar 2016 03:50:31 GMT
Ashwin,

Please see my replies inline:

On 5 March 2016 at 22:42, Ashwin Chandra Putta <ashwinchandrap@gmail.com>
wrote:

> I think the concrete implementation should contain the following to allow
> for the most common use cases.
>
> 1. Take any java object as input and get the bytes of the string returned
> from toString method on the object.
>

Yes. It would allow any java object and byte[] will be derived from the
toString(). If input is byte[]; then it would be passed on without any
conversion.


> 2. The separator should be configurable. Null separator should also be
> valid.
>

Implementation will allow any String separator. Default would be newline.
Even empty string will be supported.
Are you referring to no-separator case by Null separator? How about using
empty string for no-separator instead of Null to avoid any special handling?


> 3. Should have one time configurable file path and name.
>

Yes. Filepath and name will be configurable as a property.



> 4. Should have configurable time based and size based rotation policy.
>

Do you mean rotate based on whichever happens first?

Size based rotation policy will be inherited from
AbstractFileOutputOperator.

For time based rotation, are you referring to write one file for X windows?
OR rotate if there is no new data for X windows?

In either case, can we say that set appropriate value X for
APPLICATION_WINDOW_COUNT for this operator?
OR should we expose another property rotationWindowCount for this?


>
> Regards,
> Ashwin.



~ Yogi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message