gobblin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Tiwari <a...@apache.org>
Subject Re: compaction feature
Date Fri, 23 Mar 2018 18:45:05 GMT
Hi Ethan,

Did you modify it already? If so, please feel free to send a PR.

Abhishek

On Mon, Dec 18, 2017 at 7:28 PM, Zhang, Xiuzhu(AWF) <xiuzhzhang@paypal.com>
wrote:

> Hi guys,
>
>
>
> The compaction feature currently can handle datasets between
> ‘compaction.timebase.max.time.ago’ and ‘…min.time.age’ the time based on
> current mean the time is changeable.
>
> Could let it to handle datasets after a fixed time point presented in
> directory(e.g.2017/12/16/08 or else)?
>
>
>
> I am also know can configure ‘compaction.timebased.folder.pattern’ to
> point the time patter based on TimeBasedSubDirDatasetsFinder.java at line
> 136:
>
> String folderStructure = getFolderStructure();
>
> for (FileStatus status : this.fs.globStatus(new Path(inputPath,
> folderStructure))) {xxx}
>
> Could configure multiple time pattern in xx.pull file(e.g.YYYY/MM/dd/HH
> and YYYY/MM/dd)?
>
>
>
> Thanks,
>
> Ethan
>

Mime
View raw message