apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Priyanka Gugale <priya...@datatorrent.com>
Subject Re: [malhar-users] Re: How to find EOF - FileSplitter
Date Wed, 14 Oct 2015 13:05:29 GMT
+ dev@apex.incubator.apache.org

You don't need to set anything for dynamic partitioning. Dynamic
partitioning is by default applied on BlockReader.

-Priyanka

On Wed, Oct 14, 2015 at 6:27 PM, Chiru <chiru.vcj@gmail.com> wrote:

> Hi Priyanka,
>
> In my usecase I want to read a large file, which contains duplicate
> records. I want to find unique records by saving in collection.And once we
> reach the eof file / all block read then start processing the each record.
>
>
> I want to read the blocks in parallel as we need a good performance as
> well.Let me know how to use the dynamic partining here.
>
> thanks-Chiru
>
> On Friday, 9 October 2015 18:33:06 UTC+5:30, Chiru wrote:
>
>> Hi All,
>>
>> How i can find the entire file read when using the FileSplitter.I have to
>> wait till the EOF then start processing.
>>
>> Please share sample code if possible.
>>
>> Thanks -Chiru
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Malhar" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to malhar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to malhar-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/malhar-users.
> For more options, visit https://groups.google.com/d/optout.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message