apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Priyanka Gugale <priya...@datatorrent.com>
Subject Re: [malhar-users] Re: How to find EOF - FileSplitter
Date Wed, 14 Oct 2015 11:39:19 GMT
You can set the partition count on block reader by using following property:
<property>

<name>dt.application.<appName>.operator.<operatorName>.attr.PARTITIONER</name>
  <value>com.datatorrent.common.partitioner.StatelessPartitioner:1</value>
</property>
This will make sure your data is read sequentially but performance would be
low. Also in this case detecting EOF will be easy by checking for last
block as suggested by Chandani.

Instead if you use dynamic partitioning or even use the high partition
count your reading speed will increase. But now you will have to add one
operator to track if all blocks of file are read by your application.

-Priyanka


On Wed, Oct 14, 2015 at 5:01 PM, Priyanka Gugale <priyanka@datatorrent.com>
wrote:

> If you have only one block reader (say you set partition count to 1 using
> static partitioning) then it will be sequential. But if you have multiple
> block readers, the blocks will be read in parallel and your last block
> could arrive before intermediate blocks.
>
> Can you brief us about your usecase. May be we could help you in designing
> the application.
>
> -Priyanka
>
> On Wed, Oct 14, 2015 at 4:55 PM, Chiru <chiru.vcj@gmail.com> wrote:
>
>> Hi Priyanks,
>>
>> thanks for the reply,
>>
>> I have one small query on FileSplitter.
>>
>> After splitting the file into blocks, the block read will be sequential
>> in order or parallell? Based on this i will build the logic for middle
>> operator for tracking.
>>
>>
>>
>> Thanks-Chiru
>>
>>
>>
>>
>>
>>
>> On Friday, 9 October 2015 18:33:06 UTC+5:30, Chiru wrote:
>>>
>>> Hi All,
>>>
>>> How i can find the entire file read when using the FileSplitter.I have
>>> to wait till the EOF then start processing.
>>>
>>> Please share sample code if possible.
>>>
>>> Thanks -Chiru
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Malhar" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to malhar-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to malhar-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/malhar-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message