apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: [malhar-users] Re: How to use FileSplitter to read huge file
Date Mon, 28 Sep 2015 18:10:45 GMT
Routing to dev@apex

Amol


On Mon, Sep 28, 2015 at 1:58 AM, Chiru <chiru.vcj@gmail.com> wrote:

> Hi Pramod,
>
> thanks for the reply, it is working..
>
> And i have one more query on it, How to decide the block size?
>
> as per my understanding the
>
> noofBlocks=filesize / blocksize
>
> By this some records may be split into two blocks, when converting the
> record we dont have the complete data in one block.
>
> how to handle this?
>
> thanks in adavance.
>
> Thanks -Chiru
>
>
> On Thursday, 24 September 2015 12:45:07 UTC+5:30, Chiru wrote:
>>
>> Hi All,
>>
>> I would like to read a large file using filesplitter and emit tuples.So i
>> have writtent the code like below.
>>
>>
>> public class Reader extends
>> AbstractFSBlockReader.AbstractFSReadAheadLineReader<Data>{
>>
>> @Override
>> protected Data convertToRecord(byte[] data)  { ///
>> }
>>
>> }
>>
>>
>> In my application class  i have created the object for filesplitter and
>> Reader classes and connect through stream.
>>
>> dag.addStream("blockin", fileSplitter.blocksMetadataOutput,
>> reader.blocksMetadataInput)
>>
>>
>> In properties file iam passing the directory path
>> like <name>dt.application.MyFirstApplication.operator.fileSplitter.prop.directoryPath</name>
>>
>> when i run the application iam getting the below error:
>> *2015-09-24 11:40:03,040 [1/FileSplitter:FileSplitter] ERROR
>> engine.StreamingContainer run - Abandoning deployment of operator
>> OperatorDeployInfo[id=1,name=FileSplitter,type=INPUT,checkpoint={ffffffffffffffff,
>> 0,
>> 0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=blocksMetadataOutput,streamId=blockin,bufferServer=<null>]]]
>> due to setup failure.*
>> *java.lang.IllegalArgumentException: empty files*
>>
>>
>> Please suggest is my approach is correct or not?
>> how to read data using Filesplitter using malhar-library-3.1.0.jar ,
>> share any sample code line.
>>
>> thanks _Chiranjeevi
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Malhar" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to malhar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to malhar-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/malhar-users.
> For more options, visit https://groups.google.com/d/optout.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message