apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: Question about running yahoofinance example in apex-malhar
Date Wed, 01 Jun 2016 07:24:00 GMT
Hi Junguk,

Please check your yarn configuration settings. I normally set
minimum-allocation-mb to 256MB for single node install, you can add
following lines to your yarn-site.xml and restart the yarn.

<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>256</value>
</property>

I am able to run YahooFinanceDemo app with above setting on my laptop
with 5GB memory usage.

Regards,
-Tushar.

On Wed, Jun 1, 2016 at 12:40 PM, Sandeep Deshmukh
<sandeep@datatorrent.com> wrote:
> Thanks for details.
>
> Typically the "Waiting for pending status" is followed by "Deploy request"
> as highlighted below. Are you getting the highlighted line in your log? If
> not, the operators is not getting deployed correctly.  Please let us know.
>
>
> 2016-05-23 14:46:51,649 DEBUG
> com.datatorrent.stram.engine.StreamingContainer: buffer server memory 512
> 2016-05-23 14:46:51,655 INFO
> com.datatorrent.bufferserver.storage.DiskStorage: using
> /data/hadoop-2.7.2/tmp/nm-local-dir/usercache/sandeep/appcache/application_1463980780220_0003/container_1463980780220_0003_01_000002/tmp
> as the basepath for spooling.
> 2016-05-23 14:46:51,663 INFO com.datatorrent.bufferserver.server.Server:
> Server started listening at /0:0:0:0:0:0:0:0:42358
> 2016-05-23 14:46:51,665 DEBUG
> com.datatorrent.stram.engine.StreamingContainer: Buffer server started:
> /0:0:0:0:0:0:0:0:42358
> 2016-05-23 14:46:51,700 DEBUG
> com.datatorrent.stram.engine.StreamingContainer: Entering heartbeat loop
> (interval is 1000 ms)
> 2016-05-23 14:46:52,750 INFO
> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending
> request.
> 2016-05-23 14:46:53,255 INFO
> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending
> request.
> 2016-05-23 14:46:56,287 INFO
> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending
> request.
> 2016-05-23 14:46:56,791 INFO
> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending
> request.
> 2016-05-23 14:46:57,295 INFO
> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending
> request.
> 2016-05-23 14:46:57,809 INFO
> com.datatorrent.stram.engine.StreamingContainer: Deploy request:
> [OperatorDeployInfo[id=1,name=Input,type=INPUT,checkpoint={ffffffffffffffff,
> 0,
> 0},inputs=[],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=enrich,bufferServer=goofy]]]]
> 2016-05-23 14:46:57,811 DEBUG
> com.datatorrent.stram.engine.StreamingContainer: Restoring operator 1 to
> checkpoint ffffffffffffffff stateless=false.
> 2016-05-23 14:46:57,876 DEBUG com.datatorrent.stram.engine.Node: type=INPUT,
> operator class=class com.datatorrent.InputFileReader
> 2016-05-23 14:46:57,889 DEBUG
> com.datatorrent.stram.engine.StreamingContainer: Marking operator 1 as
> deployed.
>
>
> Regards,
> Sandeep
>
> Regards,
> Sandeep
>
> On Wed, Jun 1, 2016 at 12:29 PM, Junguk Cho <jmanbal@gmail.com> wrote:
>>
>> Hi,
>>
>> 1. I used only one machine and use docker image to run it.
>> When I access to ip:8088, it shows I use 8 GB memory.
>> (Is there a configuration file I can check?)
>>
>> 2. Hadoop version : 2.7.1
>> I tested both Apex 3.2.2 and Apex 3.3.
>>
>> Thanks,
>> Junguk
>>
>>
>> 2016-05-31 22:09 GMT-06:00 Sandeep Deshmukh <sandeep@datatorrent.com>:
>> > Hi Junguk,
>> >
>> > Could you please give us some details:
>> >
>> > Hadoop cluster configuration: no of machines and their configuration
>> > (memory
>> > in particular)
>> > Version of Hadoop and Apex
>> >
>> > Regards,
>> > Sandeep
>> >
>> > On Wed, Jun 1, 2016 at 5:04 AM, Junguk Cho <jmanbal@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I am new to apex.
>> >> I tried to run demo examples in incubator-apex-malhar.
>> >>
>> >> When I ran yahoofinance example, I think it was stuck.
>> >> Log file said
>> >> 2016-05-31 20:34:00,628 INFO
>> >> com.datatorrent.bufferserver.server.Server: Server started listening
>> >> at /0:0:0:0:0:0:0:0:45546
>> >> 2016-05-31 20:34:01,668 INFO
>> >> com.datatorrent.stram.engine.StreamingContainer: Waiting for pending
>> >> request.
>> >>
>> >> I think binding IP address has a problem.
>> >>
>> >> Does anyone help me?
>> >>
>> >>
>> >> Thanks,
>> >> Junguk
>> >
>> >
>
>

Mime
View raw message