ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chandresh pancholi <chandreshpancholi...@gmail.com>
Subject Re: After setting streamer and ignite,getting NULL
Date Tue, 29 Sep 2015 06:45:22 GMT
Gian,

is code for ignite-429 in master branch?

On Mon, Sep 28, 2015 at 5:27 PM, chandresh pancholi <
chandreshpancholi007@gmail.com> wrote:

> Gian,
>
> Lets use this and modify our tests. I will push the code by tomorrow EOD.
>
> Thanks
> Chandresh Pancholi
>
> On Mon, Sep 28, 2015 at 3:58 PM, Gianfranco Murador <
> murador.gianfranco@gmail.com> wrote:
>
>> Chandresh,
>>   I think that the suggestion of Roman is preferable, we can use  the
>> following code :
>>   Let me know if you have problems
>>
>> /** Start a simulated topology   */
>> public void startTopology(StormStreamer stormStreamer) {
>>         MkClusterParam mkClusterParam = new MkClusterParam();
>>         mkClusterParam.setSupervisors(4);
>>         Config daemonConf = new Config();
>>         daemonConf.put(Config.STORM_LOCAL_MODE_ZMQ, false);
>>         mkClusterParam.setDaemonConf(daemonConf);
>>
>>         Testing.withSimulatedTimeLocalCluster(mkClusterParam, new
>> TestJob()
>> {
>>                     @Override
>>                     public void run(ILocalCluster cluster) throws
>> IOException {
>>                         TopologyBuilder builder = new TopologyBuilder();
>>
>>                         builder.setSpout("testSpout", new StormSpout());
>>
>>                         builder.setBolt("bolt", stormStreamer)
>>                         .shuffleGrouping("spout");
>>
>>                         StormTopology topology = builder.createTopology();
>>
>>                         MockedSources mockedSources = new MockedSources();
>>
>>                          //Our spout will be processing this values.
>>                         mockedSources.addMockData("testSpout",new
>> Values(stormStream.getKeyValMap()));
>>
>>
>>                         // prepare the config
>>                         Config conf = new Config();
>>                         conf.setNumWorkers(2);
>>
>>                         CompleteTopologyParam completeTopologyParam = new
>> CompleteTopologyParam();
>>
>> completeTopologyParam.setMockedSources(mockedSources);
>>                         completeTopologyParam.setStormConf(conf);
>>
>>                         Map result = Testing.completeTopology(cluster,
>> topology, completeTopologyParam);
>>                     }
>>                 }
>>         );
>>     }
>>
>> 2015-09-28 6:11 GMT+02:00 chandresh pancholi <
>> chandreshpancholi007@gmail.com
>> >:
>>
>> > Agree,
>> > Will check for some other solution and update!!
>> >
>> > On Mon, Sep 28, 2015 at 9:12 AM, Vishal Garg <gargvish@gmail.com>
>> wrote:
>> >
>> > > Chandresh,
>> > >
>> > > Generally sleep in tests is not a good idea. Will try to look into
>> your
>> > > test this week, see if we can avoid it.
>> > > Meanwhile you can look at how storm does its own unit testing if you
>> > could
>> > > find one?
>> > > Vishal
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Sep 27, 2015, at 8:23 PM, Roman <rshtykh@yahoo.com.INVALID>
>> wrote:
>> > > >
>> > > > I also have concerns about this sleep().Storm has testing facilities
>> > you
>> > > can use for your test. How about
>> Testing.withSimulatedTimeLocalCluster()?
>> > > > Roman
>> > > >
>> > > >
>> > > >
>> > > >     On Monday, September 28, 2015 5:39 AM, Dmitriy Setrakyan <
>> > > dsetrakyan@apache.org> wrote:
>> > > >
>> > > >
>> > > > On Sun, Sep 27, 2015 at 1:36 PM, chandresh pancholi <
>> > > > chandreshpancholi007@gmail.com> wrote:
>> > > >
>> > > >> Gian,
>> > > >> I have checked the code. It looks good.
>> > > >> Just one little change make it  20 sec.
>> > > >>
>> > > >> /* Topology will run for 10sec */
>> > > >> Utils.sleep(20000);
>> > > >
>> > > > Hm... Can someone explain what is this sleep() for?
>> > > >
>> > > >
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Sun, Sep 27, 2015 at 4:30 PM, Gianfranco Murador <
>> > > >> murador.gianfranco@gmail.com> wrote:
>> > > >>
>> > > >>> Chandresh,  we're ready to do a pull request. Look here:
>> > > >>> https://github.com/murador/ignite/tree/IGNITE-429/modules/storm
>> > > >>>
>> > > >>> 2015-09-27 8:49 GMT+02:00 chandresh pancholi <
>> > > >>> chandreshpancholi007@gmail.com
>> > > >>>> :
>> > > >>>
>> > > >>>> Here is the log file.
>> > > >>>>
>> > > >>>> Thanks
>> > > >>>>
>> > > >>>> On Sat, Sep 26, 2015 at 10:40 PM, Gianfranco Murador <
>> > > >>>> murador.gianfranco@gmail.com> wrote:
>> > > >>>>
>> > > >>>>> Chan,
>> > > >>>>>   I can run the test succefully, but sometimes I get
this error:
>> > > >>>>> Caused by: class org.apache.ignite.spi.IgniteSpiException:
>> Failed
>> > to
>> > > >> add
>> > > >>>>> node to topology because remote node is configured
to use
>> loopback
>> > > >>>>> address,
>> > > >>>>> but local node is not (consider changing 'localAddress'
>> > configuration
>> > > >>>>> parameter).
>> > > >>>>>
>> > > >>>>> can you send the log file?
>> > > >>>>> Regards, Gianfranco
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> 2015-09-26 17:13 GMT+02:00 chandresh pancholi <
>> > > >>>>> chandreshpancholi007@gmail.com>:
>> > > >>>>>
>> > > >>>>>> Gian,
>> > > >>>>>> Its still giving "Ouch! Argument cannot be null:
streamer"
>> > > >>>>>>
>> > > >>>>>> On Sat, Sep 26, 2015 at 7:24 PM, Gianfranco Murador
<
>> > > >>>>>> murador.gianfranco@gmail.com> wrote:
>> > > >>>>>>
>> > > >>>>>>> Chan,
>> > > >>>>>>>   I've fixed some issue, please take a look
here:
>> > > >>
>> > >
>> >
>> https://github.com/murador/ignite/blob/IGNITE-429/modules/storm/src/test/java/org/apache/ignite/stream/storm/StormIgniteStreamerSelfTest.java
>> > > >>>>>>>
>> > > >>>>>>> 2015-09-26 8:29 GMT+02:00 chandresh pancholi
<
>> > > >>>>>>> chandreshpancholi007@gmail.com
>> > > >>>>>>>> :
>> > > >>>>>>>
>> > > >>>>>>>> Gian,
>> > > >>>>>>>>
>> > > >>>>>>>> I have followed instructions given by
you. I instantiate
>> > > >>>>>>> IgniteDateStreamer
>> > > >>>>>>>> and ignite in prepare() method of bolt.
but getting "*Cache
>> > > >>> doesn't
>> > > >>>>>>>> exist*".
>> > > >>>>>>>>
>> > > >>>>>>>> Code snippet from stormStreamer class
in prepare() method
>> > > >>>>>>>>
>> > > >>>>>>>> StormStreamer<String, String, String>
stormStreamer = null;
>> > > >>>>>>>>
>> > > >>>>>>>> Ignite ignite = Ignition.start();
>> > > >>>>>>>>
>> > > >>>>>>>> System.out.println(" ignite ====== "+ignite.toString());
>> > > >>>>>>>>
>> > > >>>>>>>> try(IgniteDataStreamer<String, String>
stmr =
>> > > >>>>>>>> ignite.dataStreamer("cache-0")) {
>> > > >>>>>>>>
>> > > >>>>>>>>     stormStreamer = new StormStreamer<>();
>> > > >>>>>>>>
>> > > >>>>>>>>     IgniteCache<String, String>
cache =
>> ignite.cache("cache-0");
>> > > >>>>>>>>
>> > > >>>>>>>>     stmr.allowOverwrite(true);
>> > > >>>>>>>>
>> > > >>>>>>>>     /*  Set ignite instance */
>> > > >>>>>>>>     stormStreamer.setIgnite(ignite);
>> > > >>>>>>>>
>> > > >>>>>>>>     /* Set streamer instance */
>> > > >>>>>>>>     stormStreamer.setStreamer(stmr);
>> > > >>>>>>>>
>> > > >>>>>>>>     /* set thread count */
>> > > >>>>>>>>     setThreads(5);
>> > > >>>>>>>>
>> > > >>>>>>>> }
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> On Thu, Sep 24, 2015 at 4:19 PM, Anton
Vinogradov <
>> > > >>>>>>>> avinogradov@gridgain.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> Chandresh,
>> > > >>>>>>>>> As far as understand IBolt implementation
should setup all
>> > > >>>>> external
>> > > >>>>>>>>> connections at .prepare() method.
>> > > >>>>>>>>> So, better way is to get existing
Ignite instance or create
>> > > >> new
>> > > >>> at
>> > > >>>>>>>>> .prepare()
>> > > >>>>>>>>> method.
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Thu, Sep 24, 2015 at 10:55 AM,
Gianfranco Murador <
>> > > >>>>>>>>> murador.gianfranco@gmail.com> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Chandresh,
>> > > >>>>>>>>>> I will launch the unit test later
this week , if the
>> problem
>> > > >>>>>>> persists.
>> > > >>>>>>>> I
>> > > >>>>>>>>>> think that you should instantiate
Ignite through the
>> start()
>> > > >>>>> method
>> > > >>>>>>> of
>> > > >>>>>>>>>> Ignition class
>> > > >>>>>>>>>> and retrieve the Ignite cache
specifying a name for the
>> > > >>> default
>> > > >>>>>>> cache.
>> > > >>>>>>>>> You
>> > > >>>>>>>>>> should provide a name also for
data streamer, not passing
>> > > >>> null.
>> > > >>>>>>>>>> Could you do this and run again
the unit test ?
>> > > >>>>>>>>>> Thank you,
>> > > >>>>>>>>>> Regards,
>> > > >>>>>>>>>>   Gianfranco
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 2015-09-24 5:40 GMT+02:00 chandresh
pancholi <
>> > > >>>>>>>>>> chandreshpancholi007@gmail.com
>> > > >>>>>>>>>>> :
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Anton,
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Clone this repo
>> > > >>> https://github.com/chandresh-pancholi/ignite
>> > > >>>>> and
>> > > >>>>>>> run
>> > > >>>>>>>>>> this
>> > > >>>>>>>>>>> class
>> > > >>
>> > >
>> >
>> https://github.com/chandresh-pancholi/ignite/blob/master/modules/storm/src/test/java/org/apache/ignite/stream/storm/IgniteStormStreamerSelfTestSuite.java
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> You will get the Error saying
"Oouch,Argument is Null" for
>> > > >>>>>>>>> getStreamer().
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Wed, Sep 23, 2015 at 9:46
PM, Anton Vinogradov <
>> > > >>>>>>>>>>> avinogradov@gridgain.com>
>> > > >>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Chandresh,
>> > > >>>>>>>>>>>> I'm trying to analize
problem.
>> > > >>>>>>>>>>>> Could you please provide
more details and explain
>> > > >>>>> step-by-step
>> > > >>>>>>> how
>> > > >>>>>>>>> can
>> > > >>>>>>>>>> I
>> > > >>>>>>>>>>>> reproduce problem?
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Tue, Sep 22, 2015 at
11:06 PM, chandresh pancholi <
>> > > >>>>>>>>>>>> chandreshpancholi007@gmail.com>
wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Not yet.
>> > > >>>>>>>>>>>>> Still doing brainstorming
why StormStreamer object is
>> > > >>>>> getting
>> > > >>>>>>>> NULLL
>> > > >>>>>>>>>>>>> in-spite setting up
ignite and streamer instances.
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> On Tue, Sep 22, 2015
at 7:22 PM, Vishal Garg <
>> > > >>>>>>> gargvish@gmail.com
>> > > >>>>>>>>>
>> > > >>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Did you figure
it out? Something related to your
>> > > >> storm
>> > > >>>>>>>>> integration?
>> > > >>>>>>>>>>>>>> Vishal
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Sent from my iPhone
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Sep
22, 2015, at 4:50 AM, chandresh pancholi <
>> > > >>>>>>>>>>>>>>> chandreshpancholi007@gmail.com>
wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Problem here
is with Storm. Its making streamer
>> > > >>> object
>> > > >>>>>> null
>> > > >>>>>>>>> once
>> > > >>>>>>>>>> it
>> > > >>>>>>>>>>>>> goes
>> > > >>>>>>>>>>>>>>> into submit
topology.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On Mon, Sep
21, 2015 at 7:13 PM, Lalit Kumar Jha <
>> > > >>>>>>>>>>>> lalitj.dev@gmail.com
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Hi Chandresh,
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> See test
suite class here, its based on
>> > > >> annotations
>> > > >>
>> > >
>> >
>> https://github.com/sylentprayer/ignite/blob/ignite-530/modules/twitter/src/test/java/org/apache/ignite/stream/twitter/IgniteTwitterStreamerTestSuite.java
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> No code
required in class body.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Mon,
Sep 21, 2015 at 6:24 PM, chandresh
>> > > >>> pancholi <
>> > > >>>>>>>>>>>>>>>> chandreshpancholi007@gmail.com>
wrote:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Hi
Gian/Vishal/Lalit,
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> You
guys have worked on Streamer integration
>> > > >> with
>> > > >>>>>>> Ignite. I
>> > > >>>>>>>>> am
>> > > >>>>>>>>>>>>> working
>> > > >>>>>>>>>>>>>> on
>> > > >>>>>>>>>>>>>>>>> Storm-ignite
integration.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> I
am setting up setStreamer() and setIgnite() in
>> > > >>>>> Test
>> > > >>>>>>> class
>> > > >>>>>>>>> and
>> > > >>>>>>>>>>>>> sending
>> > > >>>>>>>>>>>>>>>>> this
object to setBolt() method.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> When
i try to read via gettreamer() and
>> > > >>> getIgnite()
>> > > >>>>> i
>> > > >>>>>> am
>> > > >>>>>>>>>> getting
>> > > >>>>>>>>>>>>> NULL.
>> > > >>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>> am
>> > > >>>>>>>>>>>>>>>>> trying
to find out why is it happening but No
>> > > >>> Luck.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> If
any one of you find some time and look into
>> > > >> the
>> > > >>>>> code
>> > > >>>>>>> and
>> > > >>>>>>>>>> help
>> > > >>>>>>>>>>> me
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>> resolve
this. It would be great.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Git
:
>> > > >>> https://github.com/chandresh-pancholi/ignite
>> > > >>>>>>>>>>>>>>>>> Module
: storm
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> --
>> > > >>>>>>>>>>>>>>>>> Chandresh
Pancholi
>> > > >>>>>>>>>>>>>>>>> Senior
Software Engineer
>> > > >>>>>>>>>>>>>>>>> Flipkart.com
>> > > >>>>>>>>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>>>>>>>>>>>>>>> Contact:08951803660
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> --
>> > > >>>>>>>>>>>>>>> Chandresh
Pancholi
>> > > >>>>>>>>>>>>>>> Senior Software
Engineer
>> > > >>>>>>>>>>>>>>> Flipkart.com
>> > > >>>>>>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>>>>>>>>>>>>> Contact:08951803660
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> --
>> > > >>>>>>>>>>>>> Chandresh Pancholi
>> > > >>>>>>>>>>>>> Senior Software Engineer
>> > > >>>>>>>>>>>>> Flipkart.com
>> > > >>>>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>>>>>>>>>>> Contact:08951803660
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> --
>> > > >>>>>>>>>>> Chandresh Pancholi
>> > > >>>>>>>>>>> Senior Software Engineer
>> > > >>>>>>>>>>> Flipkart.com
>> > > >>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>>>>>>>>> Contact:08951803660
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> --
>> > > >>>>>>>> Chandresh Pancholi
>> > > >>>>>>>> Senior Software Engineer
>> > > >>>>>>>> Flipkart.com
>> > > >>>>>>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>>>>>> Contact:08951803660
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> --
>> > > >>>>>> Chandresh Pancholi
>> > > >>>>>> Senior Software Engineer
>> > > >>>>>> Flipkart.com
>> > > >>>>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>>>> Contact:08951803660
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> --
>> > > >>>> Chandresh Pancholi
>> > > >>>> Senior Software Engineer
>> > > >>>> Flipkart.com
>> > > >>>> Email-id:chandresh.pancholi@flipkart.com
>> > > >>>> Contact:08951803660
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Chandresh Pancholi
>> > > >> Senior Software Engineer
>> > > >> Flipkart.com
>> > > >> Email-id:chandresh.pancholi@flipkart.com
>> > > >> Contact:08951803660
>> > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Chandresh Pancholi
>> > Senior Software Engineer
>> > Flipkart.com
>> > Email-id:chandresh.pancholi@flipkart.com
>> > Contact:08951803660
>> >
>>
>
>
>
> --
> Chandresh Pancholi
> Senior Software Engineer
> Flipkart.com
> Email-id:chandresh.pancholi@flipkart.com
> Contact:08951803660
>



-- 
Chandresh Pancholi
Senior Software Engineer
Flipkart.com
Email-id:chandresh.pancholi@flipkart.com
Contact:08951803660

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message