ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chandresh pancholi <chandreshpancholi...@gmail.com>
Subject Re: After setting streamer and ignite,getting NULL
Date Mon, 28 Sep 2015 11:57:59 GMT
Gian,

Lets use this and modify our tests. I will push the code by tomorrow EOD.

Thanks
Chandresh Pancholi

On Mon, Sep 28, 2015 at 3:58 PM, Gianfranco Murador <
murador.gianfranco@gmail.com> wrote:

> Chandresh,
>   I think that the suggestion of Roman is preferable, we can use  the
> following code :
>   Let me know if you have problems
>
> /** Start a simulated topology   */
> public void startTopology(StormStreamer stormStreamer) {
>         MkClusterParam mkClusterParam = new MkClusterParam();
>         mkClusterParam.setSupervisors(4);
>         Config daemonConf = new Config();
>         daemonConf.put(Config.STORM_LOCAL_MODE_ZMQ, false);
>         mkClusterParam.setDaemonConf(daemonConf);
>
>         Testing.withSimulatedTimeLocalCluster(mkClusterParam, new TestJob()
> {
>                     @Override
>                     public void run(ILocalCluster cluster) throws
> IOException {
>                         TopologyBuilder builder = new TopologyBuilder();
>
>                         builder.setSpout("testSpout", new StormSpout());
>
>                         builder.setBolt("bolt", stormStreamer)
>                         .shuffleGrouping("spout");
>
>                         StormTopology topology = builder.createTopology();
>
>                         MockedSources mockedSources = new MockedSources();
>
>                          //Our spout will be processing this values.
>                         mockedSources.addMockData("testSpout",new
> Values(stormStream.getKeyValMap()));
>
>
>                         // prepare the config
>                         Config conf = new Config();
>                         conf.setNumWorkers(2);
>
>                         CompleteTopologyParam completeTopologyParam = new
> CompleteTopologyParam();
>
> completeTopologyParam.setMockedSources(mockedSources);
>                         completeTopologyParam.setStormConf(conf);
>
>                         Map result = Testing.completeTopology(cluster,
> topology, completeTopologyParam);
>                     }
>                 }
>         );
>     }
>
> 2015-09-28 6:11 GMT+02:00 chandresh pancholi <
> chandreshpancholi007@gmail.com
> >:
>
> > Agree,
> > Will check for some other solution and update!!
> >
> > On Mon, Sep 28, 2015 at 9:12 AM, Vishal Garg <gargvish@gmail.com> wrote:
> >
> > > Chandresh,
> > >
> > > Generally sleep in tests is not a good idea. Will try to look into your
> > > test this week, see if we can avoid it.
> > > Meanwhile you can look at how storm does its own unit testing if you
> > could
> > > find one?
> > > Vishal
> > >
> > > Sent from my iPhone
> > >
> > > > On Sep 27, 2015, at 8:23 PM, Roman <rshtykh@yahoo.com.INVALID>
> wrote:
> > > >
> > > > I also have concerns about this sleep().Storm has testing facilities
> > you
> > > can use for your test. How about
> Testing.withSimulatedTimeLocalCluster()?
> > > > Roman
> > > >
> > > >
> > > >
> > > >     On Monday, September 28, 2015 5:39 AM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org> wrote:
> > > >
> > > >
> > > > On Sun, Sep 27, 2015 at 1:36 PM, chandresh pancholi <
> > > > chandreshpancholi007@gmail.com> wrote:
> > > >
> > > >> Gian,
> > > >> I have checked the code. It looks good.
> > > >> Just one little change make it  20 sec.
> > > >>
> > > >> /* Topology will run for 10sec */
> > > >> Utils.sleep(20000);
> > > >
> > > > Hm... Can someone explain what is this sleep() for?
> > > >
> > > >
> > > >>
> > > >>
> > > >>
> > > >> On Sun, Sep 27, 2015 at 4:30 PM, Gianfranco Murador <
> > > >> murador.gianfranco@gmail.com> wrote:
> > > >>
> > > >>> Chandresh,  we're ready to do a pull request. Look here:
> > > >>> https://github.com/murador/ignite/tree/IGNITE-429/modules/storm
> > > >>>
> > > >>> 2015-09-27 8:49 GMT+02:00 chandresh pancholi <
> > > >>> chandreshpancholi007@gmail.com
> > > >>>> :
> > > >>>
> > > >>>> Here is the log file.
> > > >>>>
> > > >>>> Thanks
> > > >>>>
> > > >>>> On Sat, Sep 26, 2015 at 10:40 PM, Gianfranco Murador <
> > > >>>> murador.gianfranco@gmail.com> wrote:
> > > >>>>
> > > >>>>> Chan,
> > > >>>>>   I can run the test succefully, but sometimes I get this
error:
> > > >>>>> Caused by: class org.apache.ignite.spi.IgniteSpiException:
Failed
> > to
> > > >> add
> > > >>>>> node to topology because remote node is configured to
use
> loopback
> > > >>>>> address,
> > > >>>>> but local node is not (consider changing 'localAddress'
> > configuration
> > > >>>>> parameter).
> > > >>>>>
> > > >>>>> can you send the log file?
> > > >>>>> Regards, Gianfranco
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> 2015-09-26 17:13 GMT+02:00 chandresh pancholi <
> > > >>>>> chandreshpancholi007@gmail.com>:
> > > >>>>>
> > > >>>>>> Gian,
> > > >>>>>> Its still giving "Ouch! Argument cannot be null: streamer"
> > > >>>>>>
> > > >>>>>> On Sat, Sep 26, 2015 at 7:24 PM, Gianfranco Murador
<
> > > >>>>>> murador.gianfranco@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> Chan,
> > > >>>>>>>   I've fixed some issue, please take a look here:
> > > >>
> > >
> >
> https://github.com/murador/ignite/blob/IGNITE-429/modules/storm/src/test/java/org/apache/ignite/stream/storm/StormIgniteStreamerSelfTest.java
> > > >>>>>>>
> > > >>>>>>> 2015-09-26 8:29 GMT+02:00 chandresh pancholi <
> > > >>>>>>> chandreshpancholi007@gmail.com
> > > >>>>>>>> :
> > > >>>>>>>
> > > >>>>>>>> Gian,
> > > >>>>>>>>
> > > >>>>>>>> I have followed instructions given by you.
I instantiate
> > > >>>>>>> IgniteDateStreamer
> > > >>>>>>>> and ignite in prepare() method of bolt. but
getting "*Cache
> > > >>> doesn't
> > > >>>>>>>> exist*".
> > > >>>>>>>>
> > > >>>>>>>> Code snippet from stormStreamer class in prepare()
method
> > > >>>>>>>>
> > > >>>>>>>> StormStreamer<String, String, String>
stormStreamer = null;
> > > >>>>>>>>
> > > >>>>>>>> Ignite ignite = Ignition.start();
> > > >>>>>>>>
> > > >>>>>>>> System.out.println(" ignite ====== "+ignite.toString());
> > > >>>>>>>>
> > > >>>>>>>> try(IgniteDataStreamer<String, String>
stmr =
> > > >>>>>>>> ignite.dataStreamer("cache-0")) {
> > > >>>>>>>>
> > > >>>>>>>>     stormStreamer = new StormStreamer<>();
> > > >>>>>>>>
> > > >>>>>>>>     IgniteCache<String, String> cache
=
> ignite.cache("cache-0");
> > > >>>>>>>>
> > > >>>>>>>>     stmr.allowOverwrite(true);
> > > >>>>>>>>
> > > >>>>>>>>     /*  Set ignite instance */
> > > >>>>>>>>     stormStreamer.setIgnite(ignite);
> > > >>>>>>>>
> > > >>>>>>>>     /* Set streamer instance */
> > > >>>>>>>>     stormStreamer.setStreamer(stmr);
> > > >>>>>>>>
> > > >>>>>>>>     /* set thread count */
> > > >>>>>>>>     setThreads(5);
> > > >>>>>>>>
> > > >>>>>>>> }
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Sep 24, 2015 at 4:19 PM, Anton Vinogradov
<
> > > >>>>>>>> avinogradov@gridgain.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Chandresh,
> > > >>>>>>>>> As far as understand IBolt implementation
should setup all
> > > >>>>> external
> > > >>>>>>>>> connections at .prepare() method.
> > > >>>>>>>>> So, better way is to get existing Ignite
instance or create
> > > >> new
> > > >>> at
> > > >>>>>>>>> .prepare()
> > > >>>>>>>>> method.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Sep 24, 2015 at 10:55 AM, Gianfranco
Murador <
> > > >>>>>>>>> murador.gianfranco@gmail.com> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Chandresh,
> > > >>>>>>>>>> I will launch the unit test later
this week , if the problem
> > > >>>>>>> persists.
> > > >>>>>>>> I
> > > >>>>>>>>>> think that you should instantiate
Ignite through the start()
> > > >>>>> method
> > > >>>>>>> of
> > > >>>>>>>>>> Ignition class
> > > >>>>>>>>>> and retrieve the Ignite cache specifying
a name for the
> > > >>> default
> > > >>>>>>> cache.
> > > >>>>>>>>> You
> > > >>>>>>>>>> should provide a name also for data
streamer, not passing
> > > >>> null.
> > > >>>>>>>>>> Could you do this and run again the
unit test ?
> > > >>>>>>>>>> Thank you,
> > > >>>>>>>>>> Regards,
> > > >>>>>>>>>>   Gianfranco
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> 2015-09-24 5:40 GMT+02:00 chandresh
pancholi <
> > > >>>>>>>>>> chandreshpancholi007@gmail.com
> > > >>>>>>>>>>> :
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Anton,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Clone this repo
> > > >>> https://github.com/chandresh-pancholi/ignite
> > > >>>>> and
> > > >>>>>>> run
> > > >>>>>>>>>> this
> > > >>>>>>>>>>> class
> > > >>
> > >
> >
> https://github.com/chandresh-pancholi/ignite/blob/master/modules/storm/src/test/java/org/apache/ignite/stream/storm/IgniteStormStreamerSelfTestSuite.java
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> You will get the Error saying
"Oouch,Argument is Null" for
> > > >>>>>>>>> getStreamer().
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Sep 23, 2015 at 9:46 PM,
Anton Vinogradov <
> > > >>>>>>>>>>> avinogradov@gridgain.com>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Chandresh,
> > > >>>>>>>>>>>> I'm trying to analize problem.
> > > >>>>>>>>>>>> Could you please provide more
details and explain
> > > >>>>> step-by-step
> > > >>>>>>> how
> > > >>>>>>>>> can
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>> reproduce problem?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Tue, Sep 22, 2015 at 11:06
PM, chandresh pancholi <
> > > >>>>>>>>>>>> chandreshpancholi007@gmail.com>
wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Not yet.
> > > >>>>>>>>>>>>> Still doing brainstorming
why StormStreamer object is
> > > >>>>> getting
> > > >>>>>>>> NULLL
> > > >>>>>>>>>>>>> in-spite setting up ignite
and streamer instances.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, Sep 22, 2015 at
7:22 PM, Vishal Garg <
> > > >>>>>>> gargvish@gmail.com
> > > >>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Did you figure it
out? Something related to your
> > > >> storm
> > > >>>>>>>>> integration?
> > > >>>>>>>>>>>>>> Vishal
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Sent from my iPhone
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Sep 22,
2015, at 4:50 AM, chandresh pancholi <
> > > >>>>>>>>>>>>>>> chandreshpancholi007@gmail.com>
wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Problem here is
with Storm. Its making streamer
> > > >>> object
> > > >>>>>> null
> > > >>>>>>>>> once
> > > >>>>>>>>>> it
> > > >>>>>>>>>>>>> goes
> > > >>>>>>>>>>>>>>> into submit topology.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Mon, Sep 21,
2015 at 7:13 PM, Lalit Kumar Jha <
> > > >>>>>>>>>>>> lalitj.dev@gmail.com
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi Chandresh,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> See test suite
class here, its based on
> > > >> annotations
> > > >>
> > >
> >
> https://github.com/sylentprayer/ignite/blob/ignite-530/modules/twitter/src/test/java/org/apache/ignite/stream/twitter/IgniteTwitterStreamerTestSuite.java
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> No code required
in class body.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Mon, Sep
21, 2015 at 6:24 PM, chandresh
> > > >>> pancholi <
> > > >>>>>>>>>>>>>>>> chandreshpancholi007@gmail.com>
wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Gian/Vishal/Lalit,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> You guys
have worked on Streamer integration
> > > >> with
> > > >>>>>>> Ignite. I
> > > >>>>>>>>> am
> > > >>>>>>>>>>>>> working
> > > >>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>> Storm-ignite
integration.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I am setting
up setStreamer() and setIgnite() in
> > > >>>>> Test
> > > >>>>>>> class
> > > >>>>>>>>> and
> > > >>>>>>>>>>>>> sending
> > > >>>>>>>>>>>>>>>>> this object
to setBolt() method.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> When i
try to read via gettreamer() and
> > > >>> getIgnite()
> > > >>>>> i
> > > >>>>>> am
> > > >>>>>>>>>> getting
> > > >>>>>>>>>>>>> NULL.
> > > >>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>> am
> > > >>>>>>>>>>>>>>>>> trying
to find out why is it happening but No
> > > >>> Luck.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> If any
one of you find some time and look into
> > > >> the
> > > >>>>> code
> > > >>>>>>> and
> > > >>>>>>>>>> help
> > > >>>>>>>>>>> me
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> resolve
this. It would be great.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Git :
> > > >>> https://github.com/chandresh-pancholi/ignite
> > > >>>>>>>>>>>>>>>>> Module
: storm
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>> Chandresh
Pancholi
> > > >>>>>>>>>>>>>>>>> Senior
Software Engineer
> > > >>>>>>>>>>>>>>>>> Flipkart.com
> > > >>>>>>>>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>>>>>>>>>>>>>>> Contact:08951803660
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>> Chandresh Pancholi
> > > >>>>>>>>>>>>>>> Senior Software
Engineer
> > > >>>>>>>>>>>>>>> Flipkart.com
> > > >>>>>>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>>>>>>>>>>>>> Contact:08951803660
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> --
> > > >>>>>>>>>>>>> Chandresh Pancholi
> > > >>>>>>>>>>>>> Senior Software Engineer
> > > >>>>>>>>>>>>> Flipkart.com
> > > >>>>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>>>>>>>>>>> Contact:08951803660
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>> Chandresh Pancholi
> > > >>>>>>>>>>> Senior Software Engineer
> > > >>>>>>>>>>> Flipkart.com
> > > >>>>>>>>>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>>>>>>>>> Contact:08951803660
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> --
> > > >>>>>>>> Chandresh Pancholi
> > > >>>>>>>> Senior Software Engineer
> > > >>>>>>>> Flipkart.com
> > > >>>>>>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>>>>>> Contact:08951803660
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Chandresh Pancholi
> > > >>>>>> Senior Software Engineer
> > > >>>>>> Flipkart.com
> > > >>>>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>>>> Contact:08951803660
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Chandresh Pancholi
> > > >>>> Senior Software Engineer
> > > >>>> Flipkart.com
> > > >>>> Email-id:chandresh.pancholi@flipkart.com
> > > >>>> Contact:08951803660
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Chandresh Pancholi
> > > >> Senior Software Engineer
> > > >> Flipkart.com
> > > >> Email-id:chandresh.pancholi@flipkart.com
> > > >> Contact:08951803660
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Chandresh Pancholi
> > Senior Software Engineer
> > Flipkart.com
> > Email-id:chandresh.pancholi@flipkart.com
> > Contact:08951803660
> >
>



-- 
Chandresh Pancholi
Senior Software Engineer
Flipkart.com
Email-id:chandresh.pancholi@flipkart.com
Contact:08951803660

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message