spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chaudhary, Umesh" <Umesh.Chaudh...@searshc.com>
Subject RE: FW: Websphere MQ as a data source for Apache Spark Streaming
Date Mon, 01 Jun 2015 09:00:30 GMT
Thanks  for your  suggestion.
Yes by Dstream.SaveAsTextFile();
I was doing a mistake by writing StorageLevel.NULL while overriding the storageLevel method
in my custom receiver.
When I changed it to StorageLevel.MEMORY_AND_DISK_2() , data started to save at disk.
Now it’s running without any issue.


From: Tathagata Das [mailto:tdas@databricks.com]
Sent: Friday, May 29, 2015 3:30 AM
To: Chaudhary, Umesh
Cc: Arush Kharbanda; user@spark.apache.org
Subject: Re: FW: Websphere MQ as a data source for Apache Spark Streaming

Are you sure that the data can be saved as strings?
Another, more controlled approach is use DStream.foreachRDD , which takes a Function2 parameter,
RDD and Time. There you can explicitly do stuff with the RDD, save it to separate files (separated
by time), or whatever.  Might help you to debug what is going on.
Might help if you shows the streaming program in a pastebin.

TD


On Fri, May 29, 2015 at 12:55 AM, Chaudhary, Umesh <Umesh.Chaudhary@searshc.com<mailto:Umesh.Chaudhary@searshc.com>>
wrote:
Hi,
I have written  manual receiver for Websphere MQ and its working fine.
If I am doing JavaDStream.SaveAsTextFile(“/home/user/out.txt”)  then its generating a
directory naming out.txt appending its timestamp.
In this directory only _SUCCESS file is present. I can see data on console while running in
local mode but not able to save it as text file.
Is there any other way for saving streaming data?

From: Chaudhary, Umesh
Sent: Tuesday, May 26, 2015 2:39 AM
To: 'Arush Kharbanda'; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: RE: Websphere MQ as a data source for Apache Spark Streaming

Thanks for the suggestion, I will try and post the outcome.

From: Arush Kharbanda [mailto:arush@sigmoidanalytics.com]
Sent: Monday, May 25, 2015 12:24 PM
To: Chaudhary, Umesh; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Websphere MQ as a data source for Apache Spark Streaming

Hi Umesh,

You can write a customer receiver for Websphere MQ, using the API for websphere MQ.

https://spark.apache.org/docs/latest/streaming-custom-receivers.html

Thanks
Arush

On Mon, May 25, 2015 at 8:04 PM, Chaudhary, Umesh <Umesh.Chaudhary@searshc.com<mailto:Umesh.Chaudhary@searshc.com>>
wrote:
I have seen it but it has different configuration for connecting the MQ.
I mean for Websphere MQ we need Host, Queue Manager, Channel And Queue Name but here according
to MQTT protocol

client = new MqttClient(brokerUrl, MqttClient.generateClientId(), persistence)

It only expects Broker URL which is in appropriate for establishing connection with Websphere
MQ.

Please Suggest !


From: Arush Kharbanda [mailto:arush@sigmoidanalytics.com<mailto:arush@sigmoidanalytics.com>]
Sent: Monday, May 25, 2015 6:29 AM
To: Chaudhary, Umesh
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Websphere MQ as a data source for Apache Spark Streaming

Hi Umesh,

You can connect to Spark Streaming with MQTT  refer to the example.

https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/MQTTWordCount.scala



Thanks
Arush



On Mon, May 25, 2015 at 3:43 PM, umesh9794 <umesh.chaudhary@searshc.com<mailto:umesh.chaudhary@searshc.com>>
wrote:
I was digging into the possibilities for Websphere MQ as a data source for
spark-streaming becuase it is needed in one of our use case. I got to know
that  MQTT <http://mqtt.org/>   is the protocol that supports the
communication from MQ data structures but since I am a newbie to spark
streaming I need some working examples for the same. Did anyone try to
connect the MQ with spark streaming. Please devise the best way for doing
so.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Websphere-MQ-as-a-data-source-for-Apache-Spark-Streaming-tp23013.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>



--

[Image removed by sender. Sigmoid Analytics]<http://htmlsig.com/www.sigmoidanalytics.com>

Arush Kharbanda || Technical Teamlead

arush@sigmoidanalytics.com<mailto:arush@sigmoidanalytics.com> || www.sigmoidanalytics.com<http://www.sigmoidanalytics.com/>
This message, including any attachments, is the property of Sears Holdings Corporation and/or
one of its subsidiaries. It is confidential and may contain proprietary or legally privileged
information. If you are not the intended recipient, please delete it without reading the contents.
Thank you.



--

[Image removed by sender. Sigmoid Analytics]<http://htmlsig.com/www.sigmoidanalytics.com>

Arush Kharbanda || Technical Teamlead

arush@sigmoidanalytics.com<mailto:arush@sigmoidanalytics.com> || www.sigmoidanalytics.com<http://www.sigmoidanalytics.com/>
This message, including any attachments, is the property of Sears Holdings Corporation and/or
one of its subsidiaries. It is confidential and may contain proprietary or legally privileged
information. If you are not the intended recipient, please delete it without reading the contents.
Thank you.


This message, including any attachments, is the property of Sears Holdings Corporation and/or
one of its subsidiaries. It is confidential and may contain proprietary or legally privileged
information. If you are not the intended recipient, please delete it without reading the contents.
Thank you.
Mime
View raw message