Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E568918BAC for ; Fri, 5 Feb 2016 13:20:52 +0000 (UTC) Received: (qmail 65904 invoked by uid 500); 5 Feb 2016 13:20:39 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 65767 invoked by uid 500); 5 Feb 2016 13:20:38 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 65756 invoked by uid 99); 5 Feb 2016 13:20:38 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Feb 2016 13:20:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1D3B6C09B1 for ; Fri, 5 Feb 2016 13:20:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.429 X-Spam-Level: * X-Spam-Status: No, score=1.429 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id PQ6teswJHNwS for ; Fri, 5 Feb 2016 13:20:37 +0000 (UTC) Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 7785E2050D for ; Fri, 5 Feb 2016 13:20:36 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id p63so26779196wmp.1 for ; Fri, 05 Feb 2016 05:20:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=PGYEWFZhXmaDGVHvNDCkg1P3s81e+P/Yp6ej2Sb3820=; b=HZxstcdl5uHl/2AOcaFq1Wt9E+ejmMq3daE4+Qa8WB+Vrt3OWaeDygk93bH/mOQ7K9 l6jD3f8OA3C8vMSjbqj36epKHZ6FNTqxe0YT2S8F6FfMuqgwwf3yd36K8Wyi0Y7DbWpy nMAT1DdqEVCkMdUeJ6T5DpujMa60fu0KXQuFhhY1zvyRJSpCmOI2DzAst04uK+R9qWh3 tT7caUXcVF8ot8dtavEv/x011l6eZ1FWvz3y4es7hs99XgjRFRpqqX0IR971p0PzHNI9 CZbQswNw1tK0CxaYwuM6gSnJ+j7BAEKtpBkmDgulozL/oeTUd+AMTjaNcOV4xgJLjmUK E1BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=PGYEWFZhXmaDGVHvNDCkg1P3s81e+P/Yp6ej2Sb3820=; b=Z+XYGKn4MPmAWVEA9aaEDKr1S8HWVekA58N1Sw9aJStWGQkX2rEZ8YJ1tVFiMUeZxa lE8KE/I4ma6z1u8BMUbJHe4aTlSexc32L/+BnlNloA9TJN7Q3GoRuv4QTX/gmHcBGl5Q Zwma7BRMidCQg/LtNzpVCvj2HTqW8MFSZPpgOWnm7B+1qnbsag56X7J/Chbne4iuEDvH kpWMVRnLUEAqBVFbP5Mx6b5CP8lGawb6AqqNCZj+TOShbMNJgTc3fSVFUAO0TmJUzPS4 KzWhZCjJwbU06reWxs6QJbXfw/siY3XQC87nLdqkUQ1FNbHPGk48jVamCqXYDOM1uWIS Rlng== X-Gm-Message-State: AG10YOQbSWfXObIN3WUhaoN/R1rJlO2tiXy/EYFA7nFDSuqr0JKjO+YMNDIlsWzrfZlAqxMqcz+ECycWu6hPYw== MIME-Version: 1.0 X-Received: by 10.28.55.76 with SMTP id e73mr17003945wma.53.1454678435168; Fri, 05 Feb 2016 05:20:35 -0800 (PST) Received: by 10.28.10.1 with HTTP; Fri, 5 Feb 2016 05:20:35 -0800 (PST) Date: Fri, 5 Feb 2016 18:50:35 +0530 Message-ID: Subject: spark streaming From: siva kumar To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1143ab92640ac2052b05b759 --001a1143ab92640ac2052b05b759 Content-Type: text/plain; charset=UTF-8 Hi , Im trying to pull data from twitter and do some processing out of it. I found this code import org.apache.spark.streaming._import org.apache.spark.streaming.twitter._import org.apache.spark.storage.StorageLevelimport scala.io.Sourceimport scala.collection.mutable.HashMapimport java.io.Fileimport org.apache.log4j.Loggerimport org.apache.log4j.Levelimport sys.process.stringSeqToProcess /** Configures the Oauth Credentials for accessing Twitter */def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) { val configs = new HashMap[String, String] ++= Seq( "apiKey" -> apiKey, "apiSecret" -> apiSecret, "accessToken" -> accessToken, "accessTokenSecret" -> accessTokenSecret) println("Configuring Twitter OAuth") configs.foreach{ case(key, value) => if (value.trim.isEmpty) { throw new Exception("Error setting authentication - value for " + key + " not set") } val fullKey = "twitter4j.oauth." + key.replace("api", "consumer") System.setProperty(fullKey, value.trim) println("\tProperty " + fullKey + " set as [" + value.trim + "]") } println()} // Configure Twitter credentialsval apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxx"val apiSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"val accessToken = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"val accessTokenSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret) import org.apache.spark.streaming.twitter._val ssc = new StreamingContext(sc, Seconds(2))val tweets = TwitterUtils.createStream(ssc, None)val twt = tweets.window(Seconds(60)) case class Tweet(createdAt:Long, text:String)twt.map(status=> Tweet(status.getCreatedAt().getTime()/1000, status.getText())).foreachRDD(rdd=> // Below line works only in spark 1.3.0. // For spark 1.1.x and spark 1.2.x, // use rdd.registerTempTable("tweets") instead. rdd.toDF().registerAsTable("tweets")) twt.print ssc.start() But getting the below error import org.apache.spark.streaming.twitter._ :19: error: object twitter is not a member of package org.apache.spark.streaming import org.apache.spark.streaming.twitter._ I using spark 1.3.0 The same worked for spark 0.9.0 . Any help? --001a1143ab92640ac2052b05b759 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi ,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Im trying to pu= ll data from twitter and do some processing out of it.=C2=A0 I found this c= ode
import org.apache=
.spark.streaming._
import org.apache.spark.streaming.twitter._
import org.apache.spark.storage.StorageLevel
import scala.io.Source
import scala.collection.mutable.HashMap
import java.io.File
import org.apache.log4j.Logger
import org.apache.log4j.Level
import sys.process.stringSeqToProcess

/** Con=
figures the Oauth Credentials for accessing Twitter */
def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {
  val configs =3D new HashMap[String, String] ++=
=3D Seq(
    "apiKey" -> apiKey, "apiSecret" -> apiSecret, "accessToke=
n" -> accessToken, "=
accessTokenSecret" =
-> accessTokenSecret)
  println("=
Configuring Twitter OAuth")
  configs.foreach{ case(<=
span class=3D"" style=3D"">key, value) =3D>
    if (value=
.trim.isEmpty) {
      throw new Exception(=
"Error setting authentication - value for " + key + " not set")
    }
    val fullKey =3D<=
/span> "twitter4j.oaut=
h." + key.replace("api", "consumer")
    System=
.setProperty(=
fullKey, value.tri=
m)
    println(&quo=
t;\tProperty " + fullKey + &=
quot; set as [" + value.trim + "]")
  }
  println()
}

// Conf=
igure Twitter credentials
val apiKey =3D "xxxxxxxxxxxxxxxxxxx=
xxxxxx"
val apiSecret =3D "xxxxxxxxxxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
val accessToken =3D<=
/span> "xxxxxxxxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
val accessTokenSecret =3D "xxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
configureTwitterCredentials(apiKey=
, apiSecret,<=
/span> accessToken, accessTokenSecre=
t)

import org.apache.spark.streaming.twitter._
val ssc =3D <=
span class=3D"" style=3D"font-weight:bold">new StreamingContext(sc<=
/span>, Seconds(2))
val tweets =3D TwitterU=
tils.createStream(ssc, None)
val twt =3D <=
span class=3D"" style=3D"">tweets.window(Seconds(60))

case class Tweet(createdAt:Long, text:String)
twt.map(status=3D>
  Tweet(status.<=
span class=3D"" style=3D"">getCreatedAt().getTime()/1000,=
 status.getText())
).foreachRDD(=
rdd=3D>
  // Be=
low line works only in spark 1.3.0.
  // Fo=
r spark 1.1.x and spark 1.2.x,
  // us=
e rdd.registerTempTable("tweets") instead.
  rdd.toDF().registerAsTa=
ble("tweets")
)

twt.print

ssc.start()
But getting the below error
import org.apache.spark.streaming.twitter._
<console>:19: error: object twitter is not a member of package org.ap=
ache.spark.streaming
       import org.apache.spark.streaming.twitter._
I =C2=A0using spark 1.3.0

=
The same worked for spark 0.9.0 . Any help?
--001a1143ab92640ac2052b05b759--