Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAF=CqftENHt3WhVd_uQv9VTr7KJGU+uV7Q0RNhOpktW_5Y5gmg@mail.gmail.com>
References: <CAM8qWkCEKhBG5gi9YDhw93mj_0=zm+n17vbC1Ei75BxBCCwd0A@mail.gmail.com>
 <CAF=CqfvK1FJdY676k8nH96+DMxcEuLyUestFzcwiMdv8oLLO7Q@mail.gmail.com> <CAF=CqftENHt3WhVd_uQv9VTr7KJGU+uV7Q0RNhOpktW_5Y5gmg@mail.gmail.com>
From: Neil Jonkers <neilodini@gmail.com>
Date: Mon, 12 Dec 2016 21:22:54 +0200
Message-ID: <CABkNtuic5GuzKpxhGhyCaoraN1MOdCTp-AB8BvWvbm9bw0=UbQ@mail.gmail.com>
Subject: Re: [Spark Core] - Spark dynamoDB integration
To: Marco Mistroni <mmistroni@gmail.com>
Cc: Pratyaksh Sharma <pratyaksh.sharma.ece12@itbhu.ac.in>, User <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c033d78cdea0d05437b070d
archived-at: Mon, 12 Dec 2016 19:23:02 -0000

--94eb2c033d78cdea0d05437b070d
Content-Type: text/plain; charset=UTF-8

Hello,

Good examples on how to interface with DynamoDB from Spark here:

https://aws.amazon.com/blogs/big-data/using-spark-sql-for-etl/
https://aws.amazon.com/blogs/big-data/analyze-your-data-on-amazon-dynamodb-with-apache-spark/

Thanks

On Mon, Dec 12, 2016 at 7:56 PM, Marco Mistroni <mmistroni@gmail.com> wrote:

> Hi
>  If it can help
> 1.Check Java docs of when that method was introduced
> 2. U building a fat jar? Check which libraries have been included....some
> other dependencies might have forced an old copy to be included....
> 3. If u. Take code outside spark.....does it work successfully?
> 4. Send short sample....
> Hth
>
> On 12 Dec 2016 11:03 am, "Pratyaksh Sharma" <pratyaksh.sharma.ece12@itbhu.
> ac.in> wrote:
>
> Hey I am using Apache Spark for one streaming application. I am trying to
> store the processed data into dynamodb using java sdk. Getting the
> following exception -
> 16/12/08 23:23:43 WARN TaskSetManager: Lost task 0.0 in stage 1.0:
> java.lang.NoSuchMethodError: com.amazonaws.SDKGlobalConfigu
> ration.isInRegionOptimizedModeEnabled()Z
> at com.amazonaws.ClientConfigurationFactory.getConfig(ClientCon
> figurationFactory.java:35)
> at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init
> >(AmazonDynamoDBClient.java:374)
>
> Spark version - 1.6.1
> Scala version - 2.10.5
> aws sdk version - 1.11.33
>
> Has anyone faced this issues? Any help will be highly appreciated.
>
> --
> Regards
>
> Pratyaksh Sharma
> 12105EN013
> Department of Electronics Engineering
> IIT Varanasi
> Contact No +91-8127030223 <+91%2081270%2030223>
>
>
>

--94eb2c033d78cdea0d05437b070d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello,<div><br></div><div>Good examples on how to interfac=
e with DynamoDB from Spark here:</div><div><br></div><div><div><a href=3D"h=
ttps://aws.amazon.com/blogs/big-data/using-spark-sql-for-etl/">https://aws.=
amazon.com/blogs/big-data/using-spark-sql-for-etl/</a></div><div><a href=3D=
"https://aws.amazon.com/blogs/big-data/analyze-your-data-on-amazon-dynamodb=
-with-apache-spark/">https://aws.amazon.com/blogs/big-data/analyze-your-dat=
a-on-amazon-dynamodb-with-apache-spark/</a></div></div><div><br></div><div>=
Thanks</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"=
>On Mon, Dec 12, 2016 at 7:56 PM, Marco Mistroni <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:mmistroni@gmail.com" target=3D"_blank">mmistroni@gmail.com</a=
>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto">Hi<=
div dir=3D"auto">=C2=A0If it can help</div><div dir=3D"auto">1.Check Java d=
ocs of when that method was introduced=C2=A0</div><div dir=3D"auto">2. U bu=
ilding a fat jar? Check which libraries have been included....some other de=
pendencies might have forced an old copy to be included....</div><div dir=
=3D"auto">3. If u. Take code outside spark.....does it work successfully?</=
div><div dir=3D"auto">4. Send short sample....</div><div dir=3D"auto">Hth</=
div></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra=
"><br><div class=3D"gmail_quote">On 12 Dec 2016 11:03 am, &quot;Pratyaksh S=
harma&quot; &lt;<a href=3D"mailto:pratyaksh.sharma.ece12@itbhu.ac.in" targe=
t=3D"_blank">pratyaksh.sharma.ece12@itbhu.<wbr>ac.in</a>&gt; wrote:<br type=
=3D"attribution"><blockquote class=3D"m_4966490763746048766quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"=
ltr"><div style=3D"font-size:13px">Hey I am using Apache Spark for one stre=
aming application. I am trying to store the processed data into dynamodb us=
ing java sdk. Getting the following exception -=C2=A0</div><div style=3D"fo=
nt-size:13px">16/12/08 23:23:43 WARN TaskSetManager: Lost task 0.0 in stage=
 1.0:=C2=A0</div><div style=3D"font-size:13px">java.lang.NoSuchMethodError:=
 com.amazonaws.SDKGlobalConfigu<wbr>ration.isInRegionOptimizedMode<wbr>Enab=
led()Z</div><div style=3D"font-size:13px"><span class=3D"m_4966490763746048=
766m_-1806208422378907242gmail-m_6155966059112232388m_-7858112119017174825g=
mail-Apple-tab-span" style=3D"white-space:pre-wrap">	</span>at com.amazonaw=
s.ClientConfigurat<wbr>ionFactory.getConfig(ClientCon<wbr>figurationFactory=
.java:35)</div><div style=3D"font-size:13px"><span class=3D"m_4966490763746=
048766m_-1806208422378907242gmail-m_6155966059112232388m_-78581121190171748=
25gmail-Apple-tab-span" style=3D"white-space:pre-wrap">	</span>at com.amazo=
naws.services.dynamod<wbr>bv2.AmazonDynamoDBClient.&lt;init<wbr>&gt;(Amazon=
DynamoDBClient.java:37<wbr>4)</div><div style=3D"font-size:13px"><br></div>=
<div style=3D"font-size:13px">Spark version - 1.6.1</div><div style=3D"font=
-size:13px">Scala version - 2.10.5</div><div style=3D"font-size:13px">aws s=
dk version - 1.11.33</div><div style=3D"font-size:13px">=C2=A0</div><div st=
yle=3D"font-size:13px">Has anyone faced this issues? Any help will be highl=
y appreciated.</div><font color=3D"#888888"><div><br></div>-- <br><div clas=
s=3D"m_4966490763746048766m_-1806208422378907242gmail_signature"><div dir=
=3D"ltr">Regards<div><br></div><div>Pratyaksh Sharma</div><div>12105EN013</=
div><div>Department of Electronics Engineering</div><div>IIT Varanasi</div>=
<div>Contact No <a href=3D"tel:+91%2081270%2030223" value=3D"+918127030223"=
 target=3D"_blank">+91-8127030223</a></div></div></div>
</font></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--94eb2c033d78cdea0d05437b070d--