Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EB6C5D780 for ; Wed, 23 Jan 2013 08:00:36 +0000 (UTC) Received: (qmail 27613 invoked by uid 500); 23 Jan 2013 08:00:35 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 27312 invoked by uid 500); 23 Jan 2013 08:00:35 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 27297 invoked by uid 99); 23 Jan 2013 08:00:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 08:00:34 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nitinpawar432@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vb0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2013 08:00:25 +0000 Received: by mail-vb0-f48.google.com with SMTP id fc21so2439383vbb.7 for ; Wed, 23 Jan 2013 00:00:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=gUFAOltZH7ULA0a0LmnpqV6IlvcM4bIXhz8WMXemqyU=; b=td63XAKKFmYvpSB/Dp12VfKT15IysTNdnNW+lET7/ZduSjLhS6E9LYuk04RmGErVui SrPGeLwfYBNT0z6y9u4q+E8Ix9KDRlVAvQUoL/omm/y+qi1QZSLZU1YYYKtJqXAhmuhu KfKqRJPmFHI4krBE1pg/g829X3oqxLcDcTh+CGixb7sov6QsMIGygkYJn9whuREmpRhH Lq9IPnf0exMFuzXpHVtY/aOvG2wz72EK5E2RsjYmK0WxLN0cvNrwcOSudoniMmb20Cqk s0mS992EmjkFTfO/64S4skZXV6UzYViZUP+fofWo7xy9rxyn4NgFuXjMeAN/Q/wGsf52 WBPg== MIME-Version: 1.0 X-Received: by 10.52.97.104 with SMTP id dz8mr364411vdb.21.1358928005069; Wed, 23 Jan 2013 00:00:05 -0800 (PST) Received: by 10.59.9.67 with HTTP; Wed, 23 Jan 2013 00:00:04 -0800 (PST) In-Reply-To: <50FF9842.7040208@smartek21.com> References: <50F7E61F.5060306@smartek21.com> <50FF90E9.3080105@smartek21.com> <50FF9842.7040208@smartek21.com> Date: Wed, 23 Jan 2013 13:30:04 +0530 Message-ID: Subject: Re: Configure Hive in Cluster From: Nitin Pawar To: user@hive.apache.org Content-Type: multipart/alternative; boundary=20cf307f35f4047ccf04d3f015c1 X-Virus-Checked: Checked by ClamAV on apache.org --20cf307f35f4047ccf04d3f015c1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable this is the error on hadoop job 2013-01-23 12:15:44,884 INFO org.apache.hadoop.mapred.ReduceTask: Failed to fetch map-output from attempt_201301231151_0002_m_000001_0 even after MAX_FETCH_RETRIES_PER_MAP retries... or it is a read error, reporting to the JobTracker 2013-01-23 12:15:44,885 FATAL org.apache.hadoop.mapred.ReduceTask: Shuffle failed with too many fetch failures and insufficient progress!Killing task attempt_201301231151_0002_r_000000_0. 2013-01-23 12:15:45,220 FATAL org.apache.hadoop.mapred.Task: Failed to contact the tasktracker org.apache.hadoop.ipc.RemoteException: java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201301231151_0002_r_000000_0, with JvmId: jvm_201301231151_0002_r_1079250852 so something is a mess either your network went down or nodes went down hive tries to get the same task log from the host (savitha-vitualbox) and it can't figure out what that host is. On Wed, Jan 23, 2013 at 1:28 PM, venkatramanan wrote: > No, all the nodes are up and running. i dont know, when hive takes the > other nodes "HOST NAME" thats the error i guess.. > > revert me if am wrong > > > On Wednesday 23 January 2013 01:07 PM, Nitin Pawar wrote: > > when you ran the query, did the VM shutdown ? > > > On Wed, Jan 23, 2013 at 12:57 PM, venkatramanan < > venkatramanann@smartek21.com> wrote: > >> Hi, >> >> I got the following error while executing the "select count(1) from >> tweettrend;" >> >> Below are the exact log msg from the jobtracker Web Interface >> >> *Hive Cli Error:* >> >> Exception in thread "Thread-21" java.lang.RuntimeException: Error while >> reading from task log url >> at >> org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(Ta= skLogProcessor.java:240) >> at >> org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebug= ger.java:227) >> at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:9= 2) >> at java.lang.Thread.run(Thread.java:722) >> Caused by: java.net.UnknownHostException: savitha-VirtualBox >> at >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:17= 8) >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) >> at java.net.Socket.connect(Socket.java:579) >> at java.net.Socket.connect(Socket.java:528) >> at sun.net.NetworkClient.doConnect(NetworkClient.java:180) >> at sun.net.www.http.HttpClient.openServer(HttpClient.java:378) >> at sun.net.www.http.HttpClient.openServer(HttpClient.java:473) >> at sun.net.www.http.HttpClient.(HttpClient.java:203) >> at sun.net.www.http.HttpClient.New(HttpClient.java:290) >> at sun.net.www.http.HttpClient.New(HttpClient.java:306) >> at >> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConn= ection.java:995) >> at >> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnecti= on.java:931) >> at >> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.ja= va:849) >> at >> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnec= tion.java:1299) >> at java.net.URL.openStream(URL.java:1037) >> at >> org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(Ta= skLogProcessor.java:192) >> ... 3 more >> FAILED: Execution Error, return code 2 from >> org.apache.hadoop.hive.ql.exec.MapRedTask >> MapReduce Jobs Launched: >> Job 0: Map: 2 Reduce: 1 Cumulative CPU: 9.0 sec HDFS Read: 40867105= 3 >> HDFS Write: 0 FAIL >> Total MapReduce CPU Time Spent: 9 seconds 0 msec >> >> *syslog logs* >> >> utCopier.copyOutput(ReduceTask.java:1394) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run= (ReduceTask.java:1326) >> >> 2013-01-23 12:15:44,884 INFO org.apache.hadoop.mapred.ReduceTask: Task a= ttempt_201301231151_0002_r_000000_0: Failed fetch #10 from attempt_20130123= 1151_0002_m_000001_0 >> 2013-01-23 12:15:44,884 INFO org.apache.hadoop.mapred.ReduceTask: Failed= to fetch map-output from attempt_201301231151_0002_m_000001_0 even after M= AX_FETCH_RETRIES_PER_MAP retries... or it is a read error, reporting to t= he JobTracker >> 2013-01-23 12:15:44,885 FATAL org.apache.hadoop.mapred.ReduceTask: Shuff= le failed with too many fetch failures and insufficient progress!Killing ta= sk attempt_201301231151_0002_r_000000_0. >> 2013-01-23 12:15:44,889 WARN org.apache.hadoop.mapred.ReduceTask: attemp= t_201301231151_0002_r_000000_0 adding host savitha-VirtualBox to penalty bo= x, next contact in 137 seconds >> 2013-01-23 12:15:44,889 INFO org.apache.hadoop.mapred.ReduceTask: attemp= t_201301231151_0002_r_000000_0: Got 1 map-outputs from previous failures >> 2013-01-23 12:15:45,218 FATAL org.apache.hadoop.mapred.Task: attempt_201= 301231151_0002_r_000000_0 GetMapEventsThread Ignoring exception : org.apach= e.hadoop.ipc.RemoteException: java.io.IOException: JvmValidate Failed. Igno= ring request from task: attempt_201301231151_0002_r_000000_0, with JvmId: j= vm_201301231151_0002_r_1079250852 >> at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:32= 78) >> at org.apache.hadoop.mapred.TaskTracker.getMapCompletionEvents(TaskTrac= ker.java:3537) >> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:601) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1121) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1070) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) >> at $Proxy1.getMapCompletionEvents(Unknown Source) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.= getMapCompletionEvents(ReduceTask.java:2846) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.= run(ReduceTask.java:2810) >> >> 2013-01-23 12:15:45,220 FATAL org.apache.hadoop.mapred.Task: Failed to c= ontact the tasktracker >> org.apache.hadoop.ipc.RemoteException: java.io.IOException: JvmValidate = Failed. Ignoring request from task: attempt_201301231151_0002_r_000000_0, w= ith JvmId: jvm_201301231151_0002_r_1079250852 >> at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:32= 78) >> at org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:352= 0) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:601) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1121) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1070) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) >> at $Proxy1.fatalError(Unknown Source) >> at org.apache.hadoop.mapred.Task.reportFatalError(Task.java:298) >> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.= run(ReduceTask.java:2829) >> >> thanks, >> Venkat >> >> -------- Original Message -------- Subject: Re: Configure Hive in >> Cluster Date: Thu, 17 Jan 2013 17:23:03 +0530 From: venkatramanan >> Reply-To: >> To: >> >> >> Can you suggest me the mandatory hive parameters and clustering >> configuration steps >> >> On Thursday 17 January 2013 12:56 PM, Nitin Pawar wrote: >> >> looks like a very small cluster with very limited memory to run mapreduc= e >> jobs also number of map/reduce slots on nodes are less so at a time only >> one map is running. >> >> but still 15 min is a lot of time for 600MB memory >> >> >> On Thu, Jan 17, 2013 at 12:47 PM, venkatramanan < >> venkatramanann@smartek21.com> wrote: >> >>> Below details are the cluster configuration >>> >>> Configured Capacity : 82.8 GB >>> DFS Used : 1.16 GB >>> Non DFS Used : 31.95 GB >>> DFS Remaining : 49.69 GB >>> DFS Used% : 1.4 % >>> DFS Remaining% : 60.01 % >>> Live Nodes >>> : 2 >>> Dead Nodes >>> : 0 >>> Decommissioning Nodes: 0 >>> Number of Under-Replicated Blocks : 0 >>> >>> My Select Query is: >>> >>> "select * from tweet where Id =3D 810;" >>> >>> This query takes 15 min to complete >>> >>> >>> >>> On Thursday 17 January 2013 12:29 PM, Nitin Pawar wrote: >>> >>> how many number of nodes you have for select query? >>> whats your select query? >>> >>> if its just a select * from table then it does not run any mapreduce >>> job >>> so its just taking time to show data on your screen if you are using >>> that query >>> >>> >>> On Thu, Jan 17, 2013 at 12:24 PM, venkatramanan < >>> venkatramanann@smartek21.com> wrote: >>> >>>> I didnt set any hive parameters and my total table size is 610 MB onl= y >>>> >>>> >>>> >>>> On Thursday 17 January 2013 12:11 PM, Nitin Pawar wrote: >>>> >>>> a bit more details on size of table and select query will help >>>> also did you set any hive parameters ? >>>> >>>> >>>> On Thu, Jan 17, 2013 at 12:12 PM, venkatramanan < >>>> venkatramanann@smartek21.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> Am Newbie in apache hive. I have create a table and thats points to >>>>> the HDFS Folder path and its takes 15 min to execute the simple "* >>>>> select*" stmt, Can anyone suggest me for a best practices and >>>>> performance improvement on hive. >>>>> >>>>> Thanks in Advance >>>>> >>>>> Venkat >>>>> >>>> >>>> >>>> >>>> -- >>>> Nitin Pawar >>>> >>>> >>>> >>> >>> >>> -- >>> Nitin Pawar >>> >>> >>> >> >> >> -- >> Nitin Pawar >> >> >> >> >> > > > -- > Nitin Pawar > > > > -- > > --=20 Nitin Pawar --20cf307f35f4047ccf04d3f015c1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
this is the error on hadoop job=A0
2013-01-23 12:15:44,884 INFO org.apache.hadoop.mapred.Redu=
ceTask: Failed to fetch map-output from attempt_201301231151_0002_m_000001_=
0 even after MAX_FETCH_RETRIES_PER_MAP retries...  or it is a read error,  =
reporting to the JobTracker
2013-01-23 12:15:44,885 FATAL org.apache.hadoop.mapred.ReduceTask: Shuffle =
failed with too many fetch failures and insufficient progress!Killing task =
attempt_201301231151_0002_r_000000_0.
2013-01-23 12:15:45,220 FATAL org.apache.hadoop.mapred.Task: Failed to cont=
act the tasktracker
org.apache.hadoop.ipc.RemoteException: java.io.IOException: JvmValidate Fai=
led. Ignoring request from task: attempt_201301231151_0002_r_000000_0, with=
 JvmId: jvm_201301231151_0002_r_1079250852

so something is a mess either=
 your network went down or nodes went down 
hive tries to get the same task log from the host (savitha-vitual=
box) and it can't figure out what that host is.




On Wed, Jan 23, 2013 at 1:28 PM, venkatramanan <venkatramanann@smartek21.com> wrote:
=20 =20 =20
No, all the nodes are up and running. i dont know, when hive takes the other nodes "HOST NAME" that= s the error i guess..

revert me if am wrong


On Wednesday 23 January 2013 01:07 PM, Nitin Pawar wrote:
=20
when you ran the query, did the VM shutdown ?=A0


On Wed, Jan 23, 2013 at 12:57 PM, venkatramanan <venkatramanann@smartek21.com> wrote:
Hi,

I got the following error while executing the "select count(1) from tweettrend;"

Below are the exact log msg from the jobtracker Web Interface

Hive Cli Error:

Exception in thread "Thread-21" java.lang.RuntimeException: Error while reading from task log url
=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskL= ogProcessor.java:240)
=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger= .java:227)
=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugge= r.java:92)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.UnknownHostException: savitha-VirtualBox
=A0=A0=A0 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)<= br> =A0=A0=A0 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391= )
=A0=A0=A0 at java.net.Socket.connect(Socket.java:579)
=A0=A0=A0 at java.net.Socket.connect(Socket.java:528)
=A0=A0=A0 at sun.net.NetworkClient.doConnect(NetworkClient.java:180) =A0=A0=A0 at sun.net.www.http.HttpClient.openServer(HttpClient.java:37= 8)
=A0=A0=A0 at sun.net.www.http.HttpClient.openServer(HttpClient.java:47= 3)
=A0=A0=A0 at sun.net.www.http.HttpClient.<init>(HttpClient.java:= 203)
=A0=A0=A0 at sun.net.www.http.HttpClient.New(HttpClient.java:290)
=A0=A0=A0 at sun.net.www.http.HttpClient.New(HttpClient.java:306)
=A0=A0=A0 at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnect= ion.java:995)
=A0=A0=A0 at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.= java:931)
=A0=A0=A0 at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:= 849)
=A0=A0=A0 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnectio= n.java:1299)
=A0=A0=A0 at java.net.URL.openStream(URL.java:1037)
=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskL= ogProcessor.java:192)
=A0=A0=A0 ... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 2=A0 Reduce: 1=A0=A0 Cumulative CPU: 9.0 sec= =A0=A0 HDFS Read: 408671053 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 9 seconds 0 msec

syslog logs
utCopier.copyOutput(Reduce=
Task.java:1394)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(Re=
duceTask.java:1326)

2013-01-23 12:15:44,884 INFO org.apache.hadoop.mapred.ReduceTask: Task atte=
mpt_201301231151_0002_r_000000_0: Failed fetch #10 from attempt_20130123115=
1_0002_m_000001_0
2013-01-23 12:15:44,884 INFO org.apache.hadoop.mapred.ReduceTask: Failed to=
 fetch map-output from attempt_201301231151_0002_m_000001_0 even after MAX_=
FETCH_RETRIES_PER_MAP retries...  or it is a read error,  reporting to the =
JobTracker
2013-01-23 12:15:44,885 FATAL org.apache.hadoop.mapred.ReduceTask: Shuffle =
failed with too many fetch failures and insufficient progress!Killing task =
attempt_201301231151_0002_r_000000_0.
2013-01-23 12:15:44,889 WARN org.apache.hadoop.mapred.ReduceTask: attempt_2=
01301231151_0002_r_000000_0 adding host savitha-VirtualBox to penalty box, =
next contact in 137 seconds
2013-01-23 12:15:44,889 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01301231151_0002_r_000000_0: Got 1 map-outputs from previous failures
2013-01-23 12:15:45,218 FATAL org.apache.hadoop.mapred.Task: attempt_201301=
231151_0002_r_000000_0 GetMapEventsThread Ignoring exception : org.apache.h=
adoop.ipc.RemoteException: java.io.IOException: JvmValidate Failed. Ignorin=
g request from task: attempt_201301231151_0002_r_000000_0, with JvmId: jvm_=
201301231151_0002_r_1079250852
	at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3278)
	at org.apache.hadoop.mapred.TaskTracker.getMapCompletionEvents(TaskTracker=
.java:3537)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati=
on.java:1121)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

	at org.apache.hadoop.ipc.Client.call(Client.java:1070)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
	at $Proxy1.getMapCompletionEvents(Unknown Source)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.get=
MapCompletionEvents(ReduceTask.java:2846)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run=
(ReduceTask.java:2810)

2013-01-23 12:15:45,220 FATAL org.apache.hadoop.mapred.Task: Failed to cont=
act the tasktracker
org.apache.hadoop.ipc.RemoteException: java.io.IOException: JvmValidate Fai=
led. Ignoring request from task: attempt_201301231151_0002_r_000000_0, with=
 JvmId: jvm_201301231151_0002_r_1079250852
	at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3278)
	at org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:3520)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja=
va:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso=
rImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati=
on.java:1121)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

	at org.apache.hadoop.ipc.Client.call(Client.java:1070)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
	at $Proxy1.fatalError(Unknown Source)
	at org.apache.hadoop.mapred.Task.reportFatalError(Task.java:298)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run=
(ReduceTask.java:2829)
thanks,
Venkat

-------- Original Message --------
Subjec= t: Re: Configure Hive in Cluster
Date: = Thu, 17 Jan 2013 17:23:03 +0530
From: = venkatramanan <venkatramanann@smartek21.com>
Reply-= To: <user@hive.apache.org>
To: <user@hive.apache.org>


Can you suggest me the mandatory hive parameters and clustering configuration steps
=A0
On Thursday 17 January 2013 12:56 PM, Nitin Pawar wrote:
looks like a very small cluster with very limited memory to run mapreduce jobs also number of map/reduce slots on nodes are less so at a time only one map is running.=A0

but still 15 min is a lot of time for 600MB memory=A0


On Thu, Jan 17, 2013 at 12:47 PM, venkatramanan <venkatramana= nn@smartek21.com> wrote:
Below details are the cluster configuration

Configured Capacity =A0=A0=A0 =A0=A0=A0 : 8= 2.8 GB
DFS Used =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 =A0=A0=A0 =A0=A0=A0=A0 : 1.16 GB
Non DFS Used =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 = =A0=A0=A0=A0 : 31.95 GB
DFS Remaining =A0=A0=A0 =A0=A0=A0 =A0=A0=A0= =A0=A0 : 49.69 GB
DFS Used% =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 =A0=A0=A0=A0 : 1.4 %
DFS Remaining% =A0=A0=A0 =A0=A0=A0 =A0=A0= =A0=A0 : 60.01 %
Live Nodes =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0 : 2
Dead Nodes =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 : 0
Decommissioning Nodes : 0
Number of Under-Replicated Blocks : 0

My Select Query is:

"select * from tweet where Id =3D 810;= "

This query takes 15 min to complete



On Thursday 17 January 2013 12:29 PM, Nitin Pawar wrote:
how many number of nodes you have for select query?=A0
whats your select query?=A0

if its just a select * from table then it does not run any mapreduce job
so its just taking time to show data on your screen if you are using that query=A0


On Thu, Jan 17, 2013 at 12:24 PM, venkatramanan <= ;venkatra= manann@smartek21.com> wrote:
I didnt set any hive parameters and my total table size is 610 MB only



On Thursday 17 January 2013 12:11 PM, Nitin Pawar wrote:
a bit more details on size of table and select query will help=A0
also did you set any hive parameters ?=A0


On Thu, Jan 17, 2013 at 12:12 PM, venkatramanan <venkatramanann@smartek21.com> wrote:
Hi All,
Am Newbie in apache hive. I have create a table and thats points to the HDFS Folder path and its takes 15 min to execute the simple "s= elect" stmt, Can anyone suggest me for a best practices and performance improvement on hive.

Thanks in Advance

Venkat



--
Nitin Pawar




--
Nitin Pawar




--
Nitin Pawar






--
Nitin Pawar


--




--
Nitin Pawar<= br>
--20cf307f35f4047ccf04d3f015c1--