spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhan Zhang <zzh...@hortonworks.com>
Subject Re: OOM for HiveFromSpark example
Date Wed, 25 Mar 2015 18:06:26 GMT
I solve this by  increase the PermGen memory size in driver.

-XX:MaxPermSize=512m

Thanks.

Zhan Zhang

On Mar 25, 2015, at 10:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepujain@gmail.com<mailto:deepujain@gmail.com>>
wrote:

I am facing same issue, posted a new thread. Please respond.

On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzhang@hortonworks.com<mailto:zzhang@hortonworks.com>>
wrote:
Hi Folks,

I am trying to run hive context in yarn-cluster mode, but met some error. Does anybody know
what cause the issue.

I use following cmd to build the distribution:

 ./make-distribution.sh -Phive -Phive-thriftserver  -Pyarn  -Phadoop-2.4

15/01/13 17:59:42 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block manager cn122-10.l42scl.hortonworks.com:56157<http://cn122-10.l42scl.hortonworks.com:56157/>
with 1589.8 MB RAM, BlockManagerId(2, cn122-10.l42scl.hortonworks.com<http://cn122-10.l42scl.hortonworks.com/>,
56157)
15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src
(key INT, value STRING)
15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed
15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize called
15/01/13 17:59:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown
- will be ignored
15/01/13 17:59:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown
unknown - will be ignored
15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH
(or one of dependencies)
15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH
(or one of dependencies)
15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not
on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
15/01/13 17:59:53 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
15/01/13 17:59:53 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
15/01/13 17:59:59 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
15/01/13 17:59:59 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore
15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not found in metastore.
hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in metastore
15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in metastore
15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin role, since config
is empty
15/01/13 18:00:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src
(key INT, value STRING)
15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed
15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse start=1421190003030 end=1421190003031
duration=1 from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src position=27
15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=src
15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang  ip=unknown-ip-addr      cmd=get_table
: db=default tbl=src
15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default
15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang  ip=unknown-ip-addr      cmd=get_database:
default
15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed
15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=semanticAnalyze start=1421190003031
end=1421190003406 duration=375 from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile start=1421190002998 end=1421190003416
duration=418 from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF NOT EXISTS src (key INT,
value STRING)
15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1421190002995
end=1421190003421 duration=426 from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for table src
15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1421190003416
end=1421190005498 duration=2082 from=org.apache.hadoop.hive.ql.Driver>
Exception in thread "Driver"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Driver"
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>




--
Deepak


Mime
View raw message