tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Senia (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (TEZ-3105) Tez does not run on IBM JDK 7 or 8
Date Sat, 27 Feb 2016 07:03:18 GMT

    [ https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170402#comment-15170402
] 

Greg Senia edited comment on TEZ-3105 at 2/27/16 7:02 AM:
----------------------------------------------------------

[~gopalv] or [~hitesh] I attempted the above and Tez no longer works correctly when testing
with the above changes... I wish there was a better answer.. Looking at what the Flink folks
did they did the following:

https://github.com/apache/flink/blob/6b01a89020f2de3f7710cf72336291b1e8ca8562/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java

	public static long getMaxJvmHeapMemory() {
		long maxMemory = Runtime.getRuntime().maxMemory();

		if (maxMemory == Long.MAX_VALUE) {
			// amount of free memory unknown
			try {
				// workaround for Oracle JDK
				OperatingSystemMXBean operatingSystemMXBean = ManagementFactory.getOperatingSystemMXBean();
				Class<?> clazz = Class.forName("com.sun.management.OperatingSystemMXBean");
				Method method = clazz.getMethod("getTotalPhysicalMemorySize");
				maxMemory = (Long) method.invoke(operatingSystemMXBean) / 4;
			}
			catch (Throwable e) {
				throw new RuntimeException("Could not determine the amount of free memory.\n" +
						"Please set the maximum memory for the JVM, e.g. -Xmx512M for 512 megabytes.");
			}
		}
		
		return maxMemory;
	}


Please advise if this patch is acceptable: https://issues.apache.org/jira/secure/attachment/12787320/TEZ-3105-2.patch

And if not what can be changed.


was (Author: gss2002):
[~gopalv] or [~hitesh] I attempted the above and Tez no longer works correctly when testing
with the above changes... I wish there was a better answer.. Looking at what the Flink folks
did they did the following:

https://github.com/apache/flink/blob/6b01a89020f2de3f7710cf72336291b1e8ca8562/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java

	public static long getMaxJvmHeapMemory() {
		long maxMemory = Runtime.getRuntime().maxMemory();

		if (maxMemory == Long.MAX_VALUE) {
			// amount of free memory unknown
			try {
				// workaround for Oracle JDK
				OperatingSystemMXBean operatingSystemMXBean = ManagementFactory.getOperatingSystemMXBean();
				Class<?> clazz = Class.forName("com.sun.management.OperatingSystemMXBean");
				Method method = clazz.getMethod("getTotalPhysicalMemorySize");
				maxMemory = (Long) method.invoke(operatingSystemMXBean) / 4;
			}
			catch (Throwable e) {
				throw new RuntimeException("Could not determine the amount of free memory.\n" +
						"Please set the maximum memory for the JVM, e.g. -Xmx512M for 512 megabytes.");
			}
		}
		
		return maxMemory;
	}

> Tez does not run on IBM JDK 7 or 8
> ----------------------------------
>
>                 Key: TEZ-3105
>                 URL: https://issues.apache.org/jira/browse/TEZ-3105
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Greg Senia
>            Assignee: Greg Senia
>              Labels: ibm, ibm-jdk
>         Attachments: TEZ-3105-2.patch
>
>
> When testing Hive on Tez with IBM JDK 7 and 8. The following issue was discovered:
> 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> 	at org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225)
> 	at org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347)
> 	at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371)
> 	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> 	at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274)
> 	at java.security.AccessController.doPrivileged(AccessController.java:686)
> 	at javax.security.auth.Subject.doAs(Subject.java:569)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271)
> 	at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:436)
> 	at org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221)
> 	... 9 more
> Caused by: java.lang.ClassCastException: com.ibm.lang.management.ExtendedOperatingSystem
incompatible with com.sun.management.OperatingSystemMXBean
> 	at org.apache.tez.util.TezMxBeanResourceCalculator.<init>(TezMxBeanResourceCalculator.java:44)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message