Return-Path: X-Original-To: apmail-incubator-accumulo-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-accumulo-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 052339C06 for ; Sun, 11 Mar 2012 16:46:59 +0000 (UTC) Received: (qmail 84783 invoked by uid 500); 11 Mar 2012 16:46:59 -0000 Delivered-To: apmail-incubator-accumulo-user-archive@incubator.apache.org Received: (qmail 84754 invoked by uid 500); 11 Mar 2012 16:46:59 -0000 Mailing-List: contact accumulo-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: accumulo-user@incubator.apache.org Delivered-To: mailing list accumulo-user@incubator.apache.org Received: (qmail 84745 invoked by uid 99); 11 Mar 2012 16:46:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2012 16:46:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric.newton@gmail.com designates 209.85.210.175 as permitted sender) Received: from [209.85.210.175] (HELO mail-iy0-f175.google.com) (209.85.210.175) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2012 16:46:55 +0000 Received: by iaag37 with SMTP id g37so5130076iaa.6 for ; Sun, 11 Mar 2012 09:46:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=fS28nwJbO+LgkhE7Vl3qYAzqdbDdlgiaYfK6jaG+z5A=; b=ybvL/Hx304rps5lpPhA8Uo6TV4qPNnIjXR/OEBaSXE/zcXiAbp/OOdctlrYhAG+tch V6RzrJ7dBgg7VO+eADJ3L+61VE7dq7E+8Zb2mDs5OlzMCr5PlodHJ+SbUfJ2ZhWXv/mr +3LCrL6+N6mV8ytp8UZHVSvXgDX6OHWMVauBoeriIAjS+0hej1hU3Y32GZdTv8nbiky8 TIEeNnMW0H+W8ue2sgwJIkJnUbps4IkCguYu6RHXfXT621GUZpvvtyhr0QUJGQycEu8O vSyOBOUGGNNsQokCtyp8RRbJOe37oeHjWFD2yRPLUog8Uat+iOiVIVWr+dpNEj0UtIGc cMjQ== MIME-Version: 1.0 Received: by 10.50.188.167 with SMTP id gb7mr14742949igc.37.1331484394855; Sun, 11 Mar 2012 09:46:34 -0700 (PDT) Received: by 10.50.181.202 with HTTP; Sun, 11 Mar 2012 09:46:34 -0700 (PDT) In-Reply-To: References: <4F5C2462.1070106@gmail.com> Date: Sun, 11 Mar 2012 12:46:34 -0400 Message-ID: Subject: Re: object heap error From: Eric Newton To: accumulo-user@incubator.apache.org Content-Type: multipart/alternative; boundary=14dae9340b43611b3404bafa5e54 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9340b43611b3404bafa5e54 Content-Type: text/plain; charset=ISO-8859-1 If you type this: $ ./bin/accumulo classpath You should see a zookeeper jar in the list. See if you can figure out why it is being missed. -Eric On Sun, Mar 11, 2012 at 10:44 AM, Donald Resnik wrote: > Joey, > > Thanks for the email. I had HADOOP_HOME and ZOOKEEPER_HOME in my env, but > I added them to acumulo-env.sh as well. Now I am not getting the hadoop > class error, but I am still getting the zookeeper error and a new tserver > error. zookeeper.jar is in /usr/lib/zookeeper > > > ==> logger_localhost.err <== > Uncaught exception: org/apache/zookeeper/Watcher > java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) > at java.lang.ClassLoader.defineClass(ClassLoader.java:616) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:296) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.accumulo.start.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:348) > at > org.apache.accumulo.start.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:355) > at org.apache.accumulo.start.Main.main(Main.java:56) > Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > ... 17 more > > ==> tserver_localhost.err <== > java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) > at java.lang.ClassLoader.defineClass(ClassLoader.java:616) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.getDeclaredMethods0(Native Method) > at java.lang.Class.privateGetDeclaredMethods(Class.java:2427) > at java.lang.Class.getMethod0(Class.java:2670) > at java.lang.Class.getMethod(Class.java:1603) > at org.apache.accumulo.start.Main.main(Main.java:76) > > Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > ... 16 more > > ==> tserver_localhost.out <== > tserver must implement a public static void main(String args[]) method > > > > > On Sun, Mar 11, 2012 at 7:31 AM, Joey Echeverria wrote: > >> Did you set the ZOOKEEPER_HOME and HADOOP_HOME environment variables to >> point to /usr/lib/zookeeper and /usr/lib/hadoop respectively? >> >> -Joey >> >> Sent from my iPhone >> >> On Mar 11, 2012, at 8:10, Donald Resnik wrote: >> >> Josh, >> >> Thanks for the email. I raised the VM ram to 1G and lowered the heap >> sizes from David Medinets levels from 256 to 128 and 128 to 64. I no >> longer get the memory error, but now I am getting this: >> >> ==> logger_localhost.err <== >> Uncaught exception: org/apache/zookeeper/Watcher >> java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:616) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) >> at java.net.URLClassLoader.access$000(URLClassLoader.java:58) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:197) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:296) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:247) >> at >> org.apache.accumulo.start.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:348) >> at >> org.apache.accumulo.start.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:355) >> at org.apache.accumulo.start.Main.main(Main.java:56) >> Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> ... 17 more >> >> ==> tserver_localhost.err <== >> Uncaught exception: org/apache/hadoop/io/BinaryComparable >> java.lang.NoClassDefFoundError: org/apache/hadoop/io/BinaryComparable >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:247) >> at >> org.apache.accumulo.start.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:348) >> at >> org.apache.accumulo.start.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:355) >> at org.apache.accumulo.start.Main.main(Main.java:44) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.io.BinaryComparable >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> ... 5 more >> >> It is also haning on starting the loggers: >> >> cloudera@cloudera-vm:~/accumulo-1.3.5-incubating/conf$ >> ../bin/start-all.sh >> Starting tablet servers and loggers .... done >> Starting tablet server on localhost >> Starting logger on localhost >> >> >> Any help would be appreciated. >> >> Thanks, >> >> Don >> >> >> On Sat, Mar 10, 2012 at 8:04 PM, Josh Elser wrote: >> >>> Don, >>> >>> What are the values you tried for Xmx for the tserver process? How much >>> memory did you allocate to your vm? Is the sum of all of the Accumulo >>> processes exceeding the available memory in the vm? >>> >>> - Josh >>> >>> >>> On 3/10/2012 10:42 PM, Donald Resnik wrote: >>> >>>> Accumulo, >>>> >>>> I am having trouble staring accumulo in a cloudera VM. I am getting >>>> this error in the tserver log: >>>> >>>> Could not reserve enough space for object heap. >>>> >>>> I was using the steps from David Medinets post about configuring >>>> accumulo in a vm from 30 Dec 2011. I tried to vary some of the >>>> accumulo-env settings for Xmx and Xms, but kept getting the error. >>>> >>>> Any help would be appreciated. >>>> >>>> Thanks, >>>> >>>> -- >>>> Don Resnik >>>> Expert Consultants Inc. >>>> 410-854-1756 (desk) >>>> 240-381-9412 (cell) >>>> 301-957-2270 (ECI office) >>>> >>>> >> >> >> -- >> Don Resnik >> Expert Consultants Inc. >> 410-854-1756 (desk) >> 240-381-9412 (cell) >> 301-957-2270 (ECI office) >> >> > > > -- > Don Resnik > Expert Consultants Inc. > 410-854-1756 (desk) > 240-381-9412 (cell) > 301-957-2270 (ECI office) > > --14dae9340b43611b3404bafa5e54 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable If you type this:

=A0$ ./bin/accumulo classpath

You should see a zookeeper jar in the list. =A0See if you = can figure out why it is being missed.=A0

-Eric
On Sun, Mar 11, 2012 at 10:44 AM, Donald Resnik = <dresnik@eci-us= a.net> wrote:
Joey,

Thanks for the email.=A0 I had HADOOP_HOME and ZOOKEEPER_HOME = in my env, but I added them to acumulo-env.sh as well.=A0 Now I am not gett= ing the hadoop class error, but I am still getting the zookeeper error and = a new tserver error.=A0 zookeeper.jar is in /usr/lib/zookeeper


=3D=3D> logger_localhost.err <=3D=3D
Uncaught exception: org/a= pache/zookeeper/Watcher
java.lang.NoClassDefFoundError: org/apache/zooke= eper/Watcher
=A0=A0=A0 at java.lang.ClassLoader.defineClass1(Native Meth= od)
=A0=A0=A0 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:= 632)
=A0=A0=A0 at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
=A0= =A0=A0 at java.security.SecureClassLoader.defineClass(SecureClassLoader.jav= a:141)
=A0=A0=A0 at java.net.URLClassLoader.defineClass(URLClassLoader.j= ava:283)
=A0=A0=A0 at java.net.URLClassLoader.access$000(URLClassLoader.= java:58)
=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
=A0= =A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0= =A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
=A0=A0= =A0 at java.lang.Class.forName0(Native Method)
=A0=A0=A0 at java.lang.Cl= ass.forName(Class.java:247)
=A0=A0=A0 at org.apache.accumulo.start.classloader.AccumuloClassLoader.load= Class(AccumuloClassLoader.java:348)
=A0=A0=A0 at org.apache.accumulo.sta= rt.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:355)<= br>=A0=A0=A0 at org.apache.accumulo.start.Main.main(Main.java:56)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
= =A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
= =A0=A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
=A0=A0= =A0 ... 17 more

=3D=3D> tserver_localhost.err <=3D=3D
java.= lang.NoClassDefFoundError: org/apache/zookeeper/Watcher
=A0=A0=A0 at java.lang.ClassLoader.defineClass1(Native Method)
=A0=A0=A0= at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
=A0=A0= =A0 at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
=A0=A0=A0= at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)=
=A0=A0=A0 at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)=A0=A0=A0 at java.net.URLClassLoader.access$000(URLClassLoader.java:58)=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
= =A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0=A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
= =A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)=A0=A0=A0 at java.lang.Class.getDeclaredMethods0(Native Method)
=A0=A0=A0 at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
= =A0=A0=A0 at java.lang.Class.getMethod0(Class.java:2670)
=A0=A0=A0 at ja= va.lang.Class.getMethod(Class.java:1603)
=A0=A0=A0 at org.apache.accumul= o.start.Main.main(Main.java:76)

Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
= =A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
= =A0=A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
= =A0=A0=A0 ... 16 more

=3D=3D> tserver_localhost.out <=3D=3Dtserver must implement a public static void main(String args[]) method




On Sun, Mar 11, 2012 at 7:31 AM, Joe= y Echeverria <joey@cloudera.com> wrote:
Did you set the ZOOKEEPER_HOME and HADOOP_HOM= E environment variables to point to /usr/lib/zookeeper and /usr/lib/hadoop = respectively?

-Joey

Sent from my iPhone

On Mar 11, 2012, at 8:10, Donald Resnik <dresnik@eci-usa.net> wr= ote:

Josh,

Thanks for the email.=A0 I raised the VM ram to 1G and lowered the heap= sizes from David Medinets levels from 256 to 128 and 128 to 64.=A0 I no lo= nger get the memory error, but now I am getting this:

=3D=3D> log= ger_localhost.err <=3D=3D
Uncaught exception: org/apache/zookeeper/Watcher
java.lang.NoClassDefFou= ndError: org/apache/zookeeper/Watcher
=A0=A0=A0 at java.lang.ClassLoader= .defineClass1(Native Method)
=A0=A0=A0 at java.lang.ClassLoader.defineCl= assCond(ClassLoader.java:632)
=A0=A0=A0 at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
=A0= =A0=A0 at java.security.SecureClassLoader.defineClass(SecureClassLoader.jav= a:141)
=A0=A0=A0 at java.net.URLClassLoader.defineClass(URLClassLoader.j= ava:283)
=A0=A0=A0 at java.net.URLClassLoader.access$000(URLClassLoader.= java:58)
=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
=A0= =A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0= =A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
=A0=A0= =A0 at java.lang.Class.forName0(Native Method)
=A0=A0=A0 at java.lang.Cl= ass.forName(Class.java:247)
=A0=A0=A0 at org.apache.accumulo.start.classloader.AccumuloClassLoader.load= Class(AccumuloClassLoader.java:348)
=A0=A0=A0 at org.apache.accumulo.sta= rt.classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:355)<= br>=A0=A0=A0 at org.apache.accumulo.start.Main.main(Main.java:56)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.Watcher=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
= =A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
= =A0=A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
=A0=A0= =A0 ... 17 more

=3D=3D> tserver_localhost.err <=3D=3D
Uncau= ght exception: org/apache/hadoop/io/BinaryComparable
java.lang.NoClassDefFoundError: org/apache/hadoop/io/BinaryComparable
= =A0=A0=A0 at java.lang.Class.forName0(Native Method)
=A0=A0=A0 at java.l= ang.Class.forName(Class.java:247)
=A0=A0=A0 at org.apache.accumulo.start= .classloader.AccumuloClassLoader.loadClass(AccumuloClassLoader.java:348) =A0=A0=A0 at org.apache.accumulo.start.classloader.AccumuloClassLoader.load= Class(AccumuloClassLoader.java:355)
=A0=A0=A0 at org.apache.accumulo.sta= rt.Main.main(Main.java:44)
Caused by: java.lang.ClassNotFoundException: = org.apache.hadoop.io.BinaryComparable
=A0=A0=A0 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
=A0= =A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0= =A0=A0 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
=A0= =A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
=A0=A0=A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
=A0= =A0=A0 ... 5 more

It is also haning on starting the loggers:

= cloudera@cloudera-vm:~/accumulo-1.3.5-incubating/conf$ ../bin/start-all.sh =
Starting tablet servers and loggers .... done
Starting tablet server on localhost
Starting logger on localhost

=
Any help would be appreciated.

Thanks,

Don


On Sat, Mar 10, 2012 at 8:04 PM, Josh Elser <jo= sh.elser@gmail.com> wrote:
Don,

What are the values you tried for Xmx for the tserver process? How much mem= ory did you allocate to your vm? Is the sum of all of the Accumulo processe= s exceeding the available memory in the vm?
- Josh


On 3/10/2012 10:42 PM, Donald Resnik wrote:
Accumulo,

I am having trouble staring accumulo in a cloudera VM. =A0I am getting this= error in the tserver log:

Could not reserve enough space for object heap.

I was using the steps from David Medinets post about configuring accumulo i= n a vm from 30 Dec 2011. =A0I tried to vary some of the accumulo-env settin= gs for Xmx and Xms, but kept getting the error.

Any help would be appreciated.

Thanks,

--
Don Resnik
Expert Consultants Inc.
410-8= 54-1756=A0(desk)
240-3= 81-9412=A0(cell)
301-9= 57-2270=A0(ECI office)




--
Don Resnik<= br>Expert Consultants Inc.
410-854-1756 (desk)
240-381-9412 (cell)
301-9= 57-2270 (ECI office)




--
Don Resnik
Expert Consultants Inc.
410-854-1756 (desk)240= -381-9412 (cell)
301-9= 57-2270 (ECI office)


--14dae9340b43611b3404bafa5e54--