Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of dlmarion@comcast.net designates
 76.96.62.48 as permitted sender)
Date: Thu, 1 May 2014 17:08:57 +0000 (UTC)
From: dlmarion@comcast.net
To: dev@accumulo.apache.org
Message-ID: <524424208.1549136.1398964137840.JavaMail.root@comcast.net>
In-Reply-To: 
 <CAD-fFUL5wemZz7N8y7+kUWmiR8gO7NKkQ_U_1jwCHq=sR2f3iQ@mail.gmail.com>
References: <5mjj96v9oj21bjccdk320m4g.1398910708022@email.android.com>
 <5361B4CA.8000005@gmail.com>
 <CAD-fFUL5wemZz7N8y7+kUWmiR8gO7NKkQ_U_1jwCHq=sR2f3iQ@mail.gmail.com>
Subject: Re: [VOTE] New Blog Entry
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_1549135_376578731.1398964137839"
Thread-Topic: New Blog Entry
Thread-Index: WV8YAs53jT8brEvCAEcQDVBGnXcj1Q==

------=_Part_1549135_376578731.1398964137839
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Yeah, just email them to me.=20

----- Original Message -----

From: "Bill Havanki" <bhavanki@clouderagovt.com>=20
To: "Accumulo Dev List" <dev@accumulo.apache.org>=20
Sent: Thursday, May 1, 2014 12:47:02 PM=20
Subject: Re: [VOTE] New Blog Entry=20

Overall the article is great! I have suggested edits, so I'd like to know=
=20
where I can stick them (don't be rude now ;) ). We've used Review Board for=
=20
doc feedback in the past ... that's an OK way. Dave, I can just email them=
=20
to you to avoid spamming. Let me know.=20


On Wed, Apr 30, 2014 at 10:43 PM, Josh Elser <josh.elser@gmail.com> wrote:=
=20

> Ah ok. I was just looking through the link you provided and didn't notice=
=20
> an author at all.=20
>=20
> Just found it now in tiny letters at the bottom :)=20
>=20
>=20
> On 4/30/14, 10:18 PM, dlmarion wrote:=20
>=20
>> I believe that the author is shown. Well, at least the person who posts=
=20
>> it is shown. In this case it is one in the same.=20
>>=20
>>=20
>> Sent via the Samsung GALAXY S=C2=AE4, an AT&T 4G LTE smartphone=20
>>=20
>> -------- Original message --------=20
>> From: Josh Elser <josh.elser@gmail.com>=20
>> Date:04/30/2014 9:51 PM (GMT-05:00)=20
>> To: dev@accumulo.apache.org=20
>> Subject: Re: [VOTE] New Blog Entry=20
>>=20
>> It would be nice to include yourself as the author of the post. That=20
>> would be nice to help users identify who created the content.=20
>>=20
>> On 4/30/14, 6:51 PM, dlmarion@comcast.net wrote:=20
>>=20
>>>=20
>>> I have created a new entry for the blog. The preview feature does not=
=20
>>> appear to be working at the moment. I will submit an INFRA issue for th=
is.=20
>>> I have pasted the text below. For those that have a blog account, you=
=20
>>> should be able to see the blog at [1]. This blog entry is set to be=20
>>> published at 235959 3 May 2014 GMT pending no vetoes. This vote will re=
main=20
>>> open for 72 hours, until 2300 3 May 2014 GMT.=20
>>>=20
>>> [1] https://blogs.apache.org/roller-ui/authoring/preview/=20
>>> accumulo/?previewEntry=3Dthe_accumulo_classloader=20
>>>=20
>>> - Dave=20
>>> ------------------------------------------------------------=20
>>> ------------------------------------------------------------=20
>>> --------------------------------------------------------------=20
>>>=20
>>> Blog Title: The Accumulo Classloader=20
>>>=20
>>> Blog Text:=20
>>>=20
>>> First, some history=20
>>>=20
>>>=20
>>> The classloader in version 1.4 used a simple hierarchy of two=20
>>> classloaders that would load classes from locations specified by two=20
>>> properties. The locations specified by the "general.classpaths" propert=
y=20
>>> would be used to create a parent classloader and locations specified by=
 the=20
>>> "general.dynamic.classpaths" property were used to create a child=20
>>> classloader. The child classloader would monitor the specified location=
s=20
>>> for changes and when a change occurred it would replace the child=20
>>> classloader with a new instance. Classes that referenced the orphaned c=
hild=20
>>> classloader would continue to work and the classloader would be garbage=
=20
>>> collected when no longer referenced.=20
>>>=20
>>> The only place where the dynamic classloader would come into play is fo=
r=20
>>> user iterators and their dependencies. The general advice for using thi=
s=20
>>> classloader would be to put the jars containing your iterators in the=
=20
>>> dynamic location. Everything else that does not change very often or wo=
uld=20
>>> require a restart can be put into the non-dynamic location.=20
>>>=20
>>> There are a couple of things to note about the classloader in 1.4.=20
>>> First, if you modified the dynamic locations too often, you would run o=
ut=20
>>> of perm-gen space. This is likely due to unreferenced classes not being=
=20
>>> unloaded from the JVM. This is captured in ACCUMULO-599 . Secondly, whe=
n=20
>>> you modified files in dynamic locations within the same cycle, it would=
 on=20
>>> occasion miss the second change. Out with the old, in with the new=20
>>>=20
>>>=20
>>> The Accumulo classloader was rewritten in version 1.5. It maintains the=
=20
>>> same dynamic capability and includes a couple of new features. The=20
>>> classloader uses Commons VFS so that it can load jars and classes from =
a=20
>>> variety of sources, including HDFS. Additionally, we introduced the not=
ion=20
>>> of classloader contexts into Accumulo. This is not a new concept for an=
yone=20
>>> that has used an application server, but the implementation is a little=
=20
>>> different for Accumulo.=20
>>>=20
>>> The hierarchy set up by the new classloader uses the same property name=
s=20
>>> as the old classloader. In the most basic configuration the locations=
=20
>>> specified by "general.classpaths" are used to create the root of the=20
>>> application classloader hierarchy. This classloader is a URLClassLoader=
 and=20
>>> it does not support dynamic reloading. If you only specify this propert=
y,=20
>>> then you are loading all of your jars from the local file system and th=
ey=20
>>> will not be monitored for changes. We will call this top level applicat=
ion=20
>>> classloader the SYSTEM classloader. Next, a classloader is created that=
=20
>>> supports VFS sources and reloading. The parent of this classloader is t=
he=20
>>> SYSTEM classloader and we will call this the VFS classloader. If the=20
>>> "general.vfs.classpaths" property is set, the VFS classloader will use =
this=20
>>> location. If the property is not set, it will use the value of=20
>>> "general.dynamic.classpaths" with a default value of $ACCUMULO_HOME/lib=
/ext=20
>>> to support backwards compatibility. Running Accumulo F=20
>>>=20
>> r=20
>=20
> o=20
>> m HDFS=20
>>=20
>>>=20
>>>=20
>>> If you have defined "general.vfs.classpaths" in your Accumulo=20
>>> configuration, then you can use the bootstrap_hdfs.sh script in the bin=
=20
>>> directory to seed HDFS with the Accumulo jars. A couple of jars will re=
main=20
>>> on the local file system for starting services. Now when you start up=
=20
>>> Accumulo the master, gc, tracer, and all of the tablet servers will get=
=20
>>> their jars and classes from HDFS. The bootstrap_hdfs.sh script sets the=
=20
>>> replication on the directory, but you may want to set it higher after=
=20
>>> bootstrapping. An example configuration setting would be:=20
>>> <property>=20
>>> <name>general.vfs.classpaths</name>=20
>>> <value>hdfs://localhost:8020/accumulo/system-classpath</value>=20
>>> <description>Configuration for a system level vfs classloader.=20
>>> Accumulo jars can be configured here and loaded out of HDFS.</descripti=
on>=20
>>> </property>=20
>>> About Contexts=20
>>>=20
>>>=20
>>> You can also define classloader contexts in your accumulo-site.xml file=
.=20
>>> A context is defined by a user supplied name and it references location=
s=20
>>> like the other classloader properties. When a context is defined in the=
=20
>>> configuration, it can then be applied to one or more tables. When a con=
text=20
>>> is applied to a table, then a classloader is created for that context. =
If=20
>>> multiple tables use the same context, then they share the context=20
>>> classloader. The context classloader is a child to the VFS classloader=
=20
>>> created above.=20
>>>=20
>>> The goal here is to enable multiple tenants to share the same Accumulo=
=20
>>> instance. For example, we may have a context called 'app1' which refere=
nces=20
>>> the jars for application A. We may also have another context called app=
2=20
>>> which references the jars for application B. By default the context=20
>>> classloader delegates to the parent classloader. This behavior may be=
=20
>>> overridden as seen in the app2 example below.=20
>>> <property>=20
>>> <name>general.vfs.context.classpath.app1</name>=20
>>> <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,=20
>>> file:///opt/applicationA/lib/.*.jar</value>=20
>>> <description>Application A classpath, loads jars from HDFS and=20
>>> local file system</description>=20
>>> </property>=20
>>>=20
>>> <property>=20
>>> <name>general.vfs.context.classpath.app2.delegation=3Dpost</name>=20
>>> <value>hdfs://localhost:8020/applicationB/classpath/.*.jar,=20
>>> http://my-webserver/applicationB/.*.jar</value>=20
>>> <description>Application B classpath, loads jars from HDFS and=20
>>> HTTP, does not delegate to parent first</description>=20
>>> </property>=20
>>>=20
>>>=20
>>> Context classloaders do not have to be defined in the accumulo-site.xml=
=20
>>> file. The "general.vfs.context.classpath.{context}" property can be=20
>>> defined on the table either programatically or manually in the shell. T=
hen=20
>>> set the "table.classpath.context" property on your table. Known Issues=
=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>> Remember the two issues I mentioned above? Well, they are still a=20
>>> problem.=20
>>>=20
>>> * ACCUMULO-1507 is tracking VFS-487 for frequent modifications to=20
>>> files.=20
>>> * If you start running out of perm-gen space, take a look at=20
>>> ACCUMULO-599 and try applying the JVM settings for class unloading.=20
>>> * Additionally, there is an issue with the bootstrap_hdfs.sh=20
>>> script detailed in ACCUMULO-2761 . There is a workaround listed in the=
=20
>>> issue.=20
>>>=20
>>>=20
>>>=20
>>> I have disabled comments as I see they are being abused in other blogs.=
=20
>>> Please email the dev list for comments and questions.=20
>>>=20
>>>=20


--=20
// Bill Havanki=20
// Solutions Architect, Cloudera Govt Solutions=20
// 443.686.9283=20


------=_Part_1549135_376578731.1398964137839--