ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: Ignite File System (re)design.
Date Tue, 03 Mar 2015 04:51:45 GMT
On Mon, Mar 02, 2015 at 08:39PM, Dmitriy Setrakyan wrote:
> I agree. I would actually drop Hadoop 1 altogether, especially given that
> it does not have Ignite MapReduce acceleration component.

Even better ;)

> 
> D.
> 
> On Mon, Mar 2, 2015 at 8:07 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > Seems pretty good. Also, considering that Hadoop1.x is pretty much a
> > dead-meat, specifying the implementor for Hadoop v1 should be optional.
> >
> > Cos
> >
> > On Mon, Mar 02, 2015 at 04:12PM, Vladimir Ozerov wrote:
> > > HI all,
> > >
> > > We spend some time on discussions about file system and Hadoop APIs.
> > There
> > > were two possible ways to improve current non-obvious API.
> > > First idea was to leave API more or less the same with only some cosmetic
> > > changes, mainly class names.
> > > Second idea was to remove all secondary file system configuration
> > > parameters from IgfsConfiguration and move it to Hadoop module. Then IGFS
> > > could be wired up with Hadoop secondary file system with help of some
> > > private interface which are not exposed to users.
> > >
> > > I think the first solution is better because currently secondary file
> > > system in IGFS is a kind of extension point. User is free to implement
> > his
> > > own secondary storage and use it in pretty the same way as store is used
> > in
> > > cache. I do not see any sensible reasons why we should remove this
> > > extension point and hide it in Hadoop module. Therefore, I designed new
> > API
> > > using the first approach and the draft put into the branch ignite-386.
> > > Please feel free to review and comment it.
> > >
> > > I'll also briefly go through the new design here:
> > >
> > > Core module:
> > > 1) o.a.i.IgniteFileSystem - user interface to work with our native file
> > > system. Obtained using Ignite.fileSystem() method
> > > Based on "IgniteFs" and "Igfs" interfaces in current implementation
> > >
> > > 2) o.a.i.filesystem.SecondaryFileSystem - API for creating secondary file
> > > systems for IGFS.
> > > Based on "Igfs" interface in current implementation.
> > >
> > > Note that there is no more direct link between IgniteFileSystem and
> > > SecondaryFileSystem, as these are completely different entities.
> > >
> > > 3) o.a.i.configuration.FileSystemConfiguration - configuration bean for
> > > IgniteFileSystem. It has setter
> > > "setSecondaryFileSystem(SecondaryFileSystem)".
> > >
> > > Hadoop module:
> > > 1) There are 4 map-reduce classes under o.a.i.hadoop.mapreduce package.
> > > Their packages reflect corresponding packages in Hadoop API. E.g.:
> > > org.apache.ignite.[hadoop.mapreduce.protocol.IgniteHadoopClientProtocol]
> > > implements org.apache.[hadoop.mapreduce.protocol.ClientProtocol].
> > >
> > > 2) Two file system implementations named "IgniteHadoopFileSystem" for v1
> > > and v2 Hadoops.
> > >
> > > 3) IgniteHadoopSecondaryFileSystem - implementation of
> > SecondaryFileSystem
> > > from core module, which is capable of delegating native IGFS calls to
> > > underlying Hadoop FileSystem.
> > > It is named "IgfsHadoopFileSystemWrapper" in current implementation.
> > >
> > > Let me give you an example of how user is going to configure it now.
> > >
> > > 1) Ignite configuration:
> > > <bean class="org.apache.ignite.configuration.IgniteConfiguration">
> > >     <property name="fileSystemConfiguration">
> > >         <list>
> > >             <bean
> > > class="org.apache.ignite.configuration.FileSystemConfiguration">
> > >                 <!-- Delegate to real HDFS. -->
> > >                 <property name="secondaryFileSystem">
> > >                     <bean
> > > class="├Ârg.apache.ignite.hadoop.fs.IgniteHadoopSecondaryFileSystem">
> > >                         <constructor-arg value="hdfs://192.168.1.23"/>
> > >                     </bean>
> > >                 </property>
> > >             </bean>
> > >         </list>
> > >     </property>
> > > </bean>
> > >
> > > 2) core-site.xml:
> > > <configuration>
> > >     <property>
> > >         <name>fs.default.name</name>
> > >         <value>igfs:///</value>
> > >     </property>
> > >     <property>
> > >         <name>fs.igfs.impl</name>
> > >
> >  <value>org.apache.ignite.igfs.hadoop.v1.IgfsHadoopFileSystem</value>
> > >     </property>
> > >     <property>
> > >         <name>fs.AbstractFileSystem.igfs.impl</name>
> > >
> >  <value>org.apache.ignite.igfs.hadoop.v2.IgfsHadoopFileSystem</value>
> > >     </property>
> > > </configuration>
> > >
> > > Seems pretty clear and consistent to me.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> >

Mime
View raw message