hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Clampffer <james.clampf...@gmail.com>
Subject Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk
Date Wed, 28 Mar 2018 15:49:23 GMT
It's certainly possible to add a flag to disable the libhdfs++ build, I
opened HDFS-13362 for that.  I can put up an initial patch that includes a
flag for cmake so that you could do -Dnative_cmake_args="skipLibhdfspp" or
something similar.  I don't have a good understanding of maven and antrun
so if someone wants to help plumb a maven flag to the cmake flag I'd
appreciate it.

Is it common to do completely clean builds for interactive use?
Incremental rebuilds of libhdfs++ shouldn't be adding more than a few
seconds to the process.

On Tue, Mar 27, 2018 at 3:09 PM, Anu Engineer <aengineer@hortonworks.com>
wrote:

> Would it be possible to add a maven flag like –skipShade, that helps in
> reducing the compile time for people who does not need to build libhdfs++ ?
>
>
>
> Thanks
> Anu
>
>
>
>
>
> *From: *Jim Clampffer <james.clampffer@gmail.com>
> *Date: *Tuesday, March 27, 2018 at 11:09 AM
> *To: *Eric Badger <ebadger@oath.com>
> *Cc: *Deepak Majeti <majeti.deepak@gmail.com>, Jitendra Pandey <
> jitendra@hortonworks.com>, Anu Engineer <aengineer@hortonworks.com>,
> Mukul Kumar Singh <msingh@hortonworks.com>, Owen O'Malley <
> owen.omalley@gmail.com>, Chris Douglas <cdouglas@apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> *Subject: *Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to
> trunk
>
>
>
> Hi Eric,
>
> There isn't a way to completely skip compiling libhdfs++ as part of the
> native build.  You could pass -Dnative_cmake_args="-DHDFSPP_LIBRARY_ONLY=TRUE"
> to maven to avoid building all of the libhdfs++ tests, examples, and tools
> though.  That cut the native client build time from 4:10 to 2:20 for me.
>
> -Jim
>
>
>
> On Tue, Mar 27, 2018 at 12:25 PM, Eric Badger <ebadger@oath.com> wrote:
>
> Is there a way to skip the libhdfs++ compilation during a native build? I
> just went to build native trunk to test out some container-executor changes
> and it spent 7:49 minutes out of 14:31 minutes in Apache Hadoop HDFS Native
> Client. For me it basically doubled the compilation time.
>
>
>
> Eric
>
>
>
> On Fri, Mar 16, 2018 at 4:01 PM, Deepak Majeti <majeti.deepak@gmail.com>
> wrote:
>
> Thanks for all your hard work on getting this feature (with > 200
> sub-tasks) in James!
>
> On Fri, Mar 16, 2018 at 12:05 PM, Jim Clampffer <james.clampffer@gmail.com
> >
> wrote:
>
>
> > With 6 +1s, 0 0s, and 0 -1s the vote passes.  I'll be merging this into
> > trunk shortly.
> >
> > Thanks everyone who participated in the discussion and vote!  And many
> > thanks to everyone who contributed code and feedback throughout the
> > development process! Particularly Bob, Anatoli, Xiaowei and Deepak who
> > provided lots of large pieces of code as well as folks like Owen, Chris
> D,
> > Allen, and Stephen W who provided various support and guidance with the
> > Apache process and project design.
> >
> > On Wed, Mar 14, 2018 at 1:32 PM, Jitendra Pandey <
> jitendra@hortonworks.com
> > >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On 3/14/18, 9:57 AM, "Anu Engineer" <aengineer@hortonworks.com> wrote:
> > >
> > >     +1 (binding). Thanks for all the hard work and getting this client
> > > ready.
> > >     It is nice to have an official and supported native client for
> HDFS.
> > >
> > >     Thanks
> > >     Anu
> > >
> > >     On 3/13/18, 8:16 PM, "Mukul Kumar Singh" <msingh@hortonworks.com>
> > > wrote:
> > >
> > >         +1 (binding)
> > >
> > >         Thanks,
> > >         Mukul
> > >
> > >         On 14/03/18, 2:06 AM, "Owen O'Malley" <owen.omalley@gmail.com>
> > > wrote:
> > >
> > >             +1 (binding)
> > >
> > >             .. Owen
> > >
> > >             On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas <
> > > cdouglas@apache.org> wrote:
> > >
> > >             > +1 (binding) -C
> > >             >
> > >             > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer <
> > > james.clampffer@gmail.com>
> > >             > wrote:
> > >             > > Hi Everyone,
> > >             > >
> > >             > > The feedback was generally positive on the discussion
> > > thread [1] so I'd
> > >             > > like to start a formal vote for merging HDFS-8707
> > > (libhdfs++) into trunk.
> > >             > > The vote will be open for 7 days and end 6PM EST on
> > > 3/15/18.
> > >             > >
> > >             > > This branch includes a C++ implementation of an HDFS
> > > client for use in
> > >             > > applications that don't run an in-process JVM.  Right
> now
> > > the branch only
> > >             > > supports reads and metadata calls.
> > >             > >
> > >             > > Features (paraphrasing the list from the discussion
> > > thread):
> > >             > > -Avoiding the JVM means applications that use libhdfs++
> > > can explicitly
> > >             > > control resources (memory, FDs, threads).  The driving
> > > goal for this
> > >             > > project was to let C/C++ applications access HDFS while
> > > maintaining a
> > >             > > single heap.
> > >             > > -Includes support for Kerberos authentication.
> > >             > > -Includes a libhdfs/libhdfs3 compatible C API as well
> as
> > a
> > > C++ API that
> > >             > > supports asynchronous operations.  Applications that
> only
> > > do reads may be
> > >             > > able to use this as a drop in replacement for libhdfs.
> > >             > > -Asynchronous IO is built on top of boost::asio which
> in
> > > turn uses
> > >             > > select/epoll so many sockets can be monitored from a
> > > single thread (or
> > >             > > thread pool) rather than spawning a thread to sleep on
> a
> > > blocked socket.
> > >             > > -Includes a set of utilities written in C++ that mirror
> > > the CLI tools
> > >             > (e.g.
> > >             > > ./hdfs dfs -ls).  These have a 3 order of magnitude
> lower
> > > startup time
> > >             > than
> > >             > > java client which is useful for scripts that need to
> work
> > > with many
> > >             > files.
> > >             > > -Support for cancelable reads that release associated
> > > resources
> > >             > > immediately.  Useful for applications that need to be
> > > responsive to
> > >             > > interactive users.
> > >             > >
> > >             > > Other points:
> > >             > > -This is almost all new code in a new subdirectory. 
No
> > > Java source for
> > >             > the
> > >             > > rest of hadoop was changed so there's no risk of
> > > regressions there.  The
> > >             > > only changes outside of that subdirectory were
> > integrating
> > > the build in
> > >             > > some of the pom files and adding a couple dependencies
> to
> > > the DockerFile.
> > >             > > -The library has had plenty of burn-in time.  It's been
> > > used in
> > >             > production
> > >             > > for well over a year and is indirectly being
> distributed
> > > as part of the
> > >             > > Apache ORC project (in the form of a third party
> > > dependency).
> > >             > > -There isn't much in the way of well formatted
> > > documentation right now.
> > >             > > The documentation for the libhdfs API is applicable to
> > the
> > > libhdfs++ C
> > >             > API.
> > >             > > Header files describe various component including
> details
> > > about threading
> > >             > > and lifecycle expectations for important objects.  Good
> > > places to start
> > >             > are
> > >             > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h
> > and
> > > rpc_enginel.h.
> > >             > >
> > >             > > I'll start with my +1 (binding).
> > >             > >
> > >             > > [1]
> > >             > > http://mail-archives.apache.
> > org/mod_mbox/hadoop-hdfs-dev/
> > >             > 201803.mbox/browser
> > >             > > (second message in thread, can't figure out how to link
> > > directly to mine)
> > >             > >
> > >             > > Thanks!
> > >             >
> > >             > ------------------------------
> > ------------------------------
> > > ---------
> > >             > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.
> > > apache.org
> > >             > For additional commands, e-mail:
> > > hdfs-dev-help@hadoop.apache.org
> > >             >
> > >             >
> > >
> > >
> > >
> > >         ------------------------------------------------------------
> > > ---------
> > >         To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >         For additional commands, e-mail: hdfs-dev-help@hadoop.apache.
> org
> > >
> > >
> > >
> > >
> > >     ------------------------------------------------------------
> > ---------
> > >     To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >     For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> > >
> > >
> > >
> > >
> >
>
>
>
> --
>
> regards,
> Deepak Majeti
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message