hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Krogen <ekro...@linkedin.com>
Subject Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)
Date Fri, 21 Jul 2017 22:17:58 GMT
Hi Sanjay,

Actually I was not aware of that work… This seems to be a better way of achieving some of
the same things we do externally to the DN process. I will look into reimplementing some parts
on top of this; seems it should just require some very small extensions to DataNodeCluster.
Thank you very much for the pointer!

Erik

On 7/21/17, 11:01 AM, "sanjay Radia" <sanjayosrc@gmail.com> wrote:

    Erik
      Great stuff. 
    BTW did you build on top of the “simulated data nodes” in HDFS which has a way to
storing only the length of data (but not real data)? That work allowed supplementing  with
a matching editsLog for the NN. Your approach of using a real image has the advantage of being
able to replay traces from audit logs.
    (Ref https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)
    
    thanks
    
    sanjay
    > On Jul 20, 2017, at 10:42 AM, Erik Krogen <ekrogen@linkedin.com.INVALID> wrote:
    > 
    > forking off of the 2.7.4 release thread to answer this question about
    > Dynamometer
    > 
    > Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
    > specifically the NameNode. We have been using it for some time now and have
    > recently been making some enhancements to ease of use and reproducibility.
    > We hope to post a blog post sometime in the not-too-distant future, and
    > also to open source it. I can provide some details here given that we have
    > been leveraging it as part of our 2.7.4 release / upgrade process (in
    > addition to previous upgrades).
    > 
    > The basic idea is to get full-scale black-box testing of the HDFS NN while
    > using significantly less (~10%) hardware than a real cluster of that size
    > would require. We use real NN images from our at-scale clusters paired with
    > some logic to fake out DNs into thinking they are storing data when they
    > are not, allowing us to stuff more DNs onto each machine. Since we use a
    > real image, we can replay real traces (collected from audit logs) to
    > compare actual production performance vs. performance on this simulated
    > cluster (with additional tuning, different version, etc.). We leverage YARN
    > to manage setting up this cluster and to replay the traces.
    > 
    > Happy to answer questions.
    > 
    > Erik
    > 
    > On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko <shv.hadoop@gmail.com>
    > wrote:
    > 
    >> Hi Tianyi,
    >> 
    >> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
    >> on this project right now, I'll let him elaborate.
    >> Erik, you should probably respond on Apache dev list, as I think it could
    >> be interesting for other people as well, asince we planned to open source
    >> it. You can fork the "About 2.7.4 Release" thread with a new subject and
    >> give some details about Dynamometer there.
    >> 
    >> Thanks,
    >> --Konstantin
    >> 
    >> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 <hetianyi@bytedance.com> wrote:
    >> 
    >>> Hi, Shavachko.
    >>> 
    >>> You mentioned an internal tool called Dynamometer to test NameNode
    >>> performance earlier in the 2.7.4 release thread.
    >>> I wonder if you could share some ideas behind the tool. Or is there a
    >>> plan to bring Dynamometer to open source community?
    >>> 
    >>> Thanks.
    >>> 
    >>> BR,
    >>> Tianyi
    >>> 
    >>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko <shv.hadoop@gmail.com>
    >>> wrote:
    >>> 
    >>>> Hi everybody.
    >>>> 
    >>>> We have been doing some internal testing of Hadoop 2.7.4. The testing
is
    >>>> going well.
    >>>> Did not find any major issues on our workloads.
    >>>> Used an internal tool called Dynamometer to check NameNode performance
on
    >>>> real cluster traces. Good.
    >>>> Overall test cluster performance looks good.
    >>>> Some more testing is still going on.
    >>>> 
    >>>> I plan to build an RC next week. If there are no objection.
    >>>> 
    >>>> Thanks,
    >>>> --Konst
    >>>> 
    >>>> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
    >>>> shv.hadoop@gmail.com>
    >>>> wrote:
    >>>> 
    >>>>> Hey guys.
    >>>>> 
    >>>>> An update on 2.7.4 progress.
    >>>>> We are down to 4 blockers. There is some work remaining on those.
    >>>>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
    >>>>> Would be good if people could follow up on review comments.
    >>>>> 
    >>>>> I looked through nightly Jenkins build results for 2.7.4 both on
Apache
    >>>>> Jenkins and internal.
    >>>>> Some test fail intermittently, but there no consistent failures.
I
    >>>> filed
    >>>>> HDFS-11985 to track some of them.
    >>>>> https://issues.apache.org/jira/browse/HDFS-11985
    >>>>> I do not currently consider these failures as blockers. LMK if some
of
    >>>>> them are.
    >>>>> 
    >>>>> We started internal testing of branch-2.7 on one of our smallish
(100+
    >>>>> nodes) test clusters.
    >>>>> Will update on the results.
    >>>>> 
    >>>>> There is a plan to enable BigTop for 2.7.4 testing.
    >>>>> 
    >>>>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
    >>>>> Thank you everybody for contributing to this effort.
    >>>>> 
    >>>>> Regards,
    >>>>> --Konstantin
    >>>>> 
    >>>>> 
    >>>>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajisaka@apache.org>
    >>>>> wrote:
    >>>>> 
    >>>>>> Sure.
    >>>>>> If you want to edit the wiki, please tell me your ASF confluence
    >>>> account.
    >>>>>> 
    >>>>>> -Akira
    >>>>>> 
    >>>>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
    >>>>>> 
    >>>>>>> Couple of more JIRAs need to be back ported for 2.7.4 release.
These
    >>>> will
    >>>>>>> solve RM HA unstability issues.
    >>>>>>> https://issues.apache.org/jira/browse/YARN-5333
    >>>>>>> https://issues.apache.org/jira/browse/YARN-5988
    >>>>>>> https://issues.apache.org/jira/browse/YARN-6304
    >>>>>>> 
    >>>>>>> I will raise a JIRAs to back port it.
    >>>>>>> 
    >>>>>>> @Akira , could  you help to add these JIRAs into wiki?
    >>>>>>> 
    >>>>>>> Thanks & Regards
    >>>>>>> Rohith Sharma K S
    >>>>>>> 
    >>>>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aajisaka@apache.org>
wrote:
    >>>>>>> 
    >>>>>>> Created a page for 2.7.4 release.
    >>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
    >>>>>>>> 
    >>>>>>>> If you want to edit this wiki, please ping me.
    >>>>>>>> 
    >>>>>>>> Regards,
    >>>>>>>> Akira
    >>>>>>>> 
    >>>>>>>> 
    >>>>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
    >>>>>>>> 
    >>>>>>>> Hi Konstantin Shvachko
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> how about creating a wiki page for 2.7.4 release
status like 2.8
    >>>> and
    >>>>>>>>> trunk in following link.??
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> ________________________________
    >>>>>>>>> From: Konstantin Shvachko <shv.hadoop@gmail.com>
    >>>>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
    >>>>>>>>> To: Akira Ajisaka
    >>>>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
    >>>>>>>>> yarn-dev@hadoop.apache.org
    >>>>>>>>> Subject: Re: About 2.7.4 Release
    >>>>>>>>> 
    >>>>>>>>> Latest update on the links and filters. Here is the
correct link
    >>>> for
    >>>>>>>>> the
    >>>>>>>>> filter:
    >>>>>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
    >>>>>>>>> requestId=12340814
    >>>>>>>>> 
    >>>>>>>>> Also updated: https://s.apache.org/Dzg4
    >>>>>>>>> 
    >>>>>>>>> Had to do some Jira debugging. Sorry for confusion.
    >>>>>>>>> 
    >>>>>>>>> Thanks,
    >>>>>>>>> --Konstantin
    >>>>>>>>> 
    >>>>>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko
<
    >>>>>>>>> shv.hadoop@gmail.com>
    >>>>>>>>> wrote:
    >>>>>>>>> 
    >>>>>>>>> Hey Akira,
    >>>>>>>>> 
    >>>>>>>>>> 
    >>>>>>>>>> I didn't have private filters. Most probably
Jira caches
    >>>> something.
    >>>>>>>>>> Your filter is in the right direction, but for
some reason it
    >>>> lists
    >>>>>>>>>> only
    >>>>>>>>>> 22 issues, while mine has 29.
    >>>>>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
    >>>>>>>>>> a/browse/YARN-5543>
    >>>>>>>>>> .
    >>>>>>>>>> 
    >>>>>>>>>> Anyways, I created a Jira filter now "Hadoop
2.7.4 release
    >>>> blockers",
    >>>>>>>>>> shared it with "everybody", and updated my link
to point to that
    >>>>>>>>>> filter.
    >>>>>>>>>> So
    >>>>>>>>>> you can use any of the three methods below to
get the correct
    >>>> list:
    >>>>>>>>>> 1. Go to https://s.apache.org/Dzg4
    >>>>>>>>>> 2. Go to the filter via
    >>>>>>>>>>    https://issues.apache.org/jira/issues?filter=12340814
    >>>>>>>>>>   or by finding "Hadoop 2.7.4 release blockers"
filter in the
    >>>> jira
    >>>>>>>>>> 3. On Advanced issues search page paste this:
    >>>>>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND
labels =
    >>>>>>>>>> release-blocker
    >>>>>>>>>> AND "Target Version/s" = 2.7.4
    >>>>>>>>>> 
    >>>>>>>>>> Hope this solves the confusion for which issues
are included.
    >>>>>>>>>> Please LMK if it doesn't, as it is important.
    >>>>>>>>>> 
    >>>>>>>>>> Thanks,
    >>>>>>>>>> --Konstantin
    >>>>>>>>>> 
    >>>>>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka
<
    >>>> aajisaka@apache.org>
    >>>>>>>>>> wrote:
    >>>>>>>>>> 
    >>>>>>>>>> Hi Konstantin,
    >>>>>>>>>> 
    >>>>>>>>>>> 
    >>>>>>>>>>> Thank you for volunteering as release manager!
    >>>>>>>>>>> 
    >>>>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
    >>>>>>>>>>> 
    >>>>>>>>>>>> 
    >>>>>>>>>>>> I couldn't see the link. Maybe is it
private filter?
    >>>>>>>>>>> 
    >>>>>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
    >>>>>>>>>>> This filter includes resolved issue and excludes
fixversion ==
    >>>> 2.7.4
    >>>>>>>>>>> 
    >>>>>>>>>>> Thanks and Regards,
    >>>>>>>>>>> Akira
    >>>>>>>>>>> 
    >>>>>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko
wrote:
    >>>>>>>>>>> 
    >>>>>>>>>>> Hi Brahma Reddy Battula,
    >>>>>>>>>>> 
    >>>>>>>>>>>> 
    >>>>>>>>>>>> Actually the original link works fine:
    >>>> https://s.apache.org/Dzg4
    >>>>>>>>>>>> Your link excludes closed and resolved
issues, which needs
    >>>>>>>>>>>> backporting,
    >>>>>>>>>>>> and
    >>>>>>>>>>>> which we cannot reopen, as discussed
in this thread earlier.
    >>>>>>>>>>>> 
    >>>>>>>>>>>> Looked through the issues you proposed:
    >>>>>>>>>>>> 
    >>>>>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
    >>>>>>>>>>>> Seems like a new feature. It helps failover
to standby node when
    >>>>>>>>>>>> primary
    >>>>>>>>>>>> is
    >>>>>>>>>>>> under heavy load, but it introduces new
APIs, addresses, config
    >>>>>>>>>>>> parameters.
    >>>>>>>>>>>> And needs at least one follow up jira.
    >>>>>>>>>>>> Looks like a backward compatible change,
though.
    >>>>>>>>>>>> Did you have a chance to run it in production?
    >>>>>>>>>>>> 
    >>>>>>>>>>>> +1 on
    >>>>>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
    >>>>>>>>>>>> 
    >>>>>>>>>>>> [HDFS-10987] Make Decommission less expensive
when lot of ...<
    >>>>>>>>>>> 
    >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
    >>>>>>>>> issues.apache.org
    >>>>>>>>> When user want to decommission a node which having
50M blocks ,it
    >>>> could
    >>>>>>>>> hold the namesystem lock for long time.We've seen
it is taking 36
    >>>> sec.
    >>>>>>>>> As
    >>>>>>>>> we knew during this ...
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
    >>>>>>>>> 
    >>>>>>>>>> 
    >>>>>>>>>>>> [HDFS-9902] Support different values
of dfs.datanode.du ...<
    >>>>>>>>>>> 
    >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
    >>>>>>>>> issues.apache.org
    >>>>>>>>> Now Hadoop support different storage type for DISK,
SSD, ARCHIVE
    >>>> and
    >>>>>>>>> RAM_DISK, but they share one configuration
    >>>> dfs.datanode.du.reserved.
    >>>>>>>>> The
    >>>>>>>>> DISK size may be several ...
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
    >>>>>>>>> 
    >>>>>>>>>> 
    >>>>>>>>>>>> Trash does not descent into child directories
to check for ...<
    >>>>>>>>>>> 
    >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
    >>>>>>>>> issues.apache.org
    >>>>>>>>> HDFS trash does not descent into child directory
to check if user
    >>>> has
    >>>>>>>>> permission to delete files. For example: Run the
following command
    >>>> to
    >>>>>>>>> initialize directory ...
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
    >>>>>>>>> 
    >>>>>>>>>> 
    >>>>>>>>>>>> Upgrade Jsch jar to latest version to
fix vulnerability in ...<
    >>>>>>>>>>> 
    >>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
    >>>>>>>>> issues.apache.org
    >>>>>>>>> Recently there was on vulnerability reported on jsch
library. Its
    >>>> fixed
    >>>>>>>>> in latest 0.1.54 version before CVE was made public.
    >>>>>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> Added them to 2.7.4 release. You should see them
via the above link
    >>>>>>>>>>>> now.
    >>>>>>>>>>>> Would be good if you could attach backport
patches for some of
    >>>> them?
    >>>>>>>>>>>> 
    >>>>>>>>>>>> Appreciate your help,
    >>>>>>>>>>>> --Konstantin
    >>>>>>>>>>>> 
    >>>>>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma
Reddy Battula <
    >>>>>>>>>>>> brahmareddy.battula@huawei.com> wrote:
    >>>>>>>>>>>> 
    >>>>>>>>>>>> 
    >>>>>>>>>>>> Looks following link is not correct..
    >>>>>>>>>>>> 
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> https://s.apache.org/Dzg4
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> It should be like following..?
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> https://s.apache.org/wi3U
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> Apart from Konstantin mentioned,Following
also good to go..?
    >>>> let me
    >>>>>>>>>>>>> know
    >>>>>>>>>>>>> your thoughts on this.
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> For Large Cluster:
    >>>>>>>>>>>>> =============
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life
Line
    >>>>>>>>>>>>> Protocol
    >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommis
    >>>> sion
    >>>>>>>>>>>>> Expensive when lot's of blocks are
present
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
    >>>>>>>>>>>>> "dfs.datanode.du.reserved"  per Storage
Type
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> For Security:
    >>>>>>>>>>>>> =========
    >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash
does
    >>>> not
    >>>>>>>>>>>>> descent
    >>>>>>>>>>>>> into child directories to check for
permission
    >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade
    >>>> Jsch
    >>>>>>>>>>>>> jar
    >>>>>>>>>>>>> to
    >>>>>>>>>>>>> latest version to fix vulnerability
in old versions
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> Regards
    >>>>>>>>>>>>> Brahma Reddy Battula
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> -----Original Message-----
    >>>>>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
    >>>>>>>>>>>>> Sent: 06 May 2017 02:40
    >>>>>>>>>>>>> To: Konstantin Shvachko
    >>>>>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
    >>>>>>>>>>>>> mapreduce-dev@hadoop.apache.org
    >>>>>>>>>>>>> ;
    >>>>>>>>>>>>> yarn-dev@hadoop.apache.org
    >>>>>>>>>>>>> Subject: Re: About 2.7.4 Release
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> List LGTM Konstantin!
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> Let's say that we will only create
a new tracking JIRA for
    >>>> patches
    >>>>>>>>>>>>> which
    >>>>>>>>>>>>> do not backport cleanly, to avoid
having too many lying around.
    >>>>>>>>>>>>> Otherwise
    >>>>>>>>>>>>> we can directly attach to old ticket.
If a clean backport does
    >>>>>>>>>>>>> happen
    >>>>>>>>>>>>> to
    >>>>>>>>>>>>> break a test the nightly build will
help us catch it.
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> Erik
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin
Shvachko <
    >>>>>>>>>>>>> shv.hadoop@gmail.com>
    >>>>>>>>>>>>> wrote:
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> Great Zhe. Let's monitor the build.
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>> 
    >>>>>>>>>>>>>> I marked all jiras I knew of
for inclusion into 2.7.4 as I
    >>>>>>>>>>>>>> described
    >>>>>>>>>>>>>> before.
    >>>>>>>>>>>>>> Target Version/s: 2.7.4
    >>>>>>>>>>>>>> Label: release-blocker
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> Here is the link to the list:
https://s.apache.org/Dzg4
    >>>> Please
    >>>>>>>>>>>>>> let
    >>>>>>>>>>>>>> me
    >>>>>>>>>>>>>> know if I missed anything.
    >>>>>>>>>>>>>> And feel free to pick up any.
Most of backports are pretty
    >>>>>>>>>>>>>> straightforward, but not all.
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> We can create tracking jiras
for backporting if you need to
    >>>> run
    >>>>>>>>>>>>>> Jenkins on the patch (and since
Allen does not allow reopening
    >>>>>>>>>>>>>> them).
    >>>>>>>>>>>>>> But I think the final patch should
be attached to the original
    >>>>>>>>>>>>>> jira.
    >>>>>>>>>>>>>> Otherwise history will be hard
to follow.
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> Thanks,
    >>>>>>>>>>>>>> --Konstantin
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM,
Zhe Zhang <zhz@apache.org>
    >>>> wrote:
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> Thanks for volunteering as RM
Konstantin! The plan LGTM.
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> I've created a nightly Jenkins
job for branch-2.7 (unit
    >>>> tests):
    >>>>>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> On Wed, May 3, 2017 at 12:42
AM Konstantin Shvachko <
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> shv.hadoop@gmail.com>
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> wrote:
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> Hey guys,
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> I and a few of my colleagues
would like to help here and
    >>>> move
    >>>>>>>>>>>>>>>> 2.7.4
    >>>>>>>>>>>>>>>> release forward. A few
points in this regard.
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> 1. Reading through this
thread since March 1 I see that
    >>>> Vinod
    >>>>>>>>>>>>>>>> hinted on managing the
release. Vinod, if you still want the
    >>>>>>>>>>>>>>>> job /
    >>>>>>>>>>>>>>>> have bandwidth will be
happy to work with you.
    >>>>>>>>>>>>>>>> Otherwise I am glad to
volunteer as the release manager.
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> 2. In addition to current
blockers and criticals, I would
    >>>> like
    >>>>>>>>>>>>>>>> to
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> propose
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> a
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> few issues to be included
in the release, see the list below.
    >>>>>>>>>>>>>>>> Those
    >>>>>>>>>>>>>>>> are mostly bug fixes
and optimizations, which we already
    >>>> have in
    >>>>>>>>>>>>>>>> our
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> internal
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> branch and run in production.
Plus one minor feature "node
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> labeling", which we found
very handy, when you have
    >>>> heterogeneous
    >>>>>>>>>>>>>>>> environments and mixed
workloads, like MR and Spark.
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> 3. For marking issues
for the release I propose to
    >>>>>>>>>>>>>>>> - set the target version
to 2.7.4, and
    >>>>>>>>>>>>>>>> - add a new label "release-blocker"
    >>>>>>>>>>>>>>>> That way we will know
issues targeted for the release
    >>>> without
    >>>>>>>>>>>>>>>> reopening them for backports.
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> 4. I see quite a few
people are interested in the release.
    >>>> With
    >>>>>>>>>>>>>>>> all
    >>>>>>>>>>>>>>>> the help I think we can
target to release by the end of May.
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> Other things include
fixing CHANGES.txt and fixing Jenkins
    >>>> build
    >>>>>>>>>>>>>>>> for
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> 2.7.4
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> branch.
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> Thanks,
    >>>>>>>>>>>>>>>> --Konstantin
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> ==========  List of issue
for 2.7.4  ===========
    >>>>>>>>>>>>>>>> ------ Backports
    >>>>>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
    >>>>>>>>>>>>>>>> a/browse/HADOOP-12975>.
    >>>>>>>>>>>>>>>> Add
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> du
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> jitters
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>.
    >>>> IBR
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> batching
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715
    >>>>> .
    >>>>>>>>>>>>>> NPE
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
    >>>>>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538
    >>>>> .
    >>>>>>>>>>>>>>>> fsck
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> removal
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> of dot printing
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
    >>>>>>>>>>>>>>>> space-balanced
    >>>>>>>>>>>>>>>> policy for balancer
    >>>>>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549
    >>>>> .
    >>>>>>>>>>>>>>>> abort
    >>>>>>>>>>>>>>>> balancer if upgrade in
progress
    >>>>>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412
    >>>>> .
    >>>>>>>>>>>>>>>> skip
    >>>>>>>>>>>>>>>> small blocks in getBlocks
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471
    >>>>> .
    >>>>>>>>>>>>>>>> SLS
    >>>>>>>>>>>>>>>> simulator
    >>>>>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302
    >>>>> .
    >>>>>>>>>>>>>>>> SLS
    >>>>>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367
    >>>>> .
    >>>>>>>>>>>>>>>> SLS
    >>>>>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612
    >>>>> .
    >>>>>>>>>>>>>>>> SLS
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> ----- Node labeling
    >>>>>>>>>>>>>>>> MAPREDUCE-6304
    >>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
    >>>>>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
    >>>>>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
    >>>>>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
    >>>>>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
    >>>>>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> --
    >>>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>>> Zhe Zhang
    >>>>>>>>>>>>>>> Apache Hadoop Committer
    >>>>>>>>>>>>>>> http://zhe-thoughts.github.io/about/
| @oldcap
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>>> 
    >>>>>>>>>>>>>> 
    >>>>>>>>>>>>> 
    >>>>>>>>>>>> 
    >>>>>>>>>> 
    >>>>>>>>> ------------------------------------------------------------
    >>>> ---------
    >>>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
    >>>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
    >>>>>>>> 
    >>>>>>>> 
    >>>>>>>> 
    >>>>>>> 
    >>>>> 
    >>>> 
    >>> 
    >> 
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
    For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
    
    


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Mime
View raw message