hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Krogen <ekro...@linkedin.com.INVALID>
Subject Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)
Date Thu, 20 Jul 2017 18:02:13 GMT
Hi Anu,

Thanks for the interest!

1. Unfortunately I am very doubtful that LinkedIn security would let us
release our traces. If you collect your audit logs that's the only thing
necessary to build traces.
2. Our current approach is to use 'truncate' to create sparse files which
have the expected length. We overwrite the checksum as well. Luckily this
requires no modification to the DN itself and has worked quite well.

Erik

On Thu, Jul 20, 2017 at 10:56 AM, Anu Engineer <aengineer@hortonworks.com>
wrote:

> Hi Erik,
>
> Looking forward to the release of this tool. Thank you very much for the
> contribution.
>
> Had a couple of questions about how the tool works.
>
> 1. Would you be able to provide the traces along with this tool? In other
> words, would I be able to use this out of the box, or do I have to build up
> traces myself?
>
> 2. Could you explain how the “fake out DNs into thinking they are storing
> data” — works? Or I can be patient and read your blog post too.
>
> Thanks
> Anu
>
>
>
>
>
>
> On 7/20/17, 10:42 AM, "Erik Krogen" <ekrogen@linkedin.com.INVALID> wrote:
>
> >forking off of the 2.7.4 release thread to answer this question about
> >Dynamometer
> >
> >Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> >specifically the NameNode. We have been using it for some time now and
> have
> >recently been making some enhancements to ease of use and reproducibility.
> >We hope to post a blog post sometime in the not-too-distant future, and
> >also to open source it. I can provide some details here given that we have
> >been leveraging it as part of our 2.7.4 release / upgrade process (in
> >addition to previous upgrades).
> >
> >The basic idea is to get full-scale black-box testing of the HDFS NN while
> >using significantly less (~10%) hardware than a real cluster of that size
> >would require. We use real NN images from our at-scale clusters paired
> with
> >some logic to fake out DNs into thinking they are storing data when they
> >are not, allowing us to stuff more DNs onto each machine. Since we use a
> >real image, we can replay real traces (collected from audit logs) to
> >compare actual production performance vs. performance on this simulated
> >cluster (with additional tuning, different version, etc.). We leverage
> YARN
> >to manage setting up this cluster and to replay the traces.
> >
> >Happy to answer questions.
> >
> >Erik
> >
> >On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> >wrote:
> >
> >> Hi Tianyi,
> >>
> >> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
> >> on this project right now, I'll let him elaborate.
> >> Erik, you should probably respond on Apache dev list, as I think it
> could
> >> be interesting for other people as well, asince we planned to open
> source
> >> it. You can fork the "About 2.7.4 Release" thread with a new subject and
> >> give some details about Dynamometer there.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 <hetianyi@bytedance.com> wrote:
> >>
> >>> Hi, Shavachko.
> >>>
> >>> You mentioned an internal tool called Dynamometer to test NameNode
> >>> performance earlier in the 2.7.4 release thread.
> >>> I wonder if you could share some ideas behind the tool. Or is there a
> >>> plan to bring Dynamometer to open source community?
> >>>
> >>> Thanks.
> >>>
> >>> BR,
> >>> Tianyi
> >>>
> >>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi everybody.
> >>>>
> >>>> We have been doing some internal testing of Hadoop 2.7.4. The testing
> is
> >>>> going well.
> >>>> Did not find any major issues on our workloads.
> >>>> Used an internal tool called Dynamometer to check NameNode
> performance on
> >>>> real cluster traces. Good.
> >>>> Overall test cluster performance looks good.
> >>>> Some more testing is still going on.
> >>>>
> >>>> I plan to build an RC next week. If there are no objection.
> >>>>
> >>>> Thanks,
> >>>> --Konst
> >>>>
> >>>> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
> >>>> shv.hadoop@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Hey guys.
> >>>> >
> >>>> > An update on 2.7.4 progress.
> >>>> > We are down to 4 blockers. There is some work remaining on those.
> >>>> > https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> >>>> > Would be good if people could follow up on review comments.
> >>>> >
> >>>> > I looked through nightly Jenkins build results for 2.7.4 both on
> Apache
> >>>> > Jenkins and internal.
> >>>> > Some test fail intermittently, but there no consistent failures.
I
> >>>> filed
> >>>> > HDFS-11985 to track some of them.
> >>>> > https://issues.apache.org/jira/browse/HDFS-11985
> >>>> > I do not currently consider these failures as blockers. LMK if
some
> of
> >>>> > them are.
> >>>> >
> >>>> > We started internal testing of branch-2.7 on one of our smallish
> (100+
> >>>> > nodes) test clusters.
> >>>> > Will update on the results.
> >>>> >
> >>>> > There is a plan to enable BigTop for 2.7.4 testing.
> >>>> >
> >>>> > Akira, Brahma thank you for setting up a wiki page for 2.7.4
> release.
> >>>> > Thank you everybody for contributing to this effort.
> >>>> >
> >>>> > Regards,
> >>>> > --Konstantin
> >>>> >
> >>>> >
> >>>> > On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <
> aajisaka@apache.org>
> >>>> > wrote:
> >>>> >
> >>>> >> Sure.
> >>>> >> If you want to edit the wiki, please tell me your ASF confluence
> >>>> account.
> >>>> >>
> >>>> >> -Akira
> >>>> >>
> >>>> >> On 2017/05/30 15:31, Rohith Sharma K S wrote:
> >>>> >>
> >>>> >>> Couple of more JIRAs need to be back ported for 2.7.4 release.
> These
> >>>> will
> >>>> >>> solve RM HA unstability issues.
> >>>> >>> https://issues.apache.org/jira/browse/YARN-5333
> >>>> >>> https://issues.apache.org/jira/browse/YARN-5988
> >>>> >>> https://issues.apache.org/jira/browse/YARN-6304
> >>>> >>>
> >>>> >>> I will raise a JIRAs to back port it.
> >>>> >>>
> >>>> >>> @Akira , could  you help to add these JIRAs into wiki?
> >>>> >>>
> >>>> >>> Thanks & Regards
> >>>> >>> Rohith Sharma K S
> >>>> >>>
> >>>> >>> On 29 May 2017 at 12:19, Akira Ajisaka <aajisaka@apache.org>
> wrote:
> >>>> >>>
> >>>> >>> Created a page for 2.7.4 release.
> >>>> >>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
> >>>> >>>>
> >>>> >>>> If you want to edit this wiki, please ping me.
> >>>> >>>>
> >>>> >>>> Regards,
> >>>> >>>> Akira
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
> >>>> >>>>
> >>>> >>>> Hi Konstantin Shvachko
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> how about creating a wiki page for 2.7.4 release
status like 2.8
> >>>> and
> >>>> >>>>> trunk in following link.??
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> https://cwiki.apache.org/confluence/display/HADOOP
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> ________________________________
> >>>> >>>>> From: Konstantin Shvachko <shv.hadoop@gmail.com>
> >>>> >>>>> Sent: Saturday, May 13, 2017 3:58 AM
> >>>> >>>>> To: Akira Ajisaka
> >>>> >>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
> >>>> >>>>> yarn-dev@hadoop.apache.org
> >>>> >>>>> Subject: Re: About 2.7.4 Release
> >>>> >>>>>
> >>>> >>>>> Latest update on the links and filters. Here is
the correct link
> >>>> for
> >>>> >>>>> the
> >>>> >>>>> filter:
> >>>> >>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> >>>> >>>>> requestId=12340814
> >>>> >>>>>
> >>>> >>>>> Also updated: https://s.apache.org/Dzg4
> >>>> >>>>>
> >>>> >>>>> Had to do some Jira debugging. Sorry for confusion.
> >>>> >>>>>
> >>>> >>>>> Thanks,
> >>>> >>>>> --Konstantin
> >>>> >>>>>
> >>>> >>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko
<
> >>>> >>>>> shv.hadoop@gmail.com>
> >>>> >>>>> wrote:
> >>>> >>>>>
> >>>> >>>>> Hey Akira,
> >>>> >>>>>
> >>>> >>>>>>
> >>>> >>>>>> I didn't have private filters. Most probably
Jira caches
> >>>> something.
> >>>> >>>>>> Your filter is in the right direction, but
for some reason it
> >>>> lists
> >>>> >>>>>> only
> >>>> >>>>>> 22 issues, while mine has 29.
> >>>> >>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
> >>>> >>>>>> a/browse/YARN-5543>
> >>>> >>>>>> .
> >>>> >>>>>>
> >>>> >>>>>> Anyways, I created a Jira filter now "Hadoop
2.7.4 release
> >>>> blockers",
> >>>> >>>>>> shared it with "everybody", and updated my
link to point to
> that
> >>>> >>>>>> filter.
> >>>> >>>>>> So
> >>>> >>>>>> you can use any of the three methods below
to get the correct
> >>>> list:
> >>>> >>>>>> 1. Go to https://s.apache.org/Dzg4
> >>>> >>>>>> 2. Go to the filter via
> >>>> >>>>>>     https://issues.apache.org/jira/issues?filter=12340814
> >>>> >>>>>>    or by finding "Hadoop 2.7.4 release blockers"
filter in the
> >>>> jira
> >>>> >>>>>> 3. On Advanced issues search page paste this:
> >>>> >>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE)
AND labels =
> >>>> >>>>>> release-blocker
> >>>> >>>>>> AND "Target Version/s" = 2.7.4
> >>>> >>>>>>
> >>>> >>>>>> Hope this solves the confusion for which issues
are included.
> >>>> >>>>>> Please LMK if it doesn't, as it is important.
> >>>> >>>>>>
> >>>> >>>>>> Thanks,
> >>>> >>>>>> --Konstantin
> >>>> >>>>>>
> >>>> >>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka
<
> >>>> aajisaka@apache.org>
> >>>> >>>>>> wrote:
> >>>> >>>>>>
> >>>> >>>>>> Hi Konstantin,
> >>>> >>>>>>
> >>>> >>>>>>>
> >>>> >>>>>>> Thank you for volunteering as release manager!
> >>>> >>>>>>>
> >>>> >>>>>>> Actually the original link works fine:
> https://s.apache.org/Dzg4
> >>>> >>>>>>>
> >>>> >>>>>>>>
> >>>> >>>>>>>> I couldn't see the link. Maybe is it
private filter?
> >>>> >>>>>>>
> >>>> >>>>>>> Here is a link I generated: https://s.apache.org/ehKy
> >>>> >>>>>>> This filter includes resolved issue and
excludes fixversion ==
> >>>> 2.7.4
> >>>> >>>>>>>
> >>>> >>>>>>> Thanks and Regards,
> >>>> >>>>>>> Akira
> >>>> >>>>>>>
> >>>> >>>>>>> On 2017/05/08 19:20, Konstantin Shvachko
wrote:
> >>>> >>>>>>>
> >>>> >>>>>>> Hi Brahma Reddy Battula,
> >>>> >>>>>>>
> >>>> >>>>>>>>
> >>>> >>>>>>>> Actually the original link works fine:
> >>>> https://s.apache.org/Dzg4
> >>>> >>>>>>>> Your link excludes closed and resolved
issues, which needs
> >>>> >>>>>>>> backporting,
> >>>> >>>>>>>> and
> >>>> >>>>>>>> which we cannot reopen, as discussed
in this thread earlier.
> >>>> >>>>>>>>
> >>>> >>>>>>>> Looked through the issues you proposed:
> >>>> >>>>>>>>
> >>>> >>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
> >>>> >>>>>>>> Seems like a new feature. It helps
failover to standby node
> when
> >>>> >>>>>>>> primary
> >>>> >>>>>>>> is
> >>>> >>>>>>>> under heavy load, but it introduces
new APIs, addresses,
> config
> >>>> >>>>>>>> parameters.
> >>>> >>>>>>>> And needs at least one follow up jira.
> >>>> >>>>>>>> Looks like a backward compatible change,
though.
> >>>> >>>>>>>> Did you have a chance to run it in
production?
> >>>> >>>>>>>>
> >>>> >>>>>>>> +1 on
> >>>> >>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987
> >
> >>>> >>>>>>>>
> >>>> >>>>>>>> [HDFS-10987] Make Decommission less
expensive when lot of
> ...<
> >>>> >>>>>>>
> >>>> >>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
> >>>> >>>>> issues.apache.org
> >>>> >>>>> When user want to decommission a node which having
50M blocks
> ,it
> >>>> could
> >>>> >>>>> hold the namesystem lock for long time.We've seen
it is taking
> 36
> >>>> sec.
> >>>> >>>>> As
> >>>> >>>>> we knew during this ...
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
> >>>> >>>>>
> >>>> >>>>>>
> >>>> >>>>>>>> [HDFS-9902] Support different values
of dfs.datanode.du ...<
> >>>> >>>>>>>
> >>>> >>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
> >>>> >>>>> issues.apache.org
> >>>> >>>>> Now Hadoop support different storage type for DISK,
SSD, ARCHIVE
> >>>> and
> >>>> >>>>> RAM_DISK, but they share one configuration
> >>>> dfs.datanode.du.reserved.
> >>>> >>>>> The
> >>>> >>>>> DISK size may be several ...
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
> >>>> >>>>>
> >>>> >>>>>>
> >>>> >>>>>>>> Trash does not descent into child directories
to check for
> ...<
> >>>> >>>>>>>
> >>>> >>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
> >>>> >>>>> issues.apache.org
> >>>> >>>>> HDFS trash does not descent into child directory
to check if
> user
> >>>> has
> >>>> >>>>> permission to delete files. For example: Run the
following
> command
> >>>> to
> >>>> >>>>> initialize directory ...
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> HADOOP-14100 <https://issues.apache.org/
> jira/browse/HADOOP-14100>
> >>>> >>>>>
> >>>> >>>>>>
> >>>> >>>>>>>> Upgrade Jsch jar to latest version
to fix vulnerability in
> ...<
> >>>> >>>>>>>
> >>>> >>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
> >>>> >>>>> issues.apache.org
> >>>> >>>>> Recently there was on vulnerability reported on
jsch library.
> Its
> >>>> fixed
> >>>> >>>>> in latest 0.1.54 version before CVE was made public.
> >>>> >>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>>
> >>>> >>>>> Added them to 2.7.4 release. You should see them
via the above
> link
> >>>> >>>>>>>> now.
> >>>> >>>>>>>> Would be good if you could attach backport
patches for some
> of
> >>>> them?
> >>>> >>>>>>>>
> >>>> >>>>>>>> Appreciate your help,
> >>>> >>>>>>>> --Konstantin
> >>>> >>>>>>>>
> >>>> >>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma
Reddy Battula <
> >>>> >>>>>>>> brahmareddy.battula@huawei.com>
wrote:
> >>>> >>>>>>>>
> >>>> >>>>>>>>
> >>>> >>>>>>>> Looks following link is not correct..
> >>>> >>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> https://s.apache.org/Dzg4
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> It should be like following..?
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> https://s.apache.org/wi3U
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Apart from Konstantin mentioned,Following
also good to go..?
> >>>> let me
> >>>> >>>>>>>>> know
> >>>> >>>>>>>>> your thoughts on this.
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> For Large Cluster:
> >>>> >>>>>>>>> =============
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life
Line
> >>>> >>>>>>>>> Protocol
> >>>> >>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===
> Deecommis
> >>>> sion
> >>>> >>>>>>>>> Expensive when lot's of blocks
are present
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
> >>>> >>>>>>>>> "dfs.datanode.du.reserved"  per
Storage Type
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> For Security:
> >>>> >>>>>>>>> =========
> >>>> >>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash
> does
> >>>> not
> >>>> >>>>>>>>> descent
> >>>> >>>>>>>>> into child directories to check
for permission
> >>>> >>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===
> Upgrade
> >>>> Jsch
> >>>> >>>>>>>>> jar
> >>>> >>>>>>>>> to
> >>>> >>>>>>>>> latest version to fix vulnerability
in old versions
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Regards
> >>>> >>>>>>>>> Brahma Reddy Battula
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> -----Original Message-----
> >>>> >>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
> >>>> >>>>>>>>> Sent: 06 May 2017 02:40
> >>>> >>>>>>>>> To: Konstantin Shvachko
> >>>> >>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
> >>>> >>>>>>>>> mapreduce-dev@hadoop.apache.org
> >>>> >>>>>>>>> ;
> >>>> >>>>>>>>> yarn-dev@hadoop.apache.org
> >>>> >>>>>>>>> Subject: Re: About 2.7.4 Release
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> List LGTM Konstantin!
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Let's say that we will only create
a new tracking JIRA for
> >>>> patches
> >>>> >>>>>>>>> which
> >>>> >>>>>>>>> do not backport cleanly, to avoid
having too many lying
> around.
> >>>> >>>>>>>>> Otherwise
> >>>> >>>>>>>>> we can directly attach to old ticket.
If a clean backport
> does
> >>>> >>>>>>>>> happen
> >>>> >>>>>>>>> to
> >>>> >>>>>>>>> break a test the nightly build
will help us catch it.
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Erik
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> On Thu, May 4, 2017 at 7:21 PM,
Konstantin Shvachko <
> >>>> >>>>>>>>> shv.hadoop@gmail.com>
> >>>> >>>>>>>>> wrote:
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Great Zhe. Let's monitor the build.
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>> I marked all jiras I knew of
for inclusion into 2.7.4 as I
> >>>> >>>>>>>>>> described
> >>>> >>>>>>>>>> before.
> >>>> >>>>>>>>>> Target Version/s: 2.7.4
> >>>> >>>>>>>>>> Label: release-blocker
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>> Here is the link to the list:
https://s.apache.org/Dzg4
> >>>> Please
> >>>> >>>>>>>>>> let
> >>>> >>>>>>>>>> me
> >>>> >>>>>>>>>> know if I missed anything.
> >>>> >>>>>>>>>> And feel free to pick up any.
Most of backports are pretty
> >>>> >>>>>>>>>> straightforward, but not all.
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>> We can create tracking jiras
for backporting if you need to
> >>>> run
> >>>> >>>>>>>>>> Jenkins on the patch (and since
Allen does not allow
> reopening
> >>>> >>>>>>>>>> them).
> >>>> >>>>>>>>>> But I think the final patch
should be attached to the
> original
> >>>> >>>>>>>>>> jira.
> >>>> >>>>>>>>>> Otherwise history will be hard
to follow.
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>> Thanks,
> >>>> >>>>>>>>>> --Konstantin
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>> On Wed, May 3, 2017 at 4:53
PM, Zhe Zhang <zhz@apache.org>
> >>>> wrote:
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>> Thanks for volunteering as
RM Konstantin! The plan LGTM.
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>> I've created a nightly
Jenkins job for branch-2.7 (unit
> >>>> tests):
> >>>> >>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> On Wed, May 3, 2017 at
12:42 AM Konstantin Shvachko <
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> shv.hadoop@gmail.com>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>> wrote:
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> Hey guys,
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>> I and a few of my colleagues
would like to help here and
> >>>> move
> >>>> >>>>>>>>>>>> 2.7.4
> >>>> >>>>>>>>>>>> release forward. A
few points in this regard.
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> 1. Reading through
this thread since March 1 I see that
> >>>> Vinod
> >>>> >>>>>>>>>>>> hinted on managing
the release. Vinod, if you still want
> the
> >>>> >>>>>>>>>>>> job /
> >>>> >>>>>>>>>>>> have bandwidth will
be happy to work with you.
> >>>> >>>>>>>>>>>> Otherwise I am glad
to volunteer as the release manager.
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> 2. In addition to current
blockers and criticals, I would
> >>>> like
> >>>> >>>>>>>>>>>> to
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> propose
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>> a
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> few issues to be included
in the release, see the list
> below.
> >>>> >>>>>>>>>>>> Those
> >>>> >>>>>>>>>>>> are mostly bug fixes
and optimizations, which we already
> >>>> have in
> >>>> >>>>>>>>>>>> our
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> internal
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>> branch and run in production.
Plus one minor feature "node
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> labeling", which we found
very handy, when you have
> >>>> heterogeneous
> >>>> >>>>>>>>>>>> environments and mixed
workloads, like MR and Spark.
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> 3. For marking issues
for the release I propose to
> >>>> >>>>>>>>>>>>  - set the target version
to 2.7.4, and
> >>>> >>>>>>>>>>>>  - add a new label
"release-blocker"
> >>>> >>>>>>>>>>>> That way we will know
issues targeted for the release
> >>>> without
> >>>> >>>>>>>>>>>> reopening them for
backports.
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> 4. I see quite a few
people are interested in the
> release.
> >>>> With
> >>>> >>>>>>>>>>>> all
> >>>> >>>>>>>>>>>> the help I think we
can target to release by the end of
> May.
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> Other things include
fixing CHANGES.txt and fixing
> Jenkins
> >>>> build
> >>>> >>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> 2.7.4
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>> branch.
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>> Thanks,
> >>>> >>>>>>>>>>>> --Konstantin
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> ==========  List of
issue for 2.7.4  ===========
> >>>> >>>>>>>>>>>> ------ Backports
> >>>> >>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
> >>>> >>>>>>>>>>>> a/browse/HADOOP-12975>.
> >>>> >>>>>>>>>>>> Add
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> du
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>> jitters
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> HDFS-9710 <https://issues.apache.org/
> jira/browse/HDFS-9710>.
> >>>> IBR
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> batching
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>> HDFS-10715 <https://issues.apache.org/
> jira/browse/HDFS-10715
> >>>> >.
> >>>> >>>>>>>>>> NPE
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
> >>>> >>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/
> jira/browse/HDFS-2538
> >>>> >.
> >>>> >>>>>>>>>>>> fsck
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> removal
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>> of dot printing
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> HDFS-8131 <https://issues.apache.org/
> jira/browse/HDFS-8131>.
> >>>> >>>>>>>>>>>> space-balanced
> >>>> >>>>>>>>>>>> policy for balancer
> >>>> >>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/
> jira/browse/HDFS-8549
> >>>> >.
> >>>> >>>>>>>>>>>> abort
> >>>> >>>>>>>>>>>> balancer if upgrade
in progress
> >>>> >>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/
> jira/browse/HDFS-9412
> >>>> >.
> >>>> >>>>>>>>>>>> skip
> >>>> >>>>>>>>>>>> small blocks in getBlocks
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> YARN-1471 <https://issues.apache.org/
> jira/browse/YARN-1471
> >>>> >.
> >>>> >>>>>>>>>>>> SLS
> >>>> >>>>>>>>>>>> simulator
> >>>> >>>>>>>>>>>> YARN-4302 <https://issues.apache.org/
> jira/browse/YARN-4302
> >>>> >.
> >>>> >>>>>>>>>>>> SLS
> >>>> >>>>>>>>>>>> YARN-4367 <https://issues.apache.org/
> jira/browse/YARN-4367
> >>>> >.
> >>>> >>>>>>>>>>>> SLS
> >>>> >>>>>>>>>>>> YARN-4612 <https://issues.apache.org/
> jira/browse/YARN-4612
> >>>> >.
> >>>> >>>>>>>>>>>> SLS
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> ----- Node labeling
> >>>> >>>>>>>>>>>> MAPREDUCE-6304
> >>>> >>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> >>>> >>>>>>>>>>>> YARN-2943 <https://issues.apache.org/
> jira/browse/YARN-2943>
> >>>> >>>>>>>>>>>> YARN-4109 <https://issues.apache.org/
> jira/browse/YARN-4109>
> >>>> >>>>>>>>>>>> YARN-4140 <https://issues.apache.org/
> jira/browse/YARN-4140>
> >>>> >>>>>>>>>>>> YARN-4250 <https://issues.apache.org/
> jira/browse/YARN-4250>
> >>>> >>>>>>>>>>>> YARN-4925 <https://issues.apache.org/
> jira/browse/YARN-4925>
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> --
> >>>> >>>>>>>>>>>>
> >>>> >>>>>>>>>>>> Zhe Zhang
> >>>> >>>>>>>>>>> Apache Hadoop Committer
> >>>> >>>>>>>>>>> http://zhe-thoughts.github.io/about/
| @oldcap
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>
> >>>> >>>>>>
> >>>> >>>>> ------------------------------------------------------------
> >>>> ---------
> >>>> >>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>>> >>>> For additional commands, e-mail: common-dev-help@hadoop.apache.
> org
> >>>> >>>>
> >>>> >>>>
> >>>> >>>>
> >>>> >>>
> >>>> >
> >>>>
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message