phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: 4.7.0 RC, Bulk loading performance degradation and failed MR tasks
Date Fri, 26 Feb 2016 22:37:26 GMT
Thanks, Sergey. Were you able to confirm whether or not PHOENIX-1973 was
the root cause of the regression?

On Fri, Feb 26, 2016 at 11:55 AM, Sergey Soldatov <sergeysoldatov@gmail.com>
wrote:

> Oops. Small update. We can revert  PHOENIX-1973 (bulk load
> improvement), not PHOENIX-2649 (TableRowKeyPair comparator problem).
>
> On Fri, Feb 26, 2016 at 10:52 AM, Sergey Soldatov
> <sergeysoldatov@gmail.com> wrote:
> > Well, that's how MR bulk load works. Mappers gets all rows from the
> > file and create the corresponding pairs <rowkey, column value>. MR
> > engine sorts this stuff by rowkey and reducer sort it by value and put
> > it to the hfile. After that HBase bulkload loads it into HBase.
> > PHOENIX-2649 is just reduce the amount of data that is sending between
> > mappers and reducer. Before that was N rows * K columns after it
> > becomes N only.  Because of the bug I mentioned before the phase when
> > stuff is sorted by rowkey didn't work at all (first place why
> > performance of 4.6 was better) and all values were written with a
> > single rowkey and were received all at once by reducer (second place)
> > and during the HBase bulkload there were no reason for splitting
> > because of the same rowkey (third place).
> > But of course we can reverse PHOENIX-2649 and see whether it helps.
> >
> > Thanks,
> > Sergey
> >
> > On Fri, Feb 26, 2016 at 9:02 AM, 김영우 (Youngwoo Kim) <warwithin@gmail.com>
> wrote:
> >> Exactly! Gabriel describes the fact that I observed.
> >>
> >> Many map and reduce tasks are launched but one or two tasks are running
> at
> >> the end of the job. it  looks like the work loads are skwed on
> particular
> >> task.
> >>
> >> Thanks,
> >> Youngwoo
> >>
> >> 2016년 2월 26일 금요일, Gabriel Reid<gabriel.reid@gmail.com>님이
작성한 메시지:
> >>
> >>> I just did a quick test run on this, and it looks to me like something
> >>> is definitely wrong.
> >>>
> >>> I ran a simple ingest test for a table with 5 regions, and it appears
> >>> that only a single HFile is being created. This HFile then needs to be
> >>> recursively split during the step of handing HFiles over to the region
> >>> servers (hence the "xxx no longer fits inside a single region.
> >>> Splitting..." log messages).
> >>>
> >>> This implies that only a single reducer is actually doing any
> >>> processing, which would certainly account for a performance
> >>> degradation. My assumption is that the underlying issue is in the
> >>> partitioner (or the data being passed to the partitioner). I don't
> >>> know if this was introduced as part of PHOENIX-2649 or not.
> >>>
> >>> Sergey, are you (or someone else) able to take a look at this?
> >>> Unfortunately, I don't think there's any way I can get a serious look
> >>> at this any more today.
> >>>
> >>> - Gabriel
> >>>
> >>>
> >>> On Fri, Feb 26, 2016 at 11:21 AM, Sergey Soldatov
> >>> <sergeysoldatov@gmail.com <javascript:;>> wrote:
> >>> > I see. We will try to reproduce it. The degradation is possible
> >>> > because 4.6 had a problem described in PHOENIX-2649. In two words -
> >>> > the comparator for rowkeys was working incorrectly and reported that
> >>> > all rowkeys are the same. If the input files are relatively small and
> >>> > reducer has enough memory, all records will be written in one step
> >>> > with the same single rowkey. And that can be the reason why it was
> >>> > faster and there were no splits.
> >>> >
> >>> > Thanks,
> >>> > Sergey
> >>> >
> >>> > On Fri, Feb 26, 2016 at 1:37 AM, 김영우 (YoungWoo Kim) <
> ywkim@apache.org
> >>> <javascript:;>> wrote:
> >>> >> Sergey,
> >>> >>
> >>> >> I can't access the cluster right now, so I'll post details and
> >>> >> configurations next week. important facts as far as I remember:
> >>> >> - 8 nodes dev cluster (Hadoop 2.7.1, HBase 1.1.3, Phoenix 4.7.0
RC2
> and
> >>> >> Zookeeper 3.4.6)
> >>> >>  * 32 core / 256 GB RAM, Datanode/Nodemanager and RegionServer
@
> same
> >>> node,
> >>> >> Assigned 24 GB for heap for region server
> >>> >> - # of tables = 9
> >>> >> - Salted with 5, 10 or 20 buckets
> >>> >> - Compressed using Snappy codec
> >>> >> - Data Ingestion : 30 ~ 40 GB / day using bulk loading
> >>> >> - Schema
> >>> >>   The table that I mentioned has 10 columns and 7 columns are
> varchar
> >>> and
> >>> >> the rest are varchar[].
> >>> >>   I can see performance degradation on bulk load from other tables
> >>> >>
> >>> >> Thanks,
> >>> >>
> >>> >> Youngwoo
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Fri, Feb 26, 2016 at 6:02 PM, Sergey Soldatov <
> >>> sergeysoldatov@gmail.com <javascript:;>>
> >>> >> wrote:
> >>> >>
> >>> >>> Hi Youngwoo,
> >>> >>> Could you provide a bit more information about the table structure
> >>> >>> (DDL would be great)? Do you have indexes?
> >>> >>>
> >>> >>> Thanks,
> >>> >>> Sergey
> >>> >>>
> >>> >>> On Tue, Feb 23, 2016 at 10:18 PM, 김영우 (Youngwoo Kim)
> >>> >>> <warwithin@gmail.com <javascript:;>> wrote:
> >>> >>> > Gabriel,
> >>> >>> >
> >>> >>> > I'm using RC2.
> >>> >>> >
> >>> >>> > Youngwoo
> >>> >>> >
> >>> >>> > 2016년 2월 24일 수요일, Gabriel Reid<gabriel.reid@gmail.com
> <javascript:;>>님이
> >>> 작성한 메시지:
> >>> >>> >
> >>> >>> >> Hi Youngwoo,
> >>> >>> >>
> >>> >>> >> Which RC are you using for this? RC-1 or RC-2?
> >>> >>> >>
> >>> >>> >> Thanks,
> >>> >>> >>
> >>> >>> >> Gabriel
> >>> >>> >>
> >>> >>> >> On Tue, Feb 23, 2016 at 11:30 AM, 김영우 (YoungWoo
Kim) <
> >>> ywkim@apache.org <javascript:;>
> >>> >>> >> <javascript:;>> wrote:
> >>> >>> >> > Hi,
> >>> >>> >> >
> >>> >>> >> > I'm evaluating 4.7.0 RC on my dev cluster. Looks
like it works
> >>> fine
> >>> >>> but I
> >>> >>> >> > run into performance degradation for MR based
bulk loading.
> I've
> >>> been
> >>> >>> >> > loading a million of rows per day into Phoenix
table. From
> 4.7.0
> >>> RC,
> >>> >>> >> there
> >>> >>> >> > are failed jobs with '600 sec' time out in map
or reduce
> stage.
> >>> logs
> >>> >>> as
> >>> >>> >> > follows:
> >>> >>> >> >
> >>> >>> >> > 16/02/22 18:03:45 INFO mapreduce.Job: Task Id
:
> >>> >>> >> > attempt_1456035298774_0066_m_000002_0, Status
: FAILED
> >>> >>> >> > AttemptID:attempt_1456035298774_0066_m_000002_0
Timed out
> after
> >>> 600
> >>> >>> secs
> >>> >>> >> >
> >>> >>> >> > 16/02/22 18:05:14 INFO mapreduce.LoadIncrementalHFiles:
HFile
> at
> >>> >>> >> >
> >>> >>> >>
> >>> >>>
> >>>
> hdfs://fcbig/tmp/74da7ab1-a8ac-4ba8-9d43-0b70f08f8602/HYNIX.BIG_TRACE_SUMMARY/0/_tmp/_tmp/f305427aa8304cf98355bf01c1edb5ce.top
> >>> >>> >> > no longer fits inside a single region. Splitting...
> >>> >>> >> >
> >>> >>> >> > But, the logs have not seen before. so I'm facing
about 5 ~
> 10x
> >>> >>> >> performance
> >>> >>> >> > degradation for bulk loading. (4.6.0: 10min but
60+ min from
> >>> 4.7.0 RC)
> >>> >>> >> > furthermore, I can't find a clue from MR logs
why the tasks
> filed.
> >>> >>> >> >
> >>> >>> >> > And, I can see the hfile splitting after reduce
stage. Is it
> >>> normal?
> >>> >>> >> >
> >>> >>> >> > My envs are:
> >>> >>> >> > - Hadoop 2.7.1
> >>> >>> >> > - HBase 1.1.3
> >>> >>> >> > - Phoenix 4.7.0 RC
> >>> >>> >> >
> >>> >>> >> > Thanks,
> >>> >>> >> >
> >>> >>> >> > Youngwoo
> >>> >>> >>
> >>> >>>
> >>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message