Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 170CB20049C for ; Fri, 11 Aug 2017 14:14:01 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1583B16D2B6; Fri, 11 Aug 2017 12:14:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B116616D2B5 for ; Fri, 11 Aug 2017 14:13:59 +0200 (CEST) Received: (qmail 43823 invoked by uid 500); 11 Aug 2017 12:13:58 -0000 Mailing-List: contact user-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kylin.apache.org Delivered-To: mailing list user@kylin.apache.org Received: (qmail 43811 invoked by uid 99); 11 Aug 2017 12:13:58 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Aug 2017 12:13:58 +0000 Received: from mail-pf0-f181.google.com (mail-pf0-f181.google.com [209.85.192.181]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 565A51A0019 for ; Fri, 11 Aug 2017 12:13:57 +0000 (UTC) Received: by mail-pf0-f181.google.com with SMTP id t86so15226969pfe.2 for ; Fri, 11 Aug 2017 05:13:57 -0700 (PDT) X-Gm-Message-State: AHYfb5gHjxNRs2OvYSdJTp1xZ62JegakUItnVf/EZbXeXE1JaYxf6Pxp eXMRndNhejo4Ar+GB5yKLzlO/0SS2Q== X-Received: by 10.84.129.132 with SMTP id b4mr17320036plb.294.1502453636498; Fri, 11 Aug 2017 05:13:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.168.66 with HTTP; Fri, 11 Aug 2017 05:13:15 -0700 (PDT) In-Reply-To: References: <6B00168F-5524-460A-B799-1FF5D48B24DD@joom.it> From: ShaoFeng Shi Date: Fri, 11 Aug 2017 20:13:15 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: HFile is empty if kylin.hbase.cluster.fs is set to s3 To: user Content-Type: multipart/alternative; boundary="94eb2c11a66e4b8e7e0556793f98" archived-at: Fri, 11 Aug 2017 12:14:01 -0000 --94eb2c11a66e4b8e7e0556793f98 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable EMR enables the direct output in mapred-site.xml, while in this step it seems these settings doesn't work (althoug the job's configuration shows they are there). I disabled the direct output but the behavior has no change. I did some search but no finding. I need drop the EMR now, and may get back it later. If you have any idea or findings, please share it. We'd like to make Kylin has better support for cloud. Thanks for your feedback! 2017-08-11 19:19 GMT+08:00 Alexander Sterligov : > Any ideas how to fix that? > > On Fri, Aug 11, 2017 at 2:16 PM, ShaoFeng Shi > wrote: > >> I got the same problem as you: >> >> 2017-08-11 08:44:16,342 WARN [Job 2c86b4b6-7639-4a97-ba63-63c9dca095f6-= 2255] >> mapreduce.LoadIncrementalHFiles:422 : Bulk load operation did not find >> any files to load in directory s3://privatekeybucket-anac5h41 >> 523l/kylin/kylin_default_instance/kylin-2c86b4b6-7639- >> 4a97-ba63-63c9dca095f6/kylin_sales_cube_clone3/hfile. Does it contain >> files in subdirectories that correspond to column family names? >> >> In S3 view, I see the files exist in "_temporary" folder, seems were not >> moved to the target folder on complete. It seems EMR try to direct write= to >> otuput path, but actually not. >> >> 2017-08-11 16:34 GMT+08:00 Alexander Sterligov : >> >>> No, defaultFs is hdfs. >>> >>> I=E2=80=99ve seen such behavior when set working dir to s3, but didn=E2= =80=99t set >>> cluster-fs at all. Maybe you have a typo in the name of the property. I >>> used the old one =C2=ABkylin.hbase.cluster.fs=C2=BB >>> >>> When both working-dir and cluster-fs were set to s3 I got _temporary di= r >>> of convert job at s3, but no hfiles. Also I saw correct output path for= the >>> job in the log. But I didn=E2=80=99t check if job creates temporary fil= es in s3, >>> but then copies results to hdfs. I hardly believe it happens. >>> >>> Do you see proper arguments for the step in the log? >>> >>> >>> 11 =D0=B0=D0=B2=D0=B3. 2017 =D0=B3., =D0=B2 11:17, ShaoFeng Shi >>> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0): >>> >>> Hi Alexander, >>> >>> That makes sense. Using S3 for Cube build and storage is required for a >>> cloud hadoop environment. >>> >>> I tried to reproduce this problem. I created a EMR with S3 as HBase >>> storage, in kylin.properties, I set "kylin.env.hdfs-working-dir" >>> and "kylin.storage.hbase.cluster-fs" to the S3 bucket. But in the "Conv= ert >>> Cuboid Data to HFile" step, Kylin still writes to local HDFS; Did you >>> modify the core-site.xml to make S3 as the default FS? >>> >>> >>> >>> >>> 2017-08-10 22:53 GMT+08:00 Alexander Sterligov : >>> >>>> Yes, I workarounded this problem in such way and it works. >>>> >>>> One problem of such solution is that I have to use pretty large hdfs >>>> and it'expensive. And also I have to manually garbage collect it, beca= use >>>> it is not moved to s3, but copied. Kylin cleanup job doesn't work for = it, >>>> because main metadata folder is at s3. So it would be really nice to p= ut >>>> everything to s3. >>>> >>>> Another problem is that I had to rise hbase rpc timeout, because bulk >>>> loading from hdfs takes long. That was not trivial. 3 minutes work goo= d, >>>> but with drawback of queries or metadata writes handing for 3 minutes = if >>>> something bad happen. But that's rare event. >>>> >>>> 10 =D0=B0=D0=B2=D0=B3. 2017 =D0=B3. 17:42 =D0=BF=D0=BE=D0=BB=D1=8C=D0= =B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C "ShaoFeng Shi" < >>>> shaofengshi@apache.org> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: >>>> >>>> How about leaving empty for "kylin.hbase.cluster.fs"? This property is >>>>> for two-cluster deployment (one Hadoop for cube build, the other for >>>>> query); >>>>> >>>>> When be empty, the HFile will be written to default fs (HDFS in EMR), >>>>> and then load to HBase. I'm not sure whether EMR HBase (using S3 as >>>>> storage) can bulk load files from HDFS or not. If it can, that would = be >>>>> great as the write performance of HDFS would be better than S3. >>>>> >>>>> 2017-08-10 22:29 GMT+08:00 Alexander Sterligov : >>>>> >>>>>> I also thought about it, but no, it's not consistency. >>>>>> >>>>>> Consistency view is enabled. I use same s3 for my own map-reduce job= s >>>>>> and it's ok. >>>>>> >>>>>> I also checked if it lost consistency (emrfs diff). No problems. >>>>>> >>>>>> In case of inconsistency of s3 files disappear right after they were >>>>>> written and appear some time after. Hfiles didn't appear after a day= , but >>>>>> _template is there. >>>>>> >>>>>> It's 100% reproducable, I think I'll investigate this problem by >>>>>> running conversion job manually. >>>>>> >>>>>> 10 =D0=B0=D0=B2=D0=B3. 2017 =D0=B3. 17:18 =D0=BF=D0=BE=D0=BB=D1=8C= =D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C "ShaoFeng Shi" < >>>>>> shaofengshi@apache.org> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: >>>>>> >>>>>> Did you enable the Consistent View? This article explains the >>>>>>> challenge when using S3 directly for ETL process: >>>>>>> https://aws.amazon.com/cn/blogs/big-data/ensuring-consistenc >>>>>>> y-when-using-amazon-s3-and-amazon-elastic-mapreduce-for-etl- >>>>>>> workflows/ >>>>>>> >>>>>>> >>>>>>> 2017-08-09 18:19 GMT+08:00 Alexander Sterligov >>>>>>> : >>>>>>> >>>>>>>> Yes, it's empty. Also I see this message in the log: >>>>>>>> >>>>>>>> 2017-08-09 09:02:35,947 WARN [Job 1e436685-7102-4621-a4cb-6472b86= 6126d-7608] >>>>>>>> mapreduce.LoadIncrementalHFiles:234 : Skipping non-directory >>>>>>>> s3://joom.emr.fs/home/production/bi/kylin/kylin_metadata/kyl >>>>>>>> in-1e436685-7102-4621-a4cb-6472b866126d >>>>>>>> /main_event_1_main/hfile/_SUCCESS >>>>>>>> 2017-08-09 09:02:36,009 WARN [Job 1e436685-7102-4621-a4cb-6472b86= 6126d-7608] >>>>>>>> mapreduce.LoadIncrementalHFiles:252 : Skipping non-file >>>>>>>> FileStatusExt{path=3Ds3://joom.emr.fs/home/production/bi/kylin >>>>>>>> /kylin_metadata/kylin-1e436685-7102-4621-a4cb-6472b866126d/m >>>>>>>> ain_event_1_main/hfile/_temporary/1; isDirectory=3Dtrue; >>>>>>>> modification_time=3D0; access_time=3D0; owner=3D; group=3D; permis= sion=3Drwxrwxrwx; >>>>>>>> isSymlink=3Dfalse} >>>>>>>> 2017-08-09 09:02:36,014 WARN [Job 1e436685-7102-4621-a4cb-6472b86= 6126d-7608] >>>>>>>> mapreduce.LoadIncrementalHFiles:422 : Bulk load operation did not >>>>>>>> find any files to load in directory s3://joom.emr.fs/home/producti >>>>>>>> on/bi/kylin/kylin_metadata/kylin-1e436685-7102-4621-a4cb-647 >>>>>>>> 2b866126d/main_event_1_main/hfile. Does it contain files in >>>>>>>> subdirectories that correspond to column family names? >>>>>>>> >>>>>>>> On Wed, Aug 9, 2017 at 1:15 PM, ShaoFeng Shi < >>>>>>>> shaofengshi@apache.org> wrote: >>>>>>>> >>>>>>>>> The HFile will be moved to HBase data folder when bulk load >>>>>>>>> finished; Did you check whether the HTable has data? >>>>>>>>> >>>>>>>>> 2017-08-09 17:54 GMT+08:00 Alexander Sterligov < >>>>>>>>> sterligovak@joom.it>: >>>>>>>>> >>>>>>>>>> Hi! >>>>>>>>>> >>>>>>>>>> I set kylin.hbase.cluster.fs to s3 bucket where hbase lives. >>>>>>>>>> >>>>>>>>>> Step "Convert Cuboid Data to HFile" finished without errors. >>>>>>>>>> Statistics at the end of the job said that it has written lot's = of data to >>>>>>>>>> s3. >>>>>>>>>> >>>>>>>>>> But there is no hfiles in kylin_metadata folder (kylin_metadata >>>>>>>>>> /kylin-1e436685-7102-4621-a4cb-6472b866126d//hfile), >>>>>>>>>> but only _temporary folder and _SUCCESS file. >>>>>>>>>> >>>>>>>>>> _temporary contains hfiles inside attempt folders. it looks like >>>>>>>>>> there were not copied from _temporary to result dir. But there i= s no errors >>>>>>>>>> neither in kylin log, nor in reducers' logs. >>>>>>>>>> >>>>>>>>>> Then loading empty hfiles produces empty segments. >>>>>>>>>> >>>>>>>>>> Is that a bug or I'm doing something wrong? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best regards, >>>>>>> >>>>>>> Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B >>>>>>> >>>>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> >>>>> Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B >>>>> >>>>> >>> >>> >>> -- >>> Best regards, >>> >>> Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B >>> >>> >>> >> >> >> -- >> Best regards, >> >> Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B >> >> > --=20 Best regards, Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B --94eb2c11a66e4b8e7e0556793f98 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
EMR enables the direct output in mapred-site.xml, while in= this step it seems these settings doesn't work (althoug the job's = configuration shows they are there). I disabled the direct output but the b= ehavior has no change. I did some search but no finding. I need drop the EM= R now, and may get back it later.=C2=A0

If you have any = idea or findings, please share it. We'd like to make Kylin has better s= upport for cloud.=C2=A0

Thanks for your feedback!<= /div>

2017-0= 8-11 19:19 GMT+08:00 Alexander Sterligov <sterligovak@joom.it>:
Any ideas how to f= ix that?

On Fri, Aug 11, 2017 at 2:16 PM, ShaoF= eng Shi <shaofengshi@apache.org> wrote:
I got the same problem as you:

=
2017-08-11 08:44:16,342 WARN =C2=A0[Job 2c86b4b6-7639-4a97-ba63-= 63c9dca095f6-2255] mapreduce.LoadIncrementalHFiles:422 : Bulk loa= d operation did not find any files to load in directory s3://privatekeybuck= et-anac5h41523l/kylin/kylin_default_instance/kylin-2c86b4b6-7639-= 4a97-ba63-63c9dca095f6/kylin_sales_cube_clone3/hfile.=C2=A0 Does = it contain files in subdirectories that correspond to column family names?<= br>

In S3 view, I see the files exist in "_te= mporary" folder, seems were not moved to the target folder on complete= . It seems EMR try to direct write to otuput path, but actually not.
<= /div>

2017-08= -11 16:34 GMT+08:00 Alexander Sterligov <sterligovak@joom.it>:
= No, defaultFs is hdfs.

I=E2=80=99ve seen such behavior w= hen set working dir to s3, but didn=E2=80=99t set cluster-fs at all. Maybe = you have a typo in the name of the property. I used the old one =C2=ABkylin= .hbase.cluster.fs=C2=BB=C2=A0

When both working-di= r and cluster-fs were set to s3 I got _temporary dir of convert job at s3, = but no hfiles. Also I saw correct output path for the job in the log. But I= didn=E2=80=99t check if job creates temporary files in s3, but then copies= results to hdfs. I hardly believe it happens.

Do = you see proper arguments for the step in the log?

=

11 =D0=B0=D0=B2=D0=B3. 2017 = =D0=B3., =D0=B2 11:17, ShaoFeng Shi <shaofengshi@apache.org> =D0=BD=D0=B0=D0=BF= =D0=B8=D1=81=D0=B0=D0=BB(=D0=B0):

Hi Alexander,

That makes sense. Using S3 for Cube bu= ild and storage is required for a cloud hadoop environment.

<= /div>
I tried to reproduce this problem. I created a EMR with S3 as HBa= se storage, in kylin.properties, I set=C2=A0"kylin.env.hdfs-working-di= r" and=C2=A0"kylin.storage.hbase.cluster-fs" to th= e S3 bucket. But in the "Con= vert Cuboid Data to HFile" step, Kylin still writes to local HDFS; Did= you modify the core-site.xml to make S3 as the default FS?




2017-08-10 22:53 GMT+= 08:00 Alexander Sterligov <sterligovak@joom.it>:
Yes, I workarounded this problem= in such way and it works.

One= problem of such solution is that I have to use pretty large hdfs and it= 9;expensive. And also I have to manually garbage collect it, because it is = not moved to s3, but copied. Kylin cleanup job doesn't work for it, bec= ause main metadata folder is at s3. So it would be really nice to put every= thing to s3.=C2=A0

Another pro= blem is that I had to rise hbase rpc timeout, because bulk loading from hdf= s takes long. That was not trivial. 3 minutes work good, but with drawback = of queries or metadata writes handing for 3 minutes if something bad happen= . But that's rare event.=C2=A0

10 =D0=B0=D0=B2=D0=B3. 2017 =D0=B3. 17:4= 2 =D0=BF=D0=BE=D0=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C = "ShaoFeng Shi" <shaofengshi@apache.org> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81= =D0=B0=D0=BB:

How about leaving empty for "kylin.hbase.cluster.fs"? This property is for two-cluster = deployment (one Hadoop for cube build, the other for query);=C2=A0
When be empty, the HFile will be written to default fs (HDFS in= EMR), and then load to HBase. I'm not sure whether EMR HBase (using S3= as storage) can bulk load files from HDFS or not. If it can, that would be= great as the write performance of HDFS would be better than S3.

2017-08-1= 0 22:29 GMT+08:00 Alexander Sterligov <sterligovak@joom.it>:
I = also thought about it, but no, it's not consistency.=C2=A0

Consistency view is enabled. I use s= ame s3 for my own map-reduce jobs and it's ok.
<= br>
I also checked if it lost consistency (emrfs dif= f). No problems.=C2=A0

I= n case of inconsistency of s3 files disappear right after they were written= and appear some time after. Hfiles didn't appear after a day, but _tem= plate is there.=C2=A0

It= 's 100% reproducable, I think I'll investigate this problem by runn= ing conversion job manually.=C2=A0
10 =D0=B0=D0=B2=D0=B3. 2017 =D0=B3. 17:18 =D0= =BF=D0=BE=D0=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C "= ;ShaoFeng Shi" <shaofengshi@apache.org> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0= =B0=D0=BB:

Did you enable= the Consistent View? This article explains the challenge when using S3 dir= ectly for ETL process:


2017-08-09 18:19 GM= T+08:00 Alexander Sterligov <sterligovak@joom.it>:
Yes, it's empty. Also I see= this message in the log:

2017-08-09 09:02:35,947 W= ARN =C2=A0[Job 1e436685-7102-4621-a4cb-6472b866126d-7608] mapreduce.Lo= adIncrementalHFiles:234 : Skipping non-directory s3://joom.emr.fs/h= ome/production/bi/kylin/kylin_metadata/kylin-1e436685-7102-46= 21-a4cb-6472b866126d
/main_event_1_main/hfile/_SUCCESS<= /div>
2017-08-09 09:02:36,009 WARN =C2=A0[Job 1e436685-7102-4621-a4cb-6= 472b866126d-7608] mapreduce.LoadIncrementalHFiles:252 : Skipping = non-file FileStatusExt{path=3Ds3://joom.emr.fs/home/production/= bi/kylin/kylin_metadata/kylin-1e436685-7102-4621-a4cb-6472b866126= d/main_event_1_main/hfile/_temporary/1; isDirectory=3Dtrue; modif= ication_time=3D0; access_time=3D0; owner=3D; group=3D; permission=3Drwxrwxr= wx; isSymlink=3Dfalse}
2017-08-09 09:02:36,014 WARN =C2=A0[Job 1e= 436685-7102-4621-a4cb-6472b866126d-7608] mapreduce.LoadIncrementalHFil= es:422 : Bulk load operation did not find any files to load in directo= ry s3://joom.emr.fs/home/production/bi/kylin/kylin_metadata/kyl= in-1e436685-7102-4621-a4cb-6472b866126d/main_event_1_main/hf= ile.=C2=A0 Does it contain files in subdirectories that correspond to colum= n family names?

On Wed, Aug 9, 2017 at 1:= 15 PM, ShaoFeng Shi <shaofengshi@apache.org> wrote:
=
The HFile will be moved to = HBase data folder when bulk load finished; Did you check whether the HTable= has data?

2017-08-09 17:54 GMT+08:00 Alexande= r Sterligov <sterligovak@joom.it>:
Hi!

I set=C2=A0kylin.hbase.c= luster.fs to s3 bucket where hbase lives.

Step &qu= ot;Convert Cuboid Data to HFile&q= uot; finished without errors. Statistics at the end of the job said that it= has written lot's of data to s3.

But t= here is no hfiles in kylin_metadata folder (kylin_metadata /kylin-1e436685-= 7102-4621-a4cb-6472b866126d/<table name>/hfile), but only _tempo= rary folder and _SUCCESS file.

_temporary contains= hfiles inside attempt folders. it looks like there were not copied from _t= emporary to result dir. But there is no errors neither in kylin log, nor in= reducers' logs.

Then loading empty hfiles pro= duces empty segments.

Is that a bug or I'm doi= ng something wrong?



=



--
Best regards,
Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B





--
=
Best regards,

Shaofeng Shi = =E5=8F=B2=E5=B0=91=E9=94=8B




--
Best regards,

Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B

=



--
Best regards,

Shaofeng Shi =E5=8F=B2=E5=B0=91= =E9=94=8B





--
Best regards,

Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B

=




--
=
Best regards,

Shaofeng Sh= i =E5=8F=B2=E5=B0=91=E9=94=8B

--94eb2c11a66e4b8e7e0556793f98--