kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Petter von Dolwitz (Hem)" <petter.von.dolw...@gmail.com>
Subject Re: Data inconsistency after restart
Date Thu, 07 Dec 2017 20:34:39 GMT
Hi Andrew and Alexey,

thanks for taking the time to answer.

>Interesting. Just to be sure, was that seen on one tserver, or did you see
them across all of them?
I did not look at all tservers at the time but it was one or two out of
five so this could be a reason if one of more tservers had remaining work.
The inserted_rows metric was steady at zero though.

>Did you verified the client didn't report any errors during data
ingestion?  Most likely you did, but I just wanted to make sure. BTW, what
kind of client did you use for the data ingestion?
I don't remember seeing any errors at the client side. I used 4 parallell
impala jobs as clients. When memory was not constrained I could see
ingestion rates steady around 1.5 million rows / second. After a while
(couple of hours perhaps), the ingestion rate dropped significantly so I
decided to stop ingesting and restart kudu with more memory (after the logs
stopped spinning at the tserver(s) I was looking at). I guess this was due
to a backlog building up and that kudu throttled the incoming traffic.

I could try to repeat the exercise and try to record result more
scientifically.

Br,
Petter




2017-12-07 20:48 GMT+01:00 Alexey Serbin <aserbin@cloudera.com>:

> Hi Petter,
>
> Before going too deep in attempts to find the place where the data was
> lost, I just wanted to make sure we definitely know that the data was
> delivered from the client to the server side.
>
> Did you verified the client didn't report any errors during data
> ingestion?  Most likely you did, but I just wanted to make sure. BTW, what
> kind of client did you use for the data ingestion?
>
> Thanks!
>
>
> Kind regards,
>
> Alexey
>
>
> On 12/6/17 3:56 PM, Andrew Wong wrote:
>
>> Hi Petter,
>>
>>     Before we shut down we could only see the following in the logs.
>>     I.e., no sign that ingestion was still ongoing.
>>
>>
>> Interesting. Just to be sure, was that seen on one tserver, or did you
>> see them across all of them?
>>
>>     But if the maintenance_manager performs important jobs that are
>>     required to ensure that all data is inserted then I can understand
>>     why we ended up with inconsistent data.
>>
>>
>> The maintenance manager's role is somewhat orthogonal to writes: data is
>> first written to the on-disk write-ahead log and also kept in-memory to be
>> accessible by scans. The maintenance manager periodically shuttles this
>> in-memory data to disk, among various other tasks like cleaning up WAL
>> segments, compacting rowsets, etc. Given that, a lack of maintenance ops
>> shouldn't cause incorrectness in data, even after restarting.
>>
>>     I would assume this means that it does not guarantee consistency
>>     if new data is inserted but should give valid (and same) results
>>     if no new data is inserted?
>>
>>
>> Right, if /all/ tservers a truly caught up and done processing the
>> writes, with no tablet copies going on, and with no new data coming in,
>> then the results should be consistent.
>>
>>
>> Hope this helped,
>> Andrew
>>
>> On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin <boris@boristyukin.com
>> <mailto:boris@boristyukin.com>> wrote:
>>
>>     this is smart, we are doing the same thing but the best part that
>>     attracts me to Kudu is replacing our main HDFS storage with Kudu
>>     to enable near RT use cases and not to deal with HBase and a
>>     Lambda architecture mess so reliability and scalability is a big
>>     deal for us as we are looking to move most of our data to Kudu.
>>
>>     On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem)
>>     <petter.von.dolwitz@gmail.com
>>     <mailto:petter.von.dolwitz@gmail.com>> wrote:
>>
>>         Hi Boris,
>>
>>         we do not have a Cloudera contract at the moment. Until we
>>         gained more Kudu experience we keep our master data in parquet
>>         format so that we can rebuild Kudu-tables upon errors. We are
>>         still in the early learning phase.
>>
>>         Br,
>>         Petter
>>
>>
>>
>>         2017-12-06 14:35 GMT+01:00 Boris Tyukin <boris@boristyukin.com
>>         <mailto:boris@boristyukin.com>>:
>>
>>             this is definitely concerning thread for us looking to use
>>             Impala for storing mission-critical company data. Petter,
>>             are you paid Cloudera customer btw? I wonder if you opened
>>             support ticket as well
>>
>>             On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem)
>>             <petter.von.dolwitz@gmail.com
>>             <mailto:petter.von.dolwitz@gmail.com>> wrote:
>>
>>                 Thanks for your reply Andrew!
>>
>>                 >How did you verify that all the data was inserted and
>>                 how did you find some data missing?
>>                 This was done using Impala. We counted the rows for
>>                 groups representing the chunks we inserted.
>>
>>                 >Following up on what I posted, take a look at
>>                 https://kudu.apache.org/docs/t
>> ransaction_semantics.html#_read_operations_scans
>>                 <https://kudu.apache.org/docs/
>> transaction_semantics.html#_read_operations_scans>.
>>                 It seems definitely possible that not all of the rows
>>                 had finished inserting when counting, or that the
>>                 scans were sent to a stale replica.
>>                 Before we shut down we could only see the following in
>>                 the logs. I.e., no sign that ingestion was still ongoing.
>>
>>                 kudu-tserver.ip-xx-yyy-z-nnn.r
>> oot.log.INFO.20171201-065232.90314:I1201
>>                 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>                 a38902afefca4a85a5469d149df9b4cb: we have exceeded our
>>                 soft memory limit (current capacity is 67.52%).
>>        However, there are no ops currently runnable which
>>                 would free memory.
>>
>>                 Also the (cloudera) metric
>>                 total_kudu_rows_inserted_rate_across_kudu_replicas
>>                 showed zero.
>>
>>                 Still it seems like some data became inconsistent
>>                 after restart. But if the maintenance_manager performs
>>                 important jobs that are required to ensure that all
>>                 data is inserted then I can understand why we ended up
>>                 with inconsistent data. But, if I understand you
>>                 correct,  you are saying that these jobs are not
>>                 critical for ingestion. In the link you provided I
>>                 read "Impala scans are currently performed as
>>                 READ_LATEST and have no consistency guarantees.". I
>>                 would assume this means that it does not guarantee
>>                 consistency if new data is inserted but should give
>>                 valid (and same) results if no new data is inserted?
>>
>>                 I have not tried the ksck tool yet. Thank you for
>>                 reminding. I will have a look.
>>
>>                 Br,
>>                 Petter
>>
>>
>>                 2017-12-06 1:31 GMT+01:00 Andrew Wong
>>                 <awong@cloudera.com <mailto:awong@cloudera.com>>:
>>
>>                         How did you verify that all the data was
>>                         inserted and how did you find some data
>>                         missing? I'm wondering if it's possible that
>>                         the initial "missing" data was data that Kudu
>>                         was still in the process of inserting (albeit
>>                         slowly, due to memory backpressure or somesuch).
>>
>>
>>                     Following up on what I posted, take a look at
>>                     https://kudu.apache.org/docs/t
>> ransaction_semantics.html#_read_operations_scans
>>                     <https://kudu.apache.org/docs/
>> transaction_semantics.html#_read_operations_scans>.
>>                     It seems definitely possible that not all of the
>>                     rows had finished inserting when counting, or that
>>                     the scans were sent to a stale replica.
>>
>>                     On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong
>>                     <awong@cloudera.com <mailto:awong@cloudera.com>>
>>
>>                     wrote:
>>
>>                         Hi Petter,
>>
>>                             When we verified that all data was
>>                             inserted we found that some data was
>>                             missing. We added this missing data and on
>>                             some chunks we got the information that
>>                             all rows were already present, i.e impala
>>                             says something like Modified: 0 rows,
>>                             nnnnnnn errors. Doing the verification
>>                             again now shows that the Kudu table is
>>                             complete. So, even though we did not
>>                             insert any data on some chunks, a count(*)
>>                             operation over these chunks now returns a
>>                             different value.
>>
>>
>>                         How did you verify that all the data was
>>                         inserted and how did you find some data
>>                         missing? I'm wondering if it's possible that
>>                         the initial "missing" data was data that Kudu
>>                         was still in the process of inserting (albeit
>>                         slowly, due to memory backpressure or somesuch).
>>
>>                             Now to my question. Will data be
>>                             inconsistent if we recycle Kudu after
>>                             seeing soft memory limit warnings?
>>
>>
>>                         Your data should be consistently written, even
>>                         with those warnings. AFAIK they would cause a
>>                         bit of slowness, not incorrect results.
>>
>>                             Is there a way to tell when it is safe to
>>                             restart Kudu to avoid these issues? Should
>>                             we use any special procedure when
>>                             restarting (e.g. only restart the tablet
>>                             servers, only restart one tablet server at
>>                             a time or something like that)?
>>
>>
>>                         In general, you can use the `ksck` tool to
>>                         check the health of your cluster. See
>>                         https://kudu.apache.org/docs/c
>> ommand_line_tools_reference.html#cluster-ksck
>>                         <https://kudu.apache.org/docs/
>> command_line_tools_reference.html#cluster-ksck>
>>                         for more details. For restarting a cluster, I
>>                         would recommend taking down all tablet servers
>>                         at once, otherwise tablet replicas may try to
>>                         replicate data from the server that was taken
>>                         down.
>>
>>                         Hope this helped,
>>                         Andrew
>>
>>                         On Tue, Dec 5, 2017 at 10:42 AM, Petter von
>>                         Dolwitz (Hem) <petter.von.dolwitz@gmail.com
>>                         <mailto:petter.von.dolwitz@gmail.com>> wrote:
>>
>>                             Hi Kudu users,
>>
>>                             We just started to use Kudu
>>                             (1.4.0+cdh5.12.1). To make a baseline for
>>                             evaluation we ingested 3 month worth of
>>                             data. During ingestion we were facing
>>                             messages from the maintenance threads that
>>                             a soft memory limit were reached. It seems
>>                             like the background maintenance threads
>>                             stopped performing their tasks at this
>>                             point in time. It also so seems like the
>>                             memory was never recovered even after
>>                             stopping ingestion so I guess there was a
>>                             large backlog being built up. I guess the
>>                             root cause here is that we were a bit too
>>                             conservative when giving Kudu memory.
>>                             After a reststart a lot of maintenance
>>                             tasks were started (i.e. compaction).
>>
>>                             When we verified that all data was
>>                             inserted we found that some data was
>>                             missing. We added this missing data and on
>>                             some chunks we got the information that
>>                             all rows were already present, i.e impala
>>                             says something like Modified: 0 rows,
>>                             nnnnnnn errors. Doing the verification
>>                             again now shows that the Kudu table is
>>                             complete. So, even though we did not
>>                             insert any data on some chunks, a count(*)
>>                             operation over these chunks now returns a
>>                             different value.
>>
>>                             Now to my question. Will data be
>>                             inconsistent if we recycle Kudu after
>>                             seeing soft memory limit warnings?
>>
>>                             Is there a way to tell when it is safe to
>>                             restart Kudu to avoid these issues? Should
>>                             we use any special procedure when
>>                             restarting (e.g. only restart the tablet
>>                             servers, only restart one tablet server at
>>                             a time or something like that)?
>>
>>                             The table design uses 50 tablets per day
>>                             (times 90 days). It is 8 TB of data after
>>                             3xreplication over 5 tablet servers.
>>
>>                             Thanks,
>>                             Petter
>>
>>
>>
>>
>>
>>                         --                         Andrew Wong
>>
>>
>>
>>
>>                     --                     Andrew Wong
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Andrew Wong
>>
>
>

Mime
View raw message