Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAPd2BNnDiZZ5DOde0cUncb2WS-pKTc+ysBoyFonJ+Ew6Eniw-Q@mail.gmail.com>
References: <CAPd2BN=Ahz_1cC46O6O7kKES13J8ptgFhtrOcBxtdNOy5-QwYA@mail.gmail.com>
 <CALte62yXzdXPSMwizAWtvB2Nh6EPUoYFFPPL_KsOVcjG1So6SA@mail.gmail.com>
 <CAJQYon3f9Bkf3ZVu4bsLoGyPVd4=KD627YrzV9RKntuZm-6eow@mail.gmail.com> <CAPd2BNnDiZZ5DOde0cUncb2WS-pKTc+ysBoyFonJ+Ew6Eniw-Q@mail.gmail.com>
From: Ted Yu <yuzhihong@gmail.com>
Date: Thu, 29 Jun 2017 02:41:50 -0700
Message-ID: <CALte62y_LQb=Q06_r0CHcgSftP2U3qzNekA-2AXR5wEDOBUyoQ@mail.gmail.com>
Subject: Re: Implementation of full table scan using Spark
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c18fbda32a04f0553161cf6"
archived-at: Thu, 29 Jun 2017 09:41:57 -0000

--94eb2c18fbda32a04f0553161cf6
Content-Type: text/plain; charset="UTF-8"

Sachin:
My previous answer was inaccurate.

Please take a look at TableRecordReaderImpl where htable.getScanner() is
called to obtain ResultScanner.

The (relatively) fast table scan may be due to your table having not much
data.

Cheers

On Wed, Jun 28, 2017 at 10:27 PM, Sachin Jain <sachinjain024@gmail.com>
wrote:

> @Ted Yu If full table scan does not read memstore then why I am getting the
> recently inserted data. I am pretty sure others may have seen this earlier
> and may not didn't notice.
>
> @Jingcheng Thanks for your answer. If you are true, then my understanding
> was wrong. I will try to see the code of TableInputFormat and see if I get
> something new.
>
> On Thu, Jun 29, 2017 at 9:31 AM, Jingcheng Du <dujingch@gmail.com> wrote:
>
> > Hi Sachin,
> > The TableInputFormat should read the memstore.
> > The TableInputFormat is converted to scan to each region, the operations
> in
> > each region should be a normal scan, so the memstore should be included.
> > That's why you can always read all the data.
> >
> > bq. As per my understanding this full table scan works fast because we
> are
> > reading Hfiles directly.
> > I think the fast full table scan is because you run the scan in each
> region
> > concurrently in Spark.
> >
> > 2017-06-29 11:33 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> >
> > > TableInputFormat doesn't read memstore.
> > >
> > > bq. I am inserting 10-20 entires only
> > >
> > > You can query JMX and check the values for the following:
> > >
> > > flushedCellsCount
> > > flushedCellsSize
> > >
> > > FlushMemstoreSize_num_ops
> > >
> > > For Q2, there is no client side support for knowing where the data
> comes
> > > from.
> > >
> > > On Wed, Jun 28, 2017 at 8:15 PM, Sachin Jain <sachinjain024@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have used TableInputFormat and newAPIHadoopRDD defined on
> > sparkContext
> > > to
> > > > do a full table scan and get an rdd from it.
> > > >
> > > > Partial piece of code looks like this:
> > > >
> > > > sparkContext.newAPIHadoopRDD(
> > > >   HBaseConfigurationUtil.hbaseConfigurationForReading(table.getName.
> > > > getNameWithNamespaceInclAsString,
> > > > hbaseQuorum, hBaseFilter, versionOpt, zNodeParentOpt),
> > > >   classOf[TableInputFormat],
> > > >   classOf[ImmutableBytesWritable],
> > > >   classOf[Result]
> > > > )
> > > >
> > > >
> > > > As per my understanding this full table scan works fast because we
> are
> > > > reading Hfiles directly.
> > > >
> > > > *Q1. Does that mean we are skipping memstores ? *If yes, then we
> should
> > > > have missed some data which is present in memstore because that data
> > has
> > > > not been persisted to disk yet and hence not available via HFile.
> > > >
> > > > *In my local setup, I always get all the data*. Since I am inserting
> > > 10-20
> > > > entires only I am assuming this is present in memstore when I am
> > issuing
> > > > the full table scan spark job.
> > > >
> > > > Q2. When I issue a get command, Is there a way to know if the record
> is
> > > > served from blockCache, memstore or Hfile?
> > > >
> > > > Thanks
> > > > -Sachin
> > > >
> > >
> >
>

--94eb2c18fbda32a04f0553161cf6--