hudi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vbalaji@apache.org" <vbal...@apache.org>
Subject Re: Add checkpoint metadata while using HoodieSparkSQLWriter
Date Sat, 22 Jun 2019 13:28:57 GMT
 
Here is the correct gist link : https://gist.github.com/bvaradar/e18d96f9b99980dfb67a6601de5aa626


    On Saturday, June 22, 2019, 6:08:48 AM PDT, vbalaji@apache.org <vbalaji@apache.org>
wrote:  
 
  Hi,
I have given a sample command to set up and run deltastreamer in continuous mode and ingest
fake data in the following gist
https://gist.github.com/bvaradar/c5feec486fd4b2a3dac40c93649962c7

We will eventually get this to project wiki.
Balaji.V

    On Friday, June 21, 2019, 3:12:49 PM PDT, Netsanet Gebretsadkan <net22geb@gmail.com>
wrote:  
 
 @Vinoth, Thanks , that would be great if Balaji could share it.

Kind regards,


On Thu, Jun 20, 2019 at 11:17 PM Vinoth Chandar <vinoth@apache.org> wrote:

> Hi,
>
> We usually test with our production workloads.. However, balaji recently
> merged a DistributedTestDataSource,
>
> https://github.com/apache/incubator-hudi/commit/a0d7ab238473f22347e140b0e1e273ab80583eb7#diff-893dced90c18fd2698c6a16475f5536d
>
>
> that can generate some random data for testing..  Balaji, do you mind
> sharing a command that can be used to kick something off like that?
>
>
> On Thu, Jun 20, 2019 at 1:54 AM Netsanet Gebretsadkan <net22geb@gmail.com>
> wrote:
>
> > Dear Vinoth,
> >
> > I want to try to check out the performance comparison of upsert and bulk
> > insert.  But i couldn't find a clean data set more than 10 GB.
> > Would it be possible to get a data set from Hudi team? For example i was
> > using the stocks data that you provided on your demo. Hence, can i get
> > more GB's of that dataset for my experiment?
> >
> > Thanks for your consideration.
> >
> > Kind regards,
> >
> > On Fri, Jun 7, 2019 at 7:59 PM Vinoth Chandar <vinoth@apache.org> wrote:
> >
> > >
> >
> https://github.com/apache/incubator-hudi/issues/714#issuecomment-499981159
> > >
> > > Just circling back with the resolution on the mailing list as well.
> > >
> > > On Tue, Jun 4, 2019 at 6:24 AM Netsanet Gebretsadkan <
> net22geb@gmail.com
> > >
> > > wrote:
> > >
> > > > Dear Vinoth,
> > > >
> > > > Thanks for your fast response.
> > > > I have created a new issue called Performance Comparison of
> > > > HoodieDeltaStreamer and DataSourceAPI #714  with the screnshots of
> the
> > > > spark UI which can be found at the  following  link
> > > > https://github.com/apache/incubator-hudi/issues/714.
> > > > In the UI,  it seems that the ingestion with the data source API is
> > > > spending  much time in the count by key of HoodieBloomIndex and
> > workload
> > > > profile.  Looking forward to receive insights from you.
> > > >
> > > > Kinde regards,
> > > >
> > > >
> > > > On Tue, Jun 4, 2019 at 6:35 AM Vinoth Chandar <vinoth@apache.org>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Both datasource and deltastreamer use the same APIs underneath. So
> > not
> > > > > sure. If you can grab screenshots of spark UI for both and open a
> > > ticket,
> > > > > glad to take a look.
> > > > >
> > > > > On 2, well one of goals of Hudi is to break this dichotomy and
> enable
> > > > > streaming style (I call it incremental processing) of processing
> even
> > > in
> > > > a
> > > > > batch job. MOR is in production at uber. Atm MOR is lacking just
> one
> > > > > feature (incr pull using log files) that Nishith is planning to
> merge
> > > > soon.
> > > > > PR #692 enables Hudi DeltaStreamer to ingest continuously while
> > > managing
> > > > > compaction etc in the same job. I already knocked off some index
> > > > > performance problems and working on indexing the log files, which
> > > should
> > > > > unlock near real time ingest.
> > > > >
> > > > > Putting all these together, within a month or so near real time MOR
> > > > vision
> > > > > should be very real. Ofc we need community help with dev and
> testing
> > to
> > > > > speed things up. :)
> > > > >
> > > > > Hope that gives you a clearer picture.
> > > > >
> > > > > Thanks
> > > > > Vinoth
> > > > >
> > > > > On Mon, Jun 3, 2019 at 1:01 AM Netsanet Gebretsadkan <
> > > net22geb@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks, Vinoth
> > > > > >
> > > > > > Its working now. But i have 2 questions:
> > > > > > 1. The ingestion latency of using DataSource API with
> > > > > > the  HoodieSparkSQLWriter  is high compared to using delta
> > streamer.
> > > > Why
> > > > > is
> > > > > > it slow? Are there specific option where we could specify to
> > minimize
> > > > the
> > > > > > ingestion latency.
> > > > > >    For example: when i run the delta streamer its talking
about 1
> > > > minute
> > > > > to
> > > > > > insert some data. If i use DataSource API with
> > HoodieSparkSQLWriter,
> > > > its
> > > > > > taking 5 minutes. How can we optimize this?
> > > > > > 2. Where do we categorize Hudi in general (Is it batch processing
> > or
> > > > > > streaming)?  I am asking this because currently the copy on
write
> > is
> > > > the
> > > > > > one which is fully working and since the functionality of the
> merge
> > > on
> > > > > read
> > > > > > is not fully done which enables us to have a near real time
> > > analytics,
> > > > > can
> > > > > > we consider Hudi as a batch job?
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > >
> > > > > > On Thu, May 30, 2019 at 5:52 PM Vinoth Chandar <
> vinoth@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Short answer, by default any parameter you pass in using
> > > option(k,v)
> > > > or
> > > > > > > options() beginning with "_" would be saved to the commit
> > metadata.
> > > > > > > You can change "_" prefix to something else by using the
> > > > > > >  DataSourceWriteOptions.COMMIT_METADATA_KEYPREFIX_OPT_KEY().
> > > > > > > Reason you are not seeing the checkpointstr inside the
commit
> > > > metadata
> > > > > is
> > > > > > > because its just supposed to be a prefix for all such commit
> > > > metadata.
> > > > > > >
> > > > > > > val metaMap = parameters.filter(kv =>
> > > > > > >
> kv._1.startsWith(parameters(COMMIT_METADATA_KEYPREFIX_OPT_KEY)))
> > > > > > >
> > > > > > > On Thu, May 30, 2019 at 2:56 AM Netsanet Gebretsadkan <
> > > > > > net22geb@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I am trying to use the HoodieSparkSQLWriter to upsert
data
> from
> > > any
> > > > > > > > dataframe into a hoodie modeled table.  Its creating
> everything
> > > > > > correctly
> > > > > > > > but , i also want to save the checkpoint but i couldn't
even
> > > though
> > > > > am
> > > > > > > > passing it as an argument.
> > > > > > > >
> > > > > > > > inputDF.write()
> > > > > > > > .format("com.uber.hoodie")
> > > > > > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > > > "_row_key")
> > > > > > > > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > > > > > > "partition")
> > > > > > > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > > > > "timestamp")
> > > > > > > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > > > > > > >
> > > .option(DataSourceWriteOptions.COMMIT_METADATA_KEYPREFIX_OPT_KEY(),
> > > > > > > > checkpointstr)
> > > > > > > > .mode(SaveMode.Append)
> > > > > > > > .save(basePath);
> > > > > > > >
> > > > > > > > am using the COMMIT_METADATA_KEYPREFIX_OPT_KEY() for
> inserting
> > > the
> > > > > > > > checkpoint while using the dataframe writer but i
couldn't
> add
> > > the
> > > > > > > > checkpoint meta data in to the .hoodie meta data.
Is there a
> > way
> > > i
> > > > > can
> > > > > > > add
> > > > > > > > the checkpoint meta data while using the dataframe
writer
> API?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
    
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message