ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Ignite PDS WAL analysis with human readable records
Date Mon, 18 Sep 2017 20:22:57 GMT
Dmitriy, can you please describe your idea in the ticket?

Here is the link for easier access:
https://issues.apache.org/jira/browse/IGNITE-6421

D.

On Mon, Sep 18, 2017 at 7:02 AM, Dmitry Pavlov <dpavlov.spb@gmail.com>
wrote:

> Dmitriy,
>
> Thank you for your feedback I've created IGNITE-6421 for this change.
>
> Is it common practice to use property file in Ignite utils? I'd like to
> support parse and display data entries and its fields, but it is required
> to configure several folders: 'binary_meta' and 'marshaller' folders.
>
> My idea is to provide .conf file to configure reader once.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 18 сент. 2017 г. в 16:10, Dmitriy Setrakyan <dsetrakyan@apache.org>:
>
> > Thanks!
> >
> > After looking at the ticket, the way to launch this utility seems
> complex.
> > Does this utility have a shell script? If not, we should add one with
> > proper command line options, like we do for other scripts.
> >
> > D.
> >
> > On Mon, Sep 18, 2017 at 4:05 AM, Eduard Shangareev <
> > eduard.shangareev@gmail.com> wrote:
> >
> > > Example of output (will add to wiki):
> > >
> > > [W] InitNewPageRecord [newPageId=0001001c00000067,
> super=PageDeltaRecord
> > > [grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
> > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > [W] InitNewPageRecord [newPageId=0001001c00000068,
> super=PageDeltaRecord
> > > [grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
> > > forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
> > > [W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
> > > super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
> > > super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=25488, len=37, forceFlush=false],
> type=PAGES_LIST_ADD_PAGE]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
> > > effectivePageId=0000001c0000002a, grpId=2141373875], page = [
> > > Header [
> > > type=1 (DataPageIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
> > > ],
> > > DataPageIO [
> > > 0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
> > > 3626][][free=3552, actualFree=-544
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
> > > rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
> > > pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
> > > pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
> > > type=BTREE_PAGE_INSERT]]]
> > > [W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377,
> op=CREATE,
> > > writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
> > > nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
> > > chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
> > > forceFlush=false], type=DATA_RECORD]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
> > > effectivePageId=000000000000000a, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
> > > effectivePageId=0000000200000010, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
> > > effectivePageId=000000050000000d, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
> > > effectivePageId=0000000900000003, grpId=2141373875], page = [
> > > Header [
> > > type=12 (PagesListMetaIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
> > > ],
> > > PagesListMeta [
> > > nextMetaPageId=0000000000000000,
> > > count=230,
> > > bucketData={
> > > bucket=163, list=GridLongList [idx=6,
> > > arr=[281513631416558,281513631416559,281513631416560,281513631416561,
> > > 281513631416562,281513631416563]]
> > > bucket=166, list=GridLongList [idx=8,
> > > arr=[281513631416550,281513631416551,281513631416552,281513631416553,
> > > 281513631416554,281513631416555,281513631416556,281513631416557]]
> > > bucket=169, list=GridLongList [idx=8,
> > > arr=[281513631416542,281513631416543,281513631416544,281513631416545,
> > > 281513631416546,281513631416547,281513631416548,281513631416549]]
> > > bucket=172, list=GridLongList [idx=8,
> > > arr=[281513631416534,281513631416535,281513631416536,281513631416537,
> > > 281513631416538,281513631416539,281513631416540,281513631416541]]
> > > bucket=175, list=GridLongList [idx=8,
> > > arr=[281513631416526,281513631416527,281513631416528,281513631416529,
> > > 281513631416530,281513631416531,281513631416532,281513631416533]]
> > > bucket=178, list=GridLongList [idx=8,
> > > arr=[281513631416512,281513631416513,281513631416520,281513631416521,
> > > 281513631416522,281513631416523,281513631416524,281513631416525]]
> > > bucket=181, list=GridLongList [idx=8,
> > > arr=[281513631416510,281513631416511,281513631416514,281513631416515,
> > > 281513631416516,281513631416517,281513631416518,281513631416519]]
> > > bucket=184, list=GridLongList [idx=8,
> > > arr=[281513631416500,281513631416503,281513631416504,281513631416505,
> > > 281513631416506,281513631416507,281513631416508,281513631416509]]
> > > bucket=187, list=GridLongList [idx=8,
> > > arr=[281513631416494,281513631416495,281513631416496,281513631416497,
> > > 281513631416498,281513631416499,281513631416501,281513631416502]]
> > > bucket=190, list=GridLongList [idx=8,
> > > arr=[281513631416484,281513631416485,281513631416486,281513631416487,
> > > 281513631416488,281513631416489,281513631416492,281513631416493]]
> > > bucket=193, list=GridLongList [idx=8,
> > > arr=[281513631416448,281513631416449,281513631416480,281513631416481,
> > > 281513631416482,281513631416483,281513631416490,281513631416491]]
> > > bucket=196, list=GridLongList [idx=8,
> > > arr=[281513631416446,281513631416447,281513631416474,281513631416475,
> > > 281513631416476,281513631416477,281513631416478,281513631416479]]
> > > bucket=199, list=GridLongList [idx=8,
> > > arr=[281513631416444,281513631416445,281513631416468,281513631416469,
> > > 281513631416470,281513631416471,281513631416472,281513631416473]]
> > > bucket=202, list=GridLongList [idx=8,
> > > arr=[281513631416436,281513631416437,281513631416462,281513631416463,
> > > 281513631416464,281513631416465,281513631416466,281513631416467]]
> > > bucket=205, list=GridLongList [idx=8,
> > > arr=[281513631416434,281513631416435,281513631416456,281513631416457,
> > > 281513631416458,281513631416459,281513631416460,281513631416461]]
> > > bucket=209, list=GridLongList [idx=8,
> > > arr=[281513631416426,281513631416427,281513631416450,281513631416451,
> > > 281513631416452,281513631416453,281513631416454,281513631416455]]
> > > bucket=212, list=GridLongList [idx=8,
> > > arr=[281513631416418,281513631416419,281513631416438,281513631416439,
> > > 281513631416440,281513631416441,281513631416442,281513631416443]]
> > > bucket=215, list=GridLongList [idx=8,
> > > arr=[281513631416416,281513631416417,281513631416428,281513631416429,
> > > 281513631416430,281513631416431,281513631416432,281513631416433]]
> > > bucket=218, list=GridLongList [idx=8,
> > > arr=[281513631416414,281513631416415,281513631416420,281513631416421,
> > > 281513631416422,281513631416423,281513631416424,281513631416425]]
> > > bucket=221, list=GridLongList [idx=8,
> > > arr=[281513631416406,281513631416407,281513631416408,281513631416409,
> > > 281513631416410,281513631416411,281513631416412,281513631416413]]
> > > bucket=224, list=GridLongList [idx=8,
> > > arr=[281513631416398,281513631416399,281513631416400,281513631416401,
> > > 281513631416402,281513631416403,281513631416404,281513631416405]]
> > > bucket=227, list=GridLongList [idx=8,
> > > arr=[281513631416390,281513631416391,281513631416392,281513631416393,
> > > 281513631416394,281513631416395,281513631416396,281513631416397]]
> > > bucket=230, list=GridLongList [idx=8,
> > > arr=[281513631416381,281513631416382,281513631416383,281513631416384,
> > > 281513631416385,281513631416386,281513631416387,281513631416388]]
> > > bucket=233, list=GridLongList [idx=8,
> > > arr=[281513631416373,281513631416374,281513631416375,281513631416376,
> > > 281513631416377,281513631416378,281513631416379,281513631416380]]
> > > bucket=236, list=GridLongList [idx=8,
> > > arr=[281513631416365,281513631416366,281513631416367,281513631416368,
> > > 281513631416369,281513631416370,281513631416371,281513631416372]]
> > > bucket=239, list=GridLongList [idx=8,
> > > arr=[281513631416351,281513631416352,281513631416359,281513631416360,
> > > 281513631416361,281513631416362,281513631416363,281513631416364]]
> > > bucket=242, list=GridLongList [idx=8,
> > > arr=[281513631416348,281513631416349,281513631416353,281513631416354,
> > > 281513631416355,281513631416356,281513631416357,281513631416358]]
> > > bucket=245, list=GridLongList [idx=8,
> > > arr=[281513631416335,281513631416336,281513631416337,281513631416338,
> > > 281513631416339,281513631416340,281513631416346,281513631416347]]
> > > bucket=248, list=GridLongList [idx=8,
> > > arr=[281513631416326,281513631416327,281513631416328,281513631416329,
> > > 281513631416330,281513631416331,281513631416343,281513631416344]]
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
> > > effectivePageId=000000090000000d, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=198
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
> > > effectivePageId=0000000900000000, grpId=2141373875], page = [
> > > Header [
> > > type=14 (PagePartitionMetaIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
> > > ],
> > > PageMeta[
> > > treeRoot=281513631416323,
> > > lastSuccessfulFullSnapshotId=0,
> > > lastSuccessfulSnapshotId=0,
> > > nextSnapshotTag=1,
> > > lastSuccessfulSnapshotTag=0,
> > > lastAllocatedPageCount=0,
> > > candidatePageCount=0
> > > ],
> > > PagePartitionMeta[
> > > size=198,
> > > updateCounter=198,
> > > globalRemoveId=1505412422861000000,
> > > partitionState=1(OWNING),
> > > countersPageId=281513631416333
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
> > > effectivePageId=0000000b0000000c, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > > [W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
> > > effectivePageId=0000000c0000000f, grpId=2141373875], page = [
> > > Header [
> > > type=20 (PagePartitionCountersIO),
> > > ver=1,
> > > crc=0,
> > > pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
> > > ],
> > > PagePartitionCounters [
> > > count=1,
> > > lastFlag=true,
> > > nextCountersPageId=0000000000000000,
> > > size={
> > > -1368047377=313
> > > }
> > > ]],
> > > super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
> > > fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]
> > >
> > >
> > > On Mon, Sep 18, 2017 at 1:10 PM, Dmitry Pavlov <dpavlov.spb@gmail.com>
> > > wrote:
> > >
> > > > Hi Igniters,
> > > >
> > > > Page https://cwiki.apache.org/confluence/display/IGNITE/
> > > Ignite+WAL+reader
> > > > to
> > > > collect related information about this prototype was created.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пн, 18 сент. 2017 г. в 7:01, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >:
> > > >
> > > > > Can we get a sample print out from this utility?
> > > > >
> > > > > On Sun, Sep 17, 2017 at 3:42 PM, Eduard Shangareev <
> > > > > eduard.shangareev@gmail.com> wrote:
> > > > >
> > > > > > Dmitry,
> > > > > > thank you for starting this topic.
> > > > > >
> > > > > > The IDEA of this utility is very simple - get the possibility
to
> > look
> > > > > into
> > > > > > your WAL.
> > > > > > But it also requires understanding what's going on in WAL
> mechanism
> > > (We
> > > > > > need to describe it in more details in the wiki).
> > > > > >
> > > > > > It could be useful for a debug some weird things with recovery
> > > > mechanism.
> > > > > > Initially, it was created for dealing with the issue in
> > IGNITE-6342.
> > > It
> > > > > > helped to find out the error in how we deal with partition size.
> > > > > >
> > > > > > Best regards,
> > > > > > Ed.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 16, 2017 at 2:09 AM, Dmitry Pavlov <
> > > dpavlov.spb@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Denis,
> > > > > > >
> > > > > > > Thank you for your feedback. It is very important to us
to
> > > understand
> > > > > > that
> > > > > > > this tool may be useful not only for Ignite tests but for
all
> > > > > developers.
> > > > > > >
> > > > > > > I will create wiki page.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > сб, 16 сент. 2017 г. в 1:54, Denis Magda <dmagda@apache.org>:
> > > > > > >
> > > > > > > > Dmitriy, Eduard,
> > > > > > > >
> > > > > > > > Sounds like a useful thing. I’ll be happy to give
a feedback
> > but
> > > > > need a
> > > > > > > > documentation that explains how to use the tool, what’s
the
> > > output
> > > > > > > format,
> > > > > > > > how do decipher the output, etc (basic things). The
Wiki
> > > > description
> > > > > > > should
> > > > > > > > be enough.
> > > > > > > >
> > > > > > > > —
> > > > > > > > Denis
> > > > > > > >
> > > > > > > > > On Sep 15, 2017, at 11:36 AM, Dmitry Pavlov <
> > > > dpavlov.spb@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Igniters,
> > > > > > > > >
> > > > > > > > > Eduard Shangareev created WAL converter utility
which shows
> > WAL
> > > > > > > > > archive/work directory segments content in human
readable
> > form.
> > > > > This
> > > > > > > > > utility outputs all WAL records to output stream
(may be
> > > > redirected
> > > > > > to
> > > > > > > > > file). How to use tips can be found in issue
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-6277
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This utility does not require Ignite node to
be up and
> > running
> > > > and
> > > > > > may
> > > > > > > be
> > > > > > > > > used for offline analysis of copied files. Thanks
to Alexey
> > G.
> > > > for
> > > > > > idea
> > > > > > > > and
> > > > > > > > > review. Code is now available in modules/ignite-dev-tools.
> > > > > > > > >
> > > > > > > > > 1. Could you please write if utility can be useful
for you?
> > > > > > > > >
> > > > > > > > > 2. Please respond if you need short wiki description.
> > > > > > > > >
> > > > > > > > > 3. What is your opinion what should be next steps
to
> develop
> > > this
> > > > > > > > utility?
> > > > > > > > >
> > > > > > > > > 4. Does utility need property file to refer Ignite
Work
> > > directory
> > > > > > using
> > > > > > > > > conf file.
> > > > > > > > >
> > > > > > > > > 5. What do you think if utility remains console
non
> > > interactive,
> > > > or
> > > > > > > some
> > > > > > > > > UI/Interactive console shell may be preferable?
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > >
> > > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message