axis-c-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samisa Abeysinghe <sam...@wso2.com>
Subject Re: AW: transfer limitations
Date Mon, 05 Jul 2010 13:58:22 GMT
If you want to dig into the content and pick stuff, then you got to build
AXIOM and get the job done. So MTOM would not help in that case.

However, if you are mainly interested in getting the file to the other side,
then MTOM is the choice.

It looks to me that, you want something int he middle, as you have to pick
key/value from what you transfer.

I got to admit, that building the whole thing as AXIOM for bulky stuff is
going to be very costly, and we have seen Axis2/C degrade with larger
messages in terms of performance. So if you try and get the content as MTOM
and use some other logic to do the processing rather than AXIOM build, that
would be more efficient.

Samisa...

On Mon, Jul 5, 2010 at 6:57 PM, Stadelmann Josef <
josef.stadelmann@axa-winterthur.ch> wrote:

>  Samisa,
>
>
>
> Thank you, and yes, it is not a too good idea to build a mega AXIOM. But it
> is as it is; We mainly use AXIOM to reflect buckets full of name/value pairs
> from transaction mask fields. Transactions in long lasting sessions; So we
> send two buckets (call it workspaces) to pass the past and current workspace
> to the server and receive the output-workspace from the server. On arrival
> we de-serialize the xml into an Axiom model and from the Axiom we fill
> hash-tables, we can merge hash-table easy. Then our legacy apps gets/puts by
> key-name the value; once done, code returns to create the output axiom
> model, which gets serialized into xml.
>
>
>
> It was designed for transaction masks (VB Forms with many fields); it was
> then overused by VB programmers to transmit files! In the future I am going
> to enhance my server/service to use MTOM for file-transfers.
>
> The problem I am faced with is
>
>
>
>             Scope=”soapsession”
>
>
>
> Can you tell me how I can mix-in into my existing service 2 methods which
> runs in scope=”soapsession” to send/receive MTOM coded data.
>
>
>
> Why did I not do it until today;
>
>
>
> Because it is somewhere said, either at Microsoft .NET WCF 3.5 or AXIS2/J
> or /C that one can’t mix SOAP/XML and MTOM on the same service?
>
>
>
> If I am wrong I am more than happy to sniff into my existing larger service
> object the two MTOM methods getFile() and putFile();
>
>
>
> Josef
>
>
>
>
>
>
>
>
>
>
>
> *Von:* Samisa Abeysinghe [mailto:samisa@wso2.com]
> *Gesendet:* Montag, 5. Juli 2010 15:09
> *An:* Apache AXIS C User List
>
> *Betreff:* Re: AW: transfer limitations
>
>
>
> If you want to handle large chunks of data,  why not use MTOM?
>
>
>
> Also, there is a data source mode in Axis2/C to stream the chunk through.
>
>
>
> We use used Axis2/C to send mega bytes nearing giga range and it works for
> us for MTOM.
>
>
>
> For service case, I am not sure if it is a good idea to use AXIOM to build
> a mega range doc - that means the service design is busted.
>
>
>
> Samisa...
>
> On Mon, Jul 5, 2010 at 6:20 PM, Stadelmann Josef <
> josef.stadelmann@axa-winterthur.ch> wrote:
>
> Hope this helps you?`!
>
>
>
> Yes, how did I strike it out? I am using WSIT from HP to integrate legacy
> systems. In short, big (huge) OMElement is arriving at the JAVA side, (could
> well contain Bas64 encoded data) it is then serialized to a very large
> (huge) xml string; which is then pass to an out-of-process server via some
> sort of WsiIpcContext (IPC Interprocess Communication Channel); JNI is
> involved too.
>
>
>
> ….
>
>
>
> When my large data arrives at the C-Legacy Server passed by WSIT ToolKit, I
> get a OpenVMS Descriptor which has a Length Field and a Pointer to the data.
>
>
>
>
> ….
>
>
>
> And now comes what I did not understand so far but since I try to help you,
> I was able to fix my own problem today.
>
> So what I did not understand is as follow.
>
>
>
> My data arrives at WSI$$fktmap() in *pbIndata with a inLen of Bystes set to
> 91344
>
> The code you see below is HP-WSIT-generated-code, and it works fine.
>
>
>
> The second green code line shows how this data is passed to my own routine
> fktmap() which is declared as
>
> ‘__int32 fktmap (char * p1, int inLen, dsc_d * p2);’
>
>
>
> Given that pbInData can be passed further on via a P1 of type char* then
> please explain me the problem I am faced with.
>
>
>
> 1.  In one case when I do
>
> Unsigned __int16 scnt = strlen(P1);
>
> I get scnt back as 25808 bytes
>
> BUT
>
> inLen shows me 91344
>
> Why?
>
>
>
> 2.  When I save P1 in a file, all bytes are there
> BUT
> When I pass it to
>
> xml_reader = axiom_xml_reader_create_for_memory(env, payload_inp, scnt,
> encoding, AXIS2_XML_PARSER_TYPE_BUFFER);
>
> Then I get the following errors in my log file for one transaction, at
> different call levels.
>
>
>
> [Mon Jul  5 14:10:28 2010] [error]
> DKB3:[SW-PROJEKTE.webservices.axis2.trunk.c.axiom.src.parser]libxml2_reader_wrapper.c;36(886)
>
>
>   expected '>'
>
> ***************
>
>  -- SEVERITY_ERROR
>
> ***************
>
> [Mon Jul  5 14:10:28 2010] [error]
> DKB3:[SW-PROJEKTE.webservices.axis2.trunk.c.axiom.src.parser]libxml2_reader_wrapper.c;36(886)
>
>
>   Opening and ending tag mismatch: Fldnam line 0 and F
>
> ***************
>
>  -- SEVERITY_ERROR
>
> ***************
>
> [Mon Jul  5 14:10:28 2010] [error]
> DKB3:[SW-PROJEKTE.webservices.axis2.trunk.c.axiom.src.parser]libxml2_reader_wrapper.c;36(886)
>
>
>   Extra content at the end of the document
>
> ***************
>
>  -- SEVERITY_ERROR
>
> ***************
>
> [Mon Jul  5 14:10:28 2010] [error]
> DKB3:[SW-PROJEKTE.webservices.axis2.trunk.c.axiom.src.parser]libxml2_reader_wrapper.c;36(427)
>
>
>   error occured in reading xml stream
>
> [Mon Jul  5 14:10:28 2010] [debug]
> DKB3:[SW-PROJEKTE.SPEZSRVW.axawl.spezpla.servers.SPgServer]SPg-legacy.c;100(1593)
>
> ***************
>
>
>
>
>
> The source of the problem is now fixed;
>
>
>
> scnt receiving from strlen(P1) the byte count was of unsigned __int16, too
> short to keep a value bigger then 65535!!!!!
>
>
>
> scnt must be of type unsigned __int32 and only then it is able to keep the
> number 91344 !
>
>
>
> One can take the calculator and enter 91344 as decimal number, then convert
> to hex and subtract 10000 Hex, then convert back to decimal.
>
> A too short type of an integer maskes out what strlen() returns correctly.
>
>
>
>
>
> BUT when the proper length of bytes is passed to
> axiom_xml_reader_create_for_memory()
>
> the routine works fine and does not bail out with a -- SEVERITY ERROR
>
>
>
> Conclusion of the day! If you work with very long strings (longer then FFFF
> === 65535) it is essential that you use
>
> “unsigned __int32 OR unsigned __int64 as type for then “scnt” = strlen(P1)”
> with P1 pointing to a string longer than 65535 characters.
>
>
>
>
>
> Now back to you: How do you detect that your XML is broken? Which routine
> informs you about that?
>
>
>
> Sepp
>
>
>
>
>
> ///////////////////////////////////////////////////////////
>
> //
>
> // function: WSI$$fktmap
>
> //
>
> //
>
> //
>
> ///////////////////////////////////////////////////////////
>
> static unsigned int WSI$$fktmap (unsigned int  sessionID,
>
>                                   *MSGB          *pbInData,*
>
>                                   *int           inLen,*
>
>                                   MSGB          **ppbOutData,
>
>                                   int           *pOutLen)
>
> {
>
> #ifdef __DEBUG__
>
>     lib$signal(SS$_DEBUG);
>
> #endif
>
>     char    wsi$ret [4];
>
>     dsc_sd  *wsi$p_dx;
>
>     dsc_a   *wsi$arydsc;
>
>     int     wsi$pnum = 2;
>
>     int     wsi$size = 0;
>
>     encoder wsi$enc;
>
>     dsc_sd  wsi$ret_dx = {4, DSC$K_DTYPE_L, DSC$K_CLASS_S, wsi$ret, 0, 0,
> 0};
>
>     internal_context_t *wsi$context = (internal_context_t *) sessionID;
>
>
>
>     wsi$p_dx = (dsc_sd *) pbInData;
>
>
>
>     // All pointers are converted from relative to absolute addresses
> here.
>
>     wsi$p_dx[0].dsc$a_pointer += (int) wsi$p_dx;
>
>     wsi$p_dx[1].dsc$a_pointer += (int) wsi$p_dx;
>
>
>
>     // Special parameters, such as Floats & Dynamic Strings, are fixup
> here.
>
>     wsi$dstr_fix( &wsi$p_dx[1] );
>
>
>
>     // Structure parameters are fixed up here.
>
>
>
> #pragma names restore   // we escape to Spg-Legacy and Axis2 code here
> which is compiled AS_IS & SHORTENED
>
>     // user-side function call
>
>     *(__int32 *)wsi$ret_dx.dsc$a_pointer =  fktmap((char *)
> wsi$p_dx[0].dsc$a_pointer, inLen, (dsc_d *) &wsi$p_dx[1]);
>
> #pragma names save
>
> #pragma names uppercase, shortened        // and back in HP WSIt world go
> for UPPERCASE & SHORTENED
>
>
>
>     // OUT & IN/OUT Floats get converted back to IEEE here.
>
>     // OUT & IN/OUT structures get fixup up here.
>
>     // OUT & IN/OUT strings get fixed up here
>
>
>
>     // Determine size of output buffer, size of static + dynamic out params
>
>     wsi$size = 28 + wsi$p_dx[1].dsc$w_length;
>
>     wsi$context->mem_disp->malloc( wsi$context->mem_disp->cnxid, wsi$size,
> (void **) ppbOutData );
>
>     wsi$init_encoder( &wsi$enc, *ppbOutData, wsi$pnum );
>
>
>
>     // Package up the parameters for returning here.
>
>     wsi$add_param( &wsi$ret_dx,  &wsi$enc );
>
>     wsi$add_param( &wsi$p_dx[1], &wsi$enc );
>
>
>
>     // Deallocate any memory used by Dynamic Strings here.
>
>     wsi$dstr_free( &wsi$p_dx[1] );
>
>
>
>     // Return here
>
>     *pOutLen = wsi$size;
>
>     return error_status_ok;
>
> }
>
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Akos Marton [mailto:makos999@gmail.com]
>
> Gesendet: Montag, 5. Juli 2010 11:13
>
>
> An: Apache AXIS C User List
> Betreff: Re: AW: transfer limitations
>
>
>
> Hi Sepp,
>
>
>
> On Mon, Jul 5, 2010 at 9:24 AM, Stadelmann Josef
>
> <josef.stadelmann@axa-winterthur.ch> wrote:
>
> > Given you send ALL your XML data into a file and parse it. Does the
> parser bail out somewhere?
>
> I also thought it, but varying the sent amount of data, thus XML is
>
> not broken at the same point in every case.
>
>
>
> >
>
> > I just have similar situation. There is (in my mind) too much data
> delivered; AND - Are you working with some kind of Strings through char*?
>
> Yes, axis2_char_t*.
>
>
>
> >
>
> > How long can a char* String be? How far does this pointer reach? Writing
> all my XML data into a file and checking the XML Structure shows a very well
> formed XML file; But any attempt to parse it through the
> axis2_libxml2_reader_wrapper is a no go and it bails out with various errors
> toward the end of the files-content.- no I  memory.
>
> I may not got what you mean above about the long of char*. You can
>
> allocate arbitrary (depends on your OS) amount of space, using a char*
>
> if you use dynamic memory management. How did you strike it out?
>
>
>
> >
>
> > Pointer size? 16, 32, 64?
>
> The size of pointer depends on your hardware and operating system, so
>
> that on 32bit operating system pointer has size of 4bytes.
>
>
>
> Now, I'm experimenting and learning how to configure the .Net side
>
> using WCF-technologies and set it to send data in chunks using MTOM.
>
> It seems for me, .Net-.Net can communicate with each other very well
>
> (of course), but needs more portability configuration to make it to be
>
> able to handle MTOM.
>
> In spite this we don't understand the reason of broken XMLs.
>
>
>
> Akos
>
>
>
> >
>
> > Sepp
>
> >
>
> > -----Ursprüngliche Nachricht-----
>
> > Von: Akos Marton [mailto:makos999@gmail.com]
>
> > Gesendet: Freitag, 18. Juni 2010 10:20
>
> > An: Apache AXIS C User List
>
> > Betreff: Re: AW: transfer limitations
>
> >
>
> > The problem is still not solved, so that any idea and answer is
> appreciated.
>
> >
>
> > You can read a mail on c-user list about the same problem by Jan-fon,
>
> > except that he found the same situation on client-side.
>
> > subject:        'Client to receive large data problem'
>
> > date:           Thu, 25 Feb 2010 11:07:53 GMT
>
> >
> http://mail-archives.apache.org/mod_mbox/axis-c-user/201002.mbox/%3CCC8FF067F7F4E54796AFDF21339BA0FC134A94BA@orsmsx505.amr.corp.intel.com%3E
>
> >
>
> > Hope it helps,
>
> > Akos Marton
>
> >
>
> >
>
> > Stadelmann Josef wrote:
>
> >> Have you setup to send in chunks from CL to SV or do you expect that
> more than 4 MB will go at once, within one request object?
>
> >> Sepp
>
> >>
>
> >> -----Ursprüngliche Nachricht-----
>
> >> Von: Akos Marton [mailto:makos999@gmail.com]
>
> >> Gesendet: Freitag, 11. Juni 2010 15:31
>
> >> An: Apache AXIS C User List
>
> >> Betreff: Re: transfer limitations
>
> >>
>
> >> The XML is broken every time at different part. I tried to configure
> httpd server with switching out many known size limitation and run axis2c
> under this, but it also doesn't send the whole XML.
>
> >>
>
> >> The sent data is just broken somewhere. I watched it with tcpdump.
>
> >> Smaller XMLs can be transferred.
>
> >>
>
> >> Well, written a short tester code, which reads the necessary XML from
> file and parses, decodes it perfectly, so that the algorithm and code must
> do the right.
>
> >>
>
> >> Any idea why the sending process is being broken?
>
> >>
>
> >> Thanks,
>
> >> mAkos
>
> >>
>
> >>
>
> >> Akos Marton wrote:
>
> >>> Hi List,
>
> >>>
>
> >>> Is there any limitation about the amount of data in a certain XML-tag?
>
> >>> In case of true, could I increase it somehow?
>
> >>> For instance <image> ... </image> by the size of 4Megabytes
of base64
>
> >>> code. The whole XML data does not reach the server (have cut at
>
> >>> certain part), but a smaller amount of data reaches it fully.
>
> >>>
>
> >>> Could be httpd.conf a solution? I do not know how to locate it for
>
> >>> axis2c binary version.
>
> >>>
>
> >>> Thanks,
>
> >>> mAkos
>
> >>>
>
> >>>
>
> >>
>
> >>
>
> >>
>
> >> ---------------------------------------------------------------------
>
> >> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
>
> >> For additional commands, e-mail: c-user-help@axis.apache.org
>
> >>
>
> >>
>
> >
>
> >
>
> > ---------------------------------------------------------------------
>
> > To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
>
> > For additional commands, e-mail: c-user-help@axis.apache.org
>
> >
>
> >
>
> > ---------------------------------------------------------------------
>
> > To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
>
> > For additional commands, e-mail: c-user-help@axis.apache.org
>
> >
>
> >
>
>
>
>
>
>
>
> --
>
> People seldom notice clothes, if you wear a big smile.
>
> My grandmother uses Ubuntu, don't you think so?!
>
> OLVASD: http://www.xkcd.org/
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: c-user-unsubscribe@axis.apache.org
>
> For additional commands, e-mail: c-user-help@axis.apache.org
>
>
>
> Samisa Abeysinghe
> VP Engineering
> WSO2 Inc.
> http://wso2.com
> http://wso2.org
>
>
>
Samisa Abeysinghe
VP Engineering
WSO2 Inc.
http://wso2.com
http://wso2.org

Mime
View raw message