axis-c-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kapila Dissanayake" <kapi...@gmail.com>
Subject Re: [AXIS2] MTOM performance Measuring - http_transport_utils.c modified
Date Sat, 13 May 2006 05:14:59 GMT
Hi ,

I did another modification to improve the messege transfer speed.

http_transport_utils.c was modified. (Buffer size was changed from 1024 to
1024 * 1024)

Now the transfer speed has been changed significantly. Please see the table
below.


The attached file includes the svn diff of the change.

Thank You

Kapila


         (File Size) vs (Time Taken in milli seconds) to send binary
attachments             Environment        ----------        Processor -
Intel Celeron M 1.40GHz       RAM - 256 MB        OS - Fedora Core 4.
     Java
Server - axis2 Java       C Server - axis2        mtom C and java test
clients were used       http_transport_utils.c was modified. (Buffer size
was changed from 1024 to 1024 * 1024)     Time Taken in milli seconds
             File
Size (kb)   Java Server   Java Server    C Server  C Server  C Server  C
Server   C Server  C Server  C Server  C Server     Java Client (ms)   C
Client (ms)    C Client (ms)          Java Client (ms)
*data_handler.c
Buffer Size 1024* *data_handler.c 'Buffer Size 1024 X 1024* *Using
stat* *http_transport_utils.c
'Buffer Size 1024 X 1024* * * *Buffer Size '1024* *Buffer Size '1024 X 1024*
*Using stat* *http_transport_utils.c 'Buffer Size 1024 X 1024*
----------------- ----------------- ----------------- -----------------
----------------- ----------------- ----------------- -----------------
----------------- --------------------------------- -----------------
----------------- ----------------- -----------------
--------------------------------- 13.8   147   158   73 1884 78 76   188 143
136 136 28.2   515   488   78 1912 78 78   222 413 364 387 37.8   538   694
  131 1917 81 81   232 474 377 396 132   571   807   212 2177 102 83   263
420 381 432 154   547   672   239 2215 117 84   347 389 383 517 165   688
683   397 2380 205 85   364 400 393 602 561   1340   1024   995 2537 777 100
  637 899 775 766 762   1766   1336   1903 2890 1458 111   1248 1218 1051
1055 1162   3464   1970   4859 3602 3405 135   1702 1718 1712 1092 1761
4666   3274   10928 8028 7927 188   2780 3249 2691 2596 2081   6777   2993
15066 11008 10864 219   3595 3281 2859 3185 4241   23038   Could not
transfer (Java client Broken)   52853 33351 32957
519   16234 115777 7991 6499


On 5/12/06, Samisa Abeysinghe <samisa.abeysinghe@gmail.com> wrote:
>
> BTW,
>    If we are going to use chunking at some point in time, then the data
> handler logic for reading file too has to change to keep track of the
> amount of chunks already read, and current position of read pointer.
> There is no provision for such reading in the code as of now, neither in
> the data handler nor in the mime_output that uses the data handler API.
> So there is considerable amount of work on the part of the data hander
> and the places that uses the API of data handler to read to turn on
> chunking as far as I can see.
> Thanks,
> Samisa...
>
> Samisa Abeysinghe wrote:
>
> > Nabeel wrote:
> >
> >> Kapila,
> >>    With your latest changes you always read the file  in one go
> >> (i.e.  not in chunks) and you don't need a loop. The attached compact
> >> code does the same thing.
> >>
> >> Suggestion:
> >>    When we need to attach *very* large file, we may not be keep the
> >> whole file in memory. IMO, this where chunk reading is really useful.
> >> If you are to implement this, you'll have to change the code from
> >> data_handler -> mime_body_part -> mime_output to om_output so that
> >> you'll keep only chunk_size amount of bytes in memory at any given
> time.
> >
> >
> > Chunking is a very good idea, in fact that is the perfect solution for
> > large attachments.
> > However, AFAIK, we have chunking capability only on the client side
> > transport, not on the server side transport.
> > Hence, we got to implement chunking capability into simple axis server
> > transport first. (May be we can raise a Jira on this). Or we have to
> > see if we can piggyback on Apache for chunking.
> > Secondly, we got to research and find out the ideal chunk size. As we
> > have seen through the results sent by Kapila, too small a chunk size
> > makes too large file transmission very slow and too large chunk size
> > makes small file transmission very slow. Best is to not use chunking
> > for small files and use chunking, with a reasonable chunk size, for
> > large files. So we have several parameters to figure out.
> >    i.  Max size we can read a file in one go without chunking.
> >    ii. When chunking is turned on, ideal chunk size.
> > As I mentioned earlier, we need some research around this to decide on
> > the figures.
> >
> > Thanks,
> > Samisa...
> >
> >>
> >> -Nabeel
> >>
> >> Samisa Abeysinghe wrote:
> >>
> >>> Cool, so there sure is an improvement when using stat on read side
> >>> of data handler. Hence lets use this solution.
> >>> Still we need to figure out what goes wrong with large files beyond 1M
> >>>
> >>> Samisa...
> >>>
> >>> Kapila Dissanayake wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> I did some modification to the data handler. Insted of using fixed
> >>>> buffer size to read files
> >>>> 'stat' is used.
> >>>>
> >>>> Please see the perfomance details.
> >>>>
> >>>> Thank You
> >>>>
> >>>> Kapila
> >>>>
> >>>>
> >>>> Environment            ----------                Processor - Intel
> >>>> Celeron M 1.40GHz
> >>>> RAM - 256 MB            OS - Fedora Core 4.        Java Server -
> >>>> axis2 Java        C Server - axis2            mtom C and java test
> >>>> clients were used     (in axis2 c data_handler.c, different buffer
> >>>> sizes were used to read files)
> >>>> Time Taken in milli seconds
> >>>> File Size (kb)           Java Server     (ms)     Java Server
> >>>> (ms)      C Server (ms)                 C Server     (ms)
> >>>>             Java Client     (ms)     C Client     (ms)      C
> >>>> Client     (ms)                  Java Client     (ms)
> >>>>                                     *Buffer Size 1024 *     *Buffer
> >>>> Size 1024 X 1024 * *Using stat*     * *     *Buffer Size '1024
> >>>> *     *Buffer Size '1024 X 1024 * *Using stat *
> >>>> -----------------     -----------------     -----------------
> >>>> -----------------     -----------------     -----------------
> >>>> -----------------     -----------------     -----------------
> >>>> -----------------     -----------------     -----------------
> >>>> -----------------
> >>>> 13.8           147           158           73     1884
> >>>> 78           188     143     136
> >>>> 28.2           515           488           78     1912
> >>>> 78           222     413     364
> >>>> 37.8           538           694           131     1917
> >>>> 81           232     474     377
> >>>> 132           571           807           212     2177
> >>>> 102           263     420     381
> >>>> 154           547           672           239     2215
> >>>> 117           347     389     383
> >>>> 165           688           683           397     2380
> >>>> 205           364     400     393
> >>>> 561           1340           1024           995     2537
> >>>> 777           637     899     775
> >>>> 762           1766           1336           1903     2890
> >>>> 1458           1248     1218     1051
> >>>> 1162           3464           1970           4859     3602
> >>>> 3405           1702     1718     1712
> >>>> 1761           4666           3274           10928     8028
> >>>> 7927           2780     3249     2691
> >>>> 2081           6777           2993           15066     11008
> >>>> 10864           3595     3281     2859
> >>>> 4241           23038           Could not transfer (Java client
> >>>> Broken)           52853 33351     32957           16234
> >>>> 115777     7991
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 5/10/06, *Samisa Abeysinghe* <samisa.abeysinghe@gmail.com
> >>>> <mailto:samisa.abeysinghe@gmail.com>> wrote:
> >>>>
> >>>>     Well, from this I am certain that we should not change the
> >>>> buffer size
> >>>>     to 1024*1024, rather keep it at 1024.
> >>>>     Please try with stat, using a dynamic buffer size based on the
> >>>> file
> >>>>     size, my gut feel is that it is going to be the best solution.
> >>>>
> >>>>     Thanks,
> >>>>     Samisa...
> >>>>
> >>>>     Kapila Dissanayake wrote:
> >>>>
> >>>>     >
> >>>>     > Hi,
> >>>>     >
> >>>>     > I did some testing agian with to measure the binary file
> >>>> transfer
> >>>>     > times. In axis2 c data_handler.c, different buffer sizes
> >>>> (1024 and
> >>>>     > 1024*1024) were used to read files.
> >>>>     >
> >>>>     > I think using fixed size buffers to read binary files in
> axis2-C
> >>>>     has
> >>>>     > been affected in some cases to slow down the transfer.
> >>>>     >
> >>>>     > Thanks
> >>>>     >
> >>>>     > Kapila
> >>>>     >
> >>>>     > Please see the following test results
> >>>>     >
> >>>>     > Environment
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > ----------
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Processor - Intel Celeron M 1.40GHz
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > RAM - 256 MB
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > OS - Fedora Core 4.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Java Server - axis2 Java
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > C Server - axis2
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > mtom C and java test clients were used
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Time Taken in milli seconds
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > File Size (kb)                Java Server     (ms)    Java
> >>>>     Server     (ms)             C
> >>>>     > Server        (ms)            C Server        (ms)
> >>>>     >               Java Client     (ms)    C
> >>>>     Client        (ms)             C Client       (ms)
> >>>> Java
> >>>>     > Client        (ms)
> >>>>     >                                               *Buffer Size*
> >>>>     *1024 *         *1024 X 1024*   * *     *1024*
> >>>>     > *1024 X 1024 *
> >>>>     > -----------------     -----------------       -----------------
> >>>>     > -----------------     -----------------       -----------------
> >>>>     > -----------------     -----------------       -----------------
> >>>>     > -----------------     -----------------       -----------------
> >>>>     > 13.8          147             158
> >>>> 73      1884            188     143
> >>>>     > 28.2          515             488
> >>>> 78      1912            222     413
> >>>>     > 37.8          538             694                     131
> >>>>     1917            232     474
> >>>>     > 132           571             807                     212
> >>>>     2177            263     420
> >>>>     > 154           547             672                     239
> >>>>     2215            347     389
> >>>>     > 165           688             683                     397
> >>>>     2380            364     400
> >>>>     > 561           1340            1024                    995
> >>>>     2537            637     899
> >>>>     > 762              1766            1336
> >>>> 1903    2890            1248    1218
> >>>>     >
> >>>>     1162          3464            1970                    4859
> >>>> 3602            1702    1718
> >>>>
> >>>>     > 1761          4666            3274                    10928
> >>>>     8028            2780    3249
> >>>>     > 2081          6777            2993                    15066
> >>>>     11008           3595    3281
> >>>>     > 4241          23038           Could not transfer (Java client
> >>>>     Broken)
> >>>>     > 52853         33351           16234   115777
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > On 5/8/06, *Samisa Abeysinghe* <samisa@wso2.com
> >>>>     <mailto:samisa@wso2.com>
> >>>>     > <mailto: samisa@wso2.com <mailto:samisa@wso2.com>>>
wrote:
> >>>>     >
> >>>>     >     Hmmm, Interesting. There seem to be a problem in C client
> in
> >>>>     case of
> >>>>     >     large attachments.
> >>>>     >     This must be due to the fact that we are using a loop to
> >>>>     read the
> >>>>     >     file.
> >>>>     >     If we stat and read the file at once, there could be an
> >>>>     improvement.
> >>>>     >
> >>>>     >     Why is it so slow, in C vs C case? (Third column) This
is
> >>>> worth
> >>>>     >     investigating.
> >>>>     >
> >>>>     >     Thanks,
> >>>>     >     Samisa...
> >>>>     >
> >>>>     >     Kapila Dissanayake wrote:
> >>>>     >
> >>>>     >     > Hi,
> >>>>     >     >
> >>>>     >     > I did some testing on binary attachment transfer times,
> >>>>     with axis2
> >>>>     >     > 'java' and 'C' servers with sample mtom clients. It
took
> >>>>     very high
> >>>>     >     > transfer times for large size attachments. Some minor
> >>>>     modificaions
> >>>>     >     > were done to the data_handler.c and it was sent to
the
> >>>>     mailing list.
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > (File Size) vs (Time Taken in milli seconds) to transfer
> >>>>     binary
> >>>>     >     > attachments
> >>>>     >     >
> >>>>     >     > Environment
> >>>>     >     > ----------
> >>>>     >     > Processor - Intel Celeron M 1.40GHz
> >>>>     >     > RAM - 256 MB
> >>>>     >     > OS - Fedora Core 4.
> >>>>     >     > Java Server - axis2 Java
> >>>>     >     > C Server - axis2
> >>>>     >     > mtom C and java test clients were used
> >>>>     >     > Time Taken in milli seconds
> >>>>     >     >
> >>>>     >     > File Size (kb)
> >>>>     >     >
> >>>>     >     > Java Server
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     > Java Server
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     >  C Server
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     > C Server
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > Java Client
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     > C Client
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     >  C Client
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     >
> >>>>     >     >  Java Client
> >>>>     >     >
> >>>>     >     > (ms)
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     >
> >>>>     >     > ---------------
> >>>>     >     > 28
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 445
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 274
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2004
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1009
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 38
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 617
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 297
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1957
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 842
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 132
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 593
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 527
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2181
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 726
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 154
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 551
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 447
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2081
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 677
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 165
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 440
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 743
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2147
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 676
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 561
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1029
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 418
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2581
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1075
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 762
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2137
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1572
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 3231
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2356
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1162
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 3172
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2962
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 3981
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 3100
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 1761
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 7672
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 5494
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 8186
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 3041
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 2081
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 7768
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 6837
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 11757
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 3499
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 4241
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 14465
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 24060
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > 36168
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > Could not transfer (Error Occured in Java client)
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     >
> >>>>     >     > Thank You
> >>>>     >     >
> >>>>     >     > Kapila
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> ------------------------------------------------------------------------
> >>
> >> #include <stdio.h>
> >> #include <sys/stat.h>
> >>
> >> int read_from_file(char *file_name, char** output_stream, int
> >> *output_stream_size);
> >>
> >> int main(int argc, char **argv)
> >> {
> >>     int size;
> >>     char *output;
> >>
> >>     if (argc != 2)
> >>     {
> >>         printf("usage: read_binary <file_name>\n");
> >>         return 0;
> >>     }
> >>     if (read_from_file(argv[1], &output, &size))
> >>             printf("read successful\n");
> >>     if (output)
> >>         free(output);
> >>     return 1;
> >> }
> >>
> >> int read_from_file(char *file_name, char** output_stream, int
> >> *output_stream_size)
> >> {
> >>     FILE *f = NULL;
> >>     struct stat stat_p;
> >>     int count = 0;
> >>
> >>    if (!file_name)
> >>    {
> >>         printf("error: file not given \n");
> >>         return 0;
> >>     }
> >>           f = fopen(file_name, "rb");
> >>    if (!f)
> >>     {
> >>         printf("error:cannot open file %s\n", file_name);
> >>        return ;
> >>     }
> >>            if ( -1 ==  stat (file_name, &stat_p))
> >>     {
> >>         printf("error:error reading stat\n");
> >>         return 0;
> >>     }
> >>
> >>     *output_stream_size = stat_p.st_size;
> >>     printf("size = %d\n",  *output_stream_size);
> >>     *output_stream = malloc((*output_stream_size) * sizeof(char));
> >>
> >>     if (!(*output_stream))
> >>    {
> >>        printf("error: cannot create stream\n");
> >>        return 0;
> >>     }
> >>    count = fread(*output_stream, 1, *output_stream_size, f);
> >>    if (ferror(f) != 0)
> >>    {
> >>        printf("error:cannot read file\n");
> >>        if (*output_stream)
> >>        {
> >>            free(*output_stream);
> >>            *output_stream = NULL;
> >>        }
> >>        return 0;
> >>
> >>     }
> >>    fclose(f);
> >>    return 1;
> >> }
> >>
> >>
> >
> >
>
>

Mime
View raw message