axis-c-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samisa Abeysinghe <samisa.abeysin...@gmail.com>
Subject Re: [AXIS2] MTOM performance Measuring
Date Sun, 14 May 2006 03:17:27 GMT
Thilina Gunarathne wrote:

> Hi,
>
>        With your latest changes you always read the file  in one go (i.e.
>     not in chunks) and you don't need a loop. The attached compact
>     code does
>     the same thing.
>
>  
> My suggestion,
> Don't read the file in to the memory at all...Wrap the file with 
> something like a DataHandler which we use on Axis2/java...You can get 
> sizes and other information using the wrapper..  Stream the 
> file directly to the output stream when we are serializing the message 
> with No or very low buffering in the memory..

Well that is exactly what we have done in the current implementation. 
Identical design to Java implementation.
We have the data handler, and it does the streaming stuff. We do not 
read the file until it is time to send it out.
Only thing is that we had to write everything, including the data 
handler. We are not so lucky as Java ;-)

Samisa...

> Our policy in Axis2/java is to read it only when need it...
> HTH,
>  
> ~Thilina
>  
>
>     Suggestion:
>        When we need to attach *very* large file, we may not be keep the
>     whole file in memory. IMO, this where chunk reading is really
>     useful. If
>     you are to implement this, you'll have to change the code from
>     data_handler -> mime_body_part -> mime_output to om_output so that
>     you'll keep only chunk_size amount of bytes in memory at any given
>     time.
>
>     -Nabeel
>
>     Samisa Abeysinghe wrote:
>
>     > Cool, so there sure is an improvement when using stat on read
>     side of
>     > data handler. Hence lets use this solution.
>     > Still we need to figure out what goes wrong with large files
>     beyond 1M
>     >
>     > Samisa...
>     >
>     > Kapila Dissanayake wrote:
>     >
>     >> Hi
>     >>
>     >> I did some modification to the data handler. Insted of using fixed
>     >> buffer size to read files
>     >> 'stat' is used.
>     >>
>     >> Please see the perfomance details.
>     >>
>     >> Thank You
>     >>
>     >> Kapila
>     >>
>     >>
>     >> Environment
>     >>
>     >>
>     >> ----------
>     >>
>     >>
>     >>
>     >> Processor - Intel Celeron M 1.40GHz
>     >> RAM - 256 MB
>     >>
>     >>
>     >> OS - Fedora Core 4.
>     >>
>     >> Java Server - axis2 Java
>     >>
>     >> C Server - axis2
>     >>
>     >>
>     >> mtom C and java test clients were used     (in axis2 c
>     >> data_handler.c, different buffer sizes were used to read files)
>     >> Time Taken in milli seconds
>     >>
>     >>
>     >>
>     >> File Size (kb)           Java Server     (ms)     Java Server
>     >> (ms)      C Server (ms)                 C Server     (ms)
>     >>             Java Client     (ms)     C Client     (ms)      C
>     >> Client     (ms)                  Java Client     (ms)
>     >>                                     *Buffer Size 1024 *     *Buffer
>     >> Size 1024 X 1024 * *Using stat*     * *     *Buffer Size '1024 *
>     >> *Buffer Size '1024 X 1024 * *Using stat *
>     >> -----------------     -----------------     -----------------
>     >> -----------------     -----------------     -----------------
>     >> -----------------     -----------------     -----------------
>     >> -----------------     -----------------     -----------------
>     >> -----------------
>     >> 13.8           147           158           73     1884
>     >> 78           188     143     136
>     >> 28.2           515           488           78     1912
>     >> 78           222     413     364
>     >> 37.8           538           694           131     1917
>     >> 81           232     474     377
>     >> 132           571           807           212     2177
>     >> 102           263     420     381
>     >> 154           547           672           239     2215
>     >> 117           347     389     383
>     >> 165           688           683           397     2380
>     >> 205           364     400     393
>     >> 561           1340           1024           995     2537
>     >> 777           637     899     775
>     >> 762           1766           1336           1903     2890
>     >> 1458           1248     1218     1051
>     >> 1162           3464           1970           4859     3602
>     >> 3405           1702     1718     1712
>     >> 1761           4666           3274           10928     8028
>     >> 7927           2780     3249     2691
>     >> 2081           6777           2993           15066     11008
>     >> 10864           3595     3281     2859
>     >> 4241           23038           Could not transfer (Java client
>     >> Broken)           52853 33351     32957           16234
>     >> 115777     7991
>     >>
>     >>
>     >>
>     >>
>     >> On 5/10/06, *Samisa Abeysinghe* < samisa.abeysinghe@gmail.com
>     <mailto:samisa.abeysinghe@gmail.com>
>     >> <mailto:samisa.abeysinghe@gmail.com
>     <mailto:samisa.abeysinghe@gmail.com>>> wrote:
>     >>
>     >>     Well, from this I am certain that we should not change the
>     buffer
>     >> size
>     >>     to 1024*1024, rather keep it at 1024.
>     >>     Please try with stat, using a dynamic buffer size based on
>     the file
>     >>     size, my gut feel is that it is going to be the best solution.
>     >>
>     >>     Thanks,
>     >>     Samisa...
>     >>
>     >>     Kapila Dissanayake wrote:
>     >>
>     >>     >
>     >>     > Hi,
>     >>     >
>     >>     > I did some testing agian with to measure the binary file
>     transfer
>     >>     > times. In axis2 c data_handler.c, different buffer sizes
>     (1024 and
>     >>     > 1024*1024) were used to read files.
>     >>     >
>     >>     > I think using fixed size buffers to read binary files in
>     axis2-C
>     >>     has
>     >>     > been affected in some cases to slow down the transfer.
>     >>     >
>     >>     > Thanks
>     >>     >
>     >>     > Kapila
>     >>     >
>     >>     > Please see the following test results
>     >>     >
>     >>     > Environment
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > ----------
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > Processor - Intel Celeron M 1.40GHz
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > RAM - 256 MB
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > OS - Fedora Core 4.
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > Java Server - axis2 Java
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > C Server - axis2
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > mtom C and java test clients were used
>     >>     >
>     >>     >
>     >>     >
>     >>     > Time Taken in milli seconds
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > File Size (kb)                Java Server     (ms)    Java
>     >>     Server     (ms)             C
>     >>     > Server        (ms)            C Server        (ms)
>     >>     >               Java Client     (ms)    C
>     >>     Client        (ms)             C Client      
>     (ms)             Java
>     >>     > Client        (ms)
>     >>     >                                               *Buffer Size*
>     >>     *1024 *         *1024 X 1024*   * *     *1024*
>     >>     > *1024 X 1024 *
>     >>     > -----------------     -----------------      
>     -----------------
>     >>     > -----------------     -----------------      
>     -----------------
>     >>     > -----------------     -----------------      
>     -----------------
>     >>     > -----------------     -----------------      
>     -----------------
>     >>     > 13.8          147             158
>     >> 73      1884            188     143
>     >>     > 28.2          515             488
>     >> 78      1912            222     413
>     >>     > 37.8          538             694                     131
>     >>     1917            232     474
>     >>     > 132           571             807                     212
>     >>     2177            263     420
>     >>     > 154           547             672                     239
>     >>     2215            347     389
>     >>     > 165           688             683                     397
>     >>     2380            364     400
>     >>     > 561           1340            1024                    995
>     >>     2537            637     899
>     >>     > 762              1766            1336
>     >> 1903    2890            1248    1218
>     >>     >
>     >>     1162          3464            1970                    4859
>     >> 3602            1702    1718
>     >>
>     >>     > 1761          4666            3274                    10928
>     >>     8028            2780    3249
>     >>     > 2081          6777            2993                    15066
>     >>     11008           3595    3281
>     >>     > 4241          23038           Could not transfer (Java client
>     >>     Broken)
>     >>     > 52853         33351           16234   115777
>     >>     >
>     >>     >
>     >>     >
>     >>     >
>     >>     > On 5/8/06, *Samisa Abeysinghe* <samisa@wso2.com
>     <mailto:samisa@wso2.com>
>     >>     <mailto: samisa@wso2.com <mailto:samisa@wso2.com>>
>     >>     > <mailto: samisa@wso2.com <mailto:samisa@wso2.com>
>     <mailto:samisa@wso2.com <mailto:samisa@wso2.com>>>> wrote:
>     >>     >
>     >>     >     Hmmm, Interesting. There seem to be a problem in C
>     client in
>     >>     case of
>     >>     >     large attachments.
>     >>     >     This must be due to the fact that we are using a loop to
>     >>     read the
>     >>     >     file.
>     >>     >     If we stat and read the file at once, there could be an
>     >>     improvement.
>     >>     >
>     >>     >     Why is it so slow, in C vs C case? (Third column) This is
>     >> worth
>     >>     >     investigating.
>     >>     >
>     >>     >     Thanks,
>     >>     >     Samisa...
>     >>     >
>     >>     >     Kapila Dissanayake wrote:
>     >>     >
>     >>     >     > Hi,
>     >>     >     >
>     >>     >     > I did some testing on binary attachment transfer
>     times,
>     >>     with axis2
>     >>     >     > 'java' and 'C' servers with sample mtom clients. It
>     took
>     >>     very high
>     >>     >     > transfer times for large size attachments. Some minor
>     >>     modificaions
>     >>     >     > were done to the data_handler.c and it was sent to the
>     >>     mailing list.
>     >>     >     >
>     >>     >     >
>     >>     >     > (File Size) vs (Time Taken in milli seconds) to
>     transfer
>     >>     binary
>     >>     >     > attachments
>     >>     >     >
>     >>     >     > Environment
>     >>     >     > ----------
>     >>     >     > Processor - Intel Celeron M 1.40GHz
>     >>     >     > RAM - 256 MB
>     >>     >     > OS - Fedora Core 4.
>     >>     >     > Java Server - axis2 Java
>     >>     >     > C Server - axis2
>     >>     >     > mtom C and java test clients were used
>     >>     >     > Time Taken in milli seconds
>     >>     >     >
>     >>     >     > File Size (kb)
>     >>     >     >
>     >>     >     > Java Server
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     > Java Server
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     >  C Server
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     > C Server
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > Java Client
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     > C Client
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     >  C Client
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     >
>     >>     >     >  Java Client
>     >>     >     >
>     >>     >     > (ms)
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     >
>     >>     >     > ---------------
>     >>     >     > 28
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 445
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 274
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2004
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 1009
>     >>     >     >
>     >>     >     >
>     >>     >     > 38
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 617
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 297
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 1957
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 842
>     >>     >     >
>     >>     >     >
>     >>     >     > 132
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 593
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 527
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2181
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 726
>     >>     >     >
>     >>     >     >
>     >>     >     > 154
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 551
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 447
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2081
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 677
>     >>     >     >
>     >>     >     >
>     >>     >     > 165
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 440
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 743
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2147
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 676
>     >>     >     >
>     >>     >     >
>     >>     >     > 561
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 1029
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 418
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2581
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 1075
>     >>     >     >
>     >>     >     >
>     >>     >     > 762
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2137
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 1572
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 3231
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2356
>     >>     >     >
>     >>     >     >
>     >>     >     > 1162
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 3172
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 2962
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 3981
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 3100
>     >>     >     >
>     >>     >     >
>     >>     >     > 1761
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 7672
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 5494
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 8186
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 3041
>     >>     >     >
>     >>     >     >
>     >>     >     > 2081
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 7768
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 6837
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 11757
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 3499
>     >>     >     >
>     >>     >     >
>     >>     >     > 4241
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 14465
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 24060
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > 36168
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > Could not transfer (Error Occured in Java client)
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     >
>     >>     >     > Thank You
>     >>     >     >
>     >>     >     > Kapila
>     >>     >
>     >>     >
>     >>     >
>     >>
>     >>
>     >
>     >
>
>
>
>     #include <stdio.h>
>     #include <sys/stat.h>
>
>     int read_from_file(char *file_name, char** output_stream, int
>     *output_stream_size);
>
>     int main(int argc, char **argv)
>     {
>            int size;
>            char *output;
>
>            if (argc != 2)
>            {
>                    printf("usage: read_binary <file_name>\n");
>                    return 0;
>            }
>            if (read_from_file(argv[1], &output, &size))
>                            printf("read successful\n");
>            if (output)
>                    free(output);
>            return 1;
>     }
>
>     int read_from_file(char *file_name, char** output_stream, int
>     *output_stream_size)
>     {
>            FILE *f = NULL;
>            struct stat stat_p;
>            int count = 0;
>
>        if (!file_name)
>        {
>                    printf("error: file not given \n");
>                    return 0;
>            }
>
>        f = fopen(file_name, "rb");
>        if (!f)
>            {
>                    printf("error:cannot open file %s\n", file_name);
>            return ;
>            }
>
>            if ( -1 ==  stat (file_name, &stat_p))
>            {
>                    printf("error:error reading stat\n");
>                    return 0;
>            }
>
>            *output_stream_size = stat_p.st_size;
>            printf("size = %d\n",  *output_stream_size);
>            *output_stream = malloc((*output_stream_size) * sizeof(char));
>
>            if (!(*output_stream))
>        {
>            printf("error: cannot create stream\n");
>            return 0;
>            }
>        count = fread(*output_stream, 1, *output_stream_size, f);
>        if (ferror(f) != 0)
>        {
>            printf("error:cannot read file\n");
>            if (*output_stream)
>            {
>                    free(*output_stream);
>                *output_stream = NULL;
>            }
>            return 0;
>
>            }
>        fclose(f);
>        return 1;
>     }
>
>
>
>
>
> -- 
> "May the SourcE be with u"            
> http://webservices.apache.org/~thilina/ 
> <http://webservices.apache.org/%7Ethilina/>
> http://thilinag.blogspot.com/                 
> http://www.bloglines.com/blog/Thilina 
> <http://www.bloglines.com/blog/Thilina>               



Mime
View raw message