trafficserver-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Galić <i.ga...@brainsware.org>
Subject Re: git commit: document update for hipes and buffer_upload plugins
Date Sun, 29 Dec 2013 18:56:03 GMT


----- Original Message -----
> Updated Branches:
>   refs/heads/master 8d8507f3f -> b36a7e120
> 
> 
> document update for hipes and buffer_upload plugins
> 
> 
> Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
> Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/b36a7e12
> Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/b36a7e12
> Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/b36a7e12
> 
> Branch: refs/heads/master
> Commit: b36a7e120e4b9c216fe2f13a97015a508911e0a4
> Parents: 8d8507f
> Author: Kit Chan <kichan@apache.org>
> Authored: Sat Dec 28 09:06:04 2013 -0800
> Committer: Kit Chan <kichan@apache.org>
> Committed: Sat Dec 28 09:06:04 2013 -0800
> 
> ----------------------------------------------------------------------
>  doc/reference/plugins/buffer_upload.en.rst | 93 +++++++++++--------------
>  doc/reference/plugins/hipes.en.rst         | 22 ++++--
>  2 files changed, 58 insertions(+), 57 deletions(-)
> ----------------------------------------------------------------------
> 
> 
> http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b36a7e12/doc/reference/plugins/buffer_upload.en.rst
> ----------------------------------------------------------------------
> diff --git a/doc/reference/plugins/buffer_upload.en.rst
> b/doc/reference/plugins/buffer_upload.en.rst
> index ab5f209..022d61e 100644
> --- a/doc/reference/plugins/buffer_upload.en.rst
> +++ b/doc/reference/plugins/buffer_upload.en.rst
> @@ -18,72 +18,63 @@ Buffer Upload Plugin
>    specific language governing permissions and limitations
>    under the License.
>  
> +The Buffer Upload plugin offers the following features
>  
> -..  XXX Discribe what the heck this plugin actually does.
> +Installation
> +============
>  
> -Upload proxy specs for phase I:
> +Configuration can be explicitly specified as a parameter in
> ``plugin.config`` ::
>  
> -1. Memory buffering (buffer the entire POST data in IOBuffer before
> -   connecting to OS) 1.1. Memory buffer size is configured with
> -   "mem_buffer_size" in config file. Default and minimum value is 32K
> -   You can increase it in the config file. If the size of a request is
> -   larger than the "mem_buffer_size" value specifiied in the config
> -   file, then the upload proxy feature will be disabled for this
> -   particular request
> +    buffer_upload.so /FOOBAR/upload.conf
>  
> -2. Disk buffering (buffer the entire POST data on disk before connecting
> -   to OS) 2.1. Use disk async IO. This involved some changes in ATS core
> -   . new APIs wrapping around ink_aio_read() and ink_aio_write() .
> -   change to distinguish between api call's AIO and cache's AIO .
> -   guarantee api call's AIO only involves certain amount of threads .
> -   the number of threads is configurable in plugin's config file
> -   (default is 4)
> +Memory buffering (buffer the entire POST data in IOBuffer before connecting
> to OS)
> +==================================================================================
>  
> -3.
> +Memory buffer size is configured with "mem_buffer_size" in config file.
> Default and minimum value is 32K. You can

This should default to our POST max size, which currently, IIRC, is 64K.

> +increase it in the config file. If the size of a request is larger than the
> "mem_buffer_size" value specifiied in the
> +config file, then the upload proxy feature will be disabled for this
> particular request
>  
> -   2. Directories and files generated on disk . base directory:
> -      FOOBAR/var/buffer_upload_tmp/ (configurable in config file) .
> -      number of subdirectories: 64 (configurable in config file) .
> -      filename are randomly generated . files will be removed when the
> -      entire data have been sent out to OS . remove dangling files (left
> -      on disk due to transaction interruption or traffic server crash)
> -      at startup time
> +Disk buffering (buffer the entire POST data on disk before connecting to OS)
> +============================================================================
>  
> -4.
> +1. Disk async IO is used. AIO api call only involves certain amount of
> threads. The number of threads is configurable in
> +plugin's config file (default is 4)
>  
> -   3. Default chunk size when reading from disk: 16K, configurable in
> -      config file
> +2. Directories and files are generated on disk . Base directory is
> /FOOBAR/var/buffer_upload_tmp/ (configurable in
> +config file). Number of subdirectories is 64 (configurable in config file).
> Filename are randomly generated. Files will
> +be removed when the entire data have been sent out to OS . At startup time,
> dangling files are removed (left on disk due
> +to transaction interruption or traffic server crash)
>  
> -5. Default buffering mode: disk aio buffering mode 3.1. to turn off disk
> -   buffering, add a "use_disk_buffer 0" line in config file
> +3. Default chunk size when reading from disk is 16K, configurable in config
> file
>  
> -6. Trigger POST buffering on certain URLs 4.1. certain URLs will be
> -   provided in a plain text file (one URL each line) 4.2. specify
> -   filename in config file by "url_list_file" 4.3. max length of each
> -   URL: 4096 (configurable in config file) 4.4. use exact match, don't
> -   support regex for now
> +Trigger POST buffering on certain URLs
> +======================================
>  
> -7. URL conversion for Mail's specific URL format 5.1. for now check if
> -   the "host" part in the URL is same as the proxy server name, then
> -   will do this conversion 5.2. To turn on URL conversion feature, set
> -   "convert_url 1" in config file
> +1. Certain URLs will be provided in a plain text file (one URL each line)
> +2. Specify filename in config file by "url_list_file"
> +3. max length of each URL is 4096 (configurable in config file)
> +4. use exact match, don't support regex for now
>  
> -8. All request headers inlcuding cookies plus the entire POST data will
> -   be buffered (either in memory or on disk)
> +Other Features
> +==============
>  
> -9. Config file can be explicitly sepcified as a parameter in command
> -   line (in plugin.config file)
> +1. Default buffering mode is disk aio buffering mode. To turn off disk
> buffering, add a "use_disk_buffer 0" line in
> +config file
>  
> -a sample config file:
> +2. All request headers inlcuding cookies plus the entire POST data will be
> buffered (either in memory or on disk)
>  
> -use_disk_buffer 1 convert_url 1 chunk_size 1024 url_list_file
> -/tmp/url_list.conf max_url_length 10000 base_dir /tmp/test1
> -subdir_num 100 thread_num 10 mem_buffer_size 40000
> +Configuration File
> +==================
>  
> -default config file: FOOBAR/etc/upload.conf
> +sample config file ::
>  
> -default config values: use_disk_buffer 1 convert_url 0 chunk_size
> -16384 url_list_file none max_url_length 4096 base_dir
> -FOOBAR/var/buffer_upload_tmp subdir_num 64 thread_num 4
> -mem_buffer_size 32768
> +    use_disk_buffer 1
> +    convert_url 1
> +    chunk_size 1024
> +    url_list_file /tmp/url_list.conf
> +    max_url_length 10000
> +    base_dir /tmp/test1
> +    subdir_num 100
> +    thread_num 10
> +    mem_buffer_size 40000
>  
> 
> http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b36a7e12/doc/reference/plugins/hipes.en.rst
> ----------------------------------------------------------------------
> diff --git a/doc/reference/plugins/hipes.en.rst
> b/doc/reference/plugins/hipes.en.rst
> index e74a58a..20c847b 100644
> --- a/doc/reference/plugins/hipes.en.rst
> +++ b/doc/reference/plugins/hipes.en.rst
> @@ -62,7 +62,8 @@ only apply to the connection to the specific "service"
> host.
>  Notes on HIPES
>  ==============
>  
> -HTTP Pipes (aka HIPES and pronounced "Hippies") allows data services to be
> pipelined together, as illustrated by the example below.
> +HTTP Pipes (aka HIPES and pronounced "Hippies") allows data services to be
> pipelined together, as illustrated by the
> +example below.
>  
>  Example
>  =======
> @@ -90,25 +91,34 @@ to this URL ::
>  
>      http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt
>  
> -5. In this example, the service ``svc_case.php`` fetches and transforms the
> response of ``http://127.0.0.1/test.txt`` (which ATS proxies the request to
> a local file) to upper case. And subsequently the service
> ``svc_reverse.php`` receives the response and reverse the order before the
> response is sent back to the client by ATS.
> +5. In this example, the service ``svc_case.php`` fetches and transforms the
> response of ``http://127.0.0.1/test.txt``
> +(which ATS proxies the request to a local file) to upper case. And
> subsequently the service ``svc_reverse.php`` receives
> +the response and reverse the order before the response is sent back to the
> client by ATS.
>  
>  Notes on reducing traffic
>  =========================
>  
> -There can be significant overhead using HIPES because the data can traverse
> through ATS many times. Caching can be important to reduce traffic to
> services/through ATS and can be achieved via a proper ``Cache-Control``
> header returned by the services. Another way to reduce traffic through ATS
> is to have ATS to return 302 redirects to url for the requests made by
> service, instead of proxying the requests to that url. However, the service
> must then be able to follow the redirect. The down side is that we cannot
> use ATS to cache intermediate results. Below is an example of using
> redirect.
> +There can be significant overhead using HIPES because the data can traverse
> through ATS many times. Caching can be
> +important to reduce traffic to services/through ATS and can be achieved via
> a proper ``Cache-Control`` header returned
> +by the services. Another way to reduce traffic through ATS is to have ATS to
> return 302 redirects to url for the
> +requests made by service, instead of proxying the requests to that url.
> However, the service must then be able to follow
> +the redirect. The down side is that we cannot use ATS to cache intermediate
> results. Below is an example of using
> +redirect.
>  
>  Modification to above example to reduce traffic using redirect
>  ==============================================================
>  
> -1. The service ``svc_reverse.php`` is modified to add a header of
> ``X-HIPES-Redirect: 2`` to the request made against ``url``.
> +1. The service ``svc_reverse.php`` is modified to add a header of
> ``X-HIPES-Redirect: 2`` to the request made against
> +``url``.
>  
>  2. HIPES plugin will instruct ATS to return a redirect response to this url
>  ::
>  
>      http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt
>  
> -for the following request
> +for the following request ::
>  
>      http://127.0.0.1/svc_case;case=u/test.txt
>  
> -3.  The service ``svc_reverse.php`` is also modified to follow the redirect.
> Thus the response of the service of ``svc_case.php`` will not pass through
> ATS and will pass to ``svc_reverse.php`` service instead.
> +3.  The service ``svc_reverse.php`` is also modified to follow the redirect.
> Thus the response of the service of
> +``svc_case.php`` will not pass through ATS and will pass to
> ``svc_reverse.php`` service instead.
>  
> 
> 

-- 
Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F  4008 F266 55D6 2998 1641


Mime
View raw message