Return-Path: X-Original-To: apmail-trafficserver-users-archive@www.apache.org Delivered-To: apmail-trafficserver-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 28DF017391 for ; Wed, 18 Feb 2015 23:11:04 +0000 (UTC) Received: (qmail 75102 invoked by uid 500); 18 Feb 2015 23:11:03 -0000 Delivered-To: apmail-trafficserver-users-archive@trafficserver.apache.org Received: (qmail 75048 invoked by uid 500); 18 Feb 2015 23:11:03 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 75038 invoked by uid 99); 18 Feb 2015 23:11:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Feb 2015 23:11:03 +0000 X-ASF-Spam-Status: No, hits=3.8 required=5.0 tests=HTML_MESSAGE,HTTP_ESCAPED_HOST,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [216.82.243.209] (HELO mail1.bemta8.messagelabs.com) (216.82.243.209) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Feb 2015 23:10:30 +0000 Received: from [216.82.241.100] by server-17.bemta-8.messagelabs.com id DC/44-02755-B1B15E45; Wed, 18 Feb 2015 23:07:07 +0000 X-Env-Sender: Cong_Yue@alliedtelesis.com X-Msg-Ref: server-2.tower-220.messagelabs.com!1424300825!15638730!1 X-Originating-IP: [208.184.212.190] X-StarScan-Received: X-StarScan-Version: 6.13.4; banners=-,-,- X-VirusChecked: Checked Received: (qmail 11709 invoked from network); 18 Feb 2015 23:07:06 -0000 Received: from 208.184.212.190.t01728-01.above.net (HELO svr-ca-owa.atg.lc) (208.184.212.190) by server-2.tower-220.messagelabs.com with RC4-SHA encrypted SMTP; 18 Feb 2015 23:07:06 -0000 Received: from svr-ca-exch1.atg.lc (10.4.5.8) by svr-ca-owa.atg.lc (208.184.212.190) with Microsoft SMTP Server (TLS) id 8.1.436.0; Wed, 18 Feb 2015 15:07:03 -0800 Received: from svr-ca-exch1.atg.lc ([10.4.5.8]) by svr-ca-exch1.atg.lc ([10.4.5.8]) with mapi; Wed, 18 Feb 2015 15:07:02 -0800 From: "Yue, Cong" To: "users@trafficserver.apache.org" Date: Wed, 18 Feb 2015 15:07:01 -0800 Subject: RE: cacheurl plugin does not work for youtube Thread-Topic: cacheurl plugin does not work for youtube Thread-Index: AdBLqjYspOnwkXkYS0Wq5MJdZs+ESQAJL+Uw Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_ED08B56256B38842A463A2A0804C5AC0326B3CC2A9svrcaexch1atg_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_ED08B56256B38842A463A2A0804C5AC0326B3CC2A9svrcaexch1atg_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I add some debug code into cacheurl plug as follows, it shows the video URL= for google is with very short version. There is no parameters in the URL a= ny more. static int regex_substitute(char **buf, char *str, regex_info *info) { ... if (matchcount < 0) { switch (matchcount) { case PCRE_ERROR_NOMATCH: //TODO ADD MISMATCH URL OUTPUT HERE. if (log) { TSTextLogObjectWrite(log, "Mismatch pattern:'%s' -> URL:'%s'\n", info->pattern, str); } TSDebug(PLUGIN_NAME, "Mismatch pattern:'%s' -> URL:'%s'\n"= , info->pattern, str); break; default: The log is as 20150218.15h00m40s Mismatch pattern:'http://.*\..*\.com/images/tuiguang/([[= :digit:]]{6,6})/(.*\.mp4)' -> URL:'r10---sn-nwj7knek.googlevideo.com:443/' 20150218.15h00m51s Mismatch pattern:'http://.*\..*\.com/images/tuiguang/([[= :digit:]]{6,6})/(.*\.mp4)' -> URL:'clients4.google.com:443/' Is there any way to cache youtube video with TS? Please kindly advise! Thanks, Cong From: Yue, Cong [mailto:Cong_Yue@alliedtelesis.com] Sent: Wednesday, February 18, 2015 10:42 AM To: users@trafficserver.apache.org Subject: cacheurl plugin does not work for youtube Hi I am trying to make youtube be cached with forward proxy, but youtube URL c= an not be redirected. I configured /usr/local/libexec/trafficserver/cacheurl.config as --- http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,= 3}[^&]*/f4v/.*id=3Dtudou.itemid\=3D([0-9]*).* http://www.tudou.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,= 3}[^&]*/flv/.*id=3Dtudou.itemid\=3D([0-9]*).* http://www.tudou.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,= 3}/youku/.*/(.*-.*-.*-.*-[^?]*)(.*) http://www.youku.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,= 3}/sohu/[0-9]*/[0-9]*/[0-9]*/(.*).mp4?key=3D.* http://tv.sohu.com/$1.mp4 http://.*\..*\..*\..*/.*\.com/flvdownload/[[:digit:]]{1,3}/[[:digit:]]{1,3}= /([^?]*)(.*) http://www.56.com/$1 http://[[:digit:]]{1,3}/mp4files/.*/.*\.com/images/tuiguang/[[:digit:]]{6,6= }/(.*\.mp4) http://www.56.com/$1 http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4) http://www.56.com/tuiguang/$1/$2 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,= 3}/mov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*) http://v.163.com/$1 http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,= 3}/.*-.*-.*/.*/cemov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*) http://v.163.com/$1 #YOUTUBE https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\?= .*?\&(id=3D[a-zA-Z0-9.\-\_]*).* http://video-srv.youtube.comi.atsinternal/$3.mp4 --- >From /usr/local/var/log/trafficserver/cacheurl.log it shows for youku.com, the URL match can happen, but for youtube.com, it d= oes not happen. I checked my url with online regular expression tool, it shows the URL of "= " works. This is the log of /usr/local/var/log/trafficserver/cacheurl.log ---- 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1= ,3}[^&]*/f4v/.*id=3Dtudou.itemid\=3D([0-9]*).*' -> 'http://www.tudou.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1= ,3}[^&]*/flv/.*id=3Dtudou.itemid\=3D([0-9]*).*' -> 'http://www.tudou.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1= ,3}/youku/.*/(.*-.*-.*-.*-[^?]*)(.*)' -> 'http://www.youku.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1= ,3}/sohu/[0-9]*/[0-9]*/[0-9]*/(.*).mp4?key=3D.*' -> 'http://tv.sohu.com/$1.mp4' 20150218.10h24m54s Adding pattern/replacement pair: 'http://.*\..*\..*\..*/.*\.com/flvdownload/[[:digit:]]{1,3}/[[:digit:]]{1,3= }/([^?]*)(.*)' -> 'http://www.56.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}/mp4files/.*/.*\.com/images/tuiguang/[[:digit:]]{6,= 6}/(.*\.mp4)' -> 'http://www.56.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4)' -> 'http:= //www.56.com/tuiguang/$1/$2' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1= ,3}/mov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)' -> 'http://v.163.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1= ,3}/.*-.*-.*/.*/cemov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)' -> 'http://v. 163.com/$1' 20150218.10h24m54s Adding pattern/replacement pair: 'https:\/\/(.*\.googlevideo\.com)\/(get_video|videoplayback|videodownload)\= ?.*?\&(id=3D[a-zA-Z0-9.\-\_]*).*' -> 'http://video-srv.youtube.comi.atsinternal/ $3.mp4' 20150218.10h26m22s Rewriting cache URL for http://63.243.196.157/youku/6976A4404493A8379EE16C6BCF/03000811005447043729= FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4?nk=3D58632024139_2373806= 6383&ns=3D164654 35_23569878&special=3Dtrue to http://www.youku.com/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-= 72DE72FA6302.mp4 --------- The squid log is as follows. [root@ats1 trafficserver]# traffic_logcat squid.blog ----- 1424283922.032 880 10.0.0.45 TCP_MISS/200 105743 CONNECT r6---sn-vgqsenel.g= ooglevideo.com:443/ - DIRECT/r6---sn-vgqsenel.googlevideo.com - 1424283927.747 17076 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - = DIRECT/s.ytimg.com - 1424283927.747 17076 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - = DIRECT/s.ytimg.com - 1424283927.748 17046 10.0.0.45 TCP_MISS/200 588 CONNECT yt3.ggpht.com:443/ = - DIRECT/yt3.ggpht.com - 1424283927.748 17079 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - = DIRECT/s.ytimg.com - 1424283927.749 17079 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - = DIRECT/s.ytimg.com - 1424283927.749 17080 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - = DIRECT/s.ytimg.com - 1424283927.826 17155 10.0.0.45 TCP_MISS/200 588 CONNECT yt3.ggpht.com:443/ = - DIRECT/yt3.ggpht.com - 1424283930.226 175 10.0.0.210 TCP_MISS/200 43699 CONNECT www.youtube.com:44= 3/ - DIRECT/www.youtube.com - 1424283930.386 99 10.0.0.210 TCP_MISS/200 7901 CONNECT manifest.googlevideo= .com:443/ - DIRECT/manifest.googlevideo.com - 1424283941.575 30178 10.0.0.45 TCP_MISS/200 897 CONNECT gg.google.com:443/ = - DIRECT/gg.google.com - 1424283941.576 30899 10.0.0.45 TCP_MISS/200 1107 CONNECT yt3.ggpht.com:443/= - DIRECT/yt3.ggpht.com - 1424283941.577 30906 10.0.0.45 TCP_MISS/200 2410 CONNECT s.ytimg.com:443/ -= DIRECT/s.ytimg.com - 1424283942.571 30347 10.0.0.45 TCP_MISS/200 947 CONNECT ssl.gstatic.com:443= / - DIRECT/ssl.gstatic.com - 1424283970.120 39690 10.0.0.210 TCP_MISS/200 437526696 CONNECT r10---sn-a5m= 7lnel.googlevideo.com:443/ - DIRECT/r10---sn-a5m7lnel.googlevideo.com - 1424283980.655 671 10.0.0.45 TCP_MISS/200 252 GET 1424283983.193 12 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.15= 7/youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-F= CC1-8DAC-C94F-72DE72FA6302.mp4?nk=3D76165731532_23738066400&ns=3D16576127_2= 3459186&special=3Dtrue - NONE/- video/mp4 1424283984.199 0 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157= /youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FC= C1-8DAC-C94F-72DE72FA6302.mp4?nk=3D314386706919_23738066416&ns=3D16686819_2= 3348494&special=3Dtrue - NONE/- video/mp4 1424283985.253 4 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157= /youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FC= C1-8DAC-C94F-72DE72FA6302.mp4?nk=3D410790030197_23738066434&ns=3D16797511_2= 3237802&special=3Dtrue - NONE/- video/mp4 1424283987.279 7 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157= /youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FC= C1-8DAC-C94F-72DE72FA6302.mp4?nk=3D314386706950_23738066468&ns=3D17018895_2= 3016418&special=3Dtrue - NONE/- video/mp4 1424283997.579 9 10.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157= /youku/6976A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FC= C1-8DAC-C94F-72DE72FA6302.mp4?nk=3D410790030290_23738066639&ns=3D18125815_2= 1909498&special=3Dtrue - NONE/- video/mp4 ------- >From squid log, it seems I can not get full URL of youtube, but with youtub= e-dl, I check the url should work with my regular expression. ----- [root@test-client1 webpages]# youtube-dl -v --proxy http://10.0.0.204:80 https://www.youtube.com/watch?v=3Dq1mndAYZlio [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['-v', '--proxy', 'http://10.0.0.204:80', 'https= ://www.youtube.com/watch?v=3Dq1mndAYZlio'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] yo= utube-dl version 2015.02.18.1 [debug] Python version 2.6.6 - Linux-2.6.32-5= 04.3.3.el6.x86_64-x86_64-with-centos-6.6-Final [debug] exe versions: none [debug] Proxy map: {u'http': 'http://10.0.0.204:80', u'https': 'http://10.0.0.204:80'} [youtube] q1mndAYZlio: Downloading webpage [youtube] q1mndAYZlio: Extractin= g video information [youtube] q1mndAYZlio: Downloading DASH manifest [debug= ] Invoking downloader on u'https://r10---sn-a5m7lnel.googlevideo.com/videop= layback?signature=3D78D8EB7D568039D930E67820E5D8751C67AD3273.68A236DF130AC8= 1356A4970AFEFD7168E277111A&upn=3DypVWJMpcfaA&mime=3Dvideo%2Fmp4&initcwndbps= =3D5183750&source=3Dyoutube&pl=3D18&sver=3D3&expire=3D1424305562&mm=3D31&du= r=3D2857.052&id=3Do-AITnvpDOi0FfUY-UijIKeWK61KsdMQZSUd0E_NnKO8Od&itag=3D22&= key=3Dyt5&ip=3D208.184.212.172&fexp=3D902039%2C905657%2C927622%2C936109%2C9= 405708%2C9406015%2C9407010%2C943917%2C947225%2C948124%2C948807%2C952302%2C9= 52605%2C952612%2C952901%2C955100%2C955301%2C957201%2C959701%2C960610&mt=3D1= 424283746&mv=3Dm&ms=3Dau&ratebypass=3Dyes&sparams=3Ddur%2Cid%2Cinitcwndbps%= 2Cip%2Cipbits%2Citag%2Cmime%2Cmm%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2= Csource%2Cupn%2Cexpire&ipbits=3D0&requiressl=3Dyes' [download] Destination: Building a large scale CDN with Apache Traffic Serv= er - Jan van Doorn-q1mndAYZlio.mp4 [download] 100% of 416.52MiB in 00:39 ------ Can somebody kindly advise? Thanks, Cong ________________________________ This e-mail message is for the sole use of the intended recipient(s) and ma= y contain confidential and privileged information. Any unauthorized review,= use, disclosure or distribution is prohibited. If you are not the intended= recipient, please contact the sender by reply e-mail and destroy all copie= s of the original message. If you are the intended recipient, please be adv= ised that the content of this message is subject to access, review and disc= losure by the sender's e-mail System Administrator. ________________________________ This e-mail message is for the sole use of the intended recipient(s) and ma= y contain confidential and privileged information. Any unauthorized review,= use, disclosure or distribution is prohibited. If you are not the intended= recipient, please contact the sender by reply e-mail and destroy all copie= s of the original message. If you are the intended recipient, please be adv= ised that the content of this message is subject to access, review and disc= losure by the sender's e-mail System Administrator. --_000_ED08B56256B38842A463A2A0804C5AC0326B3CC2A9svrcaexch1atg_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I add some debug code into cacheurl plug as follows,= it shows the video URL for google is with very short version. There is no = parameters in the URL any more.

 

static int regex_subst= itute(char **buf, char *str, regex_info *info) {

…

    if = (matchcount < 0) {

   &nbs= p;    switch (matchcount) {

   &nbs= p;        case PCRE_ERROR_NOMATCH:<= /o:p>

   &nbs= p;            //TODO= ADD MISMATCH URL OUTPUT HERE.

   &nbs= p;            if (lo= g) {

   &nbs= p;            &= nbsp; TSTextLogObjectWrite(log,

   &nbs= p;            &= nbsp;      "Mismatch pattern:'%s'  ->= URL:'%s'\n",

   &nbs= p;            &= nbsp;      info->pattern, str);

   &nbs= p;             = }

   &nbs= p;            TSDebu= g(PLUGIN_NAME, "Mismatch pattern:'%s'  -> URL:'%s'\n",

   &nbs= p;            info-&= gt;pattern, str);

   &nbs= p;            break;=

   &nbs= p;        default:

 

The log is as

 

20150218.15h00m40s Mis= match pattern:'http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.m= p4)'  -> URL:'r10---sn-nwj7knek.googlevideo.com:443/'

20150218.15h00m51s Mis= match pattern:'http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6})/(.*\.m= p4)'  -> URL:'clients4.google.com:443/= '

 

Is there any way to cache youtube video with TS? Ple= ase kindly advise!

 

Thanks,

Cong=

 

 

From: Yue, Con= g [mailto:Cong_Yue@alliedtelesis.com]
Sent: Wednesday, February 18, 2015 10:42 AM
To: users@trafficserver.apache.org
Subject: cacheurl plugin does not work for youtube=

 

Hi

 

I am trying to make youtube be cached with forwar= d proxy, but youtube URL can not be redirected.

 

I configured /usr/local/libexec/trafficserver/cac= heurl.config  as

---

http://[[:digit:]]{1,3}\.[= [:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/f4v/.*id=3Dtud= ou.itemid\=3D([0-9]*).*

http://www.tudou.com/$1

http://[[:digit:]]{1,3}\.[= [:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/flv/.*id=3Dtud= ou.itemid\=3D([0-9]*).*

http://www.tudou.com/$1

= http://[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[= :digit:]]{1,3}\.[[:digit:]]{1,3}/youku/.*/(.*-.*-.*-.*-[^?]*)(.*)

http://www.youku.com/$1

http://[[:digit:]]{1,3}\.[[:= digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/sohu/[0-9]*/[0-9]*/[0-9]*= /(.*).mp4?key=3D.*

http://tv.sohu.com/$1.mp4<= /span>

http://.*= \..*\..*\..*/.*\.com/flvdownload/[[:digit:]]{1,3}/[[:digit:]]{1,3}/([^?]*)(= .*)

http://www.56.com/$1=

http://[[:digit:]]{1,3}/= mp4files/.*/.*\.com/images/tuiguang/[[:digit:]]{6,6}/(.*\.mp4)

http://www.56.com/$1=

http://.*\..*\.com/images/tuiguang/([[:digit:]]{6,6= })/(.*\.mp4)

http://www.56.com/t= uiguang/$1/$2

http://[[:digit:]]{1,3}\.[[:digi= t:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/mov.bn.netease.com/.*/.*/.*/.= */.*/([^?]*)(.*)

http://v.163.com/$1

http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/.*-.*-.*/.*/cem= ov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)

http://v.163.com/$1

 

#YOUTUBE=

https:\/\/(.*\.goog= levideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=3D[a-z= A-Z0-9.\-\_]*).*

htt= p://video-srv.youtube.comi.atsinternal/$3.mp4<= /p>

---

 

From /usr/local/var/log/trafficserver/cacheurl.lo= g

it shows for youku.com, the URL match can happen,= but for youtube.com, it does not happen.

I checked my url with online regular expression t= ool, it shows the URL of " " works.

 

This is the log of /usr/local/var/log/trafficserv= er/cacheurl.log

----

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/f4v/.*= id=3Dtudou.itemid\=3D([0-9]*).*'

-> 'http://www.t= udou.com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}[^&]*/flv/.*= id=3Dtudou.itemid\=3D([0-9]*).*'

-> 'http://www.t= udou.com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/youku/.*/(.*-.*= -.*-.*-[^?]*)(.*)'

-> 'http://www.y= ouku.com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/sohu/[0-9]*/[0-= 9]*/[0-9]*/(.*).mp4?key=3D.*'

-> 'http://tv.so= hu.com/$1.mp4'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://.*\..*\..*\= ..*/.*\.com/flvdownload/[[:digit:]]{1,3}/[[:digit:]]{1,3}/([^?]*)(.*)'=

-> 'http://www.5= 6.com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}/mp4files/.*/.*\.com/images/tuiguang/[[:digit:]]{6,6}/(.*\.mp4)'<= /o:p>

-> 'http://www.5= 6.com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://.*\..*\.com= /images/tuiguang/([[:digit:]]{6,6})/(.*\.mp4)' -> 'http://www.56.com/tui= guang/$1/$2'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/mov.bn.netease.= com/.*/.*/.*/.*/.*/([^?]*)(.*)'

-> 'http://v.163= .com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'http://[[:digit:]]= {1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/.*-.*-.*/.*/cem= ov.bn.netease.com/.*/.*/.*/.*/.*/([^?]*)(.*)'

-> 'http://v.

163.com/$1'

20150218.10h24m54s = Adding pattern/replacement pair:

'https:\/\/(.*\.goo= glevideo\.com)\/(get_video|videoplayback|videodownload)\?.*?\&(id=3D[a-= zA-Z0-9.\-\_]*).*'

-> 'http://video= -srv.youtube.comi.atsinternal/

$3.mp4'<= /span>

20150218.10h26m22s = Rewriting cache URL for

http://63.243.196.157/youku/697= 6A4404493A8379EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C9= 4F-72DE72FA6302.mp4?nk=3D58632024139_23738066383&ns=3D164654=

35_23569878&spe= cial=3Dtrue to

http://www.youku.com/03000811005447= 043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA6302.mp4

---------

 

The squid log is as follows.

[root@ats1 trafficserver]# traffic_logcat squid.b= log

-----

1424283922.032 880 = 10.0.0.45 TCP_MISS/200 105743 CONNECT r6---sn-vgqsenel.googlevideo.com:443/= - DIRECT/r6---sn-vgqsenel.googlevideo.com -

1424283927.747 1707= 6 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com = -

1424283927.747 1707= 6 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com = -

1424283927.748 1704= 6 10.0.0.45 TCP_MISS/200 588 CONNECT yt3.ggpht.com:443/ - DIRECT/yt3.ggpht.= com -

1424283927.748 1707= 9 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com = -

1424283927.749 1707= 9 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com = -

1424283927.749 1708= 0 10.0.0.45 TCP_MISS/200 588 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com = -

1424283927.826 1715= 5 10.0.0.45 TCP_MISS/200 588 CONNECT yt3.ggpht.com:443/ - DIRECT/yt3.ggpht.= com -

1424283930.226 175 = 10.0.0.210 TCP_MISS/200 43699 CONNECT www.y= outube.com:443/ - DIRECT/www.youtube.com -

1424283930.386 99 1= 0.0.0.210 TCP_MISS/200 7901 CONNECT manifest.googlevideo.com:443/ - DIRECT/= manifest.googlevideo.com -

1424283941.575 3017= 8 10.0.0.45 TCP_MISS/200 897 CONNECT gg.google.com:443/ - DIRECT/gg.google.= com -

1424283941.576 3089= 9 10.0.0.45 TCP_MISS/200 1107 CONNECT yt3.ggpht.com:443/ - DIRECT/yt3.ggpht= .com -

1424283941.577 3090= 6 10.0.0.45 TCP_MISS/200 2410 CONNECT s.ytimg.com:443/ - DIRECT/s.ytimg.com= -

1424283942.571 3034= 7 10.0.0.45 TCP_MISS/200 947 CONNECT ssl.gstatic.com:443/ - DIRECT/ssl.gsta= tic.com -

1424283970.120 3969= 0 10.0.0.210 TCP_MISS/200 437526696 CONNECT r10---sn-a5m7lnel.googlevideo.c= om:443/ - DIRECT/r10---sn-a5m7lnel.googlevideo.com -

1424283980.655 671 = 10.0.0.45 TCP_MISS/200 252 GET

1424283983.193 12 1= 0.0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379= EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA630= 2.mp4?nk=3D76165731532_23738066400&ns=3D16576127_23459186&special= =3Dtrue

- NONE/- video/mp4<= o:p>

1424283984.199 0 10= .0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379= EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA630= 2.mp4?nk=3D314386706919_23738066416&ns=3D16686819_23348494&special= =3Dtrue

- NONE/- video/mp4<= o:p>

1424283985.253 4 10= .0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379= EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA630= 2.mp4?nk=3D410790030197_23738066434&ns=3D16797511_23237802&special= =3Dtrue

- NONE/- video/mp4<= o:p>

1424283987.279 7 10= .0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379= EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA630= 2.mp4?nk=3D314386706950_23738066468&ns=3D17018895_23016418&special= =3Dtrue

- NONE/- video/mp4<= o:p>

1424283997.579 9 10= .0.0.45 TCP_MEM_HIT/200 128564 GET http://63.243.196.157/youku/6976A4404493A8379= EE16C6BCF/03000811005447043729FB19A339D634175695-FCC1-8DAC-C94F-72DE72FA630= 2.mp4?nk=3D410790030290_23738066639&ns=3D18125815_21909498&special= =3Dtrue

- NONE/- video/mp4<= o:p>

-------<= /span>

 

From squid log, it seems I can not get full URL o= f youtube, but with youtube-dl, I check the url should work with my regular= expression.

-----

[root@test-client1 = webpages]# youtube-dl -v --proxy

http://10.0.0.204:80 https://www.youtube.com/watch?v=3Dq1mndAYZlio<= /o:p>

[debug] System conf= ig: []

[debug] User config= : []

[debug] Command-lin= e args: ['-v', '--proxy', 'http://10.0.0.204:80', 'https://www.youtube.com/= watch?v=3Dq1mndAYZlio']

[debug] Encodings: = locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 20= 15.02.18.1 [debug] Python version 2.6.6 - Linux-2.6.32-504.3.3.el6.x86_64-x= 86_64-with-centos-6.6-Final

[debug] exe version= s: none

[debug] Proxy map: = {u'http': 'http://10.0.0.204:80', u'https':

'http://10.0.0.204:= 80'}

[youtube] q1mndAYZl= io: Downloading webpage [youtube] q1mndAYZlio: Extracting video information= [youtube] q1mndAYZlio: Downloading DASH manifest [debug] Invoking download= er on u'https://r10---sn-a5m7lnel.googlevideo.com/videoplayback?signature= =3D78D8EB7D568039D930E67820E5D8751C67AD3273.68A236DF130AC81356A4970AFEFD716= 8E277111A&upn=3DypVWJMpcfaA&mime=3Dvideo%2Fmp4&initcwndbps=3D51= 83750&source=3Dyoutube&pl=3D18&sver=3D3&expire=3D1424305562= &mm=3D31&dur=3D2857.052&id=3Do-AITnvpDOi0FfUY-UijIKeWK61KsdMQZS= Ud0E_NnKO8Od&itag=3D22&key=3Dyt5&ip=3D208.184.212.172&fexp= =3D902039%2C905657%2C927622%2C936109%2C9405708%2C9406015%2C9407010%2C943917= %2C947225%2C948124%2C948807%2C952302%2C952605%2C952612%2C952901%2C955100%2C= 955301%2C957201%2C959701%2C960610&mt=3D1424283746&mv=3Dm&ms=3Da= u&ratebypass=3Dyes&sparams=3Ddur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2= Citag%2Cmime%2Cmm%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2= Cexpire&ipbits=3D0&requiressl=3Dyes'

[download] Destinat= ion: Building a large scale CDN with Apache Traffic Server - Jan van Doorn-= q1mndAYZlio.mp4 [download] 100% of 416.52MiB in 00:39

------

 

Can somebody kindly advise?

 

Thanks,

Cong

 

 

This e-mail message is for the = sole use of the intended recipient(s) and may contain confidential and priv= ileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, plea= se contact the sender by reply e-mail and destroy all copies of the origina= l message. If you are the intended recipient, please be advised that the co= ntent of this message is subject to access, review and disclosure by the sender's e-mail System Administrat= or.


This e-mail message is for t= he sole use of the intended recipient(s) and may contain confidential and p= rivileged information. Any unauthorized review, use, disclosure or distribu= tion is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy = all copies of the original message. If you are the intended recipient, plea= se be advised that the content of this message is subject to access, review= and disclosure by the sender's e-mail System Administrator.
 --_000_ED08B56256B38842A463A2A0804C5AC0326B3CC2A9svrcaexch1atg_--