Curl命令算是工作中最多用的命令,因为工作中牵涉到很多http响应,以及边缘节点,中间源,回源host等等。这篇博客就是来告诉使用curl的正确姿势,真的能自己发觉cdn问题中80%的问题。

1. 使用 HEAD 方式发送HTTP请求,并获取回复的内容

~» curl -I 'http://devtools.qiniu.com/qiniu.png'                                                                                                                                                                                                                [17:10:24]
HTTP/1.1 200 OK
Date: Tue, 06 Sep 2016 09:10:27 GMT
Server: nginx
Content-Type: image/png
Content-Length: 163469
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Log, X-Reqid
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="qiniu.png"
Content-Transfer-Encoding: binary
ETag: "FpHyF0kkil3sp-SaXXX8TBJY3jDh"
Last-Modified: Thu, 28 Jul 2016 10:28:31 GMT
X-Log: mc.g:2;IO:4
X-Reqid: pREAAP95irDfsHEU
X-Qiniu-Zone: 0
X-Via: 1.1 dxxz51:4 (Cdn Cache Server V2.0), 1.1 xingdianxin36:1 (Cdn Cache Server V2.0)
Connection: keep-alive

2. 使用 GET 方法来下载文件,并保存到本地

» curl 'http://devtools.qiniu.com/qiniu.png' -o qiniu.png                                                                                                                                                                                                      [17:10:27]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  159k  100  159k    0     0  28633      0  0:00:05  0:00:05 --:--:-- 39098

3. 指定代理域名或IP及相应端口来发送请求

HEAD请求
~» curl -I 'http://devtools.qiniu.com/qiniu.png' -x 'iovip.qbox.me:80'                                                                                                                                                                                          [17:12:47]
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 06 Sep 2016 09:13:18 GMT
Content-Type: image/png
Content-Length: 163469
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Log, X-Reqid
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="qiniu.png"
Content-Transfer-Encoding: binary
Etag: "FpHyF0kkil3sp-SaXXX8TBJY3jDh"
Last-Modified: Thu, 28 Jul 2016 10:28:31 GMT
X-Log: mc.g:16;IO:62
X-Reqid: K18AAD3XulsHsXEU
X-Qiniu-Zone: 0
GET请求
» curl 'http://devtools.qiniu.com/qiniu.png' -o qiniu.png -x 'iovip.qbox.me:80'                                                                                                                                                                                [17:11:53]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  159k  100  159k    0     0  92528      0  0:00:01  0:00:01 --:--:-- 92512

4. 使用 -v 选项来查看整个请求和回复过程

» curl -I 'http://ip.taobao.com/service/getIpInfo.php?ip=100.123.199.44' -v                                                                                      [23:00:01]
*  Trying 140.205.140.33...
* Connected to ip.taobao.com (140.205.140.33) port 80 (#0)
> HEAD /service/getIpInfo.php?ip=100.123.199.44 HTTP/1.1
> Host: ip.taobao.com
> User-Agent: curl/7.43.0
> Accept: */*
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: Tengine
Server: Tengine
< Date: Tue, 06 Sep 2016 15:00:09 GMT
Date: Tue, 06 Sep 2016 15:00:09 GMT
< Content-Type: text/html
Content-Type: text/html
< Connection: keep-alive
Connection: keep-alive
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Powered-By: PHP/5.4.31
X-Powered-By: PHP/5.4.31

5. 指定请求的头部,Host

» curl -I 'http://iovip.qbox.me/qiniu.png' -H 'Host: devtools.qiniu.com'  
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 06 Sep 2016 09:42:23 GMT
Content-Type: image/png
Content-Length: 163469
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Log, X-Reqid
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="qiniu.png"
Content-Transfer-Encoding: binary
Etag: "FpHyF0kkil3sp-SaXXX8TBJY3jDh"
Last-Modified: Thu, 28 Jul 2016 10:28:31 GMT
X-Log: mc.g;IO:1
X-Reqid: JToAAGYPo6-dsnEU
X-Qiniu-Zone: 0

6. 指定referer,用来检测防盗链

$curl -I --referer http://1.sina.com/ http://img.jiemian.com/101/original/20161229/148300375266728000_a580x330.jpg 2>/dev/null|grep 'HTTP'

HTTP/1.0 403 Forbidden #表示内容不存在或者被拒绝访问

$curl -I --referer http://1.qq.com/ http://img.jiemian.com/101/original/20161229/148300375266728000_a580x330.jpg 2>/dev/null|grep 'HTTP'

HTTP/1.1 200 OK #表示Referer白名单跳转请求允许访问到资源

7. 可以通过 --resolve 选项来指定访问主机名所对应的ip地址:

CDN厂商提供一个CDN节点来验证https是否配制成功,通过resolve参数将https的请求重定向到IP节点(主要是访问特定的cdn节点可以使用)

curl -IL https://fdh5imgcdn.oz1997.com/lp/img/head/t011.jpg -H 'referer:http://csdn.kmtyjz.com' --resolve fdh5imgcdn.oz1997.com:443:58.216.109.182 2>/dev/null|grep 'HTTP'

HTTP/1.1 200 OK

8.验证点播平台拖拽功能:

$ curl -o /dev/null http://vodxuanw.sobeycache.com/vod/2016/07/04/0739e865fba446faa7fdd128b27369ef/h264_256k_mp4_001.mp4?start=50

 % Total % Received
 1 6271k 1 69556 #从50秒开始下载,总共有6271k的资源可下载

$ curl -o /dev/null http://vodxuanw.sobeycache.com/vod/2016/07/04/0739e865fba446faa7fdd128b27369ef/h264_256k_mp4_001.mp4?start=100

% Total % Received 
 23 3626k 23 839k #从100秒开始下载,总共有3626k的资源可下载

9.指定不同的UserAgent来进行网页重定向请求

userAgent 属性是一个只读的字符串,声明了浏览器用于 HTTP 请求的用户代理头的值

  • Iphone
    Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; en) AppleWebKit/534.46.0 (KHTML, like Gecko) CriOS/19.0.1084.60 Mobile/9B206 Safari/7534.48.3

  • Android
    Mozilla/5.0 (Linux; ; ) AppleWebKit/ (KHTML, like Gecko) Chrome/ Mobile Safari/

  • Windows Phone
    Mozilla/5.0 (Windows Phone 10.0; Android ; ; ) AppleWebKit/ (KHTML, like Gecko) Chrome/ Mobile Safari/ Edge/.

  • Internet Explorer 11
    https://msdn.microsoft.com/en-us/library/hh869301(v=vs.85).aspx
    Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko

curl  -IL "http://www.hsyp168.com" -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; en) AppleWebKit/534.46.0 (KHTML, like Gecko) CriOS/19.0.1084.60 Mobile/9B206 Safari/7534.48.3" -x wst.cdn.clouddn.com:80

HTTP/1.0 302 Moved Temporarily
Server: Cdn Cache Server V2.0
Date: Mon, 25 Jul 2016 10:05:27 GMT
Content-Length: 0
Location: http://www.hsyp168.com/mobile

HTTP/1.1 301 Moved Permanently
Date: Mon, 25 Jul 2016 09:37:39 GMT
Content-Type: text/html; charset=UTF-8
Location: http://www.hsyp168.com/mobile/
Server: Microsoft-IIS/7.5
X-Powered-By: WAF/2.0
Content-Length: 153
Age: 1
X-Via: 1.1 zhenjiang170:8110 (Cdn Cache Server V2.0), 1.1 shnh143:10 (Cdn Cache Server V2.0)
Connection: close

HTTP/1.1 200 OK
Date: Mon, 25 Jul 2016 10:05:27 GMT
Cache-Control: private
Content-Length: 34385
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 0
X-Powered-By: WAF/2.0
Set-Cookie: safedog-flow-item=6CE9D171A87252032B5F7AA9879F789E; expires=Mon, 25-Jul-2016 15:59:26 GMT; domain=hsyp168.com; path=/
X-Via: 1.1 zhenjiang166:8111 (Cdn Cache Server V2.0), 1.1 shnh142:6 (Cdn Cache Server V2.0)
Connection: close

其中 -A 用来指定 UserAgent,另外 -L 用来追踪重定向请求。

版权声明:本文为原创文章,版权归 heroyf 所有
本文链接:https://heroyf.club/2018/07/curl/


“苹果是给那些为了爱选择死亡的人的奖励”