使用curl断点续传下载文件 |
您所在的位置：网站首页 › timeout下载 › 使用curl断点续传下载文件 |
使用curl断点续传下载文件
|
办公网络网速不是很好，使用Chrome下载一些软件时不时会中断，恶心的是Chrome居然不支持断点续传下载(为什么chrome的下载不支持断点续传呢?)，迅雷自然是不能装的，那怎么办？还好我有大名鼎鼎的curl，看官网的介绍，是不是有点6到没朋友👍~~ Supports... DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, HTTP/3, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, SCRAM-SHA, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more. What's curl used for? curl is used in command lines or scripts to transfer data. curl is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the Internet transfer engine for thousands of software applications in over ten billion installations. curl is used daily by virtually every Internet-using human on the globe. 而且 Windows 10 已经内置支持curl(Tar and Curl Come to Windows)， 通过 PowerShell/CMD 可以直接使用，一口气上六楼，很方便😎！ C:\>curl curl: try 'curl --help' for more information C:\>curl -V -v curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL Release-Date: 2017-11-14, security patched: 2019-11-05 Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL在命令行语法中，某些字符与格式有着特殊的意义与含义，一般的约定如下 ：表示必选, 即：该参数、值或信息必传。 []：表示可选，其内的参数、值或信息可传可不传。 … : 由三个句点组成的省略号表示"等等", 表示前述元素(参数、值或信息)可以重复多次。 {}：表示枚举，所传值必须在{}中给定的范围内。 |：管道符号(竖线)表示"或者"，表示在一个集合范围中的一个选项。 /：同上，表示在一个集合范围中的一个选项。 斜体: 表示要提供的信息，要用该实际值来替换的选项或参数，该约定不常见。再结合简单说下curl的基本使用，即curl [options...] 通过curl --help列出所有命令(原列表较长，已大幅删减，若安装最新版则会更多，下面只列举了最常用的) 其中options(选项)由--加小写英文单词组成，如--help，也有少量缩写，如--ipv4, 若英文单字超过两个，则用-连接，如--progress-bar 有的选项可以有参数，help文档里以的形式展示，如--data , --user 部分常用选项有简写，由-加一个字符组成，如-d, -4, -#, -:等，需要注意的是，单个字母是区分大小写的，如-H(--head)和-h(--help)表示不同的含义 选项可以是1个到多个不等(最简单的curl 是curl --url 的缩写，所以至少有一个选项) C:\>curl --help Usage: curl [options...] -a, --append Append to target file when uploading --basic Use HTTP Basic Authentication -E, --cert Client certificate file and password -C, --continue-at Resumed transfer offset -b, --cookie Send cookies from string/file -c, --cookie-jar Write cookies to after operation --create-dirs Create necessary local directory hierarchy -d, --data HTTP POST data --data-ascii HTTP POST ASCII data --data-binary HTTP POST binary data --data-raw HTTP POST data, '@' allowed --data-urlencode HTTP POST data url encoded -f, --fail Fail silently (no output at all) on HTTP errors --fail-early Fail on first transfer error, do not continue -F, --form Specify HTTP multipart POST data --form-string Specify HTTP multipart POST data -G, --get Put the post data in the URL and use GET -I, --head Show document info only -H, --header Pass custom header(s) to server -h, --help This help text -k, --insecure Allow insecure server connections when using SSL -i, --include Include protocol response headers in the output -4, --ipv4 Resolve names to IPv4 addresses -6, --ipv6 Resolve names to IPv6 addresses -L, --location Follow redirects -:, --next Make next URL use its separate set of options -N, --no-buffer Disable buffering of the output stream --oauth2-bearer OAuth 2 Bearer Token -o, --output Write to file instead of stdout -#, --progress-bar Display transfer progress as a bar -x, --proxy [protocol://]host[:port] Use this proxy -U, --proxy-user Proxy user and password -O, --remote-name Write output to a file named as the remote file --remote-name-all Use the remote file name for all URLs --retry Retry request if transient problems occur --retry-connrefused Retry on connection refused (use with --retry) --retry-delay Wait time between retries --retry-max-time Retry only within this period -S, --show-error Show error even when -s is used -s, --silent Silent mode -Y, --speed-limit Stop transfers slower than this -y, --speed-time Trigger 'speed-limit' abort after this time -t, --telnet-option Set telnet option --trace Write a debug trace to FILE --trace-ascii Like --trace, but without hex output --trace-time Add time stamps to trace/verbose output -T, --upload-file Transfer local FILE to destination --url URL to work with -B, --use-ascii Use ASCII/text transfer -u, --user Server user and password -A, --user-agent Send User-Agent to server -v, --verbose Make the operation more talkative -V, --version Show version number and quit言归正传，下面我们使用酷酷的curl下载一些文件(注：以下演示均在高速网络⚡下进行) 最简单的使用curl --remote-name(-O) URL 或者 curl --output(-o) fileName URL， 以下载最新64位windows版本Curl为例 C:\>mkdir Software C:\>cd Software C:\Software>curl -O https://curl.se/windows/dl-7.77.0_2/curl-7.77.0_2-win64-mingw.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5154k 100 5154k 0 0 5154k 0 0:00:01 --:--:-- 0:00:01 6467k使用-continue-at(-C)断点续传下载，这次用PowerShell下载一个大文件(SSMS)，关于该选项的使用，参见以下官方用户手册注解 -C, --continue-at Continue/Resume a previous file transfer at the given offset. The given offset is the exact number of bytes that will be skipped, counting from the beginning of the source file before it is transferred to the destination. If used with uploads, the FTP server command SIZE will not be used by curl. Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out. If this option is used several times, the last one will be used. 在Powershell中使用需要带上后缀名,即curl.exe，不然不能正确识别(会被替换成Invoke-WebRequest) PS C:\Software> curl -C - -O https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe Invoke-WebRequest : 无法处理参数，因为参数名称“C”具有二义性。可能的匹配项包括: -Credential -CertificateThumbprint -Certificate -ContentType。 所在位置 行:1 字符: 6 + curl -C - -O https://download.microsoft.com/download/4/6/8/4681f3b2-f ... + ~~ + CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest]，ParameterBindingException + FullyQualifiedErrorId : AmbiguousParameter,Microsoft.PowerShell.Commands.InvokeWebRequestCommand PS C:\Software> curl --continue-at - -O https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe Invoke-WebRequest : 找不到接受实际参数“-”的位置形式参数。 所在位置 行:1 字符: 1 + curl --continue-at - -O https://download.microsoft.com/download/4/6/8 ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest]，ParameterBindingException + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.InvokeWebRequestCommand通过关闭无线/有线网络，模拟网络不稳定的情况 PS C:\Software> curl.exe -C - -O https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 16 565M 16 94.7M 0 0 4852k 0 0:01:59 0:00:20 0:01:39 0 curl: (56) Send failure: Connection was reset PS C:\Software> curl.exe -C - -O https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe ** Resuming transfer from byte position 172521856 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 9 470M 9 46.9M 0 0 1502k 0 0:05:20 0:00:32 0:04:48 3720k curl: (56) Send failure: Connection was reset可以使用--retry 选项在下载失败后自动重试(此时网络没连接，在观察到重试现象后，打开网络连接)，另外：请动手尝试前文curl --help列举的其他retry选项，这里不再逐一举例 PS C:\Software> curl.exe -C - -O --retry 10 https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe ** Resuming transfer from byte position 221771295 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 1 seconds. 10 retries left. 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 2 seconds. 9 retries left. 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 4 seconds. 8 retries left. 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 8 seconds. 7 retries left. 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 16 seconds. 6 retries left. 100 423M 100 423M 0 0 4667k 0 0:01:33 0:01:33 --:--:-- 7028k PS C:\Software>下载完成后使用断点续传及重试选项，不会覆盖已下载的文件；反之，则不成立 PS C:\Software> curl.exe -C - -O https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe ** Resuming transfer from byte position 666228072 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 237 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (33) HTTP server doesn''t seem to support byte ranges. Cannot resume. PS C:\Software> curl.exe -C - -O --retry 3 https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe ** Resuming transfer from byte position 666228072 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 237 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Warning: Transient problem: FTP error Will retry in 1 seconds. 3 retries left. 0 237 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Warning: Transient problem: FTP error Will retry in 2 seconds. 2 retries left. 0 237 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Warning: Transient problem: FTP error Will retry in 4 seconds. 1 retries left. 0 237 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (33) HTTP server doesn''t seem to support byte ranges. Cannot resume. PS C:\Software> curl.exe -O https://download.microsoft.com/download/4/6/8/4681f3b2-f327-4d3d-8617-264b20685be0/SSMS-Setup-ENU.exe % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 635M 100 635M 0 0 16.2M 0 0:00:39 0:00:39 --:--:-- 24.2M对于有些资源，网速不好的时候，甚至跳转到具体下载链接都很困难，以下载最新的 windows 64 位版本的 PostgreSQL 13.X为例，官网找到下载资源列表，页面跳转后一直不开始下载， 点击下载链接也没反应。。。
别捉急，curl一样可以帮到你。鼠标右键复制下载链接，使用--head(-I) 或者 --include(-i)选项(也可使用--verbose(-v)选项，这样，你可以看到所谓的TCP三次握手是怎样发生的😂)，具体下载链接在Location首部字段里，然后使用上面提到的断点续传下载选项，DONE✌！ C:\>curl --head https://sbp.enterprisedb.com/getfile.jsp?fileid=1257713 HTTP/1.1 302 Found Content-Type: text/html; Content-Length: 0 Connection: keep-alive Server: nginx/1.16.1 Date: Sat, 10 Jul 2021 08:50:09 GMT Location: https://get.enterprisedb.com/postgresql/postgresql-13.3-2-windows-x64.exe X-Frame-Options: DENY X-Content-Type-Options: nosniff X-Cache: Miss from cloudfront Via: 1.1 a43db2746d5ea9543e11897b6654f9b6.cloudfront.net (CloudFront) X-Amz-Cf-Pop: HKG62-C1 X-Amz-Cf-Id: KuxMKejZWH-BMtFbzdKMCXsJE1ctLNMoSIcSYHLIkWQmbFnbexDsNg== C:\>curl -v https://sbp.enterprisedb.com/getfile.jsp?fileid=1257713 * Trying 13.32.53.65... * TCP_NODELAY set * Connected to sbp.enterprisedb.com (13.32.53.65) port 443 (#0) * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 1/3) * schannel: checking server certificate revocation * schannel: sending initial handshake data: sending 191 bytes... * schannel: sent initial handshake data: sent 191 bytes * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 2/3) * schannel: failed to receive handshake, need more data * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 2/3) * schannel: encrypted data got 4096 * schannel: encrypted data buffer: offset 4096 length 4096 * schannel: encrypted data length: 3996 * schannel: encrypted data buffer: offset 3996 length 4096 * schannel: received incomplete message, need more data * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 2/3) * schannel: encrypted data got 1024 * schannel: encrypted data buffer: offset 5020 length 5020 * schannel: encrypted data length: 164 * schannel: encrypted data buffer: offset 164 length 5020 * schannel: received incomplete message, need more data * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 2/3) * schannel: encrypted data got 150 * schannel: encrypted data buffer: offset 314 length 5020 * schannel: sending next handshake data: sending 93 bytes... * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 2/3) * schannel: encrypted data got 170 * schannel: encrypted data buffer: offset 170 length 5020 * schannel: SSL/TLS handshake complete * schannel: SSL/TLS connection with sbp.enterprisedb.com port 443 (step 3/3) * schannel: stored credential handle in session cache ...写在最后，准备这篇随笔的时候，翻阅了不少资料，发现有两个资源是很不错的, 一个是官网的教程, 另一个是官网提到的Everything curl，Everything curl另有开源翻译版, 找到下载章节，一个感觉就是：我有必要写这篇博客吗？写个寂寞🤔。。。最后，附上原文地址, 原创不易，且行且珍惜🤣~~ 更新: Everything curl亦有中文版图书《cURL必知必会》, 在微信读书可免费阅读。 |
【本文地址】