C# Multi-threaded download file example, File download is a common requirement in software development. This article starts from the simplest download method step by step, tells the common problems in the file download process, and gives solutions. It also shows how to use multi-threading to improve the download speed of HTTP and call aria2 to implement file downloads with non-HTTP protocols.
C# Download file example
The easiest way to download a file in a .NET program is to use the DownloadFile method of WebClient:
var url = "https://znlive.com";
var save = @"D:\1.html";
using (var web = new WebClient())
{
web.DownloadFile(url,save);
}
C# Asyn download file
This method also provides an asynchronous implementation:
var url = "https://znlive.com";
var save = @"D:\1.html";
using (var web = new WebClient())
{
await web.DownloadFileTaskAsync(url, save);
}
C# Multi-threaded download file
Send a custom request header to the server
Send a custom request header to the server while downloading the file, If you need to customize the file download request, you can use HttpClient:
var url = "https://znlive.com";
var save = @"D:\1.html";
var http = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get,url);
//Add Auth request header
request.Headers.Add("Auth","123456");
var response = await http.SendAsync(request);
response.EnsureSuccessStatusCode();
using (var fs = File.Open(save, FileMode.Create))
{
using (var ms = response.Content.ReadAsStream())
{
await ms.CopyToAsync(fs);
}
}
C# Multi-threaded download file
How to solve the problem of incomplete download files
All of the above codes have no major problems when dealing with the download of small files, and errors are easy to introduce when the network condition is poor or the files are large. The following code is very common in development:
var url = "https://znlive.com";
var save = @"D:\1.html";
if (!File.Exists(save))
{
Console.WriteLine("File does not exist, start downloading...");
using (var web = new WebClient())
{
await web.DownloadFileTaskAsync(url, save);
}
Console.WriteLine("File downloaded successfully");
}
Console.WriteLine("Start processing file");
//TODO: process the file
C# Multi-threaded download file
If an exception occurs in the DownloadFileTaskAsync method (usually a network interruption or network timeout), the downloaded incomplete file will remain in the local system. When the task is retried, because the file already exists (although it is incomplete), it will directly enter the handler, thus introducing an exception.
A simple repair method is to introduce exception handling, but this method is invalid for incomplete files caused by unexpected termination of the application:
var url = "https://znlive.com";
var save = @"D:\1.html";
if (!File.Exists(save))
{
Console.WriteLine("File does not exist, start downloading...");
using (var web = new WebClient())
{
try
{
await web.DownloadFileTaskAsync(url, save);
}
catch
{
if (File.Exists(save))
{
File.Delete(save);
}
throw;
}
}
Console.WriteLine("File downloaded successfully");
}
Console.WriteLine("Start processing file");
//TODO: process the file
C# Multi-threaded download file
The way I prefer is to introduce a temporary file. The download operation downloads the data to a temporary file, and when it is determined that the download operation is completed, the temporary file is renamed:
var url = "https://znlive.com";
var save = @"D:\1.html";
if (!File.Exists(save))
{
Console.WriteLine("File does not exist, start downloading...");
//Download to temporary file first
var tmp = save + ".tmp";
using (var web = new WebClient())
{
await web.DownloadFileTaskAsync(url, tmp);
}
File.Move(tmp, save, true);
Console.WriteLine("File downloaded successfully");
}
Console.WriteLine("Start processing file");
//TODO: process the file
C# Multi-threaded download file
C# Multi-threaded download file
Use Downloader for HTTP multi-threaded download, In the case of sufficient network bandwidth, the efficiency of single-threaded download is not ideal. We need multi-threading and resumable downloading to get a better download speed.
Downloader is a modern, fluent, asynchronous, testable, and portable .NET library. This is a multi-threaded downloading program that contains asynchronous progress events. Downloader is compatible with .NET Standard 2.0 and above and can run on Windows, Linux, and macOS.
GitHub open source address: https://github.com/bezzad/Downloader
NuGet address: https://www.nuget.org/packages/Downloader
After installing Downloader from NuGet, create a download configuration:
var downloadOpt = new DownloadConfiguration()
{
BufferBlockSize = 10240, // Generally, the host supports a maximum of 8000 bytes, and the default value is 8000.
ChunkCount = 8, // The number of file fragments to be downloaded, the default value is 1
MaximumBytesPerSecond = 1024 * 1024, // Download speed limit is 1MB/s, the default value is zero or unlimited
MaxTryAgainOnFailover = int.MaxValue, // the maximum number of failures
OnTheFlyDownload = false, //Is it cached in memory? The default value is true
ParallelDownload = true, //Whether the downloaded file is parallel. The default value is false
TempDirectory = "C:\\temp", // Set the temporary path for buffering large files, the default path is Path.GetTempPath().
Timeout = 1000, // Timeout (milliseconds) of each stream reader, the default value is 1000
RequestConfiguration = // Custom request header file
{
Accept = "*/*",
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
CookieContainer = new CookieContainer(), // Add your cookies
Headers = new WebHeaderCollection(), // Add your custom headers
KeepAlive = false,
ProtocolVersion = HttpVersion.Version11, // Default value is HTTP 1.1
UseDefaultCredentials = false,
UserAgent = $"DownloaderSample/{Assembly.GetExecutingAssembly().GetName().Version.ToString(3)}"
}
};
C# Multi-threaded download file
Create a download service:
var downloader = new DownloadService(downloadOpt);
Configure the event handler (this step can be omitted):
// Provide `FileName` and `TotalBytesToReceive` at the start of each downloads
// Provide the "file name" and "total number of bytes to receive" at the beginning of each download.
downloader.DownloadStarted += OnDownloadStarted;
// Provide any information about chunker downloads, like progress percentage per chunk, speed, total received bytes and received bytes array to live streaming.
// Provide information about the block download, such as the progress percentage, speed, total number of bytes received, and byte array received for each block to achieve real-time streaming.
downloader.ChunkDownloadProgressChanged += OnChunkDownloadProgressChanged;
// Provide any information about download progress, like progress percentage of sum of chunks, total speed, average speed, total received bytes and received bytes array to live streaming.
// Provide any information about the download progress, such as the total number of blocks of the progress percentage, the total speed, the average speed, the total received bytes and the real-time stream of the received byte array.
downloader.DownloadProgressChanged += OnDownloadProgressChanged;
// Download completed event that can include occurred errors or cancelled or download completed successfully.
// The event of download completion can include an error or cancellation or a successful download.
downloader.DownloadFileCompleted += OnDownloadFileCompleted;
C# Multi-threaded download file
Then you can download the file:
string file = @"D:\1.html";
string url = @"https://znlive.com";
await downloader.DownloadFileTaskAsync(url, file);
Download non-HTTP protocol files
Except WebClient can download FTP protocol files, the other methods shown above can only download HTTP protocol files.
aria2 is a lightweight multi-protocol and multi-source command-line download tool. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be operated through the built-in JSON-RPC and XML-RPC interfaces.
We can call aria2 to implement the file download function.
GitHub address: https://github.com/aria2/aria2
Download link: https://github.com/aria2/aria2/releases
Copy the downloaded aria2c.exe to the application directory, if it is another system, you can download the corresponding binary file.
public static async Task Download(string url, string fn)
{
var exe = "aria2c";
var dir = Path.GetDirectoryName(fn);
var name = Path.GetFileName(fn);
void Output(object sender, DataReceivedEventArgs args)
{
if (string.IsNullOrWhiteSpace(args.Data))
{
return;
}
Console.WriteLine("Aria:{0}", args.Data?.Trim());
}
var args = $"-x 8 -s 8 --dir={dir} --out={name} {url}";
var info = new ProcessStartInfo(exe, args)
{
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
};
if (File.Exists(fn))
{
File.Delete(fn);
}
Console.WriteLine("Start aria2c: {0}", args);
using (var p = new Process {StartInfo = info, EnableRaisingEvents = true })
{
if (!p.Start())
{
throw new Exception("aria failed to start");
}
p.ErrorDataReceived += Output;
p.OutputDataReceived += Output;
p.BeginOutputReadLine();
p.BeginErrorReadLine();
await p.WaitForExitAsync();
p.OutputDataReceived -= Output;
p.ErrorDataReceived -= Output;
}
var fi = new FileInfo(fn);
if (!fi.Exists || fi.Length == 0)
{
throw new FileNotFoundException("File download failed", fn);
}
}
C# Multi-threaded download file
The above code starts a new aria2c download process through command line parameters and outputs the download progress information to the console. The calling method is as follows:
var url = "https://znlive.com";
var save = @"D:\1.html";
await Download(url, save);
Reference: https://www.coderbusy.com/archives/1773.html
Hello there!
Thanks a lot for this. I am getting an error when I try to configure the event handler. It doesn’t recognize OnDownloadStarted;
Why is this?
Hi, Rodrigo Luna
OnDownloadStarted is the DownloadStarted event method. If there is other business logic to be processed, you can implement it yourself. If not, you can omit it. Something like this:
private static void OnDownloadStarted(object sender, DownloadStartedEventArgs e)
{
}
By the way, after typing += in Visual Studio, pressing the Tab key 2 times in a row will automatically generate the event method. Of course you can also use anonymous methods.
