C# Multi-threaded download file example

C# Multi-threaded download file example, File download is a common requirement in software development. This article starts from the simplest download method step by step, tells the common problems in the file download process, and gives solutions. It also shows how to use multi-threading to improve the download speed of HTTP and call aria2 to implement file downloads with non-HTTP protocols.

C# Download file example

The easiest way to download a file in a .NET program is to use the DownloadFile method of WebClient:

    var url = "https://znlive.com";
    var save = @"D:\1.html";
    using (var web = new WebClient())
    {
        web.DownloadFile(url,save);
    }

C# Asyn download file

This method also provides an asynchronous implementation:

    var url = "https://znlive.com";
    var save = @"D:\1.html";
    using (var web = new WebClient())
    {
        await web.DownloadFileTaskAsync(url, save);
    }

C# Multi-threaded download file

Send a custom request header to the server

Send a custom request header to the server while downloading the file, If you need to customize the file download request, you can use HttpClient:

     var url = "https://znlive.com";
     var save = @"D:\1.html";
     var http = new HttpClient();
     var request = new HttpRequestMessage(HttpMethod.Get,url);
     //Add Auth request header
     request.Headers.Add("Auth","123456");
     var response = await http.SendAsync(request);
     response.EnsureSuccessStatusCode();
     using (var fs = File.Open(save, FileMode.Create))
     {
         using (var ms = response.Content.ReadAsStream())
         {
             await ms.CopyToAsync(fs);
         }
     }

C# Multi-threaded download file

How to solve the problem of incomplete download files

All of the above codes have no major problems when dealing with the download of small files, and errors are easy to introduce when the network condition is poor or the files are large. The following code is very common in development:

     var url = "https://znlive.com";
     var save = @"D:\1.html";
     if (!File.Exists(save))
     {
         Console.WriteLine("File does not exist, start downloading...");
         using (var web = new WebClient())
         {
             await web.DownloadFileTaskAsync(url, save);
         }
         Console.WriteLine("File downloaded successfully");
     }
     Console.WriteLine("Start processing file");
     //TODO: process the file

C# Multi-threaded download file

If an exception occurs in the DownloadFileTaskAsync method (usually a network interruption or network timeout), the downloaded incomplete file will remain in the local system. When the task is retried, because the file already exists (although it is incomplete), it will directly enter the handler, thus introducing an exception.

A simple repair method is to introduce exception handling, but this method is invalid for incomplete files caused by unexpected termination of the application:

     var url = "https://znlive.com";
     var save = @"D:\1.html";
     if (!File.Exists(save))
     {
         Console.WriteLine("File does not exist, start downloading...");
         using (var web = new WebClient())
         {
             try
             {
                 await web.DownloadFileTaskAsync(url, save);
             }
             catch
             {
                 if (File.Exists(save))
                 {
                     File.Delete(save);
                 }
                 throw;
             }
         }
         Console.WriteLine("File downloaded successfully");
     }
     Console.WriteLine("Start processing file");
     //TODO: process the file

C# Multi-threaded download file

The way I prefer is to introduce a temporary file. The download operation downloads the data to a temporary file, and when it is determined that the download operation is completed, the temporary file is renamed:

     var url = "https://znlive.com";
     var save = @"D:\1.html";
     if (!File.Exists(save))
     {
         Console.WriteLine("File does not exist, start downloading...");
         //Download to temporary file first
         var tmp = save + ".tmp";
         using (var web = new WebClient())
         {
             await web.DownloadFileTaskAsync(url, tmp);
         }
         File.Move(tmp, save, true);
         Console.WriteLine("File downloaded successfully");
     }
     Console.WriteLine("Start processing file");
     //TODO: process the file

C# Multi-threaded download file

C# Multi-threaded download file

Use Downloader for HTTP multi-threaded download, In the case of sufficient network bandwidth, the efficiency of single-threaded download is not ideal. We need multi-threading and resumable downloading to get a better download speed.

Downloader is a modern, fluent, asynchronous, testable, and portable .NET library. This is a multi-threaded downloading program that contains asynchronous progress events. Downloader is compatible with .NET Standard 2.0 and above and can run on Windows, Linux, and macOS.

GitHub open source address: https://github.com/bezzad/Downloader

NuGet address: https://www.nuget.org/packages/Downloader

After installing Downloader from NuGet, create a download configuration:

    var downloadOpt = new DownloadConfiguration()
    {
        BufferBlockSize = 10240, // Generally, the host supports a maximum of 8000 bytes, and the default value is 8000.
        ChunkCount = 8, // The number of file fragments to be downloaded, the default value is 1
        MaximumBytesPerSecond = 1024 * 1024, // Download speed limit is 1MB/s, the default value is zero or unlimited
        MaxTryAgainOnFailover = int.MaxValue, // the maximum number of failures
        OnTheFlyDownload = false, //Is it cached in memory? The default value is true
        ParallelDownload = true, //Whether the downloaded file is parallel. The default value is false
        TempDirectory = "C:\\temp", // Set the temporary path for buffering large files, the default path is Path.GetTempPath().
        Timeout = 1000, // Timeout (milliseconds) of each stream reader, the default value is 1000
        RequestConfiguration = // Custom request header file
        {
            Accept = "*/*",
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
            CookieContainer = new CookieContainer(), // Add your cookies
            Headers = new WebHeaderCollection(), // Add your custom headers
            KeepAlive = false,
            ProtocolVersion = HttpVersion.Version11, // Default value is HTTP 1.1
            UseDefaultCredentials = false,
            UserAgent = $"DownloaderSample/{Assembly.GetExecutingAssembly().GetName().Version.ToString(3)}"
        }
    };

C# Multi-threaded download file

Create a download service:

var downloader = new DownloadService(downloadOpt);

Configure the event handler (this step can be omitted):

// Provide `FileName` and `TotalBytesToReceive` at the start of each downloads
    // Provide the "file name" and "total number of bytes to receive" at the beginning of each download.
    downloader.DownloadStarted += OnDownloadStarted;

    // Provide any information about chunker downloads, like progress percentage per chunk, speed, total received bytes and received bytes array to live streaming.
    // Provide information about the block download, such as the progress percentage, speed, total number of bytes received, and byte array received for each block to achieve real-time streaming.
    downloader.ChunkDownloadProgressChanged += OnChunkDownloadProgressChanged;

    // Provide any information about download progress, like progress percentage of sum of chunks, total speed, average speed, total received bytes and received bytes array to live streaming.
    // Provide any information about the download progress, such as the total number of blocks of the progress percentage, the total speed, the average speed, the total received bytes and the real-time stream of the received byte array.
    downloader.DownloadProgressChanged += OnDownloadProgressChanged;

    // Download completed event that can include occurred errors or cancelled or download completed successfully.
    // The event of download completion can include an error or cancellation or a successful download.
    downloader.DownloadFileCompleted += OnDownloadFileCompleted;

C# Multi-threaded download file

Then you can download the file:

    string file = @"D:\1.html";
    string url = @"https://znlive.com";
    await downloader.DownloadFileTaskAsync(url, file);

Download non-HTTP protocol files

Except WebClient can download FTP protocol files, the other methods shown above can only download HTTP protocol files.

aria2 is a lightweight multi-protocol and multi-source command-line download tool. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be operated through the built-in JSON-RPC and XML-RPC interfaces.

We can call aria2 to implement the file download function.

GitHub address: https://github.com/aria2/aria2

Download link: https://github.com/aria2/aria2/releases

Copy the downloaded aria2c.exe to the application directory, if it is another system, you can download the corresponding binary file.

public static async Task Download(string url, string fn)
    {
        var exe = "aria2c";
        var dir = Path.GetDirectoryName(fn);
        var name = Path.GetFileName(fn);

        void Output(object sender, DataReceivedEventArgs args)
        {
            if (string.IsNullOrWhiteSpace(args.Data))
            {
                return;
            }
            Console.WriteLine("Aria:{0}", args.Data?.Trim());
        }

        var args = $"-x 8 -s 8 --dir={dir} --out={name} {url}";
        var info = new ProcessStartInfo(exe, args)
        {
            UseShellExecute = false,
            CreateNoWindow = true,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
        };
        if (File.Exists(fn))
        {
            File.Delete(fn);
        }

        Console.WriteLine("Start aria2c: {0}", args);
        using (var p = new Process {StartInfo = info, EnableRaisingEvents = true })
        {
            if (!p.Start())
            {
                throw new Exception("aria failed to start");
            }
            p.ErrorDataReceived += Output;
            p.OutputDataReceived += Output;
            p.BeginOutputReadLine();
            p.BeginErrorReadLine();
            await p.WaitForExitAsync();
            p.OutputDataReceived -= Output;
            p.ErrorDataReceived -= Output;
        }

        var fi = new FileInfo(fn);
        if (!fi.Exists || fi.Length == 0)
        {
            throw new FileNotFoundException("File download failed", fn);
        }
    }

C# Multi-threaded download file

The above code starts a new aria2c download process through command line parameters and outputs the download progress information to the console. The calling method is as follows:

    var url = "https://znlive.com";
    var save = @"D:\1.html";
    await Download(url, save);

Reference: https://www.coderbusy.com/archives/1773.html