Multi-threaded file download for Linux

Is multi-threading available in Linux?

Yes, Linux supports multi-threading. In Linux terminology, this capability is commonly referred to as Simultaneous Multithreading (SMT) or Hyper-Threading. When multi-threading is enabled, a single physical CPU core is mapped to multiple logical processors, allowing multiple threads to be executed simultaneously during each CPU cycle.

Multi-threading allows one physical processor to appear as several virtual processors, each executing parallel code streams. This improves performance by allowing concurrent execution of multiple threads, increasing system efficiency and reducing overall latency.

The implementation of multi-threading in Linux depends on the CPU architecture. Some processors, such as Intel Pentium, support SMT, while others, such as AMD Athlon, do not. Linux distributions typically detect SMT support automatically and configure it accordingly. When enabled, logical processors can execute multiple threads concurrently, allowing the operating system to efficiently distribute workloads and improve throughput and responsiveness.

Axel – a utility for faster file downloads

Axel is a download utility designed to improve download performance by using multiple connections for a single file. According to its developers, Axel can increase download speed by up to 60%.

One of Axel’s key features is its support for multiple protocols, including HTTP, HTTPS, FTP, and FTPS. This flexibility allows users to download files from a wide range of sources.

By opening several connections simultaneously, Axel significantly reduces waiting time and improves the overall download experience.

Axel uses optimized algorithms to distribute resources efficiently and eliminate bottlenecks during the download process. In addition to speed improvements, it supports resuming interrupted downloads and allows users to pause and continue downloads at any time.

Installation

Ubuntu/Debian:

apt install axel

CentOS/RHEL:

yum install epel-release && yum install axel

Fedora:

dnf install axel

To perform a simple download using Axel, run:

axel https://nl.lg.introserv.eu/1000MB.test

-a enables an alternative progress indicator:

axel -a https://nl.lg.introserv.eu/1000MB.test

You can limit the maximum download speed using --max-speed or the short option -s. The value is specified in bytes per second:

axel --max-speed=512000 https://nl.lg.introserv.eu/1000MB.test

To save the file under a different name, use the -o option:

axel -o gigabyte.test https://nl.lg.introserv.eu/1000MB.test

Specify the maximum number of connections using -n:

axel -n 10 https://nl.lg.introserv.eu/1000MB.test

aria2

aria2 is a command-line download utility that supports HTTP(S), FTP, BitTorrent, and Metalink protocols.

How does aria2 work?

aria2 uses a BitTorrent-like approach to distribute downloads across multiple connections. Instead of relying on a single connection, it splits the download into segments and transfers them in parallel, significantly increasing download speed.

Features of aria2

• High configurability: You can define the number of connections, speed limits, and many other parameters.
• Resume and retry: If a connection is interrupted, aria2 can automatically resume the download.
• File integrity verification: Ensures that downloaded files are not corrupted.
• Secure connections: Supports HTTPS and FTPS.
• Torrent support: Allows downloading large files via BitTorrent.

Installation of aria2

Ubuntu/Debian:

apt install aria2

CentOS/RHEL:

yum install aria2

Fedora:

dnf install aria2

Simple download using aria2:

aria2c https://nl.lg.introserv.eu/1000MB.test

Download the same file from two HTTP servers:

aria2c "https://nl.lg.introserv.eu/1000MB.test" "https://uk.lg.introserv.eu/1000MB.test"

Download simultaneously from HTTP and FTP servers:

aria2c --ftp-user=<USER> --ftp-passwd=<PASSWD> "ftp://<ftpserver_ip>/<file>" "https://uk.lg.introserv.eu/1000MB.test"

Download using two connections:

aria2c -x2 -k1m https://nl.lg.introserv.eu/1000MB.test

Set the number of concurrent downloads with -j:

aria2c -j2 https://nl.lg.introserv.eu/1000MB.test https://uk.lg.introserv.eu/1000MB.test

Conclusion

Axel and aria2 are powerful command-line utilities for downloading files on Linux. Both support multiple connections and resuming interrupted downloads. Axel is lightweight and easy to use, while aria2 offers advanced features such as BitTorrent support and file integrity verification.