Multi-Threaded Downloading with WgetJan 24th, 2011 | By admin | Category: Random Musings
In the process of downloading a few thousand log files from one server to the next I suddenly had the need to do some serious multithreaded downloading in BSD, preferably with Wget as that was the simplest way I could think of handling this. A little looking around led me to this little nugget:
wget -r -np -N [url] & wget -r -np -N [url] & wget -r -np -N [url] & wget -r -np -N [url]
Just repeat the wget -r -np -N [url] for as many threads as you need… Now given this isn’t pretty and there are surely better ways to do this but if you want something quick and dirty it should do the trick… Enjoy!