To hell with NTFS – running the same proc on ext4 is over twice as fast

Mar 9th, 2011 | By | Category: Linux / Freebsd

The way I have my partitions setup is 50GB of EXT4 for linux and 300GB of NTFS for general file storage so that I can access them if I dual boot in windows or anything else int he future.. Compatibility was the name of the game.  Because of this I end up storing most of my working files on the NTFS and working off that drive keeping the EXT4 for linux system files only.  Well that was until today when I happened to have top running while executing a new script I am just finishing off.. The script does a LOT of search and replacing with sed, queries with awk and some other bash fun.  This particular one was taking a 24MB file and breaking it down into 2.5MB chunks based on the contents of the file.  Searches for the 1200th instance of a marker, takes all the content above that, puts it into a new file and then deletes it from the original.  Rinse repeat, rinse repeat.. Anyways top was showing that ntfs-3g was hogging 90% of the cpu while this was going on so out of curiosity I redid the job this time piping it through PV to get a timer on it.  End result: 5 minutes 25 seconds.

Sooo I wondered how long would it take if I moved all my stuff over to my EXT4 partition and ran it there.  End result: 2 minutes and 44 seconds.   Boooya, I’ve just cut down the processing time of my scripts by half just by running them on my EXT4 partition.

Just thought I would share for all those people wanting results from a real life performance test, it’s pretty substantial.

Tags: , , , , ,

Leave a Comment