Hear me taste the running man

How To: Splitting large text files into smaller ones

Friday May 29, 2009 | Comments (View) | bash unix

Let’s say you have a huge text file with couple million lines. Now you want to split it up into 100, 200, or 300 line files. Bash makes it pretty easy with split.

In Bash:

split -l 500 file.txt

That splits up the file file.txt into 500 line chunks. What is produced are files such as xaa, xab, xac and so on.

Be sure to do man split for more options.

How To: Find the size of a directory

Thursday May 28, 2009 | Comments (View) | bash unix du awk alias update

Sometimes it’s very useful to know how much content is in a directory without opening a GUI interface.

In Bash:

du -cks * | sort -n | awk '\''BEGIN { split("KB,MB,GB,TB", Units, ","); } { u = 1;while ($1 >= 1024){$1 = $1 / 1024;u += 1;}$1 = sprintf("%.1f %s", $1, Units[u]);print $0;}'\'' | tail -11

I would suggest adding this to your bash aliases as ducks.

Output looks something like so:

~ > ducks

4.0 KB p
72.0 KB Music
24.1 MB Sites
35.0 MB Downloads
433.3 MB Dropbox
937.5 MB Movies
3.5 GB Library
6.3 GB Desktop
11.7 GB Documents
16.5 GB Pictures
39.5 GB total

~ > _

Update: My good friend Clayton suggested a much simpler way. The only problem is that the output is not sorted.

du -h -d 1

The output is the whole directory. I truncated the output to show the last 11.

~ > du -h -d 1 | tail -11

6.3G	./Desktop
 12G	./Documents
212M	./Downloads
433M	./Dropbox
3.6G	./Library
8.0K	./Movies
 72K	./Music
 17G	./Pictures
  0B	./Public
 24M	./Sites
 40G	.

~ > 

Update #2: Some systems do not accept the -d flag. This can be replaced with the --max-depth flag, like so:

du -h --max-depth 1

How To: Remove Duplicate Lines From A File

Thursday May 28, 2009 | Comments (View) | bash unix awk

Sure you can load up a bulky editor and use its tools to do that, but why not do it in much faster in bash with awk.

In Bash:

awk '!x[$0]++' in.txt > out.txt

Wasn’t that easy?

Find & Replace in Linux/Mac

Wednesday May 27, 2009 | Comments (View) | bash unix mac

This is a quick one liner to find and replace strings in *nix. You can use this in your Mac terminal or Unix prompt.

In Bash:

find ./ -type f -exec sed -i 's/FIND/REPLACE/g' {} \;

Example:

find ./ -type f -exec sed -i 's/signin/login/g' {} \;