Thursday, February 10, 2011

Bash for-each line 1-liners

Note: Examples run on Solaris 5.10. input.txt contains the following three lines:

A B
C D
E F

Suppose you want to do something for each line in file. The most obvious way is to use a for loop - it almost matches how you think about it! So lets try (input line followed by output):
(/tmp ): for line in $(cat input.txt); do echo $line; done;
A
B
C
D
E
F
Not quite ideal; for uses the Bash IFS variable (ref) to detect token boundaries. IFS defaults to any whitespace (eg space, tab, line-break) so we get each token rather than each line. Handy in some circumstances but here we specifically want each line.

We can set and unset IFS (unset will return it to default; if we were really feeling careful we could store and restore the value in case someone else set it already):
(/tmp ): IFS='\n'; for line in $(cat input.txt); do echo $line; done; unset IFS;
A B
C D
E F
(/tmp ): for line in $(cat input.txt); do echo $line; done;
A
B
C
D
E
F
Note that the first for sets & unsets IFS; this ensure the second for (which doesn't set IFS) still behaves as expected. That's ugly damnit! More important, we failed we could leave IFS set and disrupt every future script run in this terminal.

We can use the builtin read command (ref) instead and construct our for-each line using while read:
(/tmp ): while read -r line; do echo $line; done < input.txt;
A B
C D
E F
Splendiferous! We get each line and we don't have to futz about with IFS. The -r argument to read disables backslash escaping to se read the literal content of the line without any special mangling.

4 comments:

javin paul said...

This is an interesting tip :)

Javin
most useful cvs commands in linux/unix

Anonymous said...

Exactly what I was looking for.

Unknown said...

now to figure out how to do this with tcsh...

Xsoft said...

Change: IFS='\n'
for: IFS=$'\n'

Post a Comment