Bourne sh and bash Notes

Useful VAS status script here. For a later version and more useful scripts, visit path http://www.windofkeltia.com/vintela/scripts.

Bourne and bash (Bourne again) shell constructs. Stoopid to the max maybe, but at least I know where to come for all the things I spent time researching and there's a little pedagogical side to me too (or you wouldn't be reading this).

My shell scripting practices are a little weird sometimes. I like to define variables before I use them and I like to pass parameters in place of using globals. I like to indent carefully and poignantly, and I like things aligned. In short, I hate the mess that is this kind of programming and attempt to impose my own order on it.

This started out to be just about bash since I mostly work under Linux, but the need for portable scripts compels me to deal (or it is duel) with Bourne.

Unless specifically noted for an item, it is my belief that all of this is as valid for sh as for bash. As we're very cross-platform at Vintela, this point is extremely important.

Note: to save real estate, I entab my script files at 2. This is reflected in the samples below.


Learning resources...

The following places are useful...


bash history is lost after closing shell

The fault is usually because the user account was created sloppily and .bash_history got owned by root. Do this: 

$ sudo chown username:groupname .bash_history

Note that this same sloppiness may have affected other, similar files like .viminfo.


bash history options...

In .bashrc, set: 

HISTSIZE=2048
HISTFILESIZE=4096

This must be done in this file (rather than in .profile) because .bashrc is executed (not .profile) for non-login shells that just open a console.


Best bash argument parsing advice ever...

How can I handle command-line arguments (options) in my script easily?


How to set your prompt in bash

To customize your prompt, see man bash and search for “PROMPTING” where you will see a bunch of specifiers. My prompt on Tru64 is established thus... 

	PS1="\u@vastru64:\w> "

This shows up as: 

	rbateman@vastru64:~/vgp-3.1.1/src> (first command-line character comes here)


How to isolate the version of Java in a script

Obviously, this example has much wider application than just Java or versions. 

#!/bin/sh
#vers=`java -version`
JAVA_INFO=$(java -version 2>&1 >/dev/null)
echo "java -version yields $JAVA_INFO"

# isolate version...
QUOTED_VERSION=`echo $JAVA_INFO | awk '{ print $3 }'`
echo "QUOTED_VERSION=$QUOTED_VERSION"

# smoke the surrounding quotes...
VERSION=`echo $QUOTED_VERSION | sed s/\"//g`
echo "VERSION=$VERSION"

# isolate the major version...
MAJOR=`echo $VERSION | awk -F'.' '{ print $2 }'`
echo "MAJOR=$MAJOR"

The first mess...

The first mess will only occur if your scripts pass through the evil hands of Windoz on their way to Linux (Unix) during which misadventure they acquire a carriage return. Executing, you'll see something at least as mysterious and never before seen such as whether immediately or several lines down inside: 

	# ./x.sh
	+$'\r'
	: command not found:

In Vim, simple choose Edit -> File Settings -> File Format... and change the format to Unix instead of DOS.


The semicolon...

The semicolon (;) is an expression/line separator and mostly has the value of a newline (\n). For example, ... 

	if [  ]; then
	  ...
	fi

in place of... 

	if [  ]
	then
	  ...
	fi


Creating an environment variable...

Executing subscripts (other shell scripts), you can pass arguments that get into the script just as if it had been invoked directly on the command line. Another way of communicating information, however, is to create an environment variable that the subscript will inherit.

The variable can be initialized at the same time it is exported... 

	export MY_PATH                   # as already initialized
	export MY_PATH="/home/russ/fun"  # or, initialize it at the same time

Note: in Bourne, you cannot initialize and export in the same statement, but must do these steps on separate lines.

Note that in the shell script, environment variables are visibly indistinguishable from other variables. In fact, there is no difference except for the fact that a script variable isn't an environment variable until it's exported as already noted here. 

	if [ "$MY_PATH" == "/home/russ/fun" ]; then
	  ...
	fi

Ridding the environment of the variable... 

	russ@taliesin:~> unset MY_PATH

Loading variables with output from a (sub) shell...
Born-again shell only

Imagine a script, platform.sh, that emits something like “x86 Linux SuSE 10” to the console. The following will load what is emitted into variables for use by the rest of the calling script: 

	#!/bin/sh
	platform_info_array =( $(./platform.sh) )
	  hardware=${platform_info_array[0]}
	  osname=${platform_info_array[1]}
	  vendor=${platform_info_array[2]}
	  osversion=${platform_info_array[3]}

	echo "hardware=$hardware"
	echo "osname=$osname"
	echo "vendor=$vendor"
	echo "osversion=$osversion"

The output will be... 

	hardware=x86
	osname=Linux
	vendor=SuSE
	osversion=10

Of course, the script, platform.sh, had to echo these values out to the console... 

	#!/bin/sh
	echo "x86 Linux SuSE 10"

And now, for the Bourne shell...

Here is how arrays must be done in the Bourne shell in order to be portable. This can be important since prior to Solaris 8, this OS' shell was horribly broken. Here are some constructs. Check your shell: officially, this shell doesn't even support them. If so, you'll have to try something else.

There is some other stuff going on here too to show what works in Bourne.

Yeah, I know; this is as ugly as the bash equivalent was elegant. You also need to declare an array even if the shell accepts it (arrays are not formally part of the Bourne shell, but many implementations support them). 

	----------------------------- foo.sh ---------------------------------
	#!/bin/sh
	declare -a platform_info_array

	platform_info=`./platform.sh`     # invoke script platform.sh

	platform_info_array[0]=`echo "${platform_info}" | awk '{ print $1; }'`
	platform_info_array[1]=`echo "${platform_info}" | awk '{ print $2; }'`
	platform_info_array[2]=`echo "${platform_info}" | awk '{ print $3; }'`
	platform_info_array[3]=`echo "${platform_info}" | awk '{ print $4; }'`

	echo "platform_info_array=${platform_info_array[@]}"

	f=0
	while [ $f -lt 4 ]; do
	 echo "platform_info_array[$f]=${platform_info_array[$f]}"
	 f=`expr $f + 1`
	done

	echo "My arguments ($#): $@"
	echo "sizeof(platform_info_array)=${#platform_info_array[@]}"

	--------------------------- platform.sh ------------------------------
	#!/bin/sh
	echo "x86 Linux SuSE 10"

The console output: 

	# ./foo.sh Tinkerbell was a fairy too!
	platform_info_array=x86 Linux SuSE 10
	platform_info_array[0]=x86
	platform_info_array[1]=Linux
	platform_info_array[2]=SuSE
	platform_info_array[3]=10
	My arguments [5]: Tinkerbell was a fairy too!
	sizeof(platform_info_array)=4

Of course, all of this work fine in bash too.


Compound condition tests...

These are very difficult and I don't have all the kinks worked out. Moreover, coordination does not appear to work in Bourne. 

	if  [[ ! -a "file-a"
	    || ! -a "file-b"
	    || ! -a "file-c" ]]; then
	  echo "Missing one or more essential files."
	  exit 1
	fi


Gathering the return result of a subfunction...

It's practically useless to attempt to gather the return value of a function coded inside your script... 

	foo()
	{
	  return 0
	}

	bar()
	{
	  return 99
	}

	...
	if [ foo -eq 0 ]; then
	  echo "This works"
	fi

	if [ bar -eq 99 ]; then
	  echo "This works."
	fi


My approach to script variables and function returns...

Shell variables

I find shell scripts hard to read when only global variables are used. I prefer employing the age-old concepts of scope and modularity. In consequence, I use variables local to subfunctions. I believe that a variable is confined to the scope it's in as defined by the curly braces. Thus, x and y below aren't visible in the main body while z is visible in foo as well as the main script body. 

	#!/bin/sh
	z=

	foo()
	{
	  x=
	  y=

	  do stuff...
	}

	# Main script body...
	do main stuff...

Subfunction arguments

I also like passing arguments to shell subfunctions; it makes things a lot clearer for all the reasons argued so many years ago when structured programming was invented. 

	#!/bin/sh
	foo()
	{
	  address=$1
	  phone=$2
	  ...
	}

	foo "1234 South Lincoln Drive" "1 800 Eat-Dirt"

Booleans

Another peculiarity that I have is eschewing the use of 1 to mean true and 0 to mean false. So, I artificially enhance the shell language thus... 

	#!/bin/sh
	TRUE=1
	FALSE=0
	...
	done=$FALSE
	done=$TRUE
	...
	if [ $done -eq $TRUE ]; then
	  echo "We're done!"
	fi

Subfunction return values

And, because shell subfunctions cannot return values in any satisfactory sense (see attempt to do this higher up), I use a single global variable, result, to carry the return value. 

	#!/bin/sh
	result=

	foo()
	{
	  ...
	  result=$TRUE		# (or "garbanzo beans" or 3, etc.)
	}

	# main body...
	foo
	if [ $result == $TRUE ]; then
	  echo "foo returned TRUE"
	fi


String case statements...

The case construct can take strings, something that's infinitely clearer sometimes than simple integers... 

	case "$platform" in
	  "Linux")    echo "We're on Linux"; echo "We love Linux" ;;
	  "Solaris")  echo "We're on Solaris"; echo "We can't afford a Solaris box" ;;
	  "AIX")      echo "We're on AIX"; echo "We hates it, we hates it!" ;;
	  "HPUX")     echo "We're on HP-UX"; echo "We can't afford an HP-UX box" ;;
	  *)          echo "We ain't on any box we know..." ;;
	case


What's in a string...

Is the string in the variable zero-length? 

	if [ -z "$string_var" ]; then
	  echo "String is zero-length..."
	fi

Does the string in the variable have length? 

	if [ -n "$string_var" ]; then
	  echo "String is zero-length..."
	fi

How to glue two strings together: 

	#!/bin/sh
	vendor="SuSE"
	version="10"
	echo "$vendor$version"		# yields "SuSE10"


How to get the return of a command or subscript back...

Let' say you wanted to put the current working directory into a shell variable. You'd use the backtick operator to invoke the command: 

	#!/bin/sh
	cwd=`pwd`

If all you want is to see the result of the command on the console, ... 

	#!/bin/sh
	pwd

After such a command (but not a shell subfunction), the exist status is gathered thus: 

	#!/bin/sh
	pwd
	echo "$?"


Arguments to a shell script...

...are easily referenced thus and shell subfunctions see them their own arguments this way too. Not all of this works in Bourne; see section on arrays above for notions from this section's bash code that do work. 

	#!/bin/sh
	echo "$*"	# all the arguments
	echo "$1"	# the first argument

A scheme to collect and look at arguments (uses an array) 

	#!/bin/sh
	ARG_COUNT=$#                        # count of arguments (including options)
	LAST_ARG=${!ARG_COUNT}              # the last argument

	echo "Argument count is $ARG_COUNT..."
	echo "Last argument is $LAST_ARG..."

	command_list=( $* )                 # make list of arguments as an array
	arg=${command_list[0]}              # start at first index of array

	while [ "$arg" != "$LAST_ARG" ]     # while $arg isn't the last one
	do
	  echo -n "${command_list[$arg]} "  # (-n avoids a newline)
	  arg=`expr $arg + 1`               # bump $arg (redefines it, in fact)
	done

Assuming invocation... 

	# ourscript 1 2 3 charlie 5 -h cool Z

...the console output would be: 

	Argument count is 8...
	Last argument is Z...
	1 2 3 charlie 5 -h cool Z


Arguments to a shell script (continued)...

Try this out with and without arguments... 

	#!/bin/sh
	for arg; do
	  echo $arg
	done

Arguments to a shell script (continued)...

This can ensure passing nothing to a called shell if, in this case, $1 expands to nothing... 

	#!/bin/sh

	# call another script...
	./another-script.sh ${1+"$@"}

Other comments...

"$@" expands to "" if $# is 0 on some implementations (of bash or sh).

If the arguments are

$1 equal to "foo" and
$2 equal to "a bear"

then

"$*" expands to "foo a bear" and
"$@" expands to "foo" "a bear"

Sure, you have to think about it.


Arguments to a shell script (continued)...

Note carefully that functions inside a shell script cannot access arguments since $1, $2, etc. have a different meaning (they are the arguments to the function and not to the script). So, you have to pass the outer shell's arguments: 

	#!/bin/sh

	Subfunction1()
	{
	  echo "'$1' is not the first command-line argument to this shell script..."
	  echo "'$2' is not the second command-line argument to this shell script..."
	}

	Subfunction2()
	{
	  echo "$1 is the first command-line argument to this shell script..."
	  echo "$2 is the second command-line argument to this shell script..."
	  echo "$3 is the third command-line argument to this shell script..."
	}

	echo "First command-line argument is $1..."
	Subfunction1
	Subfunction2 $*

Output: 

	# ./fun.sh snagglepuss the lion
	First command-line argument is snagglepuss...
	'' is not the first command-line argument to this shell script...
	'' is not the second command-line argument to this shell script...
	snagglepuss is the first command-line argument to this shell script...
	the is the second command-line argument to this shell script...
	lion is the third command-line argument to this shell script...


Reading from the keyboard into a variable...

This is a simple matter as demonstrated here... 

	#!/bin/sh
	mothertales=
	...
	read -p "Tell me about your mother: " -r mothertales
	echo "$mothertales"

However, the nice prompting isn't supported by the Bourne shell; instead, the previous segment must be rendered as follows. The -n option to echo isn't supported by Bourne: you must use \c just before the closing quote (either single or double). 

	#!/bin/sh
	mothertales=
	...
	echo "Tell me about your mother: \c"
	read mothertales
	echo "$mothertales"

This is a good point to mention that multiple output variables to read are supported, each one getting successive white space-delimited tokens from the string typed in on standard input. The last variable gets any additional tokens that couldn't be distributed to variable (i.e.: fewer variables than words typed in). Note the interactive Bourne shell session here with # as the command-line prompt. Everything except the echoed text in this color is typed in by the user including the response to the shell read command: 

	# read first second third
	This is a test of the Emergency Broadcast System.
	# echo "$first"
	This
	# echo "$second"
	is
	# echo "$third"
	a test of the Emergency Broadcast System.


Confirm: a handy I/O function...

This function is convenient when writing interactive shells... 

	#!/bin/sh
	#-----------------------------------------------------------------------
	# Prompt the consumer for a response to the question passed as argument.
	# When it returns, $result is set to "yes" or "no."
	#-----------------------------------------------------------------------
	Confirm()
	{
	   prompt=$1
	   output=

	   while [ 1 ]; do
	      read -p "> $prompt (y/n): " -r output

	      if [ "$output" = "y" ]; then
	         result="yes"
	         break
	      elif [ "$output" = "n" ]; then
	         result="no"
	         break
	      fi
	   done
	}

Once again, in Bourne, this must be adjusted slightly... 

	Confirm()
	{
	   prompt=$1
	   output=

	   while [ 1 ]; do
	      echo "> $prompt (y/n): \c"
	      read output

	      if [ "$output" = "y" ]; then
	         result="yes"
	         break
	      elif [ "$output" = "n" ]; then
	         result="no"
	         break
	      fi
	   done
	}


Looping, command-line arguments, etc...

Looping through an unknown number of arguments is fairly simple, although there are many ways to do it. This, however, is the right way to do it. Nevertheless, I have learned that you must not attempt to put this parsing into a function. Everything I've tried in getting around this limitation has been unsuccessful. 

	#!/bin/sh
	 while getopts "hd:" opt; do
	   case $opt in
	     h) echo "Usage:"
	        exit 1
	        ;;
	     d) echo "[-d $OPTARG]"
	        ;;
	     *) echo "ERROR: invalid option"
	        exit 1
	        ;;
	   esac
	 done

	shift `expr $OPTIND - 1`

	until [ -z "1$" ]; do
	  echo "arg: $1"
	  # do stuff with $1...
	  shift
	done

Output of script: 

	# ./poop.sh -d 99 1000 2001 "star wars" poop
	[-d 99]
	arg: 1000
	arg: 2001
	arg: start wars
	arg: poop

Reusing getopts?

Note that, once you've used getopts, OPTIND, the option-parsing index, will likely have run past the end of the arguments or, if parsing stopped early, at least will not be at the beginning. So, you can't use getopts again without resetting it: 

#!/bin/sh
while getopts "hd:" opt; do
  command-line parsing here...
done

OPTIND=1
while getopts "hd:" opt; do
  more command-line parsing (reparsing, in fact)...
done

Another example of this... 

	#!/bin/sh
	set --`getopt abcd:D:z $*`

	while [ $1 != -- ]; do
	    case $1 in
	        -a)         echo "-a found..."      ;;
	        -b)         echo "-b found..."      ;;
	        -c)         echo "-c found..."      ;;
	        -d) shift ; echo "-d $1 found..."   ;;
	        -D) shift ; echo "-D $1 found..."   ;;
	        -z)         echo "-z found..."      ;;
	        *)  echo "Illegal argument..."      ;;
	    esac
	    shift
	done

Output of script: 

	russ@taliesin:~> ./s -D 4 -d 9 -abczq -r2 -s 33
	getopt: invalid option -- q
	getopt: invalid option -- r
	getopt: invalid option -- 2
	getopt: invalid option -- s
	-D 4 found...
	-d 9 found...
	-a found...
	-b found...
	-c found...
	-z found...

The for loop...

Make 100 copies of file.txt... 

	for new in `seq 1 100`; do
		cp file.txt file-$new.txt
	done

Spew the rainbow... 

	rainbow="red yellow pink green orange purple blue"
	for color in $rainbow; do
		echo $color
	done

Output of script: 

	red
	yellow
	pink
	green
	orange
	purple
	blue

The magic of IFS...

Internal field separator (IFS) is a variable that allows one to change from parsing on space to another character, in particular, the carriage return for select, for, etc. It's nothing short of freakin' magic and I happened upon it today in a script I was analyzing to get some ideas for another one I'm writing.

The shell itself uses IFS to delimit words for the read and set commands. If you change it, you should save its original contents first, then restore them right after you are finished using it with your new definition.

The bash manpage says, “If IFS is unset, the parameters (arguments) are separated by spaces. If IFS is null, the parameters are joined without intervening separators.” The default value is “<space><tab><newline>” (mere white space). The following is a sample, using for, of the difference it makes: 

	#!/bin/sh
	line="Friends:don't:let:friends:use:Active:Directory"
	echo "Using standard delimitation..."
	for i in $line; do     # using standard delimitation...
	  echo $i
	done                   # (executes just once)
	echo ""

	old_IFS=$IFS
	IFS=:                  # delimit on colon (:)
	echo "Using new delimitation..."
	for i in $line; do     # using new delimitation...
	  echo $i
	done                   # (executes 7 times)
	IFS=old_IFS            # restore state or we'll be sorry

And the output is... 

	Using standard delimitation...
	Friends:don't:let:friends:use:Active:Directory

	Using new delimitation...
	Friends
	don't
	let
	friend
	use
	Active
	Directory

How to set IFS to just carriage return: 

	IFS='
	'

It seems more important to show the example of using it with a line containing spaces since this is probably more frequent. 

	#!/bin/sh
	line="Novell's eDirectory is X.500 compliant."
	old_IFS=$IFS
	IFS='                  # delimit on carriage return
	'
	echo "Using standard delimitation..."
	for i in $line; do     # using standard delimitation...
	  echo $i
	done                   # (executes just once)

	IFS=old_IFS            # restore to normal state
	echo ""
	echo "Using new delimitation..."
	for i in $line; do     # using new delimitation...
	  echo $i
	done                   # (executes 5 times)

And the output is... 

	Using standard delimitation...
	Novell's eDirectory is X.500 compliant.

	Using new delimitation...
	Novell's
	eDirectory
	is
	X.500
	compliant.


More magic of IFS...

A sort of reverse usage to the above case: use it to preserve newlines in gathered information for later parsing in a script. For example, suppose we wanted to get disk-free information once only and then be able to parse it several time afterward.

This is what df yields: 

	root@vastru64:~> df -k
	Filesystem           1k-blocks      Used Available Use% Mounted on
	/dev/disk/dsk0a         386759    169305    178778  49% /
	/dev/disk/dsk0g        2115237   1503013    400700  79% /usr
	/dev/disk/dsk0e       15227902   3563553  10141558  26% /home
	/dev/disk/dsk0f        9481962    635882   7897883   7% /opt

opt_is_mount_point is set up to tell us whether /opt is a disk mount point, its size, available blocks, etc. 

	save_IFS=$IFS
	IFS=:
	df=`df -k`
	opt_is_mount_point=`echo $df | awk '/\/opt$/'
	tmp_is_mount_point=`echo $df | awk '/\/tmp$/'


The select construct...

A little more esoteric than while or for, in this example, select displays as a menu all the files in the current directory and asks which one you pick. It adds a file on the end of all others before listing the directory (*) of files. With little trouble, you can figure out how to apply this to lists of items, strings, etc. Also, it is influenced by the IFS variable already explanined. 

	#!/bin/sh
	PS3="Choose: "            # variable used to hold prompt
	quit="zzzz (quit)"

	touch "$quit"

	select filename in *; do
	    case $filename in
	        "$quit") echo "Exiting..."; break   ;;
	        *      ) echo "Picked $filename..." ;;
	    esac
	done

	rm "$quit"

Output from this script on my SuSE 10 host... 

	russ@taliesin:~> ./select-test.sh
	 1) aix                 15) findfile.sh         29) valgrind
	 2) aix53.sh            16) findsym.sh          30) valgrind.old
	 3) autotools.sh        17) go.sh               31) vas
	 4) bg.gif              18) images              32) vas3sp2
	 5) bin                 19) javadev             33) VASbuild
	 6) b.new               20) more-images         34) VAS-docbook
	 7) b.sav               21) NVIDIA              35) vasscript
	 8) b.sh                22) original_PATH       36) vintela-common
	 9) cfname.vim          23) Perl                37) vintela-docbook
	10) conf                24) p.sh                38) vmware-console
	11) Desktop             25) resolv.sh           39) web
	12) dev                 26) screensaver.sh      40) zzzz (quit)
	13) dev.sh              27) site-license 3.0.1
	14) Documents           28) solaris8.sh
	Choose:


A little exercise in awk...

Here's a fun little exercise to look inside text files to see if they contain a certain string of characters and, based on the answer, copy the whole file to another place. In the first use, I simply want to see if there is a line containing a version on it. If the expression matches, awk lets it through. In the second, I use it to select the third white-space grouped string of characters. The line looks like: Product Version: 3.0. 

	# Copy all licenses...
	echo "Checking for any version 3 licenses already used with version 2..."
	filelist=`ls /etc/opt/foobar/.licenses/* 2>/dev/null`
	if [ -n "$filelist" ]; then
	  version=
	  version_line=

	  echo "   Licenses found, copying..."

	  for file in $filelist; do
	    version_line=`cat $file | awk '/^Product Version/'`
	    if [ -n "$version_line" ]; then
	      version=`echo $version | awk '{ print $3 }'`
	      if [ "$version" != "2.6" ]; then
	        cp -v /et/opt/foobar2/.licenses/$file /etc/opt/foobar3/.licenses/
	      fi
	    fi
	  done
	fi

How to tell a process is loaded...

Sure, you can issue the following command... 

	ps ax | grep process-name

...however, the problem is that this will find precisely at least the grep instance of the name. Better is this if you know the pid: 

	running=`kill -s 0 pid`

	if [ $running -ne $SUCCESS ]; then
	  echo "The process isn't running!"
	fi

If you don't know the pid, you can do as this example where the process name is “vasd”: 

	ps ax | grep [v]asd

...because that keeps grep from finding the name.


~ and $HOME...

Note that inside a shell, the tilde character used so often in filesystem paths on the command line is $HOME instead. Here's a script to squirrel away other scripts in a safe place as they are created. 

	#!/bin/sh
	scriptname=$1
	subdir=$2

	if [ -d "$subdir" ]; then
	  destination="~/dev/scripts/$subdir"   # must instead be: "$HOME/dev/scripts/$subdir"
	else
	  destination="~/dev/scripts"           # must instead be: "$HOME/dev/scripts"
	fi

	cp -v ./$scriptname $destination


A little treat from an experienced shell coder...

This is advice from a friend, Kevin Harris, of whom I ask questions occasionally.

I mostly bracket variable names with braces ({}) for readability. (see #2 below).

1. I have many times done a search and replace where I changed some names that I didn't mean to change.
You have this: 

	a=$abcd$foo

if somehow you search for $foo and replace it with something (say 2), it could become 

	a=$abcd2

or 

	b=$foobar

would become 

	b=2bar

It isn't always easy to see when it happens, especially when someone else is tweaking your script (so turn on -u).


2. It makes it easier to see what the variable actually is in cases like these 

	eval foo=\"\${VALUE_${variable_name}}\"

instead of 

	eval foo=\"\$VALUE_$variable_name\"

and 

	filename=${package_base}${package_suffix}${package_version}.${ostype}.${osarch}.${package_extension}

instead of 

	filename=$package_base$package_suffix$package_version.$ostype.$osarch.$package_extension

3. You can use the braces for all sorts of things: 

	foo=a-b-c
	bar=${foo#a-}
	baz=${bar%-c}
	zzyzx=${foo%%-*}
	xyzzy=${bar:+nonempty}
	fnord=${bar:-NULL}
	relative_path=${1#${PWD}}
	extension=${1%.*}
	quux="${@:+${default_when_arguments_given}}"
	echo "${quux}##### ${@:-No arguments supplied} #####"

	nonempty()
	{
	   [ $# -gt 0 ] || return 1
	   while [ $# -gt 0 ]; do
	     eval \${$1:+true} \${$1:-false} 2>/dev/null || return 1
	     shift
	   done
	}

4. Braces are the only way to access arguments numbered larger than 9: 

	foo()
	{
	  echo $10
	  echo ${10}
	}

	foo a b c d e f g h i j

Other scripting hints...

  • I find it useful to turn on -u and -e almost all the time in my shell scripts. You'll catch problems that you couldn't easily find by scanning over your script. If something may not be set, use the ${variable:[-+]} or ${variable[-+]} syntax. If the results must be ignored, you can always append || true to the command.
  • Quote everything that could contain a space or special character. Otherwise an apparently simple shell script could produce humorous or dangerous results.
  • Use $@ instead of $*, and almost always in quotes: "$@"
  • Always quote pathnames that you didn't supply yourself (they probably contain spaces or other special characters).

On argument passing and in reference to two scripts... 

asdf.sh:
	#!/bin/sh
	echo "Calling qwer with argument uninstalling..."
	. ./qwer uninstalling

qwer.sh:
	#!/bin/sh

	argv="$*"
	arg="`echo $argv | awk '{ print $1 }'`"

	if [ "$arg" = "uninstalling" ]; then
	  echo "Uninstalling with new argument..."
	  argv="novasclnt"
	else
	  echo "Detected no argument (%s) uninstalling to qwer..."
	fi

	for arg in $argv; do
	  echo "$arg"
	  shift
	done

1. Don't pass arguments to sourced scripts. I tried to look this up, but the POSIX specification for shells apparently has no mention of sourcing. 

	$ bash -x ./asdf
	...
	+ . ./qwer uninstalling
	++ argv=uninstalling
	+++ echo uninstalling
	...

	$ /bin/sh -x ./asdf
	...
	+ . ./qwer uninstalling
	argv=
	+ echo
	...

The problem is that some shells interpret "$*" from the real command line options and a few others (bash included) allow you to pass new positional arguments to sourced scripts.

2. Don't use $*. Use $@ unless you have a good reason to flatten the arguments and make them non-recoverable.

3. Don't use awk to extract arguments. Use the positional arguments instead. Whitespace separation is normally your enemy when input from users is possible.

4. The #!/bin/sh is almost certainly the culprit for the behavior you are seeing. Your current $SHELL doesn't mean anything because you are running the shell with the busted (see #1) Solaris /bin/sh. If you really want bash, try using /usr/bin/env bash.

5. If you are really bashing it, you could do this: 

	argv=("$0" "$@")

or 

argv=("$@")

if you don't care about argv[0] and don't mind the index being shifted compared to normal programs and the numbered arguments (eg. ${argv[0]} == $1)... Which will define argv to be the array of positional arguments, the size known through ${#argv[@]}, elements accessed through ${argv[]}. Completely unsupported by almost everything except bash.

6. Functions are more portable than passing parameters to sourced scripts. With them, you only need to worry about people replacing /bin/sh with some other, horribly incompatible and POSIX non-compliant shell.


A singularly useful script, vasstatus.sh... 
	#!/bin/sh
	# Spills the state of VAS and VAS-related things on this box.
	# This seems to work fine on most platforms.
	platform=`uname`

	echo "   +------------------------------------------------------------------"

	# VAS installation present?
	products=
	if [ -x "/opt/quest/bin/vastool" ]; then
	  vas=`/opt/quest/bin/vastool -v`
	  if [ -n "$vas" ]; then
	    vas=`echo $vas | awk '{ print $4 }'`
	    if [ -n "$vas" ]; then
	      products="VAS $vas"
	    fi
	  fi
	  if [ -x "/opt/quest/sbin/vasgpd" ]; then
	    products="$products, VGP"
	  fi
	  if [ -x "/opt/quest/sbin/vasypd" ]; then
	    products="$products, YP"
	  fi
	  if [ -x "/opt/quest/sbin/vasproxyd" ]; then
	    products="$products, Proxy"
	  fi
	  echo "   | $products installed"

	# See what daemons are running
	  daemons=
	  plural=$FALSE
	  if [ -n "`ps ax | grep vasd`" ]; then
	    daemons="vasd"
	  fi
	  if [ -n "`ps ax | grep vasgpd`" ]; then
	    daemons="$daemons, vasgpd"
	    plural=$TRUE
	  fi
	  if [ -n "`ps ax | grep vasypd`" ]; then
	    daemons="$daemons, vasypd"
	    plural=$TRUE
	  fi
	  if [ -n "`ps ax | grep vasproxyd`" ]; then
	    daemons="$daemons, vasproxyd"
	    plural=$TRUE
	  fi
	  if [ $plural -eq $FALSE ]; then
	    echo "   | $daemons daemon running"
	  else
	    echo "   | $daemons daemons running"
	  fi
	else
	  echo "   | WARNING: /opt/quest/bin/vastool not installed!"
	fi

	# VAS 2.6 installation present?
	if [ -x "/opt/vas/bin/vastool" ]; then
	  vas=`/opt/vas/bin/vastool -v`
	  if [ -n "$vas" ]; then
	    vas=`echo $vas | awk '{ print $4 }'`
	    if [ -n "$vas" ]; then
	      echo "   | VAS $vas installed."
	    fi
	  fi
	#else
	#  echo "Don't say anything about it not being installed."
	fi

	# /etc/nsswitch.conf set-up: file must be readable...
	if [ -r "/etc/nsswitch.conf" ]; then
	  vas_configured=`cat /etc/nsswitch.conf | grep vas3`
	  if [ -n "$vas_configured" ]; then
	    echo "   | /etc/nsswitch is configured for VAS."
	  fi
	fi

	# /etc/resolv.conf set-up: file must be readable...
	if [ -r "/etc/resolv.conf" ]; then
	  zut=`cat /etc/resolv.conf | grep zut`
	  dev=`cat /etc/resolv.conf | grep dev`
	  testing=`cat /etc/resolv.conf | grep test`

	  # which special /etc/resolv.conf is in effect?
	  if [ -n "$zut" ]; then
	    echo "   | /etc/resolv.conf set up for zut! development domain."
	  elif [ -n "$dev" ]; then
	    echo "   | /etc/resolv.conf set up for development domain."
	  elif [ -n "$testing" ]; then
	    echo "   | /etc/resolv.conf set up for testing domain."
	  else
	    echo "   | /etc/resolv.conf probably set up for normal (vintela.com) domain."
	  fi

	  # list nameservers...
	  nameservers=`awk '/nameserver/' /etc/resolv.conf`
	  for ns in $nameservers; do
	    if [ "$ns" != "nameserver" ]; then
	      addr=`echo $ns | sed 's/nameserver //'`
	      echo "   |    nameserver $addr"
	    fi
	  done
	else
	  echo "   | WARNING: /etc/resolv.conf not readable!"
	fi

	# hostname...
	hostname=`hostname`
	if [ -n "$hostname" ]; then
	  echo "   | Hostname is $hostname."
	fi

	# Joined domain: /etc/opt/quest/vas/lastjoin must exist and be readable...
	if [ -r "/etc/opt/quest/vas/lastjoin" ]; then
	  domain=`cat /etc/opt/quest/vas/lastjoin | awk '{ print $NF; }'`
	  echo "   | Joined domain appears to be $domain."
	fi

	if [ -n "$makes_running" ]; then
	  echo "   | makes running on this host; beware."
	else
	  echo "   | No makes appear to be running on this host."
	fi

	# Running make? not certain what platforms this will really work for...
	if [ "$platform" = "SunOS" ]; then
	  makes_running=`ps -ae | grep mak[e]`
	elif [ "$platform" = "HP-UX" ]; then
	  makes_running=`ps -ef | grep mak[e]`
	else
	  makes_running=`ps ax | grep mak[e]`
	fi

	echo "   +------------------------------------------------------------------"

	# vim: set tabstop=2 shiftwidth=2 expandtab:

Sample output... 

	russ@taliesin:~/dev/scriptwork> ./vasstatus.sh
	   +------------------------------------------------------------------
	   | VAS 3.0.3.10 installed.
	   | /etc/nsswitch is configured for VAS.
	   | /etc/resolv.conf set up for development domain.
	   | Hostname is taliesin.
	   | Joined domain appears to be h.dev.
	   | No makes appear to be running on this host.
	   +------------------------------------------------------------------

Listing (copying) files...

This script can be amended to copy files from one place to another simply by modifying ls into cp and changing some of the comments and console output messages. There's some other good stuff here in utility functions AskYesNo() and DoPrompt() along with some multiplatform suppot consideration. 

	#!/bin/sh
	FALSE=0
	TRUE=1
	ACTUAL_SHELL=
	askyesno=

	DoPrompt()
	{
	  __prompt=$1

	  if [ "$ACTUAL_SHELL" = "/bin/bash" ]; then
	    echo -n "$__prompt"
	  else
	    echo "$__prompt\c"
	  fi
	}


	AskYesNo()
	{
	  yn_prompt="$1 (yes|no)?"            # the basic prompt
	  yn_default=$2                       # the default (if any)
	  yn_answer=
	  yn_echoopt=

	  if [ -n "$2" ]; then                # add the default to prompt
	    yn_prompt="$yn_prompt [$yn_default]: "
	  else
	    yn_prompt="$yn_prompt "
	  fi

	  if [ "$ACTUAL_SHELL" = "/bin/bash" ]; then
	    yn_echoopt="-n"                   # how bash does no 
	  else
	    yn_echoopt=
	    yn_prompt="$yn_prompt\c"          # how Bourne does no 
	  fi

	  until [ -z "$yn_prompt" ]; do
	    echo $yn_echoopt "$yn_prompt"
	    read yn_answer

	    case $yn_answer in
	      # we'll accept all of these as valid responses...
	      "yes") askyesno="yes" ; yn_prompt= ;;
	        "y") askyesno="yes" ; yn_prompt= ;;
	       "no") askyesno="no"  ; yn_prompt= ;;
	        "n") askyesno="no"  ; yn_prompt= ;;

	          *) # if default, don't require answer...
	             if [ -z "$yn_answer" ]; then
	               if [ -n "$yn_default" ]; then
	                yn_prompt=
	                askyesno=$yn_default
	               fi
	             fi
	             # bogus answers don't count as...
	             # ...taking the default!
	             ;;
	    esac
	  done
	}

	found=
	errors=
	osname=`uname`
	quit=$FALSE

	# Some bashes aren't compliant with echo -n...
	case "$osname" in
	  "Linux")    ACTUAL_SHELL="$SHELL"   ;;
	  "Solaris")  ACTUAL_SHELL="/bin/sh"  ;;
	  "AIX")      ACTUAL_SHELL="$SHELL"   ;;
	  "HPUX")     ACTUAL_SHELL="/bin/sh"  ;;
	esac

	AskYesNo "Do you want to list out some files" "yes"

	if [ "$askyesno" = "no" ]; then
	  quit=$TRUE
	fi

	while [ $quit -eq $FALSE ]; do
	  echo "Please specify the file or files to list (or type 'quit'): "
	  DoPrompt "> "
	  read __path

	  if [ "$__path" = "quit" ]; then
	    quit=$TRUE
	  elif [ -n "$__path" ]; then
	    found="$__path"

	    if [ -n "$found" ]; then
	      errors=0
	      filelist=`echo $found`

	      for file in $filelist; do
	        ls $file
	        if [ $? -ne 0 ]; then
	          errors=`expr ${errors} + 1`
	        fi
	      done

	      if [ $errors -gt 0 ]; then
	        found=
	        if [ $errors -eq 1 ]; then
	          echo "WARNING: A failure occurred in listing the file(s)."
	        else
	          echo "WARNING: $errors failures occurred in listing the file(s)."
	        fi
	        echo "These list errors are not fatal."
	      fi
	    fi
	  fi
	done

	echo "We found: "
	if [ -n "$found" ]; then
	    echo "something."
	else
	    echo "nothing!"
	fi


File I/O: reading lines from a file...

Here's how to open and read a file out. Obviously, there's no end to what you can do with the lines once you get them. Also, don't mistake this for all the advantages you have using awk to do similar, yet more powerful things. This example cleanly numbers the lines of files up to 1000 lines in length. 

	#!/bin/sh

	echo "Read lines from a file..."

	filename=$1
	lineno=1
	line=

	while read line; do
	  if [ $lineno -lt 10 ]; then
	    echo "$lineno   $line"
	  elif [ $lineno -lt 100 ]; then
	    echo "$lineno  $line"
	  else
	    echo "$lineno $line"
	  fi
	  lineno=`expr $lineno + 1`
	done < $filename


Dereferencing variables containing names of variables...

...also, “How to parse command-line options.”

Imagine some variables to be set in a script and you want to list them out. There are a number of cool lessons or tricks in this code sample. 

	#!/bin/sh
	TRUE=1
	FALSE=0
	FIRST_ARG=
	SECOND_ARG=
	THIRD_ARG=$FALSE
	FOURTH_ARG=
	args="FIRST_ARG SECOND_ARG THIRD_ARG FOURTH_ARG"

	ParseScriptOptions()
	{
	  while getopts "ab:c" opt; do
	    case $opt in
	      a) FIRST_ARG=$TRUE     ;;
	      b) SECOND_ARG=$OPTARG  ;;
	      c) THIRD_ARG=$TRUE     ;;
	      d) FOURTH_ARG=$OPTARG  ;;
	    esac
	  done
	}

	ParseScriptOptions $*
	shift `expr $OPTIND - 1`
	echo "Options parsed or already set..."
	for arg in $args; do
	  if eval test "x\${$arg:+set}" = xset; then
	    eval echo "$arg=\${$arg}"
	  else
	    echo "$arg not set"
	  fi
	done

Now, invoke this supplying some args... 

	# ./foo.sh -a -b poopola this is a test adding 7 arguments

The result will be... 

	russ@taliesin:~> ./foo.sh -a -b poopola this is a test adding 7 arguments
	Options parsed or already set...
	FIRST_ARG=1
	SECOND_ARG=poopola
	THIRD_ARG=0
	FOURTH_ARG not set
	Arguments: this is a test adding 7 arguments


More on command-line parsing...

I'm leaving this code here, but at this point, I'm convinced that the best technology is found in looping, above.

When you're writing a script that must run across a variety of platforms, it can get very frustrating. The operations shift `expr $OPTIND - 1` and shift $(($OPTIND - 1)) don't always work reliably. This may be because I'm still an idiot, but I've had trouble finding a totally portable syntax (other than the one I offer here). In bold are points of the mechanism that I wish to draw your attention to. 

	#!/bin/sh

	argc=$#
	skiparg=0

	ParseScriptOptions()
	{
	  pos=${OPTIND}             # command-line argument/option index

	  while getopts ":mnopq:rs" opt; do
	# while getopts "mnopq:rs" opt; do
	    case $opt in
	      m     )
	        echo "  Option -m             [OPTIND=${pos}]"
	        skiparg=`expr ${skiparg} + 1`
	        ;;
	      n | o )
	        echo "  Option -$opt             [OPTIND=${pos}]"
	        skiparg=`expr ${skiparg} + 1`
	        ;;
	      p     )
	        echo "  Option -p             [OPTIND=${pos}]"
	        skiparg=`expr ${skiparg} + 1`
	        ;;
	      q     )
				  optarg=`expr ${pos} + 1`
	        echo "  Option -q             [OPTIND=${pos} and ${optarg}]"
	        echo "     with argument \"$OPTARG\""
	        skiparg=`expr ${skiparg} + 2`
	        # if -qargument instead of -q argument, we ate 1 too many...
	        ;;
	      *     )
	        echo "  Unimplemented option  [OPTIND=${pos}]"
	        skiparg=`expr ${skiparg} + 1`
	        ;;
	    esac
	    pos=${OPTIND}
	  done
	}

	echo "Demonstrate parsing script arguments and options from the command line..."
	echo "Argument/option count: $argc"
	ParseScriptOptions $*
	until [ $skiparg -lt 0 ]; do
	   shift
	   skiparg=`expr ${skiparg} - 1`
	done
	echo "Remaining command line: $*"

This results in the following. Note that the initial colon (:) in the options list (see while getopts above) keeps unspecified/illegal option -a from causing the error in red from coming out. 

	russ@taliesin:~> ./args.sh -n -a -p -q arg-q This is a test...
	Demonstrate parsing script arguments and options from the command line...
	Argument/option count: 9
	  Option -n             [OPTIND=1]
	  ./args.sh: illegal option -- a
	  Unimplemented option  [OPTIND=2]
	  Option -p             [OPTIND=3]
	  Option -q             [OPTIND=4 and 5]
	     with argument "arg-q"
	Remaining command line: This is a test...


Yet more on command-line parsing...

There are still shells, Tru64's /bin/sh for example, that don't handle getopt. This is a sure-fire method for those. The example is from an installation script I wrote for a product named VAS. The bold option pronouns indicate those that I handle. I found that if I don't list the ones I'm not going to handle, then their presence causes the script to error out ungracefully. This way, the invalid ones calmly go through my warning statement and are forgotten. I could also error out on them—still more graceful than the default behavior. 

	#!/bin/sh
	VAS_DEBUG_LEVEL=
	VAS_UNATTENDED_MODE=
	VAS_EULA=
	CMDLINE_LICENSE_FILE=
	MIGRATE_26_3X=
	PASSIVE_SCRIPT=

	# list all possible options even invalid ones so we can print a warning...
	set -- `getopt hd:qal:mnbcefgijkoprstuvwxyz $*`
	while [ $1 != -- ]; do
	  case $1 in
	    -h) PrintUsage                          ; exit 1 ;;
	    -d) VAS_DEBUG_LEVEL=$2                  ; shift  ;;
	    -q) VAS_UNATTENDED_MODE=$TRUE                    ;;
	    -a) VAS_EULA=$TRUE                               ;;
	    -l) CMDLINE_LICENSE_FILE=$2             ; shift  ;;
	    -m) MIGRATE_26_3X="vas-client"                   ;;
	    -n) PASSIVE_SCRIPT=$TRUE                         ;;
	     *) echo "WARNING: Ignoring invalid option ($1)" ;;
	  esac
	  shift
	done
	shift

	# play around with echo here to see if we got set up right and other things...
	argv=$*
	echo "Results..."
	echo "      VAS_DEBUG_LEVEL=$VAS_DEBUG_LEVEL"
	echo "  VAS_UNATTENDED_MODE=$VAS_UNATTENDED_MODE"
	echo "             VAS_EULA=$VAS_EULA"
	echo " CMDLINE_LICENSE_FILE=$CMDLINE_LICENSE_FILE"
	echo "        MIGRATE_26_3X=$MIGRATE_26_3X"
	echo "       PASSIVE_SCRIPT=$PASSIVE_SCRIPT"

	echo " first remaining args=$1"
	echo "second remaining args=$2"
	echo "(argv)       all args=$argv"
	echo "             All args=$*"


Excluding lines in awk that match...

In fact, don't use awk, but grep -v. Here's how to remove VAS lines from the PAM files on Linux: 

	mv /etc/pam.d/common-password /tmp
	cat /tmp/common-password | grep -v vas > /etc/pam.d/common-password


Arithmetic...

To do arithmetic, use expr; output shown here. 

	#!/bin/sh
	x=6
	y=`expr ${x} - 2`
	echo "Arithmetic operation: $x - 2 = $y"

	# output:
	Arithmetic operation: 6 - 2 = 4


Deleting empty directories...

Here's how to delete (only) empty directories. 

	{
	  __dir=$1
	  count=`ls -l $__dir | wc -l`
	  if [ $count -eq 1 ]; then
	    rmdir $__dir
	    echo "$__dir removed"
	  else
	    echo "$__dir not removed"
	  fi
	}


Variable indirect expansion...

Imagine a debugger in a shell script. Any script that incorporates command-line interaction of any sort can be modified to display the value of shell variables through variable indirect expansion. This was introduced in bash 2.0; I don't know how to do it using eval, but doing precisely that is an unanswered exercise at the end of Chapter 7 in Learning the bash Shell from O'Reilly & Associates. 

	#!/bin/bash
	x=6
	y="This is a string"

	echo "x=$x, y=$y"
	echo -n "
	Of x or y, enter the name of the variable whose value you wish to display: "
	read v
	echo "$v=${!v}"

You can even allow the entry of multiple, white space-delimited variables for display: 

	#!/bin/bash
	x=6
	y="This is a string"

	echo "x=$x, y=$y"
	echo -n "Enter both x and y to display their values: "
	read var_list
	for var in $var_list; do
	  echo "$var=${!var}"
	done

This has other applications, but this is the one that interests me. Now, I found that this isn't support prior to bash, version 2. This next one purports to be: 

	#!/bin/sh
	   ...
	while echo -n "Pick a variable; just RETURN quits: "
	  read var; do
	    case "$var" in
	      "") break ;;
	       *) eval echo \$$var ;;
	    esac
	done

Quoting shell-outs...

Okay, so I don't know the terminology exactly, but doing stuff in a subshell, you need to consume variables already set with information. In order to do this, you must double- instead of single-quote the arguments thus. Output shown here. 

	#!/bin/sh
	path="linux-glibc23-ppc"
	libname="glibc23-"
	basename=`echo $path | sed 's/${libname}//'`   # no, this dog won't hunt
	echo $basename
	basename=`echo $path | sed "s/${libname}//"`   # use this instead
	echo $basename

	# output:
	linux-glibc23-ppc
	linux-ppc


Finding available commands...

Underneath, if your script goes off-platform, you may question whether the OS you find yourself running on supports a command you use. For example, chkconfig is replaced on Debian/Ubuntu more or less by update-rc.d. Which to use (if either) is a problem to solve. Here's one way I solved it—when which itself isn't a safe bet underneath. This is from a post-install packaging script for Linux use. 

	#!/bin/sh

	command_exists=0
	verify_command_exists()      # replacement for which—missing on Fedora
	{
	  cmd=$1
	  OLDIFS=$IFS
	  IFS=:
	  for d in $PATH; do
	    if [ -x "$d/$cmd" ]; then
	      command_exists=1
	      IFS=$OLDIFS
	      return
	    fi
	  done
	  command_exists=0
	  IFS=$OLDIFS
	}

	...

	MYDAEMON=my-little-daemon-d

	# How to add ourselves to the rc.d directories?
	verify_command_exists chkconfig
	__chkconfig=$command_exists
	# this is for Debian/Ubuntu
	verify_command_exists update-rc.d
	__update_rc_d=$command_exists

	if [ $__chkconfig -eq 1 ]; then
	  chkconfig --add $MYDAEMON &>/dev/null
	elif [ $__update_rc_d -eq 1 ]; then
	  update-rc.d $MYDAEMON defaults &>/dev/null
	else # support Linuces that don't have chkconfig or update-rc.d
	  ln -s /etc/init.d/$MYDAEMON /etc/rc1.d/K73$MYDAEMON
	  ln -s /etc/init.d/$MYDAEMON /etc/rc2.d/K73$MYDAEMON
	  ln -s /etc/init.d/$MYDAEMON /etc/rc3.d/S27$MYDAEMON
	  ln -s /etc/init.d/$MYDAEMON /etc/rc4.d/S27$MYDAEMON
	  ln -s /etc/init.d/$MYDAEMON /etc/rc5.d/S27$MYDAEMON
	fi


Making a string upper- or lower-case...

It's possible to change the case on a string. Here's a sample. tr is for translating or deleting characters. 

	#!/bin/sh
	echo "ThIs Is A tEsT" | tr '[:upper:]' '[:lower:]'
	this is a test


Another way to do echos with no newlines...

Assume qwer.sh... 

	#!/bin/sh
	echo1() { echo -n "$*"; }
	echo2() { echo "$*\\c"; }
	echo3() { echo "$* +"; }

	if test "x`echo1 y`z" = "xyz"; then
	  echon() { echo1 "$*"; }
	elif test "x`echo2 y`z" = "xyz"; then
	  echon() { echo2 "$*"; }
	else
	  echon() { echo3 "$*"; }
	fi

	echon "No newline here->"

	# output:
	russ@taliesin:~> ./qwer.sh
	No newline here->russ@taliesin:~>


A sort of no-op...

...can be created using the character ':' The following does not result in a syntax error: 

	#!/bin/sh

	if [ -z "$word" ]; then
	  :
	else
	  echo "This is a test"
	fi


Fun: Linux text console colors...

Try this out (Linux bash only): 

	#!/bin/bash
	# Display ANSI colours...
	esc="\033["
	echo -n " _ _ _ _ _40 _ _ _ 41_ _ _ _42 _ _ _ 43"
	echo "_ _ _ 44_ _ _ _45 _ _ _ 46_ _ _ _47 _"
	for fore in 30 31 32 33 34 35 36 37; do
	  line1="$fore  "
	  line2="    "
	  for back in 40 41 42 43 44 45 46 47; do
	    line1="${line1}${esc}${back};${fore}m Normal  ${esc}0m"
	    line2="${line2}${esc}${back};${fore};1m Bold    ${esc}0m"
	  done
	  echo -e "$line1\n$line2"
	done


Validating prompted input...

Try this out (Linux bash only). The concrete responses are 1, 2 or 3. Other numeric responses are allowed via *), but non-numeric responses are complained about. 

	#!/bin/bash
	# Get input and validate...
	echo -n "Enter 1, 2 or 3: "
	read ans
	case "$ans" in
	  1|2|3)
	    echo "It's $ans..."
	    ;;
	  *)
	    left=`echo $ans | sed 's/[0-9]*//g'`
	    if [ -n "$left" ]; then
	      echo "NOTE: Unexpected response ($ans). Please choose one of the menu items listed."
	    else
	      echo "It's some other number than listed ($ans)."
	    fi
	    ;;
	esac


Checking to see if you are root...

...is easy: 

	#!/bin/sh
	id=`id | sed 's/uid=\([0-9]*\).*/\1/'`
	if [ $? -ne 0 ] ; then
	    echo "Must be root to run."
	    exit 1
	fi

Some platforms (Solaris) document a -u option, but using it gets an illegal argument message anyway and the command fails. This option should yield what sed 's/uid=\([0-9]*\).*/\1/' does in the previous example. 

	#!/bin/sh
	id=`id -u`


Debugging Tips

Endless topic, this one.

Skipped read...

If it appears that a read is being skipped, squint closer. It might be that it is never reached. Instead, there is an uninitialized variable. 

	1   #!/bin/bash
	2
	3   ECHO_USE_N=
	4
	5   echo_n()
	6   {
	7     if [ $USE_ECHO_N -ne 0 ]; then
	8       echo -n $1
	9     else
	10      echo "$1\c"
	11    fi
	12  }
	13
	14  PromptForContainer()
	15  {
	16    loc_container=
	17
	18    echo_n "Enter container [press Enter for default]: "
	19    read loc_container
	20
	21    if [ "x$loc_container" != "x" ]; then
	22      CONTAINER=$loc_container
	23    fi
	24  }

In tracing through this code, the error: 

	./poop.sh: test: argument expected

is found at line 7 of echo_n: 

	.
	.
	.
	+ [ -ne 0]
	.
	.
	.

This is because at that line, variable $USE_ECHO_N is consumed while clearly $ECHO_USE_N was going to be the name. $USE_ECHO_N is undefined, therefore the empty string which is incompatible with the integer comparison.


Redirecting stderr to stdout...

This can be useful for getting errors into less, for example. The following gets some of the output from cvs to paginate. With cvs you get the updating messages where nothing has changed for a file on stderr which intermingles with those put to stdout. The syntax is in bold. The second command illustrates how to dump those useless messages to the bit bucket and get only the useful ones (a bit off-topic). What does come out is filtered to show only the list of files where merging occurs. 

	# cvs update 2>&1 | less
	# cvs update 2> /dev/null | grep ^M


General script debugging...

Use the -x option to sh to see where execution is going. It's not single-stepping, but it's better than nothing. Option -v is somewhat useful, but I find it less so and more wordy without supplying much addition or useful information. 

	# sh -x ./poop.sh

You can turn this on dynamically for only part of the script by inserting the line set -x on inside your script where you want to begin tracing. set -x off is supposed to turn it off, but I haven't found this to work on SuSE Linux' bash. Another place I Googled said the following bracketing works and I've verified that this is the case: 

	#!/bin/sh
	untraced statements...
	set -x
	code to trace...
	set +x
	untraced statements...

There are various line-prefixing options in links below that show how to set up your script so that you can leave debug statements in even in production.


(Early) end of file unexpected...

This is the result of a missing syntax terminator like a closing quote, missing left brace, etc. If you're coding with Vim, you find this out while in the editor because of a change of color done by the editor. 

	#!/bin/sh
	if [ $1 -eq 1 ]; then
	  echo "This is a test
	fi
	# We're showing a sample of how editor color demonstrates a missing end
	# quote by turning everything after the opening quote to the quoted string
	# color.

The correct syntax would produce this instead: 

	#!/bin/sh
	if [ $1 -eq 1 ]; then
	  echo "This is a test"
	fi
	# We're showing a sample of how editor color demonstrates correct syntax.


Linux bash special variables

Echo out variables similar to those in the C preprocessor:

  • $LINENO —script line number
  • $SHELL —shell running this script
  • $IFS —parsing character (see elsewhere in this page)
  • $PWD —current working directory
  • $RANDOM —different each time used

Some good links on this topic...


Reading and parsing a file...

Assume the following file: 

	# Quest Software, Inc. manifest for Vintela Authentication Services (VAS)
	# and Royal Crown (RC) sodas. Do not change the format of this file. Each
	# manifest line consists of a product name followed by white space, then
	# which CD, the location on the ISO and, finally, the script used to install
	# it.

	# product name  CD  location  install script         human-readable name
	  vasclnt       1   client/   client/vas-install.sh  "VAS Client"
	  vasgp         1   client/   client/vas-install.sh  "VAS Group Policy client"
	  vasyp         1   client/   client/vas-install.sh  "VAS YP Server"
	  vasproxy      1   client/   client/vas-install.sh  "VAS Proxy Daemon"
	  vasutil       1   client/   client/vas-install.sh  "VAS Ownership-alignment Tool"
	  vassc         1   client/   client/vas-install.sh  "VAS Smartcard Client"
	  vasdev        1   client/   client/vas-install.sh  "VAS SDK"

	  sudo          2   sudo/     sudo/rc-install.sh     "sudo"
	  ssh           2   ssh/      ssh/rc-install.sh      "ssh"
	  samba         2   samba/    samba/rc-install.sh    "Samba"
	  openssh       2   openssh/  openssh/rc-install.sh  "openSSH"

	  # (db2 is source code only...)
	  db2           2   db2/      db2/rc-install.sh      "DB2"

	# product name  CD  location               install script         human-readable name
	  mod_auth      2   apache2/mod_auth_vas   apache2/rc-install.sh  "Apache2 Authorization Module"

The following script parses the file above... 

	#!/bin/sh
	TRUE=1
	FALSE=0
	MANIFEST="./manifest"

	ReadManifestFile()
	{
	  filename=$1
	  stop_at_line=$2
	  do_echo=$3
	  skip_comments=$4
	  lineno=0
	  line=

	  while read line; do
	    lineno=`expr $lineno + 1`

	    if [ $stop_at_line -ne 0 -a $stop_at_line -eq $lineno ]; then
	      break;
	    fi

	    if [ $do_echo -ne $TRUE ]; then
	      continue
	    fi

	    # don't echo this line if it's a comment and we're skipping them...
	    if [ $skip_comments -eq $TRUE ]; then
	      if [ "`echo $line | awk '{ print $1 }' | grep [#]`" ]; then
	        continue
	      fi
	    fi

	    if [ $lineno -lt 10 ]; then
	      echo "$lineno   $line"
	    elif [ $lineno -lt 100 ]; then
	      echo "$lineno  $line"
	    else
	      echo "$lineno $line"
	    fi
	  done < $filename
	}

	gpi_product=
	gpi_cd_rom=
	gpi_location=
	gpi_script=
	gpi_name=
	GetProductInfo()
	{
	  gpi_product=$1
	   gpi_cd_rom=$2
	 gpi_location=$3
	   gpi_script=$4
	     gpi_name=`echo $* | awk '{ print $5 " " $6 " " $7 " " $8 " " $9 " " $10 }'`
	     gpi_name=`echo $gpi_name | sed 's/"//g'`
	}

	PresentProductInfo()
	{
	      _name=$1
	  _location=$2
	    _cd_rom=$3
	    _script=$4
	  echo "$_name is at $_location on CD$_cd_rom; install using $_script"
	}

	echo "Here is $MANIFEST:"
	ReadManifestFile $MANIFEST 0 $TRUE $FALSE
	echo

	echo "Here is the file up to line 15, then line 16 is printed separately:"
	ReadManifestFile $MANIFEST 16 $TRUE $TRUE
	line16=$line
	echo "16  $line"
	echo

	echo "Here is just line 18 without printing any of the rest of the file:"
	ReadManifestFile $MANIFEST 18 $FALSE $FALSE
	line18=$line
	echo "18  $line"
	echo

	echo "Here we just got line 10 without printing anything out:"
	ReadManifestFile $MANIFEST 10 $FALSE $FALSE
	line10=$line
	echo

	ReadManifestFile $MANIFEST 35 $FALSE $FALSE
	if [ -n "$line" ]; then
	  echo "Here is just line 35 (ERROR! there is no line 35):"
	  echo "35  \"$line\""
	  echo
	fi

	GetProductInfo $line10
	PresentProductInfo "$gpi_name" $gpi_location $gpi_cd_rom $gpi_script

	GetProductInfo $line16
	PresentProductInfo "$gpi_name" $gpi_location $gpi_cd_rom $gpi_script

	GetProductInfo $line18
	PresentProductInfo "$gpi_name" $gpi_location $gpi_cd_rom $gpi_script

	echo

...to the output that follows: 

	Here is ./manifest:
	1   # Quest Software, Inc. manifest for Vintela Authentication Services (VAS)
	2   # and Royal Crown (RC) sodas. Do not change the format of this file. Each
	3   # manifest line consists of a product name followed by white space, then
	4   # which CD, the location on the ISO and, finally, the script used to install
	5   # it.
	6
	7   # product name  CD  location  install script         human-readable name
	8   vasclnt       1   client/   client/vas-install.sh  "VAS Client"
	9   vasgp         1   client/   client/vas-install.sh  "VAS Group Policy client"
	10  vasyp         1   client/   client/vas-install.sh  "VAS YP Server"
	11  vasproxy      1   client/   client/vas-install.sh  "VAS Proxy Daemon"
	12  vasutil       1   client/   client/vas-install.sh  "VAS Ownership-alignment Tool"
	13  vassc         1   client/   client/vas-install.sh  "VAS Smartcard Client"
	14  vasdev        1   client/   client/vas-install.sh  "VAS SDK"
	15
	16  sudo          2   sudo/     sudo/rc-install.sh     "sudo"
	17  ssh           2   ssh/      ssh/rc-install.sh      "ssh"
	18  samba         2   samba/    samba/rc-install.sh    "Samba"
	19  openssh       2   openssh/  openssh/rc-install.sh  "openSSH"
	20
	21  # (db2 is source code only...)
	22  db2           2   db2/      db2/rc-install.sh      "DB2"
	23
	24  # product name  CD  location               install script         human-readable name
	25  mod_auth      2   apache2/mod_auth_vas   apache2/rc-install.sh  "Apache2 Authorization Module"

	Here is the file up to line 15, then line 16 is printed separately:
	6
	8   vasclnt       1   client/   client/vas-install.sh  "VAS Client"
	9   vasgp         1   client/   client/vas-install.sh  "VAS Group Policy client"
	10  vasyp         1   client/   client/vas-install.sh  "VAS YP Server"
	11  vasproxy      1   client/   client/vas-install.sh  "VAS Proxy Daemon"
	12  vasutil       1   client/   client/vas-install.sh  "VAS Ownership-alignment Tool"
	13  vassc         1   client/   client/vas-install.sh  "VAS Smartcard Client"
	14  vasdev        1   client/   client/vas-install.sh  "VAS SDK"
	15
	16  sudo          2   sudo/     sudo/rc-install.sh     "sudo"

	Here is just line 18 without printing any of the rest of the file:
	18  samba         2   samba/    samba/rc-install.sh    "Samba"

	Here we just got line 10 without printing anything out:

	VAS YP Server is at client/ on CD1; install using client/vas-install.sh
	sudo is at sudo/ on CD2; install using sudo/rc-install.sh
	Samba is at samba/ on CD2; install using samba/rc-install.sh


Using set -u...

set -u at the top of the (principal) script will cause the shell to execute until an uninitialized variable is consumed whereupon it exits with the name of the variable and the line number on which it was consumed. 

	#!/bin/sh
	set -u

	foo()
	{
	  x=$1
	}

	y=`echo foo | sed 's/foo//'  # $y will be nothing and the same thing...
	foo $y                       # ...as calling foo without an argument

The above code, in a file named foo.sh does this: 

	russ@taliesin:~> ./foo.sh
	foo.sh: line 6: $1: unbound variable

To allow, in the same place, for an argument that is optional, use the following: 

	#!/bin/sh
	set -u

	foo()
	{
	  x=${1:-}
	  .
	  .
	  .

Or, between the - and the }, a default value to assign in case of the argument not being initialized may be set: 

	  x=${1:I'm a blue seraph}


Copying files and directories with spaces in their name

Here's a quickie I wrote to solve some of this. It's not exhaustively tested, but it does point out some solutions including a) gathering the rest of the arguments on the command line (spaces in directory name that is argument 2) and b) surrounding names likely to contain spaces with double-quotes. 

	#!/bin/sh
	set -u
	TRUE=1
	FALSE=0

	       verbose=
	    DEBUG_MODE=$FALSE
	PASSIVE_SCRIPT=$FALSE
	        ERRORS=$FALSE

	DoUsage()
	{
	  echo "
	$0 [-h] [-dnv]  

	  Copy files.

	Arguments
	  <target> is necessarily a directory. <source> is file or directory. If a
	  directory, subsumed contents are all copied to target intact including
	  directory hierarchy.

	Options
	  -d   engage debug mode
	  -h   show this usage blurb and exit
	  -n   perform no operations, only show what would be done
	  -v   echo progress
	"
	}

	# ============================================================================
	# Main entry point.
	#

	set -- `getopt dhnv $*`
	while [ $1 != -- ]; do
	  case $1 in
	    -d) DEBUG_MODE=$TRUE                                      ;;
	    -h) DoUsage                                      ; exit 0 ;;
	    -n) PASSIVE_SCRIPT=$TRUE                                  ;;
	    -v) verbose="v"                                           ;;
	     *) echo "WARNING: Ignoring invalid option ($1)" ; exit 0 ;;
	  esac
	  shift
	done
	shift

	src="${1:-}"
	shift
	tgt="${@:-}"
	src_is_dir=$FALSE

	# Ensure there are two arguments...
	if [ -z "$src" -o -z "$tgt" ]; then
	  DoUsage
	  exit 0
	fi

	if [ -d "$src" ]; then
	  src_is_dir=$TRUE
	elif [ -f "$src" ]; then
	  src_is_dir=$FALSE
	else
	  echo "  $src is neither file nor directory"
	  ERRORS=$TRUE
	fi
	if [ ! -d "$tgt" ]; then
	  echo "  $tgt is not a directory"
	  ERRORS=$TRUE
	fi

	# Need here to determine if $tgt is full path already, if so, don't combine!
	pos=`expr index "$tgt" /`
	fullpath=

	if [ $pos -eq 1 ]; then
	  # the first character is /, so this is not a relative path
	  dest=$tgt
	else
	  # what's given to us is a relative path from current working directory
	  fullpath=`pwd`
	  dest="$fullpath/$tgt"
	fi

	if [ -n "$verbose" ]; then
	  echo "  Path to target is: $dest"
	fi

	if [ $DEBUG_MODE -eq $TRUE ]; then
	  echo "---------------------------------------------------------------------"
	  echo "  Script variables:"
	  echo "      DEBUG_MODE = $DEBUG_MODE"
	  echo "  PASSIVE_SCRIPT = $PASSIVE_SCRIPT"
	  echo "             src = $src"
	  echo "             tgt = $tgt"
	  echo "      src_is_dir = $src_is_dir"
	  echo "         verbose = $verbose"
	  echo "        fullpath = $fullpath"
	  echo "             pos = $pos (position of slash in target path)"
	  echo "            dest = $dest"
	  echo "---------------------------------------------------------------------"
	  echo
	fi

	if [ $ERRORS -eq $TRUE ]; then
	  exit -1
	fi

	# Now do it...
	if [ $src_is_dir -eq $TRUE ]; then
	  # source is directory
	  if [ $PASSIVE_SCRIPT -ne $TRUE ]; then
	    pushd "$src"
	    cp -R$verbose * "$dest"
	    popd
	  else
	    echo "  pushd \"$src\""
	    echo "  cp -R$verbose * \"$dest\""
	    echo "  popd"
	  fi
	else    # it's just a file we're copying...
	  if [ $PASSIVE_SCRIPT -ne $TRUE ]; then
	    cp -$verbose "$src" "$dest"
	  else
	    echo "  cp -$verbose \"$src\" \"$dest\""
		fi
	fi

Color in bash script output...

This is pretty cool stuff if you're on Linux. 

	#!/bin/bash
	# Display ANSI colours.
	#
	esc="\033["
	echo -e "\t  40\t   41\t   42\t    43\t      44       45\t46\t 47"
	for fore in 30 31 32 33 34 35 36 37; do
	  line1="$fore  "
	  line2="    "
	  for back in 40 41 42 43 44 45 46 47; do
	    line1="${line1}${esc}${back};${fore}m Normal  ${esc}0m"
	    line2="${line2}${esc}${back};${fore};1m Bold    ${esc}0m"
	  done
	  echo -e "$line1\n$line2"
	done

A script that loops through a list containing variables

So, this is one of those variable containing the name of a variable things that make you scratch your head, then look to Google, but what do you search for?

In the sample below, pathlist contains the names of the variables already defined just above it. The script loops through that list one at a time, translates the variable name from a string to a bonafide script variable, "evaluates" it, then uses what's inside it as a path that is validated (to see if it exists or not).

The magic happens where bolded below. You can easily imagine other uses. 

	#!/bin/sh
	# This script checks the paths in local.properties to see if they are valid.
	# Obviously, these paths must be identical to those in that file.
	# Russ, 6 April 2011
	 dna_binaries=/home/russ/acme/software/dna_binaries
	    templates=/home/russ/acme/Templates/
	import_config=/home/russ/acme/console/sfprod/
	      install=/home/russ/acme/deploy
	      modules=/home/russ/acme/modules
	        swift=/home/russ/acme/swift
	     pathlist="dna_binaries templates import_config install modules swift"

	for path in $pathlist; do
	  the_path=${!path}

	  if [ ! -d "$the_path" ]; then
	    echo "ERROR: $path does not point anywhere real: $the_path"
	  else
	    echo " GOOD: $path points to existing subdirectory: $the_path"
	  fi
	done

Zap something in a file... 
	#!/bin/sh
	# -------------------------------------------------------------------------
	# Play around with obtaining a value from a file, .acctOid, that will be
	# substituted for another in one or more other files. It's much easier to
	# do this if we wimp out by saying there can't be anything else on the
	# line, but:
	#
	# "acctOid:10000129001,
	#
	# Actually, it's just that we'll only replace the first 10000129001 found
	# on the line since we don't want to add the additional magnitude of
	# complexity that would be to parse the line itself.
	# -------------------------------------------------------------------------
	acctOid=`cat .acctOid`

	for arg in $*; do
		file=$arg
		echo -n "$file"
	  mv $file "$file.old"
	  touch $file

	  while read line; do
	    modify=`echo $line | grep '"acctOid"'`
	    if [ -n "$modify" ]; then
	      echo `echo $line | sed s/[0-9][0-9]*/${acctOid}/` >> $file
	    else
	      echo "$line" >> $file
	    fi
	    echo -n "."
	  done < $file.old
	  rm $file.old
	  echo
	done
	# vim: set tabstop=2 shiftwidth=2 noexpandtab:

Parse out of file in a loop...

This parses stuff out of a file and, based on what it finds, will know what to do with it. Show a bunch of other interesting stuff. 

#!/bin/bash
#--------------------------------------------------------------------------
# New client run script for testing the User Management APIs
#--------------------------------------------------------------------------
TRUE=1
FALSE=0

     VERBOSE=$FALSE
     PASSIVE=$FALSE
   CLASSPATH=com.acme/iws/client.TestClient
   HOSTNAME=localhost
       PORT=7243
        LIB=bin2:lib/*

DoHelp()
{
  echo "Run Test Client

`basename $0` [--help]
`basename $0` [-hnpv]  [  ]

Options
  help    (this help blurb)
  h       Proxy host specification (default: localhost)
  n       Passive mode: show what will be run, but do not run it.
  p       Proxy port specification (default: 7243)
  v       Verbose mode: prompt with information, what's going to be run, etc.

Arguments
  file    name of a file containing an JSON request payload.
  command command in the command section of this script (add your own).

Alternative means of specification
  Proxy host may be specified by means of PROXY_HOSTNAME.
  Proxy port may be specified by means of PROXY_PORTNUMBER.
  "
}

# Look through a JSON payload for "typename":"userUpdatePayment" and
# parse out "userUpdatePayment".
GetOperationType()
{
  filename=${1:-}

  while read line; do
    typename_line=`echo $line | grep '"typeName"'`
    if [ -n "$typename_line" ]; then
      tokenized=`echo $typename_line | tr '":,' '   '`
      type=`echo $tokenized | awk '{ print $2 }'`
      simple_type=`echo $type | sed 's/iws//'`
      simple_type=`echo $simple_type | sed 's/Request//'`
      operation_type=`echo $simple_type | sed 's/^User/user/'`
      break
    fi
  done < $filename
}

GetOperationString()
{
  op=${1:-}

  case $op in
    "userRegister")      operation="iws/1/userRegister"      ;;
    "userLogin")         operation="iws/1/userLogin"         ;;
    "userExists")        operation="iws/1/userExists"        ;;
    "userDetail")        operation="iws/1/userDetail"        ;;
    "userResetPass")     operation="iws/1/userResetPass"     ;;
    "userUpdatePass")    operation="iws/1/userUpdatePass"    ;;
    "userUpdateData")    operation="iws/1/userUpdateData"    ;;
    "userUpdateAddress") operation="iws/1/userUpdateAddress" ;;
    "userUpdatePayment") operation="iws/1/userUpdatePayment" ;;
  esac
}

ExecutePayload()
{
     type=${1:-}
  payload=${2:-}

  GetOperationString $type

  if [ $VERBOSE -eq $TRUE ]; then
    echo "Operation $type from JSON payload in file $1..."
  fi
  if [ $PASSIVE -eq $TRUE ]; then
    echo "java -cp \"$LIB\" $CLASSPATH $HOSTNAME $PORT $operation $payload"
  else
    java -cp "$LIB" $CLASSPATH $HOSTNAME $PORT $operation $payload
  fi
}

# ========================================================================
# M a i n   s c r i p t   b o d y . . .
# ========================================================================

# Accept these from environment, but allow them to be overridden by the
# command line.
if [ -n "${PROXY_HOSTNAME}" ]; then
  HOSTNAME=${PROXY_HOSTNAME}
fi
if [ -n "${PROXY_PORTNUMBER}" ]; then
  PORT=${PROXY_PORTNUMBER}
fi

# Process command line.
set -- `getopt hnp:v $*`
while [ $1 != -- ]; do
  case $1 in
    -v) VERBOSE=$TRUE                                ;;
    -n) PASSIVE=$TRUE                                ;;
    -h) HOSTNAME=$2                       ; shift    ;;
    -p) PROXY_PORT=$2                     ; shift    ;;
     *) echo "WARNING: Ignoring invalid option ($1)" ;;
  esac
  shift
done
shift

# Process command line--may have filenames or commands...
# The script has choked up on the arguments. Those that remain are
# bonafide filename or command arguments. Loop through these and
# process.
for arg in $*; do
  if [ -f "$arg" ]; then
    file=$arg
    # ----------- Filename section ---------------------------------------
    # Pass the name of a JSON file with a payload in it, we open it,
    # ascertain which operation is involved (by parsing for "typename",
    # then the command line is built and issued in consequence.
    GetOperationType $file
    ExecutePayload $operation_type $file
    shift
    continue
  else
    # ----------- Command section ----------------------------------------
  fi
done

Output 

russ@tuonela:~/acme/tools/iws/TestClient> ./run.sh -n cc-ip-numeric.json  bogus-oid.json
java -cp "bin2:lib/*" com.snapfish/iws/client.TestClient localhost 7243 iws/1/userUpdatePayment cc-ip-numeric.json
java -cp "bin2:lib/*" com.snapfish/iws/client.TestClient localhost 7243 iws/1/userExists bogus-oid.json

Source files

bogus-oid.json: 
{
  "typeName":"iwsUserUpdatePaymentRequest",
  "acctOid":10000153001,
  "cardHolderName":"Lilian Munster",
  "cardType":"VIS",
  "cardNumber":"4111111111111111",
  "expiration":"12/2020",
  "street1":"1313 Mockingbird Lane",
  "street2":"Suite 120",
  "street3":"c/o Herman Munster",
  "city":"Mockingbird Heights",
  "state":"CA",
  "zipcode":"90210",
  "countryCode":"US",
  "phoneNumber1":"555 555-1212",
  "phoneNumber2":"555 555-1213",
  "cvv2":"333",
  "clientIpAddress":"127.0.0.1"
}
bogus-oid.json: 
{
  "typeName":"iwsUserExistsRequest",
  "acctOid":10000999001
}

Counting in a loop...

Here's a little exercise of looping n times, where n is supplied by command-line argument, then padding the index out to however many places is necessary to express it in the same number as the total. This would be useful in renaming files with a base name and an index for example (see last echo). 

#!/bin/sh

count=$1
number=0
width=${#count}

echo "count=$count"
echo "width=$width"

while [ $number -lt $count ]; do
  number=`expr $number + 1`
  nth=$number
  numberwidth=${#nth}
  w=`expr $width - $numberwidth`
  while [ $w -ne 0 ]; do
    w=`expr $w - 1`
    nth=0$nth
  done
  echo -n "$nth: "
  # if we were going to rename files...
  echo "newfile-$nth.ext"
done

Console output

If the script were named x.sh, ... 

$ ./x.sh 11
count=11
width=2
01: newfile-01.ext
02: newfile-02.ext
03: newfile-03.ext
04: newfile-04.ext
05: newfile-05.ext
06: newfile-06.ext
07: newfile-07.ext
08: newfile-08.ext
09: newfile-09.ext
10: newfile-10.ext
11: newfile-11.ext

$ ./x.sh 3
count=3
width=1
1: newfile-1.ext
2: newfile-2.ext
3: newfile-3.ext

cp --parent: create hierarchy while copying

To save a file on a deep hierarchical path somewhere else while preserving its deep hierarchy, in other words, while creating all the parent hierarchy as part of the copy, do this. Imagine my current working directory is /home/russ/dev/jms-queue-admin and the other place is /home/russ/dev/jms-queue. 

cp --parents jms-queue-admin-acceptance-tests/.classpath ../jms-queue
cp -R --parents jms-queue-admin-acceptance-tests/.settings ../jms-queue

Pipe stdin to browser

See Pipe stdin to browser.


Pipe stdin to browser

Links


Loop and count

Here's how to create and increment a counter. Here, we're counting just the JSON files in the current subdirectory. There are more modern (read: bash) ways to increment a counter, but when I started out, bash wasn't universally available on UNIX operating systems. It was safer to remain in the more ubiquitous Bourne shell for my work. And so I have done ever since even though I only ever work on Linux. 

#!/bin/sh
FILES=./*.json
COUNT=0
for file in $FILES; do
	COUNT=$( expr $COUNT + 1 )
	echo "${COUNT} $file"
done

Output: 

1 file1.json
2 file2.json
3 file3.json
etc...

find: How to list all XML files 
$ find . -name '*.xml' [ -type f ]

Here's how to read this command:

find filesystem entities
    . beginning in the current subdirectory (recurs below it too),
-type f (if there is a subdirectory named something.xml, it would show up in your list without specifying this, but maybe you don't care, so I made this optional above—leave it out to simplify the command)

How to look for "apple" inside all XML files found: 

$ find . -name '*.xml' -exec fgrep -Hn apple {} \;
    ...
-exec means "execute this command" after you've found something
fgrep is the command-line binary to execute
  -Hn fgrep options to show filename and line number
  {} means the (repeated) instances of what's found, in this case, filenames
  \; magic incantation that escapes in a semi-colon to end the exec clause

How to list out everything in a directory except XML files 
$ find . ! -name '*.xml' -type f

Here's how to read this command:

find filesystem entities
    . beginning in the current subdirectory (recurs below it too)
    ! not files whose extension is .xml
-type f and look among only entities under this subdirectory of type "file"

How to delete everything that isn't an XML file from the current subdirectory on down: 

$ find . ! -name '*.xml' -exec rm {} \;
    ...
-exec means "execute this command" after you've found something
  rm is the command-line binary to execute
  {} means the (repeated) instances of what's found that doesn't match the negated template
  \; magic incantation that escapes in a semi-colon to end the exec clause

How to find the smallest (shortest) file in the subdirectory

I have a directory full of 600+ JSON (and XML) files. I'm looking for shorter example to use.

Let's find the shorted JSON file in the current subdirectory. This is mostly about using awk. 

$ find . -maxdepth 1 -name '*.json' -type f -printf '%s\t%p\n' | sort -n | awk '
  NR == 1 { s = $1 }
  $1 != s { exit }
  1'

188199	./630.json      # (and the winner is!)

It's much easier to figure out how to discover the largest (longest) file: 

$ find . -name '*.json' -type f -exec ls -al {} \; | sort -nr -k5 | head -n 1
-rw-rw-r-- 1 russ russ 7955015 Jun 25 08:34 ./160.json

Using getopts for optional and mandatory arguments

First, getopts only parses options until it sees the first argument that does not start with -. getopt implements a more flexible, GNU-style parsing.

Here is a scheme that consumes all the optional arguments using getopt and all the non-optional ones with getopt a second time: 

















 Replacing the extension on a lot of files 



For example, I replace .jpeg with .jpg in the current working
directory.




#!/bin/sh
for file in *.jpeg; do
  mv -v "$file" "${file%.jpeg}.jpg"
done













 Executing last command with a space in it (from your history) 



I want to repeat my last $ docker run ... command, but I have issued a
pile of Docker commands ($ docker ps -a, $ docker rm ...,
etc.). Here's how:




$ !?docker run?













 Rename files named *_ixml.xml to _ixml 


#!/bin/sh
# First, rename such files to have the extension .ixml...
for file in *_ixml.xml ; do
	mv -- "$file" "${file%.xml}.ixml"
done

# Then, remove the _ixml subcomponent in the name...
rename 's/_ixml//' *.ixml













 fgrep/grep to match, then display next line too 


$ fgrep --after-context=n search-string file-list





...where n is the number of lines after the match you wish to display.





There is also --before-context=n.