bash-basic

Posted on 2019-12-23 Edited on 2024-02-22 In bash , demo

Bash

Introduction

As bash is old or native way to run commands, there are lots of scripts written with bash, so it’s a must to know bash and do some basic work.

Better use python instead of bash as you can call bash in python and use python powerful features

printing

There are several ways for printing and writing multiple lines, echo is the easy way to do

# print multiple lines
$ echo -e "a\nb"
$ echo "a
b"

# write multiple lines to a file
$ echo -e "a\nb" >file
$ echo "a
b" >file
$ cat <<END >file
a
b
END

%%bash
echo "a b"

# printing in multiple lines
echo "a
b"
# write in one line, printing in multiple lines
echo -e "a\nb"

a b
a
b
a
b

special characters

Special characters here mean shell sees them special but not treated as regular pattern, if shell treats them as pattern, its meaning may be different. see an example.

$ls
a.c abd.c abc.c
$ grep "ab*" ab*
abd.c:`a`
abd.c:`ab`c
abd.c:`ab`
abc.c:`ab`b
abc.c:`ab`d

grep "ab*" ab* search pattern ab*(a, ab matched, abc not match) from file which starts with ab(abd.c, abc.c all match)

whitespace The most special char in shell is whitespace, whitespace is used as separator for parameters, like function parameter, test parameter etc.

; semi-colon is used to separate instructions if at same line, if each instruction at each line, ; can be ignored. $ls;date

* any more(0-)characters #ls ab*—>file ab matches

? any character, just one #ls ab? —>file ab not match!!

[] any character inside it becomes normal except \ [] ! #ls test.[ch]

- indicates different meanings depends on its location
[a-z] – linker
[-az] – normal
[az-] - normal

! must be a start to negate the condition, can't be other place
[!a-z]
[\!a-z]
[a!z] error
[a\!z] ok

| command pipe, #ls ab* | xargs cat

() group command, run command in subshell, #msg=$(echo hello)

& run in background

‘’ keep all(characters) as literal inside, Any char in '' are escaped, echo 'I\'m a boy' #error

“” quotes string and evaluates variable #echo "$var"

most chars in “” (except [$, `, \]) are escaped

\ escape character

`` run command and get its output, #msg=`echo hello`

Standard wildcards(globing)

Standard wildcards (also known as globing patterns) are used by various command-line utilities to work with multiple files. Standard wildcards are used by nearly any command (including mv, cp, rm and many others).

? (question mark)

this can represent any single character. If you specified something at the command line like “hd?” GNU/Linux would look for hda, hdb, hdc and every other letter/number between a-z, 0-9.
* (asterisk)

this can represent any number of characters (including zero, in other words, zero or more characters). If you specified a “cd*” it would use “cda”, “cdrom”, “cdrecord” and anything that starts with “cd” also including “cd” itself. “m*l” could by mill, mull, ml, and anything that starts with an m and ends with an l.
[ ] (square brackets)

specifies a range. If you did m[a,o,u]m it can become: mam, mum, mom if you did: m[a-d]m it can become anything that starts and ends with m and has any character a to d in between. For example, these would work: mam, mbm, mcm, mdm. This kind of wildcard specifies an “or” relationship (you only need one to match).

{ } (curly brackets, also called brace)

terms are separated by commas and each term must be the name of something or a wildcard. This wildcard will copy anything that matches either wildcard(s), or exact name(s) (an “or” relationship, one or the other).
[!]

This construct is similar to the [ ] construct, except rather than matching any characters inside the brackets, it'll match any character, as long as it is not listed between the [ and ]. This is a logical NOT. For example rm myfile[!9] will remove all myfiles* (ie. myfiles1, myfiles2 etc) but won’t remove a file with the number 9 anywhere within it’s name.

variable

Like Python, variable has week type(no need to declare it first), shell checks its type only when it runs, like others, shell has local, global, env variable, if no keyword is specified, default is global!

%%bash
na="hi"
name="jason    kk" # must wihout space at both side of =
echo $name         # max match, space supressed

# {} is needed to when link with others, to ensure where the edge of variable!!!!
echo ${na}me # string link, nothing needed.

# "" is needed to prevent space supressed.
echo "$name" # keep orignal, no space supressed.
# "" {} are necessary in some case, but can be ignore in others.
echo $name, ${name}, "${name}"

echo "${#name}" # the length of string variable, count of array ${#name[@]} if name is an array

jason kk
hime
jason    kk
jason kk, jason kk, jason    kk
11

local /global var

by default, variable is global, but you can only add local keyword in function to strict its scope, local keyword can’t be used outside of a function. but all variables(local global) can be seen in sushell(forked process) as well, but it's another copy of these variables, changed in subshell not see by parent!!!

special var

1
2
3

$?	Exit status of last task
$$	PID of shell process
$0	Filename of the shell script

%%bash
var_func() {
  local a=12
  b=20 # global, can be seen outside of the function

  c=$(
    let a=a+1
    echo $a
  ) # run in subshell, still see a, get sushell output by $()
  echo "in function c=$c"

  echo "in function a=$a" # a in unchanged in parent
}

# local d=23 error!  local can't be used outside of function
var_func
echo "in main: a=$a" #a is undefined with null value(as a in var_func is local)
echo "in main: b=$b"

in function c=13
in function a=12
in main: a=
in main: b=20

env var

Env variable is defined outside of the a script, or you can define it in the script, so that all subshells can have it.

%%bash
declare env_va="hello"
echo $env_va

echo $PATH # access default env variable

hello
/opt/llvm/bin:/home/data/Anaconda3/envs/py3.9/bin:/home/data/Anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/go:/home/go/bin:/root/.yarn_pkg/bin:/usr/lib64:/usr/local/go/bin:/root/.local/bin:/home/data/Anaconda3/bin:/home/data/Anaconda3/sbin

array[pay attention]

you can define an array with one line Fruits=('Apple' 'Banana' 'Orange') or add item at any slot Fruits[1]="Ab" or remove one element unset Fruits[1], then you can access one item or all items or range items like below

1
2
3

- all items---echo ${Fruits[@]}
- one item----echo ${Fruits[0]}
- range-------echo ${Fruits[@]:0:2}# Range (from position 0, length 2)

%%bash
# Fruits=('Apple' 'Banana' 'Orange')
# OR
# no need declare array, just use it directly!!!
Fruits[0]="Apple"
Fruits[1]="Banana"
Fruits[3]="Orange"

echo ${Fruits[0]}       # Element #0
echo "${Fruits[@]}"     # All elements, space-separated
echo ${#Fruits[@]}      # 3: Number of elements, no count null slot
echo ${#Fruits}         # String length of the 1st element
echo ${#Fruits[2]}      # String length of the Nth element, no skip, index 2 is null slot
echo "${Fruits[@]:0:3}" # Range (from position 0, length 3), no count null slot

Fruits=("${Fruits[@]}" "Watermelon")    # Push
Fruits+=('Watermelon')                  # Also Push
echo "${Fruits[@]}"

# Fruits is reset with new index
Fruits=(${Fruits[@]/Ap*/})              # Remove by regex match
echo "${Fruits[@]}" 

# index is not change, slot 1 is null!!
unset "Fruits[1]"                         # Remove one item
# Fruits=("${Fruits[@]}")                 # Duplicate
# Fruits=("${Fruits[@]}" "${Veggies[@]}") # Concatenate
#lines=(`cat "logfile"`)                  # Read from file

for i in "${Fruits[@]}"; do
  echo -n "$i "
done

Apple
Apple Banana Orange
3
5
0
Apple Banana Orange
Apple Banana Orange Watermelon Watermelon
Banana Orange Watermelon Watermelon
Banana Watermelon Watermelon

function

Unsupported declare function parameter like function(a,b), use $1, $2 etc

parameter

$#----number of parameters from command line(not count script name)
$1----the first parameter
# above rule apply function as well

$0----script name(no parameter) with path together (basename $0 ,just the script name)

$ ./myspt.sh h1 h2

1
2
3

$#-----2
$1-----"hi"
$0-----./myspt.sh

#!/bin/bash

# the parameters are regarded as one long quoted string
echo "Using \"\$*\":"
for a in "$*"; do
    echo $a;
done

# the string is broken into words by the for loop
echo -e "\nUsing \$*:"
for a in $*; do
    echo $a;
done

# it treats each element of $@ as a quoted string:
# this is mostly what you want!!!
echo -e "\nUsing \"\$@\":"
for a in "$@"; do
    echo $a;
done

# it treats each element as an unquoted string, so the last one is again split by what amounts to for three four
echo -e "\nUsing \$@:"

for a in $@; do
    echo $a;
done

$ ./myspt.sh h1 h2
Using "$*":
one two three four

Using $*:
one
two
three
four

Using "$@":
one
two
three four

Using $@:
one
two
three
four

The implementation of “$“ has always been a problem and realistically should have been replaced with the behavior of “$@”. In almost every case where coders use “$“, they mean “$@”. “$*” can cause bugs and even security holes in your software.

always use “$@” if you want to check all parameters as it’s mostly what you want

definition

no return keyword, output as return
no explicit parameter list
get the return value by command group () or command substitution ``
$() is the suggested way

%%bash
max () {
    if (($1 > $2 )); then
        echo "$1"
    else
        echo "$2"
    fi
}

# get function output
echo "max is: $(max 1 2)"
let m=`max 1 2`
let m=$(max 1 2) # run in subshell

max is: 2

Subshell

()(group command) runs in subshell, all others run in main shell like {}, (()).

%%bash
(a=12) # subshell

{
  b=12; # main shell
}

((c=13)) # main shell
echo $a, $b, $c

, 12, 13

substitution[pay attention]

command substitution

1	files=`ls *`

variable substitution

${var:-word} #if var null or unset, return word, unchanged var
${var:=word} #if var null or unset, return word and set var=word
${var:?word} $if var null or unset, print word, unchanged var

${var:+word} $if var is set, return word, unchanged var

wildcard substitution

1
2
3

$ ls *
$ ls [a]*
$ ls *.[ch]

# The pattern matching is always greedy!!
# FOO is variable
FOO="hello"
${FOO%suffix}	Remove suffix
${FOO#prefix}	Remove prefix
${FOO%%suffix}	Remove long suffix
${FOO##prefix}	Remove long prefix

${FOO/from/to}	Replace first match
${FOO//from/to}	Replace all
${FOO/%from/to}	Replace suffix
${FOO/#from/to}	Replace prefix

%%bash

STR="/path/to/foo.cpp"
echo ${STR%.cpp}    # /path/to/foo
echo ${STR%.cpp}.o  # /path/to/foo.o

echo ${STR##*.}     # cpp (extension)
echo ${STR##*/}     # foo.cpp (basepath)

echo ${STR#*/}      # path/to/foo.cpp
echo ${STR##*/}     # foo.cpp

echo ${STR/foo/bar} # /path/to/bar.cpp

SRC="/path/to/foo.cpp"
BASE=${SRC##*/}   #=> "foo.cpp" (basepath)
echo $BASE
DIR=${SRC%$BASE}  #=> "/path/to/" (dirpath)
echo $DIR

/path/to/foo
/path/to/foo.o
cpp
foo.cpp
path/to/foo.cpp
foo.cpp
/path/to/bar.cpp
foo.cpp
/path/to/

compare or condition

[] is an old way, used for string and number, file, here are options for it,

- file     -d 	-c 	-b 	     -f 	-e(exist) 	-r 	-w 	-x(executable)
- string   != 	= 	-n(non-zero string) 	-z(zero string)
- integer(two characters for each operator!!!)  -eq -ne -lt -le -gt -ge(great than)
- logical  -a(and) -o(or) !(negative)

[[]] supports all options as [], As a rule of thumb,

[[ is used for strings and files.
numbers, use an (())

  = Set Equal to
 *= Multiply
 /= divide
 %= Modulo
 += Add
 -= Subract
<<= bitwise shift left
>>= bitwise shift right
 &= bitwise AND
 ^= bitwise XOR
 |= bitwise NOT
 == Test Equality
!= Test Inequality
 < Less than
 > Greater than
<= Less than or equal
>= Greater than or equal

&& and || used for logical
[[ support pattern matching [[ STRING =~ PATTERN ]], it’s regular pattern not globing

Difference:

file="file name"
[[ -f $file ]] && echo "$file is a regular file"
# will work even though $file is not quoted and contains whitespace.

# "" is a must as file has whitespace
[ -f "$file" ] && echo "$file is a regular file"

#Parentheses() in [[ do not need to be escaped:
[[ -f $file1 && (-d $dir1 || -d $dir2) ]]
[ -f "$file1" -a \( -d "$dir1" -o -d "$dir2" \) ]

Implicit Conversion

The -eq(integer operators) causes the strings to be interpreted as integers if possible including base conversion

%%bash
# implicit conversion
if [ '1' -eq 1 ]; then
  echo "equal"
fi

# comment it out to see implicit conversion
# if [ '1' -eq "a" ]; then # bash: line 7: [: a: integer expression expected
#    echo "equal"
# fi

# pattern matching for [[ ]]
dig=123
if [[ $dig =~ ^[0-9]+$ ]]; then
  echo Numeric
else
  echo Non-numeric
fi

equal
Numeric

flow control

There are several ways to do flow control, like if/while/for, let’s see each of them with example.

%%bash
let a=1

if [ $a -gt 2 ]; then
  echo "a is greater than 2"
elif [ $a -gt 1 ]; then
  echo "a is greater than 1"
else
  echo "a equal 0 or 1"
fi

while [[ $a -gt 0 ]]; do
  echo $a
  let a=a-1
done


VAR="a b"
for i in $VAR # VAR separated by space by default
do
  echo $i
done

for i in "$VAR" # VAR as a whole
do
  echo $i
done

# -----for loop in Linux treats pattern as filename when no files exist---
# for file in /tmp/*
# do
#   echo $file // if there is no file under /tmp, echo will print /tmp/*
# done

# for file in /tmp/*
# do
#   if [ -e $file ]; then
#     echo $file
#   fi
# done

# for file in `ls`
# for .. in expanded_variable 
for file in $(ls); do
  echo $file
done

# (())
for ((i = 0; i < 5; i++)); do
  echo $i
done

a=1
# case for shell pattern matching
case $a in
  "1" | "2") echo "\$a matches pattern" ;;     # ;; is a must like break
  "12*") echo "\$a matches another pattern" ;; # with ;; outside of case, as ;; is a break!
esac

a equal 0 or 1
1
a
b
a b
bash-basic.ipynb
0
1
2
3
4
$a matches pattern

signal

Set signal handler with trap command

trap quit 2 3 9 #signal handler
quit(){
  echo "You can't kill me!!!!!!!!!!!!!!!!"
}

string(slice of string)[pay attention]

%%bash
# ${parameter:offset}
# ${parameter:offset:length}

# string from index 0 !!!
name="John"
# [1-2]
echo ${name:1:2} #=> "oh" (slicing)
# [0-2)
echo ${name::2} #=> "Jo" (slicing)
# [0-end)
echo ${name::-1} #=> "Joh" (slicing)

# [-3-end] (end is -1, must add a space before negative number)
echo ${name: -3} #=> "n" (slicing from right)
# count two from index -3
echo ${name: -3:2} #=> "n" (slicing from right)

oh
Jo
Joh
ohn
oh

tips

debug shell

check syntax

$bash -n test.sh just check syntax of test.sh

debug shell

set -x #enable tracing will display all commands and their arguments as they execute.
if [ -z "$1" ] ; then
    echo "ERROR: Insufficient Args."
    exit 1
fi

set +x #disable tracing

redirect

stdin——————-0
stdout——————1
stderr——————2

> just redirects the stdout to file without stderr.

$ls no_file.txt >log.t 2>&1

cd vs pushd/popd

With cd you need to write the path explicitly, while pushd/popd, you don't need remember the original path.

#pushd /home/lzq 
(save the current dir on stack and go to /home/lzq)
#popd
(return to the saved dir)

() vs (()) vs let vs expr

() is used for command groups with subshell, you can run any command in subshell
(()) and let are just for integer, not in subshell
expr is only for integer operation as well as expr is a command, so $ is a must for variable!!!
$[] is for integer operation as well

NOTE: bash only supports integer operation when do integer operation $ can be omitted for variable

%%bash
(
  echo hello
  var1="subshell"
)
echo $var1 #never see var1

echo $(
  echo hello
  var1="subshell"
) # get output from subshell!!!

# (()) only for integer, $ can be omitted.
((var2 = "ab"))
echo $var2 #integer value 0!!!

((var2 = 10))
echo $var2

# let
let var2=10
echo $var2
let var2="ab"
echo $var2

a="1"
b="2"
if [[ "$a" = "$b" ]]; then
  echo "equal"
fi

# must add space for each operator!!!
expr $RANDOM % 10
b=`expr $RANDOM % 10 + 3`
echo $b
echo `expr 2 + 3`

echo '============$(())========='
# the last value is used as `return` value
echo $((2+3))
echo $((2+3,4+5,a=15))

echo '============$[]========='
echo $[2+3]
echo $[2+3,4+5]
echo $[a=15,5+5]
echo $a

hello

hello
0
10
10
0
1
11
5
============$(())=========
5
15
============$[]=========
5
9
10
15

[] vs [[]]

[ ("test" command) and [[ ("new test" command) are used to evaluate expressions. [[ works only in Bash, Zsh and the Korn shell, and is more powerful; [ and test are available in POSIX shells.

difference

brace expansion

Used to generate list of string or number with prefix or suffix for each item, prefix or suffix is optional.

{} linked each item with prefix and suffix if has, print it as a single string
use case

_{a..f}_
{a,b,c}

%%bash
echo {a,b} # no space between each item echo {a, b} will get unexpected!!!

echo _{a, b}_ # see the result
# if you want to use space between item, escape it
echo _{a,\ b}_ # see the result
echo not_prefix prefix_{a,b}_suffix not_suffix # see which part is prefix and suffix
echo {a..f}
echo 1.{1..10}
for i in {1..5..2}; do # with step 2 default 1
  echo $i
done

# print each item(item seperated by whitespace) on a new line
printf "%s\n" {item1,item2}

a b
_{a, b}_
_a_ _ b_
not_prefix prefix_a_suffix prefix_b_suffix not_suffix
a b c d e f
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10
1
3
5
item1
item2

read a file

you can read a whole file at one time or read it line by line

# read whole into memory at once
for line in $(cat file.txt); do
  echo $line
done

# read line by line
# read is keyword !!!
while read line; do
  echo $line
done < file.txt

dict in shell [pay attention]

In order to use dict, you must declare it first like this

declare -A sounds

%%bash
# dict requires declare first!!!
declare -A sounds
sounds[dog]="bark" # same: sounds['dog']="bark" '' is only needed when has space in key
sounds[cow]="moo"
sounds[bird]="tweet"
sounds[wolf]="howl"

echo ${sounds[dog]}  # same: echo ${sounds['dog']}
echo "${sounds[@]}"  # All values
echo "${!sounds[@]}" # All keys
echo ${#sounds[@]}   # Number of elements

for key in "${!sounds[@]}"; do
  echo $key
done

for val in "${sounds[@]}"; do
  echo $val
done

unset sounds[dog] # Delete dog

bark
bark howl moo tweet
dog wolf cow bird
4
dog
wolf
cow
bird
bark
howl
moo
tweet

difference with or without “” for variable expansion

In most shells, leaving a variable expansion unquoted is like invoking some sort of implicit split+glob operator.

$var
In another language would be written something like:
glob(split($var))

$var is first split into a list of words according to complex rules involving the $IFS special parameter (the split part) and then each word resulting of that splitting is considered as a pattern which is expanded.

%%bash
wds="hello boy"
for wd in $wds
do
echo $wd
done

for wd in "$wds"
do
echo $wd
done

o=$(ls -l)
for lt in $o
do
echo $lt
done

for lt in "$o"
do
echo "count"
echo $lt
done

hello
boy
hello boy
total
40
-rw-r--r--
1
jaluo
root
40733
Feb
22
02:04
bash-basic.ipynb
count
total 40 -rw-r--r-- 1 jaluo root 40733 Feb 22 02:04 bash-basic.ipynb

IFS

IFS: The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline>.

%%bash
# defautl IFS
str="hello boy"
for elm in $str
do
echo $elm
done

IFS=","
str="hello,boy"
for elm in $str
do
echo $elm
done

hello
boy
hello
boy

include file

Use source or .

1
2
3

source /path/s.sh
. /path/s.sh
#. (dot) command is an alias to source

random number

$RANDOM is the random number

1
2
3

%%bash
echo $RANDOM
echo $RANDOM

15887
27804

do another operation if first command runs ok

%%bash

# one way
if ls /tmp >/dev/null;then
  echo "ls runs ok"
else
  echo "ls runs badly"
fi

if ls /not_exist >/dev/null 2>&1;then
  echo "ls runs ok"
else
  echo "ls runs badly"
fi

# this way only support ok, no more control
ls /tmp>/dev/null 2>&1 && echo "ls runs ok"

# same as the first way
ls /tmp>/dev/null 2>&1
if [ $? -eq 0 ];then
  echo "ls runs ok"
else
  echo "ls runs badly"
fi

ls /not_exist>/dev/null 2>&1
if [ $? -eq 0 ];then
  echo "ls runs ok"
else
  echo "ls runs badly"
fi

ls runs ok
ls runs badly
ls runs ok
ls runs ok
ls runs badly

quotes

for assignment in bash, the righ must be a single word, so S="a b" has the same with S=a\ b, “” to make a string as a whole, like below

for i in "a b" # "a b" as a whole
do
  echo $i # execute only once
done

S="a b"
for i in $S  # same as for i in a b("" is removed for var expansion)
do
  echo $i # execute twice!!!
done

check string contains

To check if a string contains a substring

%%bash
# wildcard way1
STR='GNU/Linux is an operating system'
SUB='Linux'
if [[ "$STR" == *"$SUB"* ]]; then
  echo "It's there."
fi

# case operator way2
STR='GNU/Linux is an operating system'
SUB='Linux'

case $STR in

  *"$SUB"*)
    echo  "It's there."
    ;;
esac

# Regex Operator way3
STR='GNU/Linux is an operating system'
SUB='Linux'

if [[ "$STR" =~ "$SUB" ]]; then
  echo "It's there."
fi

# grep way4
STR='GNU/Linux is an operating system'
SUB='Linux'

if grep -q "$SUB" <<< "$STR"; then
  echo "It's there"
fi

It's there.
It's there.
It's there.
It's there

repeat char(str) n times

%%bash
printf '=%.0s' {1..10}

echo

for i in `seq 1 10`
do
echo -n "="
done

echo

for i in {1..10}
do
echo -n "="
done

==========
==========
==========

echo with color

%%bash

RED='\033[0;31m'
NC='\033[0m' # No Color
echo -e "I ${RED}love${NC} Stack Overflow"

# for simple, refer to https://svelte.dev/repl/1b3f49696f0c44c881c34587f2537aa2?version=4.0.5

I [0;31mlove[0m Stack Overflow

Invert boolean variable

%%bash

[ ${FOO:-false} == false ] && FOO=true || FOO=false
echo $FOO
[ ${FOO:-false} == false ] && FOO=true || FOO=false
echo $FOO

true
false

compile bash to binary

$ sudo yum install -y shc

$ cat hello.sh
#!/usr/bin/bash
echo hello

$ shc -f hello.sh
$ ls hello*
hello.sh  hello.sh.x  hello.sh.x.c 

# .sh is the original script.
# sh.x is the compiled binary.
# .sh.x.c is the C source code generated from the .sh file prior to compiling to .sh.x.

Kubernetes

Pod