Download Unix Operating System Fundamentals: Lecture 3 - Using Basic UNIX Utilities and Scripts and more Study notes Computer Science in PDF only on Docsity!
CS CS-- 344344 - - Unix OperatingUnix Operating
System Fundamentals System Fundamentals
Lecture 3
Using Basic UNIX Utilities
and
Using Multiple Utilities in Scripts
Based on slides created by
Dr. Bangalore for the
Spring 2005 offering of
the course
6/16/2005 3 Department of Com
puter and Informati
on Sciences
UAB
Determine which shell you currently have
One option, use โpspsโ command
How to determine where the command or utility
is present?
Use โwhichwhichโ command
Few points about the Shell (I)Few points about the Shell (I)
$ which bash /usr/bin/bash
$ ps PID TTY TIME CMD 2665 pts/2 0:00 bash
6/16/2005 4 Department of Com
puter and Informati
on Sciences
Few points about the Shell (II)Few points about the Shell (II)
How does the shell know that a command is in a
particular directory?
Using an environment variable โ PATHPATH
How to determine what PATH variable is set to?
What other environment variables are there?
Use โenvenvโ (also โprintenvprintenvโ on Linux) to display all the environment variables and their corresponding values
$ echo $PATH /usr/bin:/bin:/usr/sbin:/sbin:/hf/local/bin
6/16/2005 5 Department of Com
puter and Informati
on Sciences
UAB
How to change one of the environment variable?
For bash, type: PS1=PS1=โโsomepromptsomepromptโโ
In general, type: variable=valuevariable=value
How to make these changes be persistent?
Include these changes in the file .bash_profile or .bashrc (for bash shell)
Few points about the Shell (III)Few points about the Shell (III)
puri@blazer1:~[501]$ PS1='[hostname:date] $ ' [blazer1:Sun Feb 13 19:14:06 CST 2005] $
$ echo $PATH /usr/bin:/bin:/usr/sbin:/sbin:/hf/local/bin $ PATH=$PATH:$HOME/bin $ echo $PATH /usr/bin:/bin:/usr/sbin:/sbin:/hf/local/bin:/mz/mb/puri/bin
6/16/2005 6 Department of Com
puter and Informati
on Sciences
UAB
cutcut
To cut out fields from each line of a file
Each field must be separated by a delimiter
default delimiter is a tab other delimiters use โ โd (d (--ddโโ โโ oror โ โd:)d:)
Other options include:
By fields: cutcut โ โf1,4 myfilef1,4 myfile By character position: cutcut โ โc5c5--10 myfile10 myfile
What do you see when you type:
cutcut โ โf5,6f5,6 โ โd: /etc/passwdd: /etc/passwd ^ lsls^ โ โl | cutl | cut^ โ โc55c55โโ
6/16/2005 10 Department of Com
puter and Informati
on Sciences
sort (I)sort (I)
Sort all input lines based on specified order
(default in ASCII order)
Usage: sort
To sort:
Dictionary order: sortsort โ โd myfiled myfile Ignore case: sortsort โ โf myfilef myfile Numerical value: sortsort โ โn myfilen myfile Reverse sort: sort โ โr myfiler myfile
Output sent to standard output by default, use
- โoo option to send output to a specified file sort myfile.insort myfile.in โ โo myfile.outo myfile.out
6/16/2005 11 Department of Com
puter and Informati
on Sciences
UAB
sort (II)sort (II)
To sort using data after specific fields use:
sortsort โ โk n filek n file
To sort using a specific field range use:
sort +nsort +n^ โ โm filem file^ (start after n delimiters and stop after m delimiters)
To specify a secondary key for sorting use:
sort +nsort +n โ โm +pm +p โ โq fileq file (field range n-m for primary key, p- q for secondary key)
A different sort order can be specified for the
secondary key (see pages 265-266)
Useful when sorting data based on a specific field
and breaking ties with a secondary key
6/16/2005 12 Department of Com
puter and Informati
on Sciences
UAB
joinjoin
To combine two files based on a common field
(input files must be sorted)
Usage: join file1 file2join file1 file
By default all fields from both files are sent to the
standard output after the common field
To display only specific fields from each file the โ
o option can be used:
joinjoin โ โo 2.2 1.2 1.1 file1 file2o 2.2 1.2 1.1 file1 file
By default field 1 is used as the common key, a
different field can be specified using โj option:
joinjoin^ โ โj1 nj1 n^ โ โj2 m file1 file2j2 m file1 file
6/16/2005 13 Department of Com
puter and Informati
on Sciences
sedsed
((sstreamtream^ ededitoritor))
Stream editor โ works on individual lines instead
of reading the entire file
Useful when working on large files or making
changes from a script file
Sample usage scenarios (output sent to standard
output):
Replace all instances of a specific string: sedsed โโs/abc/ABC/gs/abc/ABC/gโโ myfilemyfile Search specific string and then make a replacement: sedsed^ โโ/xyz/s/abc/ABC/g/xyz/s/abc/ABC/gโโ^ myfilemyfile Delete lines: sedsed โโ/abc/d/abc/dโโ myfilemyfile
6/16/2005 14 Department of Com
puter and Informati
on Sciences
UAB
trtr
Reads standard input, deletes or translate
characters based on the input options, and
displays output to standard output
Usage: tr string1 string2tr string1 string
Examples:
tr a A < myfiletr a A < myfile ^ tr abc XYZ < myfiletr abc XYZ < myfile^ or^ cat myfile | tr abc XYZcat myfile | tr abc XYZ trtr โ โdd โโxyzxyzโโ < myfile< myfile cat /etc/cat /etc/passwdpasswd | tr| tr โโ::โโ โโ โโ trtr โโ\nnโโ โโ โโ < myfile< myfile
6/16/2005 15 Department of Com
puter and Informati
on Sciences
UAB
teetee
Note that output redirection with > and | we can
send output to either a file or a utility not both
โteeteeโ is used to send output to both a file and
another utility (similar to a plumberโs tee)
The data is not modified in anyway by tee utility
Example:
lsls^ โ โl | tee myfiles | wcl | tee myfiles | wc^ - -ll
6/16/2005 19 Department of Com
puter and Informati
on Sciences
Shell Scripts (II)Shell Scripts (II)
To execute a script:
The script must have execute permission
File permissions can be set using โchmodchmodโ
or
Use source script_namesource script_name
6/16/2005 20 Department of Com
puter and Informati
on Sciences
UAB
Creating a complex scriptCreating a complex script
Read a file โ myfile.in
Output to the screen the total number of unique
words
Output the list of unique words to the file
words.out along with the number of times each
word appears ordered with the most-used words
listed first
Solution:
tr -d '?."!:,;' < myfile.in | tr 'A-Z' 'a-z' | tr ' \t' '\n\n'
| sed '/^$/d'
| sort | uniq -c | sort -rn
| tee words.out | wc -l
6/16/2005 21 Department of Com
puter and Informati
on Sciences
UAB
AlgorithmAlgorithm
Delete punctuation characters
Convert all characters to lowercase
Move each word to a separate line
Remove blank lines (if any)
Sort the lines
Remove duplicates
Compute word frequency and output to file
Compute the total number of unique words
6/16/2005 22 Department of Com
puter and Informati
on Sciences
Implementation (I)Implementation (I)
Delete punctuation characters
trtr โ โdd โโ?.?.โโ!:,();!:,();โโ < file< file
Convert all characters to lowercase
trtr^ โโAA--ZZโโ^ โโaa--zzโโ^ < file< file
Move each word to a separate line
replace space and tab with a new line trtr โโ \ttโโ โโ\nn\nnโโ < file< file
Remove blank lines (if any)
sedsed โโ/^$/d/^$/dโโ filefile search for lines starting with ^ and ending $ with to text in between, and then delete those lines
6/16/2005 23 Department of Com
puter and Informati
on Sciences
UAB
Implementation (II)Implementation (II)
Sort the lines
sort filesort file
Remove duplicates and compute word frequency
uniquniq โ โc filec file
Sort based on word frequency
sortsort โ โrn filern file (-n numerical sort, -r reverse sort order)
Output to file and another utility
| tee words.outtee words.out
Compute the total number of unique words
wcwc - -ll
6/16/2005 24 Department of Com
puter and Informati
on Sciences
UAB
SolutionSolution
tr -d '?."!:,;' < myfile.in \
| tr 'A-Z' 'a-z' \
| tr ' \t' '\n\n' \
| sed '/^$/d' \
| sort | uniq -c \
| sort -rn \
| tee words.out | wc -l
continuation character โ no new line
