Download CYBR 4423 - Chapter 2: Scripting and the Shell and more Exams Nursing in PDF only on Docsity!
CYBR 4423 - Chapter 2, Scripting and the
Shell
Scripts - standardize and automate the performance of administrative chores free up admins' time for more important and more interesting tasks low-rent documentation run the gamut from simple ones that encapsulate a few static commands to major software projects Administrative scripts should emphasize programmer efficiency and code clarity typically used for light tasks such as automating a sequence of commands or assembling several filters to process data relatively portable and have few dependencies For a long time, the standard language for administrative scripts was the one defined by the shell - Most systems' default shell is bash sh (the original Bourne shell) and ksh (the Korn shell) are used on a few UNIX systems More sophisticated scripts - Perl or Python main drawback to Perl and Python is that their environments can be a bit fussy to set up, 2.1 SHELL BASICS - 2.1 SHELL BASICS Command editing - emacs - the basic emacs commands are available to you when you're editing history. <Control-E> goes to the end of the line
<Control-A> to the beginning. <Control-P> steps backward through recently executed commands and recalls them for editing. <Control-R> searches incrementally through your history to find old commands. vi mode: $ set -o vi
- to leave input mode and "i" to reenter it.
- "w" takes you forward a word
- "fX" finds the next X in the line emacs editing mode $ set -o emacs Pipes and redirection - Every process has at least three communication channels available to it
- "standard input" (STDIN)
- "standard output" (STDOUT)
- "standard error" (STDERR). kernel sets up these channels on the process's behalf, UNIX has a unified I/O model in which each channel is named with a small integer called a file descriptor STDIN, STDOUT, and STDERR are guaranteed to correspond to file descriptors 0, 1, and 2, STDIN normally reads from the keyboard and both STDOUT and STDERR write their output to the screen Most commands accept their input from STDIN and write their output to STDOUT. They write error messages to STDERR shell interprets the symbols <, >, and >> as instructions to reroute a command's input or output to or from a file. < symbol connects the command's STDIN to the contents of an existing file
and >> symbols redirect STDOUT; replaces the file's existing contents, >> appends to them
Use export varname to promote a shell variable to an environment variable quoting - The shell treats strings enclosed in single and double quotes similarly, double-quoted strings are subject to globbing Back quotes, also known as back-ticks, are treated similarly to double quotes Common filter commands - Most filter commands accept one or more filenames on the command line cut command - separate lines into fields prints selected portions of its input lines most commonly used to extract delimited fields can return segments defined by column boundaries as well default delimiter is , change delimiters with the -d option -f options specifies which fields to include in the output. sort command - sort lines Opt Meaning -b Ignore leading whitespace -f Case insensitive sorting -k Specify the columns that form the sort key -n Compare fields as integer numbers -r Reverse sort order -t Set field separator (the default is whitespace) -u Output unique records only
uniq - print unique lines -c to count the number of instances of each line -d to show only duplicated lines, -u to show only nonduplicated lines. The input must be presorted, usually by being run through sort. wc - count lines, words, and characters Counting the number of lines, words, and characters in a file is another common operation word count Run without options, it displays all three counts: it is more common to supply a -l, -w, or -c option to make wc's output consist of a single number tee command - copy input to two places tap into the data stream and send a copy to a file or to the terminal window sends its standard input both to standard out and to a file that you specify on the command line A common idiom is to terminate a pipeline that will take a long time to run with a tee command head and tail - read the beginning or end of a file These commands display ten lines by default, but you can include a command-line option to specify how many lines you want to see head is more or less obsoleted by the less command tail -f waits for new lines to be added to the end of the file and prints them as they appear -the program writing the file may be buffering its output
break a single logical line onto multiple physical lines by escaping the newline with a backslash put more than one statement on a line by separating the statements with semicolons bash script - may consist of nothing but a series of command lines To prepare the file for running, just turn on its execute bit $ chmod +x helloworld $ ./helloworld Hello, world! "shebang" statement - #!/bin/bash declares the text file to be a script for interpretation by /bin/bash kernel looks for this syntax when deciding how to execute the file shebang line is just a comment give your bash scripts a .sh suffix to remind you what they are, but you'll then have to type out the .sh when you run the command From commands to scripts - Most people write bash scripts the same way they write Perl or Python scripts: with a text editor. simply add a pipe that sends the output to bash -x.
- -x option to bash prints each command before executing it. bash's built-in command fc transfers the command to your editor of choice Add a shebang line and usage comment, write the file to a plausible location (~/bin or /usr/local/bin, perhaps), make the file executable, and you have a script. To summarize this approach: -
- Develop the script (or script component) as a pipeline, one step at a time, entirely on the command line.
- Send output to standard output and check to be sure it looks right.
- At each step, use the shell's command history to recall pipelines and the shell's editing features to tweak them.
- Until the output looks right, you haven't actually done anything, so there's nothing to undo if the command is incorrect.
- Once the output is correct, execute the actual commands and verify that they worked as you intended.
- Use fc to capture your work, then clean it up and save it. Input and output - The echo command is crude but easy -n in the echo command suppresses the usual newline use printf use the read command to prompt for input. printf - you must explicitly put newlines where you want them (use "\n") gives you the option to use tabs and enhanced number formatting in your the output read command - reads input Command-line arguments and functions - Command-line arguments to a script become variables whose names are numbers $1 is the first command-line argument, $2 is the second, and so on $0 is the name by which the script was invoked $# contains the number of command-line arguments that were supplied, and the variable $* contains all the arguments at once Arguments to bash functions are treated much like command-line arguments You can define useful functions in your ~/.bash_profile file and then use them on the command line as if they were commands. Variable scope - Variables are global within a script
any whitespace-separated list of things, including the contents of a variable, works as a target of for...in. bash also has the more familiar for loop bash's while loop, which is useful for processing command-line arguments and for reading the lines of a file $((...)) notation forces numeric evaluation Arrays and arithmetic - All bash variables are string valued, so bash does not distinguish between the number 1 and the character string "1" in assignments the plus sign in the assignment to $c does not even act as a concatenation operator for strings To force numeric evaluation, you enclose an expression in $((...)), bash has the usual assortment of arithmetic, logical, and relational operators Arrays in bash are a bit strange, and they're not often used Literal arrays are delimited by parentheses, and the elements are separated by whitespace Use ${array_name[subscript]} to access individual elements Subscripting begins at zero. subscripts * and @ refer to the array as a whole the special forms ${#array_name[*]} and ${#array_name[@]} yield the number of elements in the array. Always include the curly braces when referring to array variables—no exceptions some of the features and pitfalls of array management in bash - all bash variables are still essentially strings REGULAR EXPRESSIONS - supported by most modern languages used by UNIX commands such as grep and vi.
the name is usually shortened to "regex." The filename matching and expansion performed by the shell when it interprets command lines is not a form of regex matching cannot recognize nested delimiters. Regular expressions reached the apex of their power and perfection in Perl are not themselves a scripting language so useful that they merit featured coverage in any discussion of scripting The matching process - Code that evaluates a regular expression attempts to match a single given text string to a single given pattern often convenient to use a regex to match the contents of an entire file or HTML document the entire search pattern must match a contiguous section of the search text. the pattern can match at any position the evaluator returns the text of the match along with a list of matches for any specially delimited subsections of the pattern Literal characters - characters in a regular expression match themselves Matching is case sensitive. Special characters in regular expressions (common ones) - Symbol What it matches or does
. Matches any character [chars] Matches any character from a given set [^chars] Matches any character not in a given set ^ Matches the beginning of a line $ Matches the end of a line \w Matches any "word" character (same as [A-Za-z0-9_]) \s Matches any whitespace character (same as [ \f\t\n\r])a
digit matches. Captures - When a match succeeds, every set of parentheses becomes a "capture group" that records the actual text that it matched Since parentheses can nest, how do you know which match is which? Easy—the matches arrive in the same order as the opening parentheses There are as many captures as there are opening parentheses When a parenthesized group is not used its corresponding capture is empty Greediness, laziness, and catastrophic backtracking - Regular expressions match from left to right. Each component of the pattern matches the longest possible string before yielding to the next component, a characteristic known as greediness If the regex evaluator reaches a state from which a match cannot be completed, it unwinds a bit of the candidate match and makes one of the greedy atoms give up some of its text. and there is no more input text that can match an a; time to backtrack. greedy matching plus backtracking makes it expensive to match apparently simple patterns the > that this pattern binds to is the last possible valid match in the input, which is probably not what you want. You can also use lazy (as opposed to greedy) wild card operators: *? instead of *, and +? instead of +.
- match as few characters of the input as they can.
- they can produce different matches than the greedy operators Patterns with multiple wild-card sections can cause exponential behavior in the regex evaluator,
- "catastrophic backtracking" take-home points - If you can do pattern matching line-by-line rather than file-at-a-time, there is much less risk of poor performance
Even though regex notation makes greedy operators the default, they probably shouldn't be. Use lazy operators All instances of .* are inherently suspicious and should be scrutinized PERL PROGRAMMING - created by Larry Wall the first of the truly great scripting languages offers vastly more power than bash, does not impose much stylistic discipline accused of being a write-only language better choice for system administration work than traditional programming languages do more, in fewer lines of code, with less painful debugging, and without the hassle of compilation offer libraries of community- written modules and language extensions. catch phrase is that "there's more than one way to do it." Perl syntax - Perl statements are separated by semicolons Comments start with a hash mark (#) and continue to the end of the line Blocks of statements are enclosed in curly braces. you must either chmod +x the executable file or invoke the Perl interpreter directly. Lines in a Perl script are not shell commands provides many of the same conventions as bash, such as the use of back-ticks Variables and arrays - Perl has three fundamental data types: scalars (that is, unitary values such as numbers and strings), arrays, and hashes Hashes are also known as associative arrays.
Hashes - A hash (also known as an associative array) represents a set of key/value pairs. an array whose subscripts (keys) are arbitrary scalar values; they do not have to be numbers numbers and strings are the usual keys. Hash variables have % as their first character individual values are scalar and so begin with a $ Subscripting is indicated with curly braces rather than square brackets an important tool for system administrators. The while ($_ = <>) reads input one line at a time and assigns it to the variable named $_ To interpret <>, Perl checks the command line to see if you named any files there
- if yes, it opens each file in sequence and runs the file's contents through the loop.
- if no, Perl takes the input to the loop from standard input. You can process lines without ever making an explicit reference to the variable in which they're stored split - a function that chops up its input string by using the regular expression passed to it as the field separator. The string that split is to divide at colons is never explicitly specified When split's second argument is missing, Perl assumes you want to split the value of $_. reverse - The reverse function reverses the order of the list, yielding (valueN, keyN, ..., value1, key1). References and autovivification - Arrays and hashes can only hold scalar values You can't store arrays and hashes, but you can store references to arrays and hashes, which are themselves scalars
To create a reference to an array or hash, you precede the variable name with a backslash (e.g., @array) or use reference-to-array or reference-to-hash literal syntax The square brackets return a reference to an array containing the results of the split This lack of warnings may seem like a problem, but it's arguably one of Perl's nicest features, a feature known as "autovivification." Simply make an assignment at the lowest possible level, and the intervening structures materialize automatically Regular expressions in Perl - You use regular expressions in Perl by "binding" strings to regex operations with the =~ operator To operate on the default string, $_, you can simply omit the variable name and binding operator qq is another name for the double-quote operator Input and output - When you open a file for reading or writing, you define a "filehandle" to identify the channel It reads lines from the filehandle INFILE until the end of file, at which time the while loop ends open - returns a true value if the file is successfully opened, short-circuiting (rendering unnecessary) the evaluation of the die clauses or operator - similar to || (which Perl also has), but at lower precedence generally a better choice when you want to emphasize that everything on the left will be fully evaluated before Perl turns its attention to the consequences of failure Control flow - Perl's if construct has no then keyword or terminating word, just a block of statements enclosed in curly braces You can also add a postfix if clause (or its negated version, unless) to an individual
data types, Perl as a filter - You can use Perl without a script by putting isolated expressions on the command line Use the -pe command-line option to loop through STDIN, run a simple expression on each line, and print the result. If you use paired delimiters, you must use four of them instead of the normal three, e.g., s(foo)(bar). Perl's -a option turns on autosplit mode, which separates input lines into fields that are stored in the array named @F. Autosplit is handy to use in conjunction with -p or its nonautoprinting variant, -n. use -i in conjunction with -pe to edit files in place; Perl reads the files in, presents their lines for editing, and saves the results out to the original files. Add-on modules for Perl - CPAN, the Comprehensive Perl Archive Network at cpan.org, is the warehouse for user-contributed Perl libraries Installation of new modules is greatly facilitated by the cpan command, On systems that don't have a cpan command, try running perl -MCPAN -e shell as an alternate route to the same feature It's possible for users to install Perl modules in their home directories for personal use Many Perl modules use components written in C for better performance the most common error found in Perl programs is the reimplementation of features that are already provided by community-written modules 2.5 PYTHON SCRIPTING - well onto the downhill side of the adoption curve at this point Ruby, by contrast, is still primarily associated with web development created by Guido van Rossum
easier to code and more readable than Perl. simple-to-understand syntax that is easy to follow even if you didn't develop the code additional data types that some system administrators find useful you can get Python source code from python.org Mark Pilgrim's Dive Into Python is a great place to start Python quick start - "Hello, world!" is almost identical to Perl's. #!/usr/bin/python print "Hello, world!" To get it running, set the execute bit or invoke the python interpreter directly Python does not use braces, brackets, or begin and end to delineate blocks Statements at the same level of indentation automatically form blocks. You can suppress the newline by including an extra comma at the end of the print line; less flexibility in the formatting of code, Comments are introduced with a hash mark (#) split long lines by backslashing the end of line breaks never to mix tabs and spaces; use one or the other for indentation Objects, strings, numbers, lists, dictionaries, tuples, and files - All data types in Python are objects lists are enclosed in square brackets instead of parentheses Arrays index from zero, "tuples," which are essentially immutable lists
