Introduction to UNIX, Shell basics, and Shell programming

Introduction to UNIX, shell basics, and shell programming

Tom Ferrin

April 1, 2013

Portions Copyright © 2005-06 Python Software Foundation.

Part I: Shell Basics

  • Most modern tools have a graphical user interface (GUI)
    • They're easier to use
    • A picture is worth a thousand words
  • But command-line user interfaces (CLUIs) still have their place
    • Easier to build a simple CLUI than a simple GUI
    • Higher action-to-keystroke ratio
      • Once you're over the learning curve
    • Easier to see and understand what the computer is doing on your behalf
      • Which is part of what this course is about
    • Most important: it's easier to combine CLUI tools than GUI tools
      • Small tools, combined in many ways, can be very powerful
  • This lecture focuses on Linux (aka Unix)

The Shell

  • The most important command-line tool is the command shell
    • Usually just called "the shell"
    • Looks (and works) like an interactive terminal circa 1980
    • [A Shell in Action]

    Figure 3.1: A Shell in Action

  • Manages a user's interactions with the operating system by:
    • Reading commands from the keyboard
    • Executing those commands...
    • ...or running another program
    • Displaying the output
  • The shell is just one program among many
    • Many different shells have been written
    • The Bourne shell, called sh, is an ancestor of many of them
      • It's still a lowest common denominator that you can always rely on
    • We'll use bash (the Bourne Again Shell) in this course

The Shell is Not the Operating System

  • The operating system is not just another program
    • Automatically loaded when the computer boots up
    • Runs everything else (including shells)
    • [Operating System]

    Figure 3.2: Operating System

  • The OS manages the computer's hardware
    • Provides a common interface to different chips, disks, network cards, etc.
    • So that user-level applications can run anywhere
  • The OS also keeps track of what programs are running, what privileges they have, etc.
    • Which makes it crucial to security

The File System

  • The file system is the set of files and directories the computer can access
    • "Everything that doesn't go away when you reboot"
    • [A Directory Tree]

    Figure 3.3: A Directory Tree

  • Data is stored in files
    • By convention, files have two part names, like notes.txt or home.html
    • Most operating systems allow you to associate a filename extension with an application
      • E.g., .txt is associated with an editor, and .html with a web browser
    • But this is all just convention: you can call files (almost) anything you want
  • Files are stored in directories (often called folders)
    • Directories can contain other directories, too
    • Results in the familiar directory tree
    • Everything in a particular directory must have a unique name
      • Otherwise, how would you identify it?
    • But items in different directories can have the same name
  • On Unix, the file system has a unique root directory called /
    • Every other directory is a child of it, or a child of a child, etc.

Paths

  • A path is a description of how to find something in a file system
    • An absolute path describes a location from the root directory down
      • Equivalent to a street address
      • Always starts with "/"
      • /home/hpotter is Harry Potter's home directory
      • /courses/swc/wlec/shell.html is this file
    • A relative path describes how to find something from some other location
      • Equivalent to saying, "Four blocks north, and seven east"
      • From /courses/swc, the relative path to this file is wlec/shell.html
  • Every program (including the shell) has a current working directory
    • "Where am I?"
    • Relative paths are deciphered relative to this location
    • It can change while a program is running

Paths - continued

[Parent Directories]

Figure 3.4: Parent Directories

  • Two special directory names:
    • "." (pronounced "dot") is the current directory
    • ".." (pronounced "dot dot") is the directory one level up
      • Also called the parent directory
      • In /courses/swc/data, .. is /courses/swc
      • In /courses/swc/data/elements, .. is /courses/swc/data

    • Navigating the File System

    • Easiest way to learn basic Unix commands is to see them in action
      • Type pwd (short for "print working directory") to find out where you are
        • Unfortunately, most Unix commands have equally cryptic names
          • $ pwd/home/hpotter/swc
      • Then type ls (for "listing") to see what's in the current directory 
          $ lsLICENSE.txt     conf        data        docs    index.swc   license.swc     print.css   swc.css     testsMakefile        config.mk   depend.mk   img     lec         press           sites       swc.dtd     util
      • To see what's in the data directory, type ls data 
          $ ls databio     elements    haiku.txt   morse.txt   pdb         solarsystem.txt
      • Or:
        • Type cd data to "go into" data
          • I.e., change the current working directory to data
        • Type ls on its own
        • Type cd .. to go back to where you started
          • $ cd data$ pwd/home/hpotter/swc/data$ lsbio     elements    haiku.txt   morse.txt   pdb         solarsystem.txt$ cd ..$ pwd/home/hpotter/swc

    Execution Cycle

    • When you type a command like ls, the OS:
      • Reads characters from the keyboard
        • [Running a Program]

        Figure 3.5: Running a Program

      • Passes them to the shell (because it's the currently active window)
    • The shell:
      • Breaks the line of text it receives into words
      • Looks for a program with the same name as the first word
        • See in a moment how the shell knows where to look
      • Runs that program
    • That program's output goes back to the shell...
    • ...which gives it to the OS...
    • ...which displays it on the screen
      • (Actually, the OS hands it to the window manager, which takes care of the display)
    • All well-designed software systems work this way
      • Break the task down into pieces
      • Write a tool that solves each sub-problem
      • Hook 'em up
    • Allows you to:
      • Develop and test components independently
      • Replace or re-use components incrementally
      • Add new components as you go along

    Providing Options

    • Can make ls produce more informative output by giving it some flags
      • By convention, flags for Unix tools start with "-", as in "-c" or "-l"
      • Some flags take arguments (such as filenames)
    • Show directories with trailing slash
      • $ ls -FLICENSE.txt     conf/       data/       docs/   index.swc   print.css   swc.css     tests/
Makefile        config.mk   depend.mk   img/    lec/        sites/      swc.dtd     util/
    • Show all files and directories, including those whose names begin with .
      • By default, ls doesn't show things whose names begin with .
        • So that . and .. don't always show up
      • $ ls -a.   .svn        Makefile    config.mk   depend.mk   img         lec         sites   swc.dtd util
..  LICENSE.txt conf        data        docs        index.swc   print.css   swc.css tests
    • We'll see what the .svn directory is for later
    • Flags can be combined
      • $ ls -a -F.   .svn/       Makefile    config.mk   depend.mk   img/        lec/        sites/  swc.dtd util/
..  LICENSE.txt conf/       data/       docs/       index.swc   print.css   swc.css tests/

    Creating Files and Directories

    • Rather than messing with the course files, let's create a temporary directory and play around in there 
        $ mkdir tmp
      • Note: no output (but -v ("verbose") would tell mkdir to print a confirmation message)
    • Go into that directory: no files there yet 
        $ cd tmp$ ls
    • Use the editor of your choice to create a file called earth.txt with the following contents: 
        Name: Earth
Period: 365.26 days
Inclination: 0.00
Eccentricity: 0.02
      • TextEdit (on OS X) also runs in a window of its own
      • Pico or Vim (on Unix) takes over the shell window temporarily
      • We'll look at more advanced programming editors later
    • Easiest way to create a similar file venus.txt is to copy earth.txt and edit it 
        $ cp earth.txt venus.txt$ edit venus.txt$ ls -tvenus.txt   earth.txt
      • -t tells ls to list by modification time, instead of alphabetically

    Looking at Files

    • Check the contents of the file using cat (short for "concatenate")
      • Prints the contents of a file to the screen
        • $ cat venus.txt
Name: Venus
Period: 224.70 days
Inclination: 3.39
Eccentricity: 0.01
    • Compare the sizes of the two files using:
      • ls -l ("-l" meaning "long form")
          $ ls -l
total 2
-rwxr-xr-x  1 gvwilson bmi280 73 Jan  4 15:58 earth.txt
-rwxr-xr-x  1 gvwilson bmi280 73 Jan  4 15:58 venus.txt
        • Fifth column is size in bytes
      • wc (for "word count")
          $ wc earth.txt venus.txt
  4   9  73 earth.txt
  4   9  73 venus.txt
  8  18 146 total
        • Columns show lines, words, and characters

    Basic Tools



      manDocumentation for commands. mkdirMake directories.
      catConcatenate and display text files. morePage through a text file.
      cdChange working directory. mvMove (rename) files and directories.
      clearClear the screen. odDisplay the bytes in a file.
      cpCopy files and directories. pwdPrint current working directory.
      dateDisplay the current date and time. rmRemove files.
      diffShow differences between two text files. rmdirRemove directories.
      echoPrint arguments. sortSort lines.
      grepPrint lines matching a pattern. tailDisplay the last few lines of a file.
      headDisplay the first few lines of a file. uniqRemove adjacent duplicate lines.
      lsList files and directories. wcCount lines, words, and characters in a file.
      Table 3.1: Basic Command-Line Tools

    Part I: Summary

    • Command-line tools will be with us for a long time
      • Easiest way to do many simple tasks
      • Easiest way to see what the computer is actually doing
    • Often the only thing you can rely on having on a new machine
      • A handful of basic commands will get you a long way

    Part I: Exercises

    Exercise 3.1:

    • Suppose ls shows you this: 
      Makefile    biography.txt   data    enrolment.txt   programs    thesis
    • What argument(s) will make it print the names in reverse, like this: 
      thesis  programs    enrolment.txt   data    biography.txt   Makefile

    Exercise 3.2:

    • What does the command cd ~ do? What about cd ~hpotter?

    Exercise 3.3:

    • What command will show you the first 10 lines of a file? The first 25? The last 12?

    Exercise 3.4:

    • What do the commands pushd, popd, and dirs do? Where do their names come from?

    Exercise 3.5:

    • How would you send the file earth.txt to the default printer? How would you check it made it (other than wandering over to the printer and standing there)?

    Exercise 3.6:

    • The instructor wants you to use a hitherto unknown command for manipulating files. How would you get help on this command?

    Part I: Exercises (continued)

    Exercise 3.7:

    • diff finds and displays the differences between two text files. For example, if you modify earth.txt to create a new file earth2.txt that contains: 
      Name: Earth
Period: 365.26 days
Inclination: 0.00 degrees
Eccentricity: 0.02
Satellites: 1

      you can then compare the two files like this:

      $ diff earth.txt earth2.txt
3c3
< Inclination: 0.00
---
> Inclination: 0.00 degrees
4a5
> Satellites: 1

      (The rather cryptic header "3c3" means that line 3 of the first file must be changed to get line 3 of the second; "4a5" means that a line is being added after line 4 of the original file.)

    • What flag(s) should you give diff to tell it to ignore changes that just insert or delete blank lines? What if you want to ignore changes in case (i.e., treat lowercase and uppercase letters as the same)?

    Part II: Shell Programming
    • The shell is more than just a clumsy way to move around a file system
    • It's a component-based programming environment
      • Small tools that each do one job...
      • ...can be connected together to create ad hoc solutions to larger problems
    • A good model, even when you're building large GUI or web applications

    Wildcards

    • Some characters (called wildcards) mean special things to the shell
      • * matches zero or more characters
        • So ls bio/*.txt lists all the text files in the bio directory
        • $ ls bio/*.txt
bio/albus.txt   bio/ginny.txt   bio/harry.txt   bio/hermione.txt    bio/ron.txt
      • ? matches any single character
        • So ls jan-??.txt lists text files whose names start with “jan-” followed by two characters
        • You can probably guess what ls jan-??.* does
    • Note:
      • The shell expands wildcards, not individual applications
        • ls can't tell whether it was invoked as ls *.txt or as ls earth.txt venus.txt
      • Wildcards only work with filenames, not with command names
        • ta* does not find the tabulate command

    Redirecting Input and Output

    • A running program is called a process
      • [Redirecting Standard Input and Output]

      Figure 4.1: Redirecting Standard Input and Output

    • Every process automatically has three connections to the outside world:
    • You can tell the shell to connect standard input and standard output to files instead
      • command < input_file reads from input_file instead of from the keyboard
        • Don't need to use this very often, because most Unix commands let you specify the input file (or files) as command-line arguments
      • command > output_file writes to output_file instead of to the screen
        • Only "normal" output goes to the file, not error messages
      • command < input_file > output_file does both
      • Note that redirection takes effect command-by-command, rather than permanently

    Redirection Examples

    • Save number of words in all text files to words.len: 
        $ cd bio
$ wc *.txt > words.len
      • Nothing appears on the screen because output is being sent to the file words.len
      • Check contents using cat
        • $ cat words.len
   7   66  468 albus.txt
   5   46  311 ginny.txt
   5   49  342 harry.txt
   5   49  331 hermione.txt
   6   54  364 ron.txt
  28  264 1816 total
    • Try typing cat > junk.txt
      • No input file specified, so cat reads from the keyboard
      • Output sent to a file
      • The world's dumbest text editor
    • When you're done, use rm junk.txt to get rid of the file
      • Don't type rm * unless you're really, really, REALLY sure that's what you want to do...
    • Don't redirect out to the same file, e.g. sort words >words
      • The shell sets up redirection before running the command
      • Redirecting out to an existing file truncates it make it empty
      • sort then goes and reads the empty file
      • Contents of words are lost

    Pipes

    • Suppose you want to use the output of one program as the input of another
      • E.g., use wc -w *.txt to count the words in some files, then sort -n to sort numerically
    • The obvious solution is to send output of first command to a temporary file, then read from that file
        $ wc -w *.txt > words.tmp
$ sort -n words.tmp
  46 ginny.txt
  49 harry.txt
  49 hermione.txt
  54 ron.txt
  66 albus.txt
 264 total
$ rm words.tmp

    Pipes (continued)

    • The right answer is to use a pipe
      • Written as "|"
        • [Pipes]

        Figure 4.2: Pipes

      • Tells the operating system to connect the standard output of the first program to the standard input of the second
        • $ wc -w *.txt | sort -n
  46 ginny.txt
  49 harry.txt
  49 hermione.txt
  54 ron.txt
  66 albus.txt
 264 total
      • More convenient and less error prone than temporary files
    • Can chain any number of commands together
      • And combine with input and output redirection
        • $ grep 'Title' spells.txt | sort | uniq -c | sort -n -r | head -10 > popular_spells.txt
    • Any program that reads from standard input and writes to standard output can use redirection and pipes
      • Such programs are often called filters
      • If your programs work like filters, you (and other people) can combine them with standard tools
      • A combinatorial explosion of goodness

    Environment Variables

    • The OS stores some environment variables for every process
      • Like variables in a program, each has a name and a value
        • By convention, names are all upper case
        • Values are always strings
    • Type set at the command prompt to get a listing:
        $ set
BASH=/usr/bin/bash
BASH_VERSION='2.05b.0(1)-release'
COLUMNS=120
HISTFILE=/home/.bash_history
HISTFILESIZE=500
HISTSIZE=500
HOME=/home/rweasley
HOSTNAME=hogwarts
HOSTTYPE=x86_64
LINES=60
NUMBER_OF_PROCESSORS=1
OSTYPE=linux-gnu
PATH='/usr/local/bin:/usr/bin:/bin:/Python24:/home/rweasley/bin'
PWD=/home/rweasley
SHELL=/bin/bash
UID=1003
USER=rweasley

    Environment Variables (continued)

  • Get a variable's value by putting "$" in front of its name
    • So ls $HOME is the same as ls /home/rweasley (if you're Ron Weasley)
    • Use the echo command to print out a variable's value
        $ echo $HOME
/home/socr/a/tef
      • Question: why must you type echo $HOME, and not just $HOME?
    • If a variable hasn't been defined, its value is the empty string (rather than an error)

    • Important Environment Variables



    NameTypical ValueNotes
    COLUMNS80The width in characters of the current display window
    EDITOR/bin/editPreferred editor
    HOME/home/rweasleyThe current user's home directory
    HOSTNAME"ishad"This computer's name
    HOSTTYPE"i686"What kind of computer this is
    LINES60The height in characters of the current display
    OS"Windows_NT"What operating system is running
    PATH"/home/rweasley/bin:/usr/local/bin:/usr/bin:/bin:/Python24/"Where to look for programs
    PWD/home/rweasley/swc/lecPresent working directory (sometimes CWD, for current working directory)
    SHELL/bin/bashWhat shell is being run
    TEMP/tmpWhere to store temporary files
    USER"rweasley"The current user's ID
    Table 4.1: Important Environment Variables

    Setting Environment Variables

    • Different shells have different syntaxes for setting environment variables
      • [Setting a Variable Without Export It]

      Figure 4.3: Setting a Variable Without Export It

    • For Bash, use this: 
        $ VILLAIN="Lord Voldemort"
      • Notice that values with spaces in them have to be quoted
    • Setting an environment variable only affects that program (i.e., that shell)
      • To see this, set a variable, then run a new shell, and echo the variable's value
        • $ VILLAIN="Lord Voldemort"
$ bash
$ echo $VILLAIN

$ exit
      • [Exporting a Variable's Value]

      Figure 4.4: Exporting a Variable's Value

    • If you want programs run from that shell to inherit the value, you must export it:
        $ VILLAIN="Lord Voldemort"
$ export VILLAIN
$ bash
$ echo $VILLAIN
Lord Voldemort
$ exit
      • Can perform both operations in a single step
        • $ export VILLAIN="Lord Voldemort"
$ bash
$ echo $VILLAIN
Lord Voldemort
$ exit

    Configuration

    • To set a variable's value automatically when you log in, edit ~/.bashrc
      • "~" is a shortcut meaning "your home directory"
        • # Add personal tools directory to PATH.
PATH=$HOME/bin:$PATH

# Personal settings.
export EDITOR=/local/bin/emacs
export PRINTER=gryffindor-laserwriter

# Change default behavior of commands.
alias ls="ls -F"
      • Note: .bashrc files can become very complex...
    • Many applications look for personal configuration files in the user's home directory
      • By convention, their names begin with "." so that a normal ls won't show them
      • Once upon a time, the "rc" at the end meant "run commands"

    How the Shell Finds Programs

    • The PATH environment variables defines the shell's search path
    • When you run a command like broom, the shell:
      • Splits $PATH into components to get a list of directories
        • Unix uses ":" as a separator
        • Looks for the program in each directory in left-to-right order
        • Runs the first one that it finds
      • Example
        • PATH is /home/rweasley/bin:/usr/local/bin:/usr/bin:/bin:/Python24
        • Both /usr/local/bin/broom and /home/rweasley/bin/broom exist
        • /home/rweasley/bin/broom will be run when you type broom at the command prompt
        • Can run the other one by specifying the path, instead of just the command name

    Common Search Path Entries

    • /bin, /usr/bin: core tools like ls
      • Note: the word "bin" comes from "binary", which is geekspeak for "a compiled program"
    • /usr/local/bin: optional (but common) tools, like the gcc C compiler
    • $HOME/bin: tools you have built for yourself
      • Remember, $HOME is your home directory
    • It is also common to include . (the current working directory) in your path
      • Allows you to run a program in the current directory using whatever, instead of ./whatever

    File Ownership and Permissions

    • On Unix, every user belongs to one or more groups
      • The groups command will show you which ones you are in
    • Every file is owned by a particular user and a particular group
      • Can assign read (r), write (w), and execute (x) permissions independently to user, group, and others
      • Read: can look at contents, but not modify them
      • Write: can modify contents
      • Execute: can run the file (e.g., it's a program)
    • ls -l shows this information
      • Along with the file's size and a few other things
    • Permissions displayed as three rwx triples
      • "Missing" permissions shown by "-"
      • So rw-rw-r-- means:
        • User and group can read and write
        • Everyone else can read, but not write
        • No one can execute

    Directory Permissions

    • Execute permission means something different for directories
      • Allows you to "go into" a directory, but does not mean you can read its contents
    • If tools has permission rwx--x--x, then:
      • If someone other than the owner does ls tools, permission is denied
      • But anyone who wants to can run tools/pfold

    Changing Permissions

    • Change permissions using chmod
      • chmod u+x broom allows broom's owner to run it
      • chmod o-r notes.txt takes away the world's read permission for notes.txt
    • Any set of shell commands can be turned into a program!
      • If it's worth doing again, it's worth automating
    • Create a file called nojunk
        #!/usr/bin/bash
rm -f *.junk
      • Use man rm to find out what the "-f" flag does
      • #!/usr/bin/bash means "run this using the Bash shell"
        • Any program name can follow the #!
        • We'll see some possibilities later
    • Change permissions to rwxr-xr-x
    • Run it with ./nojunk
      • Or if $HOME/bin is in your search path, move it there
    • Don't call your temporary test programs test
      • There's already /usr/bin/test
      • Your PATH may cause that program to run instead of yours
      • Confusion results, so use something else, e.g. ./try

    More Advanced Tools



    chmodChange file and directory permissions.
    duPrint the disk space used by files and directories.
    findFind files with names that match patterns, that are of a certain age or size, etc.
    grepPrint lines matching a pattern.
    gunzipUncompress a file.
    gzipCompress a file.
    lprSend a file to a printer.
    lprmRemove a print job from a printer's queue.
    lpqCheck the status of a printer's queue.
    psDisplay running processes.
    tarArchive files.
    whichFind the path to a program.
    whoSee who is logged in.
    xargsExecute a command for each line of input.
    Table 4.2: Advanced Command-Line Tools

    Part II: Summary

    Part II: Exercises

    Exercise 4.1:

      -rwxr-xr-x   1 aturing   cambridge  69 Jul 12 09:17 mars.txt
-rwxr-xr-x   1 ghopper   usnavy     71 Jul 12 09:15 venus.txt
    • According to the listing of the data directory above, who can read the file earth.txt? Who can write it (i.e., change its contents or delete it)? When was earth.txt last changed? What command would you run to allow everyone to edit or delete the file?

    Exercise 4.2:

    • Suppose you want to remove all files whose names (not including their extensions) are of length 3, start with the letter a, and have .txt as extension. What command would you use? For example, if the directory contains three files a.txt, abc.txt, and abcd.txt, the command should remove abc.txt , but not the other two files.

    Exercise 4.3:

    • You're worried your data files can be read by your nemesis, Dr. Evil. How would you check whether or not he can, and if necessary change permissions so only you can read or write the files?

    Exercise 4.4:

    • What's the difference between the commands cd HOME and cd $HOME?

    Exercise 4.5:

    • Suppose you want to list the names of all the text files in the data directory that contain the word "carpentry". What command or commands could you use?

    Part II: Exercises (continued)

    Exercise 4.6:

    • Suppose you have written a program called analyze. What command or commands could you use to display the first ten lines of its output? What would you use to display lines 50-100? To send lines 50-100 to a file called tmp.txt?

    Exercise 4.7:

    • The command ls data > tmp.txt writes a listing of the data directory's contents into tmp.txt. Anything that was in the file before the command was run is overwritten. What command could you use to append the listing to tmp.txt instead?

    Exercise 4.8:

    • What command(s) would you use to find out how many subdirectories there are in the lectures directory?

    Exercise 4.9:

    • What does rm *.ch? What about rm *.[ch]?

    Exercise 4.10:

    • What command(s) could you use to find out how many instances of a program are running on your computer at once? For example, what would you do to find out how many instances of bash are running?

    Exercise 4.11:

    • A colleague asks for your data files. How would you archive them to send as one file? How could you compress them?

    Part II: Exercises (continued)

    Exercise 4.12:

    • You have changed a text file on your home PC, and mailed it to the university terminal. What steps can you take to see what changes you may have made, compared with a master copy in your home directory?

    Exercise 4.13:

    • How would you change your password?

    Exercise 4.14:

    • grep is one of the more useful tools in the toolbox. It finds lines in files that match a pattern and prints them out. For example, assume the files earth.txt and venus.txt contain lines like this:
      • Name: Earth
Period: 365.26 days
Inclination: 0.00
Eccentricity: 0.02
    • grep can extract lines containing the text "Period" from all the files:
      • $ grep Period *.txt
earth.txt:Period: 365.26 days
venus.txt:Period: 224.70 days
    • Search strings can use regular expressions, which will be discussed in a later lecture. grep takes many options as well; for example, grep -c /bin/bash /etc/passwd reports how many lines in /etc/passwd (the Unix password file) that contain the string /bin/bash, which in turn tells me how many users are using bash as their shell.
    • Suppose all you wanted was a list of the files that contained lines matching a pattern, rather than the matches themselves-what flag or flags would you give to grep? What if you wanted the line numbers of matching lines?

    Part II: Exercises (continued)

    Exercise 4.15:

    • Suppose you wanted ls to sort its output by filename extension, i.e., to list all .cmd files before all .exe files, and all .exe's before all .txt files. What command or commands would you use?

    Exercise 4.16:

    • What does the alias command do? When would you use it?
    Copyright ©2005-06 Python Software Foundation.