Chapter 1. GNU/Linux tutorials

Here are a few basic methods to gain the root shell prompt by using the root password.

You type Ctrl-D, i.e., the left-Ctrl-key and the d-key pressed together, at the command prompt to close the shell activity. If you are at the character console, you return to the login prompt with this. Even though these control characters are referred as "control D" with the upper case, you do not need to press the Shift-key. The short hand expression, ^D, is also used for Ctrl-D. Alternately, you can type "exit".

If you are at x-terminal-emulator(1), you can close x-terminal-emulator window with this.

When the screen goes berserk after doing some funny things such as "cat <some-binary-file>", type "reset" at the command prompt. You may not be able to see the command echoed as you type. You may also issue "clear" to clean up the screen.

Although even the minimal installation of the Debian system without any desktop environment tasks provides the basic Unix functionality, it is a good idea to install few additional commandline and curses based character terminal packages such as mc and vim with apt-get(8) for beginners to get started by the following.

# apt-get update
 ...
# apt-get install mc vim sudo
 ...

If you already had these packages installed, no new packages are installed.

It may be a good idea to read some informative documentations.

You can install some of these packages by the following.

# apt-get install package_name

If you do not want to use your main user account for the following training activities, you can create a training user account, e.g. fish by the following.

# adduser fish

Answer all questions.

This creates a new account named as fish. After your practice, you can remove this user account and its home directory by the following.

# deluser --remove-home fish

For the typical single user workstation such as the desktop Debian system on the laptop PC, it is common to deploy simple configuration of sudo(8) as follows to let the non-privileged user, e.g. penguin, to gain administrative privilege just with his user password but without the root password.

# echo "penguin  ALL=(ALL) ALL" >> /etc/sudoers

Alternatively, it is also common to do as follows to let the non-privileged user, e.g. penguin, to gain administrative privilege without any password.

# echo "penguin  ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

This trick should only be used for the single user workstation which you administer and where you are the only user.

	Warning
	Do not set up accounts of regular users on multiuser workstation like this because it would be very bad for system security.

	Caution
	The password and the account of the penguin in the above example requires as much protection as the root password and the root account.

	Caution
	Administrative privilege in this context belongs to someone authorized to perform the system administration task on the workstation. Never give some manager in the Admin department of your company or your boss such privilege unless they are authorized and capable.

	Note
	For providing access privilege to limited devices and limited files, you should consider to use group to provide limited access instead of using the root privilege via sudo(8).

	Note
	With more thoughtful and careful configuration, sudo(8) can grant limited administrative privileges to other users on a shared system without sharing the root password. This can help with accountability with hosts with multiple administrators so you can tell who did what. On the other hand, you might not want anyone else to have such privileges.

Now you are ready to play with the Debian system without risks as long as you use the non-privileged user account.

Here are some Unix file basics.

chown(1) is used from the root account to change the owner of the file. chgrp(1) is used from the file's owner or root account to change the group of the file. chmod(1) is used from the file's owner or root account to change file and directory access permissions. Basic syntax to manipulate a foo file is the following.

# chown <newowner> foo
# chgrp <newgroup> foo
# chmod  [ugoa][+-=][rwxXst][,...] foo

For example, you can make a directory tree to be owned by a user foo and shared by a group bar by the following.

# cd /some/location/
# chown -R foo:bar .
# chmod -R ug+rwX,o=rX .

There are three more special permission bits.

Here the output of "ls -l" for these bits is capitalized if execution bits hidden by these outputs are unset.

Setting set user ID on an executable file allows a user to execute the executable file with the owner ID of the file (for example root). Similarly, setting set group ID on an executable file allows a user to execute the executable file with the group ID of the file (for example root). Because these settings can cause security risks, enabling them requires extra caution.

This sounds complicated but it is actually quite simple. If you look at the first few (2-10) columns from "ls -l" command output and read it as a binary (radix=2) representation of file permissions ("-" being "0" and "rwx" being "1"), the last 3 digit of the numeric mode value should make sense as an octal (radix=8) representation of file permissions to you.

For example, try the following

$ touch foo bar
$ chmod u=rw,go=r foo
$ chmod 644 bar
$ ls -l foo bar
-rw-r--r-- 1 penguin penguin 0 Oct 16 21:39 bar
-rw-r--r-- 1 penguin penguin 0 Oct 16 21:35 foo

	Tip
	If you need to access information displayed by "ls -l" in shell script, you should use pertinent commands such as test(1), stat(1) and readlink(1). The shell builtin such as "[" or "test" may be used too.

What permissions are applied to a newly created file or directory is restricted by the umask shell builtin command. See dash(1), bash(1), and builtins(7).

 (file permissions) = (requested file permissions) & ~(umask value)

The Debian system uses a user private group (UPG) scheme as its default. A UPG is created whenever a new user is added to the system. A UPG has the same name as the user for which it was created and that user is the only member of the UPG. UPG scheme makes it safe to set umask to 0002 since every user has their own private group. (In some Unix variants, it is quite common to setup all normal users belonging to a single users group and is a good idea to set umask to 0022 for security in such cases.)

	Tip
	Enable UPG by putting "umask 002" in the ~/.bashrc file.

In order to make group permissions to be applied to a particular user, that user needs to be made a member of the group using "sudo vigr" for /etc/group and "sudo vigr -s" for /etc/gshadow. You need to login after logout (or run "exec newgrp") to enable the new group configuration.

	Note
	Alternatively, you may dynamically add users to groups during the authentication process by adding "auth optional pam_group.so" line to "/etc/pam.d/common-auth" and setting "/etc/security/group.conf". (See Chapter 4, Authentication.)

The hardware devices are just another kind of file on the Debian system. If you have problems accessing devices such as CD-ROM and USB memory stick from a user account, you should make that user a member of the relevant group.

Some notable system-provided groups allow their members to access particular files and devices without root privilege.

	Tip
	You need to belong to the dialout group to reconfigure modem, dial anywhere, etc. But if root creates pre-defined configuration files for trusted peers in "/etc/ppp/peers/", you only need to belong to the dip group to create Dialup IP connection to those trusted peers using pppd(8), pon(1), and poff(1) commands.

Some notable system-provided groups allow their members to execute particular commands without root privilege.

For the full listing of the system provided users and groups, see the recent version of the "Users and Groups" document in "/usr/share/doc/base-passwd/users-and-groups.html" provided by the base-passwd package.

See passwd(5), group(5), shadow(5), newgrp(1), vipw(8), vigr(8), and pam_group(8) for management commands of the user and group system.

There are three types of timestamps for a GNU/Linux file.

	Note
	ctime is not file creation time.

	Note
	Even simply reading a file on the Debian system normally causes a file write operation to update atime information in the inode. Mounting a filesystem with "noatime" or "relatime" option makes the system skip this operation and results in faster file access for the read. This is often recommended for laptops, because it reduces hard drive activity and saves power. See mount(8).

Use touch(1) command to change timestamps of existing files.

For timestamps, the ls command outputs different strings under non-English locale ("fr_FR.UTF-8") from under the old one ("C").

$ LANG=fr_FR.UTF-8  ls -l foo
-rw-rw-r-- 1 penguin penguin 0 oct. 16 21:35 foo
$ LANG=C  ls -l foo
-rw-rw-r-- 1 penguin penguin 0 Oct 16 21:35 foo

	Tip
	See Section 9.2.5, “Customized display of time and date” to customize "ls -l" output.

There are two methods of associating a file "foo" with a different filename "bar".

There are some special device files.

In order to make MC to change working directory upon exit and cd to the directory, I suggest to modify "~/.bashrc" to include a script provided by the mc package.

. /usr/lib/mc/mc.sh

See mc(1) (under the "-P" option) for the reason. (If you do not understand what exactly I am talking here, you can do this later.)

MC can be started by the following.

$ mc

MC takes care of all file operations through its menu, requiring minimal user effort. Just press F1 to get the help screen. You can play with MC just by pressing cursor-keys and function-keys.

	Note
	In some consoles such as gnome-terminal(1), key strokes of function-keys may be stolen by the console program. You can disable these features by "Edit" → "Keyboard Shortcuts" for gnome-terminal.

If you encounter character encoding problem which displays garbage characters, adding "-a" to MC's command line may help prevent problems.

The default is two directory panels containing file lists. Another useful mode is to set the right window to "information" to see file access privilege information, etc. Following are some essential keystrokes. With the gpm(8) daemon running, one can use a mouse on Linux character consoles, too. (Make sure to press the shift-key to obtain the normal behavior of cut and paste in MC.)

The internal editor has an interesting cut-and-paste scheme. Pressing F3 marks the start of a selection, a second F3 marks the end of selection and highlights the selection. Then you can move your cursor. If you press F6, the selected area is moved to the cursor location. If you press F5, the selected area is copied and inserted at the cursor location. F2 saves the file. F10 gets you out. Most cursor keys work intuitively.

This editor can be directly started on a file using one of the following commands.

$ mc -e filename_to_edit

$ mcedit filename_to_edit

This is not a multi-window editor, but one can use multiple Linux consoles to achieve the same effect. To copy between windows, use Alt-F<n> keys to switch virtual consoles and use "File→Insert file" or "File→Copy to file" to move a portion of a file to another file.

This internal editor can be replaced with any external editor of choice.

Also, many programs use the environment variables "$EDITOR" or "$VISUAL" to decide which editor to use. If you are uncomfortable with vim(1) or nano(1) initially, you may set these to "mcedit" by adding the following lines to "~/.bashrc".

export EDITOR=mcedit
export VISUAL=mcedit

I do recommend setting these to "vim" if possible.

If you are uncomfortable with vim(1), you can keep using mcedit(1) for most system maintenance tasks.

MC is a very smart viewer. This is a great tool for searching words in documents. I always use this for files in the "/usr/share/doc" directory. This is the fastest way to browse through masses of Linux information. This viewer can be directly started using one of the following commands.

$ mc -v path/to/filename_to_view

$ mcview path/to/filename_to_view

In order to allow these viewer and virtual file features to function, viewable files should not be set as executable. Change their status using chmod(1) or via the MC file menu.

You can select your login shell with chsh(1).

	Tip
	Although POSIX-like shells share the basic syntax, they can differ in behavior for things as basic as shell variables and glob expansions. Please check their documentation for details.

In this tutorial chapter, the interactive shell always means bash.

You can customize bash(1) behavior by "~/.bashrc".

For example, try the following.

# enable bash-completion
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi

# CD upon exiting MC
. /usr/lib/mc/mc.sh

# set CDPATH to a good one
CDPATH=.:/usr/share/doc:~:~/Desktop:~
export CDPATH

PATH="${PATH+$PATH:}/usr/sbin:/sbin"
# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
  PATH="~/bin${PATH+:$PATH}"
fi
export PATH

EDITOR=vim
export EDITOR

	Tip
	You can find more bash customization tips, such as Section 9.2.7, “Colorized commands”, in Chapter 9, System tips.

	Tip
	The bash-completion package enables programmable completion for bash.

Unix style mouse operations are based on the 3 button mouse system.

The center wheel on the modern wheel mouse is considered middle mouse button and can be used for middle-click. Clicking left and right mouse buttons together serves as the middle-click under the 2 button mouse system situation. In order to use a mouse in Linux character consoles, you need to have gpm(8) running as daemon.

The less(1) command is the enhanced pager (file content browser). It reads the file specified by its command argument or its standard input. Hit "h" if you need help while browsing with the less command. It can do much more than more(1) and can be supercharged by executing "eval $(lesspipe)" or "eval $(lessfile)" in the shell startup script. See more in "/usr/share/doc/lessf/LESSOPEN". The "-R" option allows raw character output and enables ANSI color escape sequences. See less(1).

Debian comes with a number of different editors. We recommend to install the vim package, as mentioned above.

Debian provides unified access to the system default editor via command "/usr/bin/editor" so other programs (e.g., reportbug(1)) can invoke it. You can change it by the following.

$ sudo update-alternatives --config editor

The choice "/usr/bin/vim.basic" over "/usr/bin/vim.tiny" is my recommendation for newbies since it supports syntax highlighting.

	Tip
	Many programs use the environment variables "$EDITOR" or "$VISUAL" to decide which editor to use (see Section 1.3.5, “The internal editor in MC” and Section 9.3.11, “Customizing program to be started”). For the consistency on the Debian system, set these to "/usr/bin/editor". (Historically, "$EDITOR" was "ed" and "$VISUAL" was "vi".)

You can customize vim(1) behavior by "~/.vimrc".

For example, try the following

" -------------------------------
" Local configuration
"
set nocompatible
set nopaste
set pastetoggle=<f2>
syn on
if $USER == "root"
 set nomodeline
 set noswapfile
else
 set modeline
 set swapfile
endif
" filler to avoid the line above being recognized as a modeline
" filler
" filler

The output of the shell command may roll off your screen and may be lost forever. It is a good practice to log shell activities into the file for you to review them later. This kind of record is essential when you perform any system administration tasks.

The basic method of recording the shell activity is to run it under script(1).

For example, try the following

$ script
Script started, file is typescript

Do whatever shell commands under script.

Press Ctrl-D to exit script.

$ vim typescript

Let's learn basic Unix commands. Here I use "Unix" in its generic sense. Any Unix clone OSs usually offer equivalent commands. The Debian system is no exception. Do not worry if some commands do not work as you wish now. If alias is used in the shell, its corresponding command outputs are different. These examples are not meant to be executed in this order.

Try all following commands from the non-privileged user account.

	Note
	Unix has a tradition to hide filenames which start with ".". They are traditionally files that contain configuration information and user preferences.

	Note
	For cd command, see builtins(7).

	Note
	The default pager of the bare bone Debian system is more(1) which cannot scroll back. By installing the less package using command line "apt-get install less", less(1) becomes default pager and you can scroll back with cursor keys.

Note

The "[" and "]" in the regular expression of the "ps aux | grep -e "[e]xim4*"" command above enable grep to avoid matching itself. The "4*" in the regular expression means 0 or more repeats of character "4" thus enables grep to match both "exim" and "exim4". Although "*" is used in the shell filename glob and the regular expression, their meanings are different. Learn the regular expression from grep(1).

Please traverse directories and peek into the system using the above commands as training. If you have questions on any of console commands, please make sure to read the manual page.

For example, try the following

$ man man
$ man bash
$ man builtins
$ man grep
$ man ls

The style of man pages may be a little hard to get used to, because they are rather terse, particularly the older, very traditional ones. But once you get used to it, you come to appreciate their succinctness.

Please note that many Unix-like commands including ones from GNU and BSD display brief help information if you invoke them in one of the following ways (or without any arguments in some cases).

$ <commandname> --help
$ <commandname> -h

The full locale value given to "$LANG" variable consists of 3 parts: "xx_YY.ZZZZ".

For language codes and country codes, see pertinent description in the "info gettext".

For the codeset on the modern Debian system, you should always set it to UTF-8 unless you specifically want to use the historic one with good reason and background knowledge.

Typical command execution uses a shell line sequence as the following.

$ date
Sun Jun  3 10:27:39 JST 2007
$ LANG=fr_FR.UTF-8 date
dimanche 3 juin 2007, 10:27:33 (UTC+0900)

Here, the program date(1) is executed with different values of the environment variable "$LANG".

When you type a command into the shell, the shell searches the command in the list of directories contained in the "$PATH" environment variable. The value of the "$PATH" environment variable is also called the shell's search path.

In the default Debian installation, the "$PATH" environment variable of user accounts may not include "/sbin" and "/usr/sbin". For example, the ifconfig command needs to be issued with full path as "/sbin/ifconfig". (Similar ip command is located in "/bin".)

You can change the "$PATH" environment variable of Bash shell by "~/.bash_profile" or "~/.bashrc" files.

Many commands stores user specific configuration in the home directory and changes their behavior by their contents. The home directory is identified by the environment variable "$HOME".

Some commands take arguments. Arguments starting with "-" or "--" are called options and control the behavior of the command.

$ date
Mon Oct 27 23:02:09 CET 2003
$ date -R
Mon, 27 Oct 2003 23:02:40 +0100

Often you want a command to work with a group of files without typing all of them. The filename expansion pattern using the shell glob, (sometimes referred as wildcards), facilitate this need.

For example, try the following

$ mkdir junk; cd junk; touch 1.txt 2.txt 3.c 4.h .5.txt ..6.txt
$ echo *.txt
1.txt 2.txt
$ echo *
1.txt 2.txt 3.c 4.h
$ echo *.[hc]
3.c 4.h
$ echo .*
. .. .5.txt ..6.txt
$ echo .*[^.]*
.5.txt ..6.txt
$ echo [^1-3]*
4.h
$ cd ..; rm -rf junk

See glob(7).

	Note
	Unlike normal filename expansion by the shell, the shell pattern "*" tested in find(1) with "-name" test etc., matches the initial "." of the filename. (New POSIX feature)

	Note
	BASH can be tweaked to change its glob behavior with its shopt builtin options such as "dotglob", "noglob", "nocaseglob", "nullglob", "extglob", etc. See bash(1).

Each command returns its exit status (variable: "$?") as the return value.

For example, try the following.

$ [ 1 = 1 ] ; echo $?
0
$ [ 1 = 2 ] ; echo $?
1

	Note
	Please note that, in the logical context for the shell, success is treated as the logical TRUE which has 0 (zero) as its value. This is somewhat non-intuitive and needs to be reminded here.

Let's try to remember following shell command idioms typed in one line as a part of shell command.

The Debian system is a multi-tasking system. Background jobs allow users to run multiple programs in a single shell. The management of the background process involves the shell builtins: jobs, fg, bg, and kill. Please read sections of bash(1) under "SIGNALS", and "JOB CONTROL", and builtins(1).

For example, try the following

$ </etc/motd pager

$ pager </etc/motd

$ pager /etc/motd

$ cat /etc/motd | pager

Although all 4 examples of shell redirections display the same thing, the last example runs an extra cat command and wastes resources with no reason.

The shell allows you to open files using the exec builtin with an arbitrary file descriptor.

$ echo Hello >foo
$ exec 3<foo 4>bar  # open files
$ cat <&3 >&4       # redirect stdin to 3, stdout to 4
$ exec 3<&- 4>&-    # close files
$ cat bar
Hello

The file descriptor 0-2 are predefined.

You can set an alias for the frequently used command.

For example, try the following

$ alias la='ls -la'

Now, "la" works as a short hand for "ls -la" which lists all files in the long listing format.

You can list any existing aliases by alias (see bash(1) under "SHELL BUILTIN COMMANDS").

$ alias
...
alias la='ls -la'

You can identity exact path or identity of the command by type (see bash(1) under "SHELL BUILTIN COMMANDS").

For example, try the following

$ type ls
ls is hashed (/bin/ls)
$ type la
la is aliased to ls -la
$ type echo
echo is a shell builtin
$ type file
file is /usr/bin/file

Here ls was recently searched while "file" was not, thus "ls" is "hashed", i.e., the shell has an internal record for the quick access to the location of the "ls" command.

	Tip
	See Section 9.2.7, “Colorized commands”.

There are few standard text processing tools which are used very often on the Unix-like system.

The regular expression of emacs is basically BRE but has been extended to treat "+"and "?" as the metacharacters as in ERE. Thus, there are no needs to escape them with "\" in the regular expression of emacs.

grep(1) can be used to perform the text search using the regular expression.

For example, try the following

$ egrep 'GNU.*LICENSE|Yoyodyne' /usr/share/common-licenses/GPL
GNU GENERAL PUBLIC LICENSE
GNU GENERAL PUBLIC LICENSE
Yoyodyne, Inc., hereby disclaims all copyright interest in the program

	Tip
	See Section 9.2.7, “Colorized commands”.

For the replacement expression, some characters have special meanings.

For Perl replacement string, "$n" is used instead of "\n" and "&" has no special meaning.

For example, try the following

$ echo zzz1abc2efg3hij4 | \
sed -e 's/\(1[a-z]*\)[0-9]*\(.*\)$/=&=/'
zzz=1abc2efg3hij4=
$ echo zzz1abc2efg3hij4 | \
sed -e 's/\(1[a-z]*\)[0-9]*\(.*\)$/\2===\1/'
zzzefg3hij4===1abc
$ echo zzz1abc2efg3hij4 | \
perl -pe 's/(1[a-z]*)[0-9]*(.*)$/$2===$1/'
zzzefg3hij4===1abc
$ echo zzz1abc2efg3hij4 | \
perl -pe 's/(1[a-z]*)[0-9]*(.*)$/=&=/'
zzz=&=

Here please pay extra attention to the style of the bracketed regular expression and how the matched strings are used in the text replacement process on different tools.

These regular expressions can be used for cursor movements and text replacement actions in some editors too.

The back slash "\" at the end of line in the shell commandline escapes newline as a white space character and continues shell command line input to the next line.

Please read all the related manual pages to learn these commands.

The ed(1) command can replace all instances of "FROM_REGEX" with "TO_TEXT" in "file".

$ ed file <<EOF
,s/FROM_REGEX/TO_TEXT/g
w
q
EOF

The sed(1) command can replace all instances of "FROM_REGEX" with "TO_TEXT" in "file".

$ sed -i -e 's/FROM_REGEX/TO_TEXT/g' file

The vim(1) command can replace all instances of "FROM_REGEX" with "TO_TEXT" in "file" by using ex(1) commands.

$ vim '+%s/FROM_REGEX/TO_TEXT/gc' '+w' '+q' file

	Tip
	The "c" flag in the above ensures interactive confirmation for each substitution.

Multiple files ("file1", "file2", and "file3") can be processed with regular expressions similarly with vim(1) or perl(1).

$ vim '+argdo %s/FROM_REGEX/TO_TEXT/ge|update' '+q' file1 file2 file3

	Tip
	The "e" flag in the above prevents the "No match" error from breaking a mapping.

$ perl -i -p -e 's/FROM_REGEX/TO_TEXT/g;' file1 file2 file3

In the perl(1) example, "-i" is for the in-place editing of each target file, and "-p" is for the implicit loop over all given files.

	Tip
	Use of argument "-i.bak" instead of "-i" keeps each original file by adding ".bak" to its filename. This makes recovery from errors easier for complex substitutions.

	Note
	ed(1) and vim(1) are BRE; perl(1) is ERE.

Let's consider a text file called "DPL" in which some pre-2004 Debian project leader's names and their initiation date are listed in a space-separated format.

Ian     Murdock   August  1993
Bruce   Perens    April   1996
Ian     Jackson   January 1998
Wichert Akkerman  January 1999
Ben     Collins   April   2001
Bdale   Garbee    April   2002
Martin  Michlmayr March   2003

	Tip
	See "A Brief History of Debian" for the latest Debian leadership history.

Awk is frequently used to extract data from these types of files.

For example, try the following

$ awk '{ print $3 }' <DPL                   # month started
August
April
January
January
April
April
March
$ awk '($1=="Ian") { print }' <DPL          # DPL called Ian
Ian     Murdock   August  1993
Ian     Jackson   January 1998
$ awk '($2=="Perens") { print $3,$4 }' <DPL # When Perens started
April 1996

Shells such as Bash can be also used to parse this kind of file.

For example, try the following

$ while read first last month year; do
    echo $month
  done <DPL
... same output as the first Awk example

Here, the read builtin command uses characters in "$IFS" (internal field separators) to split lines into words.

If you change "$IFS" to ":", you can parse "/etc/passwd" with shell nicely.

$ oldIFS="$IFS"   # save old value
$ IFS=':'
$ while read user password uid gid rest_of_line; do
    if [ "$user" = "bozo" ]; then
      echo "$user's ID is $uid"
    fi
  done < /etc/passwd
bozo's ID is 1000
$ IFS="$oldIFS"   # restore old value

(If Awk is used to do the equivalent, use "FS=':'" to set the field separator.)

IFS is also used by the shell to split results of parameter expansion, command substitution, and arithmetic expansion. These do not occur within double or single quoted words. The default value of IFS is <space>, <tab>, and <newline> combined.

Be careful about using this shell IFS tricks. Strange things may happen, when shell interprets some parts of the script as its input.

$ IFS=":,"                        # use ":" and "," as IFS
$ echo IFS=$IFS,   IFS="$IFS"     # echo is a Bash builtin
IFS=  , IFS=:,
$ date -R                         # just a command output
Sat, 23 Aug 2003 08:30:15 +0200
$ echo $(date -R)                 # sub shell --> input to main shell
Sat  23 Aug 2003 08 30 36 +0200
$ unset IFS                       # reset IFS to the default
$ echo $(date -R)
Sat, 23 Aug 2003 08:30:50 +0200

The following scripts do nice things as a part of a pipe.