Chapter 9. System tips

screen(1) not only allows one terminal window to work with multiple processes, but also allows remote shell process to survive interrupted connections. Here is a typical use scenario of screen(1).

	Tip
	You can save connection fees with screen for metered network connections such as dial-up and packet ones, because you can leave a process active while disconnected, and then re-attach it later when you connect again.

In a screen session, all keyboard inputs are sent to your current window except for the command keystroke. All screen command keystrokes are entered by typing ^A ("Control-A") plus a single key [plus any parameters]. Here are important ones to remember.

See screen(1) for details.

Many programs record their activities under the "/var/log/" directory.

Here are notable log analyzers ("~Gsecurity::log-analyzer" in aptitude(8)).

Colorized commands are handy for inspecting their output in the interactive environment. I include the following in my "~/.bashrc".

if [ "$TERM" != "dumb" ]; then
    eval "`dircolors -b`"
    alias ls='ls --color=always'
    alias ll='ls --color=always -l'
    alias la='ls --color=always -A'
    alias less='less -R'
    alias ls='ls --color=always'
    alias grep='grep --color=always'
    alias egrep='egrep --color=always'
    alias fgrep='fgrep --color=always'
    alias zgrep='zgrep --color=always'
else
    alias ll='ls -l'
    alias la='ls -A'
fi

The use of alias limits color effects to the interactive command usage. It has advantage over exporting environment variable "export GREP_OPTIONS='--color=auto'" since color can be seen under pager programs such as less(1). If you wish to suppress color when piping to other programs, use "--color=auto" instead in the above example for "~/.bashrc".

	Tip
	You can turn off these colorizing aliases in the interactive environment by invoking shell with "TERM=dumb bash".

You can record the editor activities for complex repeats.

There are few ways to record the graphic image of an X application, including an xterm display.

There are specialized tools to record changes in configuration files with help of DVCS system.

I recommend to use the etckeeper package with git(1) which put entire "/etc" under VCS control. Its installation guide and tutorial are found in "/usr/share/doc/etckeeper/README.gz".

Display time used by the process invoked by the command.

# time some_command >/dev/null
real    0m0.035s       # time on wall clock (elapsed real time)
user    0m0.000s       # time in user mode
sys     0m0.020s       # time in kernel mode

A nice value is used to control the scheduling priority for the process.

# nice  -19 top                                      # very nice
# nice --20 wodim -v -eject speed=2 dev=0,0 disk.img # very fast

Sometimes an extreme nice value does more harm than good to the system. Use this command carefully.

The ps(1) command on a Debian system support both BSD and SystemV features and helps to identify the process activity statically.

For the zombie (defunct) children process, you can kill them by the parent process ID identified in the "PPID" field.

The pstree(1) command display a tree of processes.

top(1) on the Debian system has rich features and helps to identify what process is acting funny dynamically.

Its is an interactive full screen program. You can get its usage help press by pressing the "h"-key and terminate it by pressing the "q"-key.

You can list all files opened by a process with a process ID (PID), e.g. 1, by the following.

$ sudo lsof -p 1

PID=1 is usually the init program.

You can trace program activity with strace(1), ltrace(1), or xtrace(1) for system calls and signals, library calls, or communication between X11 client and server.

You can trace system calls of the ls command as the following.

$ sudo strace ls

You can also identify processes using files by fuser(1), e.g. for "/var/log/mail.log" by the following.

$ sudo fuser -v /var/log/mail.log
                     USER        PID ACCESS COMMAND
/var/log/mail.log:   root       2946 F.... rsyslogd

You see that file "/var/log/mail.log" is open for writing by the rsyslogd(8) command.

You can also identify processes using sockets by fuser(1), e.g. for "smtp/tcp" by the following.

$ sudo fuser -v smtp/tcp
                     USER        PID ACCESS COMMAND
smtp/tcp:            Debian-exim   3379 F.... exim4

watch(1) executes a program repeatedly with a constant interval while showing its output in fullscreen.

$ watch w

This displays who is logged on to the system updated every 2 seconds.

There are several ways to repeat a command looping over files matching some condition, e.g. matching glob pattern "*.ext".

Some programs start another program automatically. Here are check points for customizing this process.

Use kill(1) to kill (or send a signal to) a process by the process ID.

Use killall(1) or pkill(1) to do the same by the process command name and other attributes.

Run the at(1) command to schedule a one-time job by the following.

$ echo 'command -args'| at 3:40 monday

Use cron(8) to schedule tasks regularly. See crontab(1) and crontab(5).

You can schedule to run processes as a normal user, e.g. foo by creating a crontab(5) file as "/var/spool/cron/crontabs/foo" with "crontab -e" command.

Here is an example of a crontab(5) file.

# use /bin/sh to run commands, no matter what /etc/passwd says
SHELL=/bin/sh
# mail any output to paul, no matter whose crontab this is
MAILTO=paul
# Min Hour DayOfMonth Month DayOfWeek command (Day... are OR'ed)
# run at 00:05, every day
5  0  *  * *   $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
# run at 14:15 on the first of every month -- output mailed to paul
15 14 1  * *   $HOME/bin/monthly
# run at 22:00 on weekdays(1-5), annoy Joe. % for newline, last % for cc:
0 22 *   * 1-5 mail -s "It's 10pm" joe%Joe,%%Where are your kids?%.%%
23 */2 1 2 *   echo "run 23 minutes after 0am, 2am, 4am ..., on Feb 1"
5  4 *   * sun echo "run at 04:05 every Sunday"
# run at 03:40 on the first Monday of each month
40 3 1-7 * *   [ "$(date +%a)" == "Mon" ] && command -args

	Tip
	For the system not running continuously, install the anacron package to schedule periodic commands at the specified intervals as closely as machine-uptime permits. See anacron(8) and anacrontab(5).

	Tip
	For scheduled system maintenance scripts, you can run them periodically from root account by placing such scripts in "/etc/cron.hourly/", "/etc/cron.daily/", "/etc/cron.weekly/", or "/etc/cron.monthly/". Execution timings of these scripts can be customized by "/etc/crontab" and "/etc/anacrontab".

	Tip
	Read the signal(7), kill(1), and sync(1) manpages to understand the description above.

The combination of "Alt-SysRq s", "Alt-SysRq u", and "Alt-SysRq r" is good for getting out of really bad situations and gaining usable keyboard access without stopping the system.

See "/usr/share/doc/linux-doc-3.*/Documentation/sysrq.txt.gz".

	Caution
	The Alt-SysRq feature may be considered a security risk by allowing users access to root-privileged functions. Placing "echo 0 >/proc/sys/kernel/sysrq" in "/etc/rc.local" or "kernel.sysrq = 0" in "/etc/sysctl.conf" disables the Alt-SysRq feature.

	Tip
	From SSH terminal etc., you can use the Alt-SysRq feature by writing to the "/proc/sysrq-trigger". For example, "echo s > /proc/sysrq-trigger; echo u > /proc/sysrq-trigger" from the root shell prompt syncs and umounts all mounted filesystems.

You can check who is on the system by the following.

	Tip
	"/var/run/utmp", and "/var/log/wtmp" hold such user information. See login(1) and utmp(5).

You can send message to everyone who is logged on to the system with wall(1) by the following.

$ echo "We are shutting down in 1 hour" | wall

Although most of the hardware configuration on modern GUI desktop systems such as GNOME and KDE can be managed through accompanying GUI configuration tools, it is a good idea to know some basics methods to configure them.

The following sets system and hardware time to MM/DD hh:mm, CCYY.

# date MMDDhhmmCCYY
# hwclock --utc --systohc
# hwclock --show

There are several components to configure character console and ncurses(3) system features.

If the terminfo entry for xterm doesn't work with a non-Debian xterm, change your terminal type, "$TERM", from "xterm" to one of the feature-limited versions such as "xterm-r6" when you log in to a Debian system remotely. See "/usr/share/doc/libncurses5/FAQ" for more. "dumb" is the lowest common denominator for "$TERM".

There is usually a common sound engine for each popular desktop environment. Each sound engine used by the application can choose to connect to different sound servers.

For disabling the screen saver, use following commands.

One can always unplug the PC speaker to disable beep sounds. Removing pcspkr kernel module does this for you.

The following prevents the readline(3) program used by bash(1) to beep when encountering an alert character (ASCII=7).

$ echo "set bell-style none">> ~/.inputrc

There are 2 resources available for you to get the memory usage situation.

Here is an example.

# grep '\] Memory' /var/log/dmesg
[    0.004000] Memory: 990528k/1016784k available (1975k kernel code, 25868k reserved, 931k data, 296k init)
$ free -k
             total       used       free     shared    buffers     cached
Mem:        997184     976928      20256          0     129592     171932
-/+ buffers/cache:     675404     321780
Swap:      4545576          4    4545572

You may be wondering "dmesg tells me a free of 990 MB, and free -k says 320 MB is free. More than 600 MB missing …".

Do not worry about the large size of "used" and the small size of "free" in the "Mem:" line, but read the one under them (675404 and 321780 in the example above) and relax.

For my MacBook with 1GB=1048576k DRAM (video system steals some of this), I see the following.

Poor system maintenance may expose your system to external exploitation.

For system security and integrity check, you should start with the following.

Here is a simple script to check for typical world writable incorrect file permissions.

# find / -perm 777 -a \! -type s -a \! -type l -a \! \( -type d -a -perm 1777 \)

	Caution
	Since the debsums package uses MD5 checksums stored locally, it can not be fully trusted as the system security audit tool against malicious attacks.

The disk space usage can be evaluated by programs provided by the mount, coreutils, and xdu packages:

	Tip
	You can feed the output of du(8) to xdu(1x) to produce its graphical and interactive presentation with "du -k . |xdu", "sudo du -k -x / |xdu", etc.

Although reconfiguration of your partition or activation order of removable storage media may yield different names for partitions, you can access them consistently. This is also helpful if you have multiple disks and your BIOS doesn't give them consistent device names.

The mkfs(8) command creates the filesystem on a Linux system. The fsck(8) command provides the filesystem integrity check and repair on a Linux system.

Debian now defaults to no periodic fsck after filesystem creation.

	Caution
	It is generally not safe to run fsck on mounted filesystems.

	Tip
	You can run the fsck(8) command safely on all filesystems including root filesystem on reboot by setting "enable_periodic_fsck" in "/etc/mke2fs.conf" and the max mount count to 0 using "tune2fs -c0 /dev/<partition_name>". See mke2fs.conf(5) and tune2fs(8).

	Tip
	Check files in "/var/log/fsck/" for the result of the fsck(8) command run from the boot script.

The basic static filesystem configuration is given by "/etc/fstab". For example,

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
UUID=709cbe4c-80c1-56db-8ab1-dbce3146d2f7 / ext4 noatime,errors=remount-ro 0 1
UUID=817bae6b-45d2-5aca-4d2a-1267ab46ac23 none swap sw  0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto 0       0

	Tip
	UUID (see Section 9.5.3, “Accessing partition using UUID”) may be used to identify a block device instead of normal block device names such as "/dev/sda1", "/dev/sda2", …

Performance and characteristics of a filesystem can be optimized by mount options used on it (see fstab(5) and mount(8)). Notable ones are the following.

	Tip
	You need to provide kernel boot parameter (see Section 3.1.2, “Stage 2: the boot loader”), e.g. "rootflags=data=journal" to deploy a non-default journaling mode for the root filesystem. For lenny, the default jounaling mode is "rootflags=data=ordered". For squeeze, it is "rootflags=data=writeback".

Characteristics of a filesystem can be optimized via its superblock using the tune2fs(8) command.

	Warning
	Please check your hardware and read manpage of hdparam(8) before playing with hard disk configuration because this may be quite dangerous for the data integrity.

You can test disk access speed of a hard disk, e.g. "/dev/hda", by "hdparm -tT /dev/hda". For some hard disk connected with (E)IDE, you can speed it up with "hdparm -q -c3 -d1 -u1 -m16 /dev/hda" by enabling the "(E)IDE 32-bit I/O support", enabling the "using_dma flag", setting "interrupt-unmask flag", and setting the "multiple 16 sector I/O" (dangerous!).

You can test write cache feature of a hard disk, e.g. "/dev/sda", by "hdparm -W /dev/sda". You can disable its write cache feature with "hdparm -W 0 /dev/sda".

You may be able to read badly pressed CDROMs on modern high speed CD-ROM drive by slowing it down with "setcd -x 2".

Applications create temporary files normally under the temporary storage directory "/tmp". If "/tmp" does not provide enough space, you can specify such temporary storage directory via the $TMPDIR variable for well-behaving programs.

If you have an empty partition (e.g., "/dev/sdx"), you can format it with mkfs.ext4(1) and mount(8) it to a directory where you need more space. (You need to copy original data contents.)

$ sudo mv work-dir old-dir
$ sudo mkfs.ext4 /dev/sdx
$ sudo mount -t ext4 /dev/sdx work-dir
$ sudo cp -a old-dir/* work-dir
$ sudo rm -rf old-dir

	Tip
	You may alternatively mount an empty disk image file (see Section 9.6.5, “Making the empty disk image file”) as a loop device (see Section 9.6.3, “Mounting the disk image file”). The actual disk usage grows with the actual data stored.

If you have an empty directory (e.g., "/path/to/emp-dir") on another partition with usable space, you can mount(8) it with "--bind" option to a directory (e.g., "work-dir") where you need more space.

$ sudo mount --bind /path/to/emp-dir work-dir

	Tip
	This is a deprecated method. Use Section 9.5.15, “Expansion of usable storage space by bind-mounting another directory” instead, if possible.

If you have an empty directory (e.g., "/path/to/emp-dir") in another partition with usable space, you can create a symlink to the directory with ln(8).

$ sudo mv work-dir old-dir
$ sudo mkdir -p /path/to/emp-dir
$ sudo ln -sf /path/to/emp-dir work-dir
$ sudo cp -a old-dir/* work-dir
$ sudo rm -rf old-dir

	Warning
	Do not use "symlink to a directory" for directories managed by the system such as "/opt". Such a symlink may be overwritten when the system is upgraded.

	Caution
	Some software may not function well with "symlink to a directory".

The disk image file, "disk.img", of an unmounted device, e.g., the second SCSI or serial ATA drive "/dev/sdb", can be made using cp(1) or dd(1) by the following.

# cp /dev/sdb disk.img
# dd if=/dev/sdb of=disk.img

The disk image file, "disk.img" can be written to an unmounted device, e.g., the second SCSI drive "/dev/sdb" with matching size, by the following.

# dd if=disk.img of=/dev/sdb

Similarly, the disk partition image file, "partition.img" can be written to an unmounted partition, e.g., the first partition of the second SCSI drive "/dev/sdb1" with matching size, by the following.

# dd if=partition.img of=/dev/sdb1

A disk image file, "disk.img" can be cleaned of all removed files into clean sparse image "new.img" by the following.

# mkdir old; mkdir new
# mount -t auto -o loop disk.img old
# dd bs=1 count=0 if=/dev/zero of=new.img seek=5G
# mount -t auto -o loop new.img new
# cd old
# cp -a --sparse=always ./ ../new/
# cd ..
# umount new.img
# umount disk.img

If "disk.img" is in ext2, ext3 or ext4, you can also use zerofree(8) from the zerofree package as follows.

# losetup -f -v disk.img
Loop device is /dev/loop3
# zerofree /dev/loop3
# cp --sparse=always disk.img new.img

The empty disk image "disk.img" which can grow up to 5GiB can be made using dd(1) as follows.

$ dd bs=1 count=0 if=/dev/zero of=disk.img seek=5G

	Tip
	DVD is only a large CD to wodim(1) provided by cdrkit.

You can find a usable device by the following.

# wodim --devices

Then the blank CD-R is inserted to the CD drive, and the ISO9660 image file, "cd.iso" is written to this device, e.g., "/dev/hda", using wodim(1) by the following.

# wodim -v -eject dev=/dev/hda cd.iso

If CD-RW is used instead of CD-R, do this instead by the following.

# wodim -v -eject blank=fast dev=/dev/hda cd.iso

	Tip
	If your desktop system mounts CDs automatically, unmount it by "sudo umount /dev/hda" from console before using wodim(1).

If "cd.iso" contains an ISO9660 image, then the following manually mounts it to "/cdrom".

# mount -t iso9660 -o ro,loop cd.iso /cdrom

	Tip
	Modern desktop system may mount removable media such as ISO9660 formatted CD automatically (see Section 10.1.7, “Removable storage device”).

The most basic viewing method of binary data is to use "od -t x1" command.

There are tools to read and write files without mounting disk.

There are tools for data file recovery and forensic analysis.

When a data is too big to backup as a single file, you can backup its content after splitting it into, e.g. 2000MiB chunks and merge those chunks back into the original file later.

$ split -b 2000m large_file
$ cat x* >large_file

	Caution
	Please make sure you do not have any files starting with "x" to avoid name crashes.

In order to clear the contents of a file such as a log file, do not use rm(1) to delete the file and then create a new empty file, because the file may still be accessed in the interval between commands. The following is the safe way to clear the contents of the file.

$ :>file_to_be_cleared

The following commands create dummy or empty files.

$ dd if=/dev/zero    of=5kb.file bs=1k count=5
$ dd if=/dev/urandom of=7mb.file bs=1M count=7
$ touch zero.file
$ : > alwayszero.file

You should find following files.

There are several ways to completely erase data from an entire hard disk like device, e.g., USB memory stick at "/dev/sda".

	Caution
	Check your USB memory stick location with mount(8) first before executing commands here. The device pointed by "/dev/sda" may be SCSI hard disk or serial-ATA hard disk where your entire system resides.

Erase all the disk content by resetting data to 0 with the following.

# dd if=/dev/zero of=/dev/sda

Erase everything by overwriting with random data as follows.

# dd if=/dev/urandom of=/dev/sda

Erase everything by overwriting with random data very efficiently as follows.

# shred -v -n 1 /dev/sda

Since dd(1) is available from the shell of many bootable Linux CDs such as Debian installer CD, you can erase your installed system completely by running an erase command from such media on the system hard disk, e.g., "/dev/hda", "/dev/sda", etc.

Unused area on an hard disk (or USB memory stick), e.g. "/dev/sdb1" may still contain erased data themselves since they are only unlinked from the filesystem. These can be cleaned by overwriting them.

# mount -t auto /dev/sdb1 /mnt/foo
# cd /mnt/foo
# dd if=/dev/zero of=junk
dd: writing to `junk': No space left on device
...
# sync
# umount /dev/sdb1

	Warning
	This is usually good enough for your USB memory stick. But this is not perfect. Most parts of erased filenames and their attributes may be hidden and remain in the filesystem.

Even if you have accidentally deleted a file, as long as that file is still being used by some application (read or write mode), it is possible to recover such a file.

For example, try the following

$ echo foo > bar
$ less bar
$ ps aux | grep ' less[ ]'
bozo    4775  0.0  0.0  92200   884 pts/8    S+   00:18   0:00 less bar
$ rm bar
$ ls -l /proc/4775/fd | grep bar
lr-x------ 1 bozo bozo 64 2008-05-09 00:19 4 -> /home/bozo/bar (deleted)
$ cat /proc/4775/fd/4 >bar
$ ls -l
-rw-r--r-- 1 bozo bozo 4 2008-05-09 00:25 bar
$ cat bar
foo

Execute on another terminal (when you have the lsof package installed) as follows.

$ ls -li bar
2228329 -rw-r--r-- 1 bozo bozo 4 2008-05-11 11:02 bar
$ lsof |grep bar|grep less
less 4775 bozo 4r REG 8,3 4 2228329 /home/bozo/bar
$ rm bar
$ lsof |grep bar|grep less
less 4775 bozo 4r REG 8,3 4 2228329 /home/bozo/bar (deleted)
$ cat /proc/4775/fd/4 >bar
$ ls -li bar
2228302 -rw-r--r-- 1 bozo bozo 4 2008-05-11 11:05 bar
$ cat bar
foo

Files with hardlinks can be identified by "ls -li".

$ ls -li
total 0
2738405 -rw-r--r-- 1 root root 0 2008-09-15 20:21 bar
2738404 -rw-r--r-- 2 root root 0 2008-09-15 20:21 baz
2738404 -rw-r--r-- 2 root root 0 2008-09-15 20:21 foo

All deleted but open files consume disk space although they are not visible from normal du(1). They can be listed with their size by the following.

# lsof -s -X / |grep deleted

Let's assume that your original "/etc/fstab" contains the following.

/dev/sda7 swap sw 0 0

An encrypted disk partition created with dm-crypt/LUKS on "/dev/sdc5" can be mounted onto "/mnt" as follows:

$ sudo cryptsetup open /dev/sdc5 ninja --type luks
Enter passphrase for /dev/sdc5: ****
$ sudo lvm
lvm> lvscan
  inactive          '/dev/ninja-vg/root' [13.52 GiB] inherit
  inactive          '/dev/ninja-vg/swap_1' [640.00 MiB] inherit
  ACTIVE            '/dev/goofy/root' [180.00 GiB] inherit
  ACTIVE            '/dev/goofy/swap' [9.70 GiB] inherit
lvm> lvchange -a y /dev/ninja-vg/root
lvm> exit
  Exiting.
$ sudo mount /dev/ninja-vg/root /mnt

There are few notable features on Linux kernel 2.6/3.x compared to 2.4.

Many Linux features are configurable via kernel parameters as follows.

Most normal programs don't need kernel headers and in fact may break if you use them directly for compiling. They should be compiled against the headers in "/usr/include/linux" and "/usr/include/asm" provided by the libc6-dev package (created from the glibc source package) on the Debian system.

	Note
	For compiling some kernel-specific programs such as the kernel modules from the external source and the automounter daemon (amd), you must include path to the corresponding kernel headers, e.g. "-I/usr/src/linux-particular-version/include/", to your command line. module-assistant(8) (or its short form m-a) helps users to build and install module package(s) easily for one or more custom kernels.

Debian has its own method of compiling the kernel and related modules.

For building custom kernel binary packages from the upstream kernel source, you should use the "deb-pkg" target provided by it.

$ sudo apt-get build-dep linux
$ cd /usr/src
$ wget http://www.kernel.org/pub/linux/kernel/v3.11/linux-<version>.tar.bz2
$ tar -xjvf linux-<version>.tar.bz2
$ cd linux-<version>
$ cp /boot/config-<version> .config
$ make menuconfig
 ...
$ make deb-pkg

	Tip
	The linux-source-<version> package provides the Linux kernel source with Debian patches as "/usr/src/linux-<version>.tar.bz2".

For building specific binary packages from the Debian kernel source package, you should use the "binary-arch_<architecture>_<featureset>_<flavour>" targets in "debian/rules.gen".

$ sudo apt-get build-dep linux
$ apt-get source linux
$ cd linux-3.*
$ fakeroot make -f debian/rules.gen binary-arch_i386_none_686

See further information:

The hardware driver is the code running on the target system. Most hardware drivers are available as free software now and are included in the normal Debian kernel packages in the main area.

chroot(8) offers most basic way to run different instances of the GNU/Linux environment on a single system simultaneously without rebooting.

	Caution
	Examples below assumes both parent system and chroot system share the same CPU architecture.

You can learn how to setup and use chroot(8) by running pbuilder(8) program under script(1) as follows.

$ sudo mkdir /sid-root
$ sudo pbuilder --create --no-targz --debug --buildplace /sid-root

You see how debootstrap(8) or cdebootstrap(1) populate system data for sid environment under "/sid-root".

	Tip
	These debootstrap(8) or cdebootstrap(1) are used to install Debian by the Debian Installer. These can also be used to install Debian to a system without using a Debian install disk, but instead from another GNU/Linux distribution.

$ sudo pbuilder --login --no-targz  --debug --buildplace /sid-root

You see how a system shell running under sid environment is created as the following.

	Note
	Some programs under chroot may require access to more files from the parent system to function than pbuilder provides. For example, "/sys", "/etc/passwd", "/etc/group", "/var/run/utmp", "/var/log/wtmp", etc. may need to be bind-mounted or copied.

	Note
	The "/usr/sbin/policy-rc.d" file prevents daemon programs to be started automatically on the Debian system. See "/usr/share/doc/sysv-rc/README.policy-rc.d.gz".

	Tip
	The original purpose of the specialized chroot package, pbuilder is to construct a chroot system and builds a package inside the chroot. It is an ideal system to use to check that a package's build-dependencies are correct, and to be sure that unnecessary and wrong build dependencies do not exist in the resulting package.

	Tip
	Similar schroot package may give you an idea to run i386 chroot system under amd64 parent system.