Chapter 6. Network applications

Debian offers many free browser plugin packages in the main archive area which can handle not only Java (software platform) and Flash but also MPEG, MPEG2, MPEG4, DivX, Windows Media Video (.wmv), QuickTime (.mov), MP3 (.mp3), Ogg/Vorbis files, DVDs, VCDs, etc. Debian also offers helper programs to install non-free browser plugin packages as contrib or non-free archive area.

	Tip
	Although use of above Debian packages are much easier, browser plugins can be still manually enabled by installing "*.so" into plugin directories (e.g., "/usr/lib/iceweasel/plugins/") and restarting browsers.

Some web sites refuse to be connected based on the user-agent string of your browser. You can work around this situation by spoofing the user-agent string. For example, you can do this by adding following line into user configuration files such as "~/.gnome2/epiphany/mozilla/epiphany/user.js" or "~/.mozilla/firefox/*.default/user.js".

user_pref{"general.useragent.override","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"};

Alternatively, you can add and reset this variable by typing "about:config" into URL and right clicking its display contents.

	Caution
	Spoofed user-agent string may cause bad side effects with Java.

An email message consists of three components, the message envelope, the message header, and the message body.

The "To" and "From" information in the message envelope is used by the SMTP to deliver the email. (The "From" information in the message envelope is also called bounce address, From_, etc.).

The "To" and "From" information in the message header is displayed by the email client. (While it is most common for these to be the same as ones in the message envelope, such is not always the case.)

The email client (MUA) needs to interpret the message header and body data using Multipurpose Internet Mail Extensions (MIME) to deal the content data type and encoding.

When configuring your mail system or resolving mail delivery problems, you must consider these new limitations.

In light of these hostile Internet situation and limitations, some independent Internet mail ISPs such as Yahoo.com and Gmail.com offer the secure mail service which can be connected from anywhere on the Internet using Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL).

	Caution
	It is not realistic to run SMTP server on consumer grade network to send mail directly to the remote host reliably. They are very likely to be rejected. You must use some smarthost services offered by your connection ISP or independent mail ISPs.

For the simplicity, I assume that the smarthost is located at "smtp.hostname.dom", requires SMTP Authentication, and uses the message submission port (587) with the STARTTLS in the following text.

The most simple mail configuration is that the mail is sent to the ISP's smarthost and received from ISP's POP3 server by the MUA (see Section 6.4, “Mail user agent (MUA)”) itself. This type of configuration is popular with full featured GUI based MUA such as icedove(1), evolution(1), etc. If you need to filter mail by their types, you use MUA's filtering function. For this case, the local MTA (see Section 6.3, “Mail transport agent (MTA)”) need to do local delivery only (when sender and receiver are on the same host).

Please note that the Debian system is the multiuser system. Even if you are the only user, there are many programs running as root and they may send you a mail.

The alternative mail configuration is that the mail is sent via local MTA to the ISP's smarthost and received from ISP's POP3 by the mail retriever (see Section 6.5, “The remote mail retrieval and forward utility”) to the local mailbox. If you need to filter mail by their types, you use MDA with filter (see Section 6.6, “Mail delivery agent (MDA) with filter”) to filter mail into separate mailboxes. This type of configuration is popular with simple console based MUA such as mutt(1), mew(1), etc., although this is possible with any MUAs (see Section 6.4, “Mail user agent (MUA)”). For this case, the local MTA (see Section 6.3, “Mail transport agent (MTA)”) need to do both smarthost delivery and local delivery. Since mobile workstation does not have valid FQDN, you must configure the local MTA to hide and spoof the real local mail name in outgoing mail to avoid mail delivery errors (see Section 6.3.3, “The mail address configuration”).

	Tip
	You may wish to configure MUA/MDA to use Maildir for storing email messages somewhere under your home directory.

$ sudo /etc/init.d/exim4 stop
$ sudo dpkg-reconfigure exim4-config

Set "System mail name:" to its default as the FQDN (see Section 5.1.1, “The hostname resolution”).

Set "IP-addresses to listen on for incoming SMTP connections:" to its default as "127.0.0.1 ; ::1".

Unset contents of "Other destinations for which mail is accepted:".

Unset contents of "Machines to relay mail for:".

Set "IP address or host name of the outgoing smarthost:" to "smtp.hostname.dom:587".

Select "<No>" for "Hide local mail name in outgoing mail?". (Use "/etc/email-addresses" as in Section 6.3.3, “The mail address configuration”, instead.)

Reply to "Keep number of DNS-queries minimal (Dial-on-Demand)?" as one of the following.

Set "Delivery method for local mail:" to "mbox format in /var/mail/".

Select "<Yes>" for "Split configuration into small files?:".

Create password entries for the smarthost by editing "/etc/exim4/passwd.client".

$ sudo vim /etc/exim4/passwd.client
 ...
$ cat /etc/exim4/passwd.client
^smtp.*\.hostname\.dom:[email protected]:password

Start exim4 by the following.

$ sudo /etc/init.d/exim4 start

The host name in "/etc/exim4/passwd.client" should not be the alias. You check the real host name with the following.

$ host smtp.hostname.dom
smtp.hostname.dom is an alias for smtp99.hostname.dom.
smtp99.hostname.dom has address 123.234.123.89

I use regex in "/etc/exim4/passwd.client" to work around the alias issue. SMTP AUTH probably works even if the ISP moves host pointed by the alias.

You can manually update exim4 configuration by the following:

Please read the official guide at: "/usr/share/doc/exim4-base/README.Debian.gz" and update-exim4.conf(8).

	Caution
	Starting exim4 takes long time if "No" (default value) was chosen for the debconf query of "Keep number of DNS-queries minimal (Dial-on-Demand)?" and the system is not connected to the Internet while booting.

	Warning
	It is insecure to use plain text password without encryption even if your ISP allows it.

	Tip
	Although use of SMTP with STARTTLS on port 587 is recommended, some ISPs still use deprecated SMTPS (SSL on port 465). Exim4 after 4.77 supports this deprecated SMTPS protocol for both as client and as server.

	Tip
	If you are looking for a light weight MTA that respects "/etc/aliases" for your laptop PC, you should consider to configure exim4(8) with "QUEUERUNNER='queueonly'", "QUEUERUNNER='nodaemon'", etc. in "/etc/default/exim4".

For the Internet mail via smarthost, you should first read postfix documentation and key manual pages.

$ sudo /etc/init.d/postfix stop
$ sudo dpkg-reconfigure postfix

$ sudo postconf -e 'smtp_sender_dependent_authentication = yes'
$ sudo postconf -e 'smtp_sasl_auth_enable = yes'
$ sudo postconf -e 'smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd'
$ sudo postconf -e 'smtp_sasl_type = cyrus'
$ sudo vim /etc/postfix/sasl_passwd

$ cat /etc/postfix/sasl_passwd
[smtp.hostname.dom]:587     username:password
$ sudo postmap hush:/etc/postfix/sasl_passwd

$ sudo /etc/init.d/postfix start

There are a few mail address configuration files for mail transport, delivery and user agents.

The mailname in the "/etc/mailname" file is usually a fully qualified domain name (FQDN) that resolves to one of the host's IP addresses. For the mobile workstation which does not have a hostname with resolvable IP address, set this mailname to the value of "hostname -f". (This is safe choice and works for both exim4-* and postfix.)

	Tip
	The contents of "/etc/mailname" is used by many non-MTA programs for their default behavior. For mutt, set "hostname" and "from" variables in ~/muttrc file to override the mailname value. For programs in the devscripts package, such as bts(1) and dch(1), export environment variables "$DEBFULLNAME" and "$DEBEMAIL" to override it.

Tip

The popularity-contest package normally send mail from root account with FQDN. You need to set MAILFROM in /etc/popularity-contest.conf as described in the /usr/share/popularity-contest/default.conf file. Otherwise, your mail will be rejected by the smarthost SMTP server. Although this is tedious, this approach is safer than rewriting the source address for all mails from root by MTA and should be used for other daemons and cron scripts.

When setting the mailname to "hostname -f", the spoofing of the source mail address via MTA can be realized by the following.

• "/etc/email-addresses" file for exim4(8) as explained in the exim4-config_files(5)

• "/etc/postfix/generic" file for postfix(1) as explained in the generic(5)

For postfix, the following extra steps are needed.

# postmap hash:/etc/postfix/generic
# postconf -e 'smtp_generic_maps = hash:/etc/postfix/generic'
# postfix reload

You can test mail address configuration using the following.

• exim(8) with -brw, -bf, -bF, -bV, … options

• postmap(1) with -q option.

	Tip
	Exim comes with several utility programs such as exiqgrep(8) and exipick(8). See "dpkg -L exim4-base|grep man8/" for available commands.

#
# User configuration file to override /etc/Muttrc
#
# spoof source mail address
set use_from
set hostname=example.dom
set from="Name Surname <[email protected]>"
set signature="~/.signature"

# vim: "gq" to reformat quotes
set editor="vim -c 'set tw=72 et ft=mail'"

# "mutt" goes to Inbox, while "mutt -y" lists mailboxes
set mbox_type=Maildir           # use qmail Maildir format for creating mbox
set mbox=~/Mail                 # keep all mail boxes in $HOME/Mail/
set spoolfile=+Inbox            # mail delivered to $HOME/Mail/Inbox
set record=+Outbox              # save fcc mail to $HOME/Mail/Outbox
set postponed=+Postponed        # keep postponed in $HOME/Mail/postponed
set move=no                     # do not move Inbox items to mbox
set quit=ask-yes                # do not quit by "q" only
set delete=yes                  # always delete w/o asking while exiting
set fcc_clear                   # store fcc as non encrypted

# Mailboxes in Maildir (automatic update)
mailboxes `cd ~/Mail; /bin/ls -1|sed -e 's/^/+/' | tr "\n" " "`
unmailboxes Maillog *.ev-summary

## Default
#set index_format="%4C %Z %{%b %d} %-15.15L (%4l) %s"
## Thread index with senders (collapse)
set index_format="%4C %Z %{%b %d} %-15.15n %?M?(#%03M)&(%4l)? %s"

## Default
#set folder_format="%2C %t %N %F %2l %-8.8u %-8.8g %8s %d %f"
## just folder names
set folder_format="%2C %t %N %f"

Add the following to "/etc/mailcap" or "~/.mailcap" to display HTML mail and MS Word attachments inline.

text/html; lynx -force_html %s; needsterminal;
application/msword; /usr/bin/antiword '%s'; copiousoutput; description="Microsoft Word Text"; nametemplate=%s.doc

	Tip
	Mutt can be used as the IMAP client and the mailbox format converter. You can tag messages with "t", "T", etc. These tagged messages can be copied with ";C" between different mailboxes and deleted with ";d" in one action.

getmail(1) configuration is described in getmail documentation. Here is my set up to access multiple POP3 accounts as user.

Create "/usr/local/bin/getmails" as the following.

#!/bin/sh
set -e
if [ -f $HOME/.getmail/running ]; then
  echo "getmail is already running ... (if not, remove $HOME/.getmail/running)" >&2
  pgrep -l "getmai[l]"
  exit 1
else
  echo "getmail has not been running ... " >&2
fi
if [ -f $HOME/.getmail/stop ]; then
  echo "do not run getmail ... (if not, remove $HOME/.getmail/stop)" >&2
  exit
fi
if [ "x$1" = "x-l" ]; then
  exit
fi
rcfiles="/usr/bin/getmail"
for file in $HOME/.getmail/config/* ; do
  rcfiles="$rcfiles --rcfile $file"
done
date -u > $HOME/.getmail/running
eval "$rcfiles $@"
rm $HOME/.getmail/running

Configure it as the following.

$ sudo chmod 755 /usr/local/bin/getmails
$ mkdir -m 0700 $HOME/.getmail
$ mkdir -m 0700 $HOME/.getmail/config
$ mkdir -m 0700 $HOME/.getmail/log

Create configuration files "$HOME/.getmail/config/pop3_name" for each POP3 accounts as the following.

[retriever]
type = SimplePOP3SSLRetriever
server = pop.example.com
username =  [email protected]
password = <your-password>

[destination]
type = MDA_external
path = /usr/bin/maildrop
unixfrom = True

[options]
verbose = 0
delete = True
delivered_to = False
message_log = ~/.getmail/log/pop3_name.log

Configure it as the following.

$ chmod 0600 $HOME/.getmail/config/*

Schedule "/usr/local/bin/getmails" to run every 15 minutes with cron(8) by executing "sudo crontab -e -u <user_name>" and adding following to user's cron entry.

5,20,35,50 * * * * /usr/local/bin/getmails --quiet

Tip

Problems of POP3 access may not come from getmail. Some popular free POP3 services may be violating the POP3 protocol and their SPAM filter may not be perfect. For example, they may delete messages just after receiving RETR command before receiving DELE command and may quarantined messages into Spam mailbox. You should minimize damages by configuring them to archive accessed messages and not to delete them. See also "Some mail was not downloaded".

maildrop(1) configuration is described in maildropfilter documentation. Here is a configuration example for "$HOME/.mailfilter".

# Local configuration
MAILROOT="$HOME/Mail"
# set this to /etc/mailname contents
MAILHOST="example.dom"
logfile $HOME/.maildroplog

# rules are made to override the earlier value by the later one.

# mailing list mails ?
if (     /^Precedence:.*list/:h || /^Precedence:.*bulk/:h )
{
    # rules for mailing list mails
    # default mailbox for mails from mailing list
    MAILBOX="Inbox-list"
    # default mailbox for mails from debian.org
    if ( /^(Sender|Resent-From|Resent-Sender): .*debian.org/:h )
    {
        MAILBOX="service.debian.org"
    }
    # default mailbox for mails from bugs.debian.org (BTS)
    if ( /^(Sender|Resent-From|Resent-sender): .*@bugs.debian.org/:h )
    {
        MAILBOX="bugs.debian.org"
    }
    # mailbox for each properly maintained mailing list with "List-Id: foo" or "List-Id: ...<foo.bar>"
    if ( /^List-Id: ([^<]*<)?([^<>]*)>?/:h )
    {
        MAILBOX="$MATCH2"
    }
}
else
{
    # rules for non-mailing list mails
    # default incoming box
    MAILBOX="Inbox-unusual"
    # local mails
    if ( /Envelope-to: .*@$MAILHOST/:h )
    {
        MAILBOX="Inbox-local"
    }
    # html mails (99% spams)
    if ( /DOCTYPE html/:b ||\
         /^Content-Type: text\/html/ )
    {
        MAILBOX="Inbox-html"
    }
    # blacklist rule for spams
    if ( /^X-Advertisement/:h ||\
         /^Subject:.*BUSINESS PROPOSAL/:h ||\
         /^Subject:.*URGENT.*ASISSTANCE/:h ||\
         /^Subject: *I NEED YOUR ASSISTANCE/:h )
    {
        MAILBOX="Inbox-trash"
    }
    # whitelist rule for normal mails
    if ( /^From: .*@debian.org/:h ||\
         /^(Sender|Resent-From|Resent-Sender): .*debian.org/:h ||\
         /^Subject: .*(debian|bug|PATCH)/:h )
    {
        MAILBOX="Inbox"
    }
    # whiltelist rule for BTS related mails
    if ( /^Subject: .*Bug#.*/:h ||\
         /^(To|Cc): .*@bugs.debian.org/:h )
    {
        MAILBOX="bugs.debian.org"
    }
    # whitelist rule for getmails cron mails
    if ( /^Subject: Cron .*getmails/:h )
    {
        MAILBOX="Inbox-getmails"
    }
}

# check existance of $MAILBOX
`test -d $MAILROOT/$MAILBOX`
if ( $RETURNCODE == 1 )
{
    # create maildir mailbox for $MAILBOX
    `maildirmake $MAILROOT/$MAILBOX`
}
# deliver to maildir $MAILBOX
to "$MAILROOT/$MAILBOX/"
exit

	Warning
	Unlike procmail, maildrop does not create missing maildir directories automatically. You must create them manually using maildirmake(1) in advance as in the example "$HOME/.mailfilter".

MAILDIR=$HOME/Maildir
DEFAULT=$MAILDIR/Inbox/
LOGFILE=$MAILDIR/Maillog
# clearly bad looking mails: drop them into X-trash and exit
:0
* 1^0 ^X-Advertisement
* 1^0 ^Subject:.*BUSINESS PROPOSAL
* 1^0 ^Subject:.*URGENT.*ASISSTANCE
* 1^0 ^Subject: *I NEED YOUR ASSISTANCE
X-trash/

# Delivering mailinglist messages
:0
* 1^0 ^Precedence:.*list
* 1^0 ^Precedence:.*bulk
* 1^0 ^List-
* 1^0 ^X-Distribution:.*bulk
{
:0
* 1^0 ^Return-path:.*[email protected]
jp-debian-devel/

:0
* ^Resent-Sender.*[email protected]
debian-user/

:0
* ^Resent-Sender.*[email protected]
debian-devel/

:0
* ^Resent-Sender.*[email protected]
debian-announce

:0
mailing-list/
}

:0
Inbox/

	Warning
	"/etc/ssh/sshd_not_to_be_run" must not be present if one wishes to run the OpenSSH server.

SSH has two authentication protocols.

	Caution
	Be careful about these differences if you are using a non-Debian system.

See "/usr/share/doc/ssh/README.Debian.gz", ssh(1), sshd(8), ssh-agent(1), and ssh-keygen(1) for details.

Following are the key configuration files.

	Tip
	See ssh-keygen(1), ssh-add(1) and ssh-agent(1) for how to use public and secret SSH keys.

	Tip
	Make sure to verify settings by testing the connection. In case of any problem, use "ssh -v".

	Tip
	You can change the pass phrase to encrypt local secret SSH keys later with "ssh-keygen -p".

	Tip
	You can add options to the entries in "~/.ssh/authorized_keys" to limit hosts and to run specific commands. See sshd(8) for details.

The following starts an ssh(1) connection from a client.

If you use the same user name on the local and the remote host, you can eliminate typing "username@". Even if you use different user name on the local and the remote host, you can eliminate it using "~/.ssh/config". For Debian Alioth service with account name "foo-guest", you set "~/.ssh/config" to contain the following.

Host alioth.debian.org svn.debian.org git.debian.org
    User foo-guest

For the user, ssh(1) functions as a smarter and more secure telnet(1). Unlike telnet command, ssh command does not stop on the telnet escape character (initial default CTRL-]).

There are some free SSH clients available for other platforms.

You need to protect the process doing "shutdown -h now" (see Section 1.1.8, “How to shutdown the system”) from the termination of SSH using the at(1) command (see Section 9.3.13, “Scheduling tasks once”) by the following.

# echo "shutdown -h now" | at now

Running "shutdown -h now" in screen(1) (see Section 9.1, “The screen program”) session is another way to do the same.