1.1 How to Run awk Programs

There are several ways to run an awk program. If the program is short, it is easiest to include it in the command that runs awk, like this:

awk 'program' input-file1 input-file2 …

When the program is long, it is usually more convenient to put it in a file and run it with a command like this:

awk -f program-file input-file1 input-file2 …

This section discusses both mechanisms, along with several variations of each.

• One-Shot Throwaway awk Programs
• Running awk Without Input Files
• Running Long Programs
• Executable awk Programs
• Comments in awk Programs
• Shell Quoting Issues

awk 'program' input-file1 input-file2 …

awk 'program'

$ awk 'BEGIN { print "Don\47t Panic!" }'
-| Don't Panic!

$ awk '{ print }'
Now is the time for all good men
-| Now is the time for all good men
to come to the aid of their country.
-| to come to the aid of their country.
Four score and seven years ago, ...
-| Four score and seven years ago, ...
What, me worry?
-| What, me worry?
Ctrl-d

awk -f source-file input-file1 input-file2 …

BEGIN { print "Don't Panic!" }

awk -f advice

awk 'BEGIN { print "Don\47t Panic!" }'

#! /bin/awk -f
BEGIN { print "Don't Panic!" }

$ chmod +x advice
$ ./advice
-| Don't Panic!

# This program prints a nice, friendly message.  It helps
# keep novice users from being afraid of the computer.
BEGIN    { print "Don't Panic!" }

awk 'program text' input-file1 input-file2 …

$ echo ""

$ awk 'BEGIN { print "Here is a single quote <'"'"'>" }'
-| Here is a single quote <'>

$ awk 'BEGIN { print "Here is a single quote <'\''>" }'
-| Here is a single quote <'>

$ awk "BEGIN { print \"Here is a single quote <'>\" }"
-| Here is a single quote <'>

$ awk 'BEGIN { print "Here is a single quote <\47>" }'
-| Here is a single quote <'>
$ awk 'BEGIN { print "Here is a double quote <\42>" }'
-| Here is a double quote <">

$ awk -v sq="'" 'BEGIN { print "Here is a single quote <" sq ">" }'
-| Here is a single quote <'>

{ print "\"" $0 "\"" }

gawk "{ print \"\042\" $0 \"\042\" }" file

gawk "{ print \"\\\"\" $0 \"\\\"\" }" file

1.2 Data files for the Examples

Many of the examples in this Web page take their input from two sample data files. The first, mail-list, represents a list of peoples’ names together with their email addresses and information about those people. The second data file, called inventory-shipped, contains information about monthly shipments. In both files, each line is considered to be one record.

In mail-list, each record contains the name of a person, his/her phone number, his/her email address, and a code for his/her relationship with the author of the list. The columns are aligned using spaces. An ‘A’ in the last column means that the person is an acquaintance. An ‘F’ in the last column means that the person is a friend. An ‘R’ means that the person is a relative:

Amelia       555-5553     [email protected]    F
Anthony      555-3412     [email protected]   A
Becky        555-7685     [email protected]      A
Bill         555-1675     [email protected]       A
Broderick    555-0542     [email protected] R
Camilla      555-2912     [email protected]     R
Fabius       555-1234     [email protected]    F
Julie        555-6699     [email protected]   F
Martin       555-6480     [email protected]    A
Samuel       555-3430     [email protected]        A
Jean-Paul    555-2127     [email protected]     R

The data file inventory-shipped represents information about shipments during the year. Each record contains the month, the number of green crates shipped, the number of red boxes shipped, the number of orange bags shipped, and the number of blue packages shipped, respectively. There are 16 entries, covering the 12 months of last year and the first four months of the current year. An empty line separates the data for the two years:

Jan  13  25  15 115
Feb  15  32  24 226
Mar  15  24  34 228
Apr  31  52  63 420
May  16  34  29 208
Jun  31  42  75 492
Jul  24  34  67 436
Aug  15  34  47 316
Sep  13  55  37 277
Oct  29  54  68 525
Nov  20  87  82 577
Dec  17  35  61 401
Jan  21  36  64 620
Feb  26  58  80 652
Mar  24  75  70 495
Apr  21  70  74 514

The sample files are included in the gawk distribution, in the directory awklib/eg/data.

1.3 Some Simple Examples

The following command runs a simple awk program that searches the input file mail-list for the character string ‘li’ (a grouping of characters is usually called a string; the term string is based on similar usage in English, such as “a string of pearls” or “a string of cars in a train”):

awk '/li/ { print $0 }' mail-list

When lines containing ‘li’ are found, they are printed because ‘print $0’ means print the current line. (Just ‘print’ by itself means the same thing, so we could have written that instead.)

You will notice that slashes (‘/’) surround the string ‘li’ in the awk program. The slashes indicate that ‘li’ is the pattern to search for. This type of pattern is called a regular expression, which is covered in more detail later (see Regular Expressions). The pattern is allowed to match parts of words. There are single quotes around the awk program so that the shell won’t interpret any of it as special shell characters.

Here is what this program prints:

$ awk '/li/ { print $0 }' mail-list
-| Amelia       555-5553     [email protected]    F
-| Broderick    555-0542     [email protected] R
-| Julie        555-6699     [email protected]   F
-| Samuel       555-3430     [email protected]        A

In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.

Thus, we could leave out the action (the print statement and the braces) in the previous example and the result would be the same: awk prints all lines matching the pattern ‘li’. By comparison, omitting the print statement but retaining the braces makes an empty action that does nothing (i.e., no lines are printed).

Many practical awk programs are just a line or two long. Following is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven’t been covered yet. (The description of the program will give you a good idea of what is going on, but you’ll need to read the rest of the Web page to become an awk expert!) Most of the examples use a data file named data. This is just a placeholder; if you use these programs yourself, substitute your own file names for data.

Some of the following examples use the output of ‘ls -l’ as input. ls is a system command that gives you a listing of the files in a directory. With the -l option, this listing includes each file’s size and the date the file was last modified. Its output looks like this:

-rw-r--r--  1 arnold   user   1933 Nov  7 13:05 Makefile
-rw-r--r--  1 arnold   user  10809 Nov  7 13:03 awk.h
-rw-r--r--  1 arnold   user    983 Apr 13 12:14 awk.tab.h
-rw-r--r--  1 arnold   user  31869 Jun 15 12:20 awkgram.y
-rw-r--r--  1 arnold   user  22414 Nov  7 13:03 awk1.c
-rw-r--r--  1 arnold   user  37455 Nov  7 13:03 awk2.c
-rw-r--r--  1 arnold   user  27511 Dec  9 13:07 awk3.c
-rw-r--r--  1 arnold   user   7989 Nov  7 13:03 awk4.c

The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the file’s owner. The fourth field identifies the file’s group. The fifth field contains the file’s size in bytes. The sixth, seventh, and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the file name.

For future reference, note that there is often more than one way to do things in awk. At some point, you may want to look back at these examples and see if you can come up with different ways to do the same things shown here:

• Print every line that is longer than 80 characters:

  awk 'length($0) > 80' data
  

  The sole rule has a relational expression as its pattern and has no action—so it uses the default action, printing the record.

• Print the length of the longest input line:

  awk '{ if (length($0) > max) max = length($0) }
       END { print max }' data
  

  The code associated with END executes after all input has been read; it’s the other side of the coin to BEGIN.

• Print the length of the longest line in data:

  expand data | awk '{ if (x < length($0)) x = length($0) }
                     END { print "maximum line length is " x }'
  

  This example differs slightly from the previous one: the input is processed by the expand utility to change TABs into spaces, so the widths compared are actually the right-margin columns, as opposed to the number of input characters on each line.

• Print every line that has at least one field:

  awk 'NF > 0' data
  

  This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed).

• Print seven random numbers from 0 to 100, inclusive:

  awk 'BEGIN { for (i = 1; i <= 7; i++)
                   print int(101 * rand()) }'
  

• Print the total number of bytes used by files:

  ls -l files | awk '{ x += $5 }
                     END { print "total bytes: " x }'
  

• Print the total number of kilobytes used by files:

  ls -l files | awk '{ x += $5 }
     END { print "total K-bytes:", x / 1024 }'
  

• Print a sorted list of the login names of all users:

  awk -F: '{ print $1 }' /etc/passwd | sort
  

• Count the lines in a file:

  awk 'END { print NR }' data
  

• Print the even-numbered lines in the data file:

  awk 'NR % 2 == 0' data
  

  If you used the expression ‘NR % 2 == 1’ instead, the program would print the odd-numbered lines.

1.4 An Example with Two Rules

The awk utility reads the input files one line at a time. For each line, awk tries the patterns of each rule. If several patterns match, then several actions execute in the order in which they appear in the awk program. If no patterns match, then no actions run.

After processing all the rules that match the line (and perhaps there are none), awk reads the next line. (However, see The next Statement and also see The nextfile Statement.) This continues until the program reaches the end of the file. For example, the following awk program contains two rules:

/12/  { print $0 }
/21/  { print $0 }

The first rule has the string ‘12’ as the pattern and ‘print $0’ as the action. The second rule has the string ‘21’ as the pattern and also has ‘print $0’ as the action. Each rule’s action is enclosed in its own pair of braces.

This program prints every line that contains the string ‘12’ or the string ‘21’. If a line contains both strings, it is printed twice, once by each rule.

This is what happens if we run this program on our two sample data files, mail-list and inventory-shipped:

$ awk '/12/ { print $0 }
>      /21/ { print $0 }' mail-list inventory-shipped
-| Anthony      555-3412     [email protected]   A
-| Camilla      555-2912     [email protected]     R
-| Fabius       555-1234     [email protected]    F
-| Jean-Paul    555-2127     [email protected]     R
-| Jean-Paul    555-2127     [email protected]     R
-| Jan  21  36  64 620
-| Apr  21  70  74 514

Note how the line beginning with ‘Jean-Paul’ in mail-list was printed twice, once for each rule.

1.5 A More Complex Example

Now that we’ve mastered some simple tasks, let’s look at what typical awk programs do. This example shows how awk can be used to summarize, select, and rearrange the output of another utility. It uses features that haven’t been covered yet, so don’t worry if you don’t understand all the details:

ls -l | awk '$6 == "Nov" { sum += $5 }
             END { print sum }'

This command prints the total number of bytes in all the files in the current directory that were last modified in November (of any year).

As a reminder, the output of ‘ls -l’ gives you a listing of the files in a directory, including each file’s size and the date the file was last modified. The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the file’s owner. The fourth field identifies the file’s group. The fifth field contains the file’s size in bytes. The sixth, seventh, and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the file name.

The ‘$6 == "Nov"’ in our awk program is an expression that tests whether the sixth field of the output from ‘ls -l’ matches the string ‘Nov’. Each time a line has the string ‘Nov’ for its sixth field, awk performs the action ‘sum += $5’. This adds the fifth field (the file’s size) to the variable sum. As a result, when awk has finished reading all the input lines, sum is the total of the sizes of the files whose lines matched the pattern. (This works because awk variables are automatically initialized to zero.)

After the last line of output from ls has been processed, the END rule executes and prints the value of sum. In this example, the value of sum is 80600.

These more advanced awk techniques are covered in later sections (see Actions). Before you can move on to more advanced awk programming, you have to know how awk interprets your input and displays your output. By manipulating fields and using print statements, you can produce some very useful and impressive-looking reports.

1.6 awk Statements Versus Lines

Most often, each line in an awk program is a separate statement or separate rule, like this:

awk '/12/  { print $0 }
     /21/  { print $0 }' mail-list inventory-shipped

However, gawk ignores newlines after any of the following symbols and keywords:

,    {    ?    :    ||    &&    do    else

A newline at any other point is considered the end of the statement.⁹

If you would like to split a single statement into two lines at a point where a newline would terminate it, you can continue it by ending the first line with a backslash character (‘\’). The backslash must be the final character on the line in order to be recognized as a continuation character. A backslash followed by a newline is allowed anywhere in the statement, even in the middle of a string or regular expression. For example:

awk '/This regular expression is too long, so continue it\
 on the next line/ { print $1 }'

We have generally not used backslash continuation in our sample programs. gawk places no limit on the length of a line, so backslash continuation is never strictly necessary; it just makes programs more readable. For this same reason, as well as for clarity, we have kept most statements short in the programs presented throughout the Web page.

Backslash continuation is most useful when your awk program is in a separate source file instead of entered from the command line. You should also note that many awk implementations are more particular about where you may use backslash continuation. For example, they may not allow you to split a string constant using backslash continuation. Thus, for maximum portability of your awk programs, it is best not to split your lines in the middle of a regular expression or a string.

CAUTION: Backslash continuation does not work as described with the C shell. It works for awk programs in files and for one-shot programs, provided you are using a POSIX-compliant shell, such as the Unix Bourne shell or Bash. But the C shell behaves differently! There you must use two backslashes in a row, followed by a newline. Note also that when using the C shell, every newline in your awk program must be escaped with a backslash. To illustrate:

% awk 'BEGIN { \
?   print \\
?       "hello, world" \
? }'
-| hello, world

Here, the ‘%’ and ‘?’ are the C shell’s primary and secondary prompts, analogous to the standard shell’s ‘$’ and ‘>’.

Compare the previous example to how it is done with a POSIX-compliant shell:

$ awk 'BEGIN {
>   print \
>       "hello, world"
> }'
-| hello, world

awk is a line-oriented language. Each rule’s action has to begin on the same line as the pattern. To have the pattern and action on separate lines, you must use backslash continuation; there is no other option.

Another thing to keep in mind is that backslash continuation and comments do not mix. As soon as awk sees the ‘#’ that starts a comment, it ignores everything on the rest of the line. For example:

$ gawk 'BEGIN { print "dont panic" # a friendly \
>                                    BEGIN rule
> }'
error→ gawk: cmd. line:2:                BEGIN rule
error→ gawk: cmd. line:2:                ^ syntax error

In this case, it looks like the backslash would continue the comment onto the next line. However, the backslash-newline combination is never even noticed because it is “hidden” inside the comment. Thus, the BEGIN is noted as a syntax error.

If you’re interested, see https://lists.gnu.org/archive/html/bug-gawk/2022-10/msg00025.html for a source code patch that allows lines to be continued when inside parentheses. This patch was not added to gawk since it would quietly decrease the portability of awk programs.

When awk statements within one rule are short, you might want to put more than one of them on a line. This is accomplished by separating the statements with a semicolon (‘;’). This also applies to the rules themselves. Thus, the program shown at the start of this section could also be written this way:

/12/ { print $0 } ; /21/ { print $0 }

NOTE: The requirement that states that rules on the same line must be separated with a semicolon was not in the original awk language; it was added for consistency with the treatment of statements within an action.

1.7 Other Features of awk

The awk language provides a number of predefined, or built-in, variables that your programs can use to get information from awk. There are other variables your program can set as well to control how awk processes your data.

In addition, awk provides a number of built-in functions for doing common computational and string-related operations. gawk provides built-in functions for working with timestamps, performing bit manipulation, for runtime string translation (internationalization), determining the type of a variable, and array sorting.

As we develop our presentation of the awk language, we will introduce most of the variables and many of the functions. They are described systematically in Predefined Variables and in Built-in Functions.

1.8 When to Use awk

Now that you’ve seen some of what awk can do, you might wonder how awk could be useful for you. By using utility programs, advanced patterns, field separators, arithmetic statements, and other selection criteria, you can produce much more complex output. The awk language is very useful for producing reports from large amounts of raw data, such as summarizing information from the output of other utility programs like ls. (See A More Complex Example.)

Programs written with awk are usually much smaller than they would be in other languages. This makes awk programs easy to compose and use. Often, awk programs can be quickly composed at your keyboard, used once, and thrown away. Because awk programs are interpreted, you can avoid the (usually lengthy) compilation part of the typical edit-compile-test-debug cycle of software development.

Complex programs have been written in awk, including a complete retargetable assembler for eight-bit microprocessors (see Glossary, for more information), and a microcode assembler for a special-purpose Prolog computer. The original awk’s capabilities were strained by tasks of such complexity, but modern versions are more capable.

If you find yourself writing awk scripts of more than, say, a few hundred lines, you might consider using a different programming language. The shell is good at string and pattern matching; in addition, it allows powerful use of the system utilities. Python offers a nice balance between high-level ease of programming and access to system facilities.¹⁰

1.9 Summary

• Programs in awk consist of pattern–action pairs.
• An action without a pattern always runs. The default action for a pattern without one is ‘{ print $0 }’.
• Use either ‘awk 'program' files’ or ‘awk -f program-file files’ to run awk.
• You may use the special ‘#!’ header line to create awk programs that are directly executable.
• Comments in awk programs start with ‘#’ and continue to the end of the same line.
• Be aware of quoting issues when writing awk programs as part of a larger shell script (or MS-Windows batch file).
• You may use backslash continuation to continue a source line. Lines are automatically continued after a comma, open brace, question mark, colon, ‘||’, ‘&&’, do, and else.

2.1 Invoking awk

There are two ways to run awk—with an explicit program or with one or more program files. Here are templates for both of them; items enclosed in […] in these templates are optional:

awk [options] -f progfile [--] file …
awk [options] [--] 'program' file …

In addition to traditional one-letter POSIX-style options, gawk also supports GNU long options.

It is possible to invoke awk with an empty program:

awk '' datafile1 datafile2

Doing so makes little sense, though; awk exits silently when given an empty program. (d.c.) If --lint has been specified on the command line, gawk issues a warning that the program is empty.

2.2 Command-Line Options

Options begin with a dash and consist of a single character. GNU-style long options consist of two dashes and a keyword. The keyword can be abbreviated, as long as the abbreviation allows the option to be uniquely identified. If the option takes an argument, either the keyword is immediately followed by an equals sign (‘=’) and the argument’s value, or the keyword and the argument’s value are separated by whitespace (spaces or TABs). If a particular option with a value is given more than once, it is (usually) the last value that counts.

Each long option for gawk has a corresponding POSIX-style short option. The long and short options are interchangeable in all contexts. The following list describes options mandated by the POSIX standard:

-F fs ¶

--field-separator fs

Set the FS variable to fs (see Specifying How Fields Are Separated).

-f source-file ¶

--file source-file

Read the awk program source from source-file instead of in the first nonoption argument. This option may be given multiple times; the awk program consists of the concatenation of the contents of each specified source-file.

Files named with -f are treated as if they had ‘@namespace "awk"’ at their beginning. See Changing The Namespace, for more information on this advanced feature.

-v var=val ¶

--assign var=val

Set the variable var to the value val before execution of the program begins. Such variable values are available inside the BEGIN rule (see Other Command-Line Arguments).

The -v option can only set one variable, but it can be used more than once, setting another variable each time, like this: ‘awk -v foo=1 -v bar=2 …’.

CAUTION: Using -v to set the values of the built-in variables may lead to surprising results. awk will reset the values of those variables as it needs to, possibly ignoring any initial value you may have given.

-W gawk-opt ¶

Provide an implementation-specific option. This is the POSIX convention for providing implementation-specific options. These options also have corresponding GNU-style long options. Note that the long options may be abbreviated, as long as the abbreviations remain unique. The full list of gawk-specific options is provided next.

-- ¶

Signal the end of the command-line options. The following arguments are not treated as options even if they begin with ‘-’. This interpretation of -- follows the POSIX argument parsing conventions.

This is useful if you have file names that start with ‘-’, or in shell scripts, if you have file names that will be specified by the user that could start with ‘-’. It is also useful for passing options on to the awk program; see Processing Command-Line Options.

The following list describes gawk-specific options:

-b ¶

--characters-as-bytes

Cause gawk to treat all input data as single-byte characters. In addition, all output written with print or printf is treated as single-byte characters.

Normally, gawk follows the POSIX standard and attempts to process its input data according to the current locale (see Where You Are Makes a Difference). This can often involve converting multibyte characters into wide characters (internally), and can lead to problems or confusion if the input data does not contain valid multibyte characters. This option is an easy way to tell gawk, “Hands off my data!”

-c ¶

--traditional

Specify compatibility mode, in which the GNU extensions to the awk language are disabled, so that gawk behaves just like BWK awk. See Extensions in gawk Not in POSIX awk, which summarizes the extensions. Also see Downward Compatibility and Debugging.

-C ¶

--copyright

Print the short version of the General Public License and then exit.

-d[file] ¶

--dump-variables[=file]

Print a sorted list of global variables, their types, and final values to file. If no file is provided, print this list to a file named awkvars.out in the current directory. No space is allowed between the -d and file, if file is supplied.

Having a list of all global variables is a good way to look for typographical errors in your programs. You would also use this option if you have a large program with a lot of functions, and you want to be sure that your functions don’t inadvertently use global variables that you meant to be local. (This is a particularly easy mistake to make with simple variable names like i, j, etc.)

-D[file] ¶

--debug[=file]

Enable debugging of awk programs (see Introduction to the gawk Debugger). By default, the debugger reads commands interactively from the keyboard (standard input). The optional file argument allows you to specify a file with a list of commands for the debugger to execute noninteractively. No space is allowed between the -D and file, if file is supplied.

-e program-text ¶

--source program-text

Provide program source code in the program-text. This option allows you to mix source code in files with source code that you enter on the command line. This is particularly useful when you have library functions that you want to use from your command-line programs (see The AWKPATH Environment Variable).

Note that gawk treats each string as if it ended with a newline character (even if it doesn’t). This makes building the total program easier.

CAUTION: Prior to version 5.0, there was no requirement that each program-text be a full syntactic unit. I.e., the following worked:

$ gawk -e 'BEGIN { a = 5 ;' -e 'print a }'
-| 5

However, this is no longer true. If you have any scripts that rely upon this feature, you should revise them.

This is because each program-text is treated as if it had ‘@namespace "awk"’ at its beginning. See Changing The Namespace, for more information.

-E file ¶

--exec file

Similar to -f, read awk program text from file. There are two differences from -f:

• This option terminates option processing; anything else on the command line is passed on directly to the awk program.
• Command-line variable assignments of the form ‘var=value’ are disallowed.

This option is particularly necessary for World Wide Web CGI applications that pass arguments through the URL; using this option prevents a malicious (or other) user from passing in options, assignments, or awk source code (via -e) to the CGI application.¹¹ This option should be used with ‘#!’ scripts (see Executable awk Programs), like so:

#! /usr/local/bin/gawk -E
awk program here …

-g ¶

--gen-pot

Analyze the source program and generate a GNU gettext portable object template file on standard output for all string constants that have been marked for translation. See Internationalization with gawk, for information about this option.

-h ¶

--help

Print a “usage” message summarizing the short- and long-style options that gawk accepts and then exit.

-i source-file ¶

--include source-file

Read an awk source library from source-file. This option is completely equivalent to using the @include directive inside your program. It is very similar to the -f option, but there are two important differences. First, when -i is used, the program source is not loaded if it has been previously loaded, whereas with -f, gawk always loads the file. Second, because this option is intended to be used with code libraries, gawk does not recognize such files as constituting main program input. Thus, after processing an -i argument, gawk still expects to find the main source code via the -f option or on the command line.

Files named with -i are treated as if they had ‘@namespace "awk"’ at their beginning. See Changing The Namespace, for more information.

-I ¶

--trace

Print the internal byte code names as they are executed when running the program. The trace is printed to standard error. Each “op code” is preceded by a + sign in the output.

-l ext ¶

--load ext

Load a dynamic extension named ext. Extensions are stored as system shared libraries. This option searches for the library using the AWKLIBPATH environment variable. The correct library suffix for your platform will be supplied by default, so it need not be specified in the extension name. The extension initialization routine should be named dl_load(). An alternative is to use the @load keyword inside the program to load a shared library. This advanced feature is described in detail in Writing Extensions for gawk.

-L[value] ¶

--lint[=value]

Warn about constructs that are dubious or nonportable to other awk implementations. No space is allowed between the -L and value, if value is supplied. Some warnings are issued when gawk first reads your program. Others are issued at runtime, as your program executes. The optional argument may be one of the following:

fatal

Cause lint warnings become fatal errors. This may be drastic, but its use will certainly encourage the development of cleaner awk programs.

invalid

Only issue warnings about things that are actually invalid are issued. (This is not fully implemented yet.)

no-ext

Disable warnings about gawk extensions.

Some warnings are only printed once, even if the dubious constructs they warn about occur multiple times in your awk program. Thus, when eliminating problems pointed out by --lint, you should take care to search for all occurrences of each inappropriate construct. As awk programs are usually short, doing so is not burdensome.

-M ¶

--bignum

Select arbitrary-precision arithmetic on numbers. This option has no effect if gawk is not compiled to use the GNU MPFR and MP libraries (see Arithmetic and Arbitrary-Precision Arithmetic with gawk).

As of version 5.2, the arbitrary precision arithmetic features in gawk are “on parole.” The primary maintainer is no longer willing to support this feature, but another member of the development team has stepped up to take it over. As long as this situation remains stable, MPFR will be supported. If it changes, the MPFR support will be removed from gawk.

-n ¶

--non-decimal-data

Enable automatic interpretation of octal and hexadecimal values in input data (see Allowing Nondecimal Input Data).

CAUTION: This option can severely break old programs. Use with care. Also note that this option may disappear in a future version of gawk.

-N ¶

--use-lc-numeric

Force the use of the locale’s decimal point character when parsing numeric input data (see Where You Are Makes a Difference).

-o[file] ¶

--pretty-print[=file]

Enable pretty-printing of awk programs. Implies --no-optimize. By default, the output program is created in a file named awkprof.out (see Profiling Your awk Programs). The optional file argument allows you to specify a different file name for the output. No space is allowed between the -o and file, if file is supplied.

NOTE: In the past, this option would also execute your program. This is no longer the case.

-O ¶

--optimize

Enable gawk’s default optimizations on the internal representation of the program. At the moment, this includes just simple constant folding.

Optimization is enabled by default. This option remains primarily for backwards compatibility. However, it may be used to cancel the effect of an earlier -s option (see later in this list).

-p[file] ¶

--profile[=file]

Enable profiling of awk programs (see Profiling Your awk Programs). Implies --no-optimize. By default, profiles are created in a file named awkprof.out. The optional file argument allows you to specify a different file name for the profile file. No space is allowed between the -p and file, if file is supplied.

The profile contains execution counts for each statement in the program in the left margin, and function call counts for each function.

-P ¶

--posix

Operate in strict POSIX mode. This disables all gawk extensions (just like --traditional) and disables all extensions not allowed by POSIX. See Common Extensions Summary for a summary of the extensions in gawk that are disabled by this option. Also, the following additional restrictions apply:

• Newlines are not allowed after ‘?’ or ‘:’ (see Conditional Expressions).
• Specifying ‘-Ft’ on the command line does not set the value of FS to be a single TAB character (see Specifying How Fields Are Separated).
• The locale’s decimal point character is used for parsing input data (see Where You Are Makes a Difference).

If you supply both --traditional and --posix on the command line, --posix takes precedence. gawk issues a warning if both options are supplied.

-r ¶

--re-interval

Allow interval expressions (see Regular Expression Operators) in regexps. This is now gawk’s default behavior. Nevertheless, this option remains for backward compatibility.

-s ¶

--no-optimize

Disable gawk’s default optimizations on the internal representation of the program.

-S ¶

--sandbox

Disable the system() function, input redirections with getline, output redirections with print and printf, and dynamic extensions. Also, disallow adding file names to ARGV that were not there when gawk started running. This is particularly useful when you want to run awk scripts from questionable sources and need to make sure the scripts can’t access your system (other than the specified input data files).

-t ¶

--lint-old

Warn about constructs that are not available in the original version of awk from Version 7 Unix (see Major Changes Between V7 and SVR3.1).

-V ¶

--version

Print version information for this particular copy of gawk. This allows you to determine if your copy of gawk is up to date with respect to whatever the Free Software Foundation is currently distributing. It is also useful for bug reports (see Reporting Problems and Bugs).

--

Mark the end of all options. Any command-line arguments following -- are placed in ARGV, even if they start with a minus sign.

In compatibility mode, as long as program text has been supplied, any other options are flagged as invalid with a warning message but are otherwise ignored.

In compatibility mode, as a special case, if the value of fs supplied to the -F option is ‘t’, then FS is set to the TAB character ("\t"). This is true only for --traditional and not for --posix (see Specifying How Fields Are Separated).

The -f option may be used more than once on the command line. If it is, awk reads its program source from all of the named files, as if they had been concatenated together into one big file. This is useful for creating libraries of awk functions. These functions can be written once and then retrieved from a standard place, instead of having to be included in each individual program. The -i option is similar in this regard. (As mentioned in Function Definition Syntax, function names must be unique.)

With standard awk, library functions can still be used, even if the program is entered at the keyboard, by specifying ‘-f /dev/tty’. After typing your program, type Ctrl-d (the end-of-file character) to terminate it. (You may also use ‘-f -’ to read program source from the standard input, but then you will not be able to also use the standard input as a source of data.)

Because it is clumsy using the standard awk mechanisms to mix source file and command-line awk programs, gawk provides the -e option. This does not require you to preempt the standard input for your source code, and it allows you to easily mix command-line and library source code (see The AWKPATH Environment Variable). As with -f, the -e and -i options may also be used multiple times on the command line.

If no -f option (or -e option for gawk) is specified, then awk uses the first nonoption command-line argument as the text of the program source code. Arguments on the command line that follow the program text are entered into the ARGV array; awk does not continue to parse the command line looking for options.

If the environment variable POSIXLY_CORRECT exists, then gawk behaves in strict POSIX mode, exactly as if you had supplied --posix. Many GNU programs look for this environment variable to suppress extensions that conflict with POSIX, but gawk behaves differently: it suppresses all extensions, even those that do not conflict with POSIX, and behaves in strict POSIX mode. If --lint is supplied on the command line and gawk turns on POSIX mode because of POSIXLY_CORRECT, then it issues a warning message indicating that POSIX mode is in effect. You would typically set this variable in your shell’s startup file. For a Bourne-compatible shell (such as Bash), you would add these lines to the .profile file in your home directory:

POSIXLY_CORRECT=true
export POSIXLY_CORRECT

For a C shell-compatible shell,¹² you would add this line to the .login file in your home directory:

setenv POSIXLY_CORRECT true

Having POSIXLY_CORRECT set is not recommended for daily use, but it is good for testing the portability of your programs to other environments.

2.3 Other Command-Line Arguments

Any additional arguments on the command line are normally treated as input files to be processed in the order specified. However, an argument that has the form var=value, assigns the value value to the variable var—it does not specify a file at all. (See Assigning Variables on the Command Line.) In the following example, ‘count=1’ is a variable assignment, not a file name:

awk -f program.awk file1 count=1 file2

As a side point, should you really need to have awk process a file named count=1 (or any file whose name looks like a variable assignment), precede the file name with ‘./’, like so:

awk -f program.awk file1 ./count=1 file2

All the command-line arguments are made available to your awk program in the ARGV array (see Predefined Variables). Command-line options and the program text (if present) are omitted from ARGV. All other arguments, including variable assignments, are included. As each element of ARGV is processed, gawk sets ARGIND to the index in ARGV of the current element. (gawk makes the full command line, including program text and options, available in PROCINFO["argv"]; see Built-in Variables That Convey Information.)

Changing ARGC and ARGV in your awk program lets you control how awk processes the input files; this is described in more detail in Using ARGC and ARGV.

The distinction between file name arguments and variable-assignment arguments is made when awk is about to open the next input file. At that point in execution, it checks the file name to see whether it is really a variable assignment; if so, awk sets the variable instead of reading a file.

Therefore, the variables actually receive the given values after all previously specified files have been read. In particular, the values of variables assigned in this fashion are not available inside a BEGIN rule (see The BEGIN and END Special Patterns), because such rules are run before awk begins scanning the argument list.

The variable values given on the command line are processed for escape sequences (see Escape Sequences). (d.c.)

In some very early implementations of awk, when a variable assignment occurred before any file names, the assignment would happen before the BEGIN rule was executed. awk’s behavior was thus inconsistent; some command-line assignments were available inside the BEGIN rule, while others were not. Unfortunately, some applications came to depend upon this “feature.” When awk was changed to be more consistent, the -v option was added to accommodate applications that depended upon the old behavior.

The variable assignment feature is most useful for assigning to variables such as RS, OFS, and ORS, which control input and output formats, before scanning the data files. It is also useful for controlling state if multiple passes are needed over a data file. For example:

awk 'pass == 1  { pass 1 stuff }
     pass == 2  { pass 2 stuff }' pass=1 mydata pass=2 mydata

Given the variable assignment feature, the -F option for setting the value of FS is not strictly necessary. It remains for historical compatibility.

f=""
awk '{ print("hi") }' $f

$ f=""
$ awk 'BEGIN { print ARGC }' $f
-| 1
$ awk 'BEGIN { print ARGC }' "$f"
-| 2

2.4 Naming Standard Input

Often, you may wish to read standard input together with other files. For example, you may wish to read one file, read standard input coming from a pipe, and then read another file.

The way to name the standard input, with all versions of awk, is to use a single, standalone minus sign or dash, ‘-’. For example:

some_command | awk -f myprog.awk file1 - file2

Here, awk first reads file1, then it reads the output of some_command, and finally it reads file2.

You may also use "-" to name standard input when reading files with getline (see Using getline from a File). And, you can even use "-" with the -f option to read program source code from standard input (see Command-Line Options).

In addition, gawk allows you to specify the special file name /dev/stdin, both on the command line and with getline. Some other versions of awk also support this, but it is not standard. (Some operating systems provide a /dev/stdin file in the filesystem; however, gawk always processes this file name itself.)

2.5 The Environment Variables gawk Uses

A number of environment variables influence how gawk behaves.

• The AWKPATH Environment Variable
• The AWKLIBPATH Environment Variable
• Other Environment Variables

2.6 gawk’s Exit Status

If the exit statement is used with a value (see The exit Statement), then gawk exits with the numeric value given to it.

Otherwise, if there were no problems during execution, gawk exits with the value of the C constant EXIT_SUCCESS. This is usually zero.

If an error occurs, gawk exits with the value of the C constant EXIT_FAILURE. This is usually one.

If gawk exits because of a fatal error, the exit status is two. On non-POSIX systems, this value may be mapped to EXIT_FAILURE.

2.7 Including Other Files into Your Program

This section describes a feature that is specific to gawk.

The @include keyword can be used to read external awk source files. This gives you the ability to split large awk source files into smaller, more manageable pieces, and also lets you reuse common awk code from various awk scripts. In other words, you can group together awk functions used to carry out specific tasks into external files. These files can be used just like function libraries, using the @include keyword in conjunction with the AWKPATH environment variable. Note that source files may also be included using the -i option.

Let’s see an example. We’ll start with two (trivial) awk scripts, namely test1 and test2. Here is the test1 script:

BEGIN {
    print "This is script test1."
}

and here is test2:

@include "test1"
BEGIN {
    print "This is script test2."
}

Running gawk with test2 produces the following result:

$ gawk -f test2
-| This is script test1.
-| This is script test2.

gawk runs the test2 script, which includes test1 using the @include keyword. So, to include external awk source files, you just use @include followed by the name of the file to be included, enclosed in double quotes.

NOTE: Keep in mind that this is a language construct and the file name cannot be a string variable, but rather just a literal string constant in double quotes.

The files to be included may be nested; e.g., given a third script, namely test3:

@include "test2"
BEGIN {
    print "This is script test3."
}

Running gawk with the test3 script produces the following results:

$ gawk -f test3
-| This is script test1.
-| This is script test2.
-| This is script test3.

The file name can, of course, be a pathname. For example:

@include "../io_funcs"

and:

@include "/usr/awklib/network"

are both valid. The AWKPATH environment variable can be of great value when using @include. The same rules for the use of the AWKPATH variable in command-line file searches (see The AWKPATH Environment Variable) apply to @include also.

This is very helpful in constructing gawk function libraries. If you have a large script with useful, general-purpose awk functions, you can break it down into library files and put those files in a special directory. You can then include those “libraries,” either by using the full pathnames of the files, or by setting the AWKPATH environment variable accordingly and then using @include with just the file part of the full pathname. Of course, you can keep library files in more than one directory; the more complex the working environment is, the more directories you may need to organize the files to be included.

Given the ability to specify multiple -f options, the @include mechanism is not strictly necessary. However, the @include keyword can help you in constructing self-contained gawk programs, thus reducing the need for writing complex and tedious command lines. In particular, @include is very useful for writing CGI scripts to be run from web pages.

The @include directive and the -i/--include command line option are completely equivalent. An included program source is not loaded if it has been previously loaded.

The rules for finding a source file described in The AWKPATH Environment Variable also apply to files loaded with @include.

Finally, files included with @include are treated as if they had ‘@namespace "awk"’ at their beginning. See Changing The Namespace, for more information.

2.8 Loading Dynamic Extensions into Your Program

This section describes a feature that is specific to gawk.

The @load keyword can be used to read external awk extensions (stored as system shared libraries). This allows you to link in compiled code that may offer superior performance and/or give you access to extended capabilities not supported by the awk language. The AWKLIBPATH variable is used to search for the extension. Using @load is completely equivalent to using the -l command-line option.

If the extension is not initially found in AWKLIBPATH, another search is conducted after appending the platform’s default shared library suffix to the file name. For example, on GNU/Linux systems, the suffix ‘.so’ is used:

$ gawk '@load "ordchr"; BEGIN {print chr(65)}'
-| A

This is equivalent to the following example:

$ gawk -lordchr 'BEGIN {print chr(65)}'
-| A

For command-line usage, the -l option is more convenient, but @load is useful for embedding inside an awk source file that requires access to an extension.

Writing Extensions for gawk, describes how to write extensions (in C or C++) that can be loaded with either @load or the -l option. It also describes the ordchr extension.

2.9 Obsolete Options and/or Features

This section describes features and/or command-line options from previous releases of gawk that either are not available in the current version or are still supported but deprecated (meaning that they will not be in a future release).

The arbitrary precision arithmetic feature is deprecated as of gawk version 5.2. Use of -M/--bignum produces a warning message. The feature will be removed in the release of 2024.

2.10 Undocumented Options and Features

Use the Source, Luke!

This section intentionally left blank.

2.11 Summary

• gawk parses arguments on the command line, left to right, to determine if they should be treated as options or as non-option arguments.
• gawk recognizes several options which control its operation, as described in Command-Line Options. All options begin with ‘-’.
• Any argument that is not recognized as an option is treated as a non-option argument, even if it begins with ‘-’.
  • However, when an option itself requires an argument, and the option is separated from that argument on the command line by at least one space, the space is ignored, and the argument is considered to be related to the option. Thus, in the invocation, ‘gawk -F x’, the ‘x’ is treated as belonging to the -F option, not as a separate non-option argument.
• Once gawk finds a non-option argument, it stops looking for options. Therefore, all following arguments are also non-option arguments, even if they resemble recognized options.
• If no -e or -f options are present, gawk expects the program text to be in the first non-option argument.
• All non-option arguments, except program text provided in the first non-option argument, are placed in ARGV as explained in Using ARGC and ARGV, and are processed as described in Other Command-Line Arguments. Adjusting ARGC and ARGV affects how awk processes input.
• The three standard options for all versions of awk are -f, -F, and -v. gawk supplies these and many others, as well as corresponding GNU-style long options.
• Nonoption command-line arguments are usually treated as file names, unless they have the form ‘var=value’, in which case they are taken as variable assignments to be performed at that point in processing the input.
• You can use a single minus sign (‘-’) to refer to standard input on the command line. gawk also lets you use the special file name /dev/stdin.
• gawk pays attention to a number of environment variables. AWKPATH, AWKLIBPATH, and POSIXLY_CORRECT are the most important ones.
• gawk’s exit status conveys information to the program that invoked it. Use the exit statement from within an awk program to set the exit status.
• gawk allows you to include other awk source files into your program using the @include statement and/or the -i and -f command-line options.
• gawk allows you to load additional functions written in C or C++ using the @load statement and/or the -l option. (This advanced feature is described later, in Writing Extensions for gawk.)

3.1 How to Use Regular Expressions

A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record. (Normally, it only needs to match some part of the text in order to succeed.) For example, the following prints the second field of each record where the string ‘li’ appears anywhere in the record:

$ awk '/li/ { print $2 }' mail-list
-| 555-5553
-| 555-0542
-| 555-6699
-| 555-3430

Regular expressions can also be used in matching expressions. These expressions allow you to specify the string to match against; it need not be the entire current input record. The two operators ‘~’ and ‘!~’ perform regular expression comparisons. Expressions using these operators can be used as patterns, or in if, while, for, and do statements. (See Control Statements in Actions.) For example, the following is true if the expression exp (taken as a string) matches regexp:

exp ~ /regexp/

This example matches, or selects, all input records with the uppercase letter ‘J’ somewhere in the first field:

$ awk '$1 ~ /J/' inventory-shipped
-| Jan  13  25  15 115
-| Jun  31  42  75 492
-| Jul  24  34  67 436
-| Jan  21  36  64 620

So does this:

awk '{ if ($1 ~ /J/) print }' inventory-shipped

This next example is true if the expression exp (taken as a character string) does not match regexp:

exp !~ /regexp/

The following example matches, or selects, all input records whose first field does not contain the uppercase letter ‘J’:

$ awk '$1 !~ /J/' inventory-shipped
-| Feb  15  32  24 226
-| Mar  15  24  34 228
-| Apr  31  52  63 420
-| May  16  34  29 208
…

When a regexp is enclosed in slashes, such as /foo/, we call it a regexp constant, much like 5.27 is a numeric constant and "foo" is a string constant.

3.2 Escape Sequences

Some characters cannot be included literally in string constants ("foo") or regexp constants (/foo/). Instead, they should be represented with escape sequences, which are character sequences beginning with a backslash (‘\’). One use of an escape sequence is to include a double-quote character in a string constant. Because a plain double quote ends the string, you must use ‘\"’ to represent an actual double-quote character as a part of the string. For example:

$ awk 'BEGIN { print "He said \"hi!\" to her." }'
-| He said "hi!" to her.

The backslash character itself is another character that cannot be included normally; you must write ‘\\’ to put one backslash in the string or regexp. Thus, the string whose contents are the two characters ‘"’ and ‘\’ must be written "\"\\".

Other escape sequences represent unprintable characters such as TAB or newline. There is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, but they may look ugly.

The following list presents all the escape sequences used in awk and what they represent. Unless noted otherwise, all these escape sequences apply to both string constants and regexp constants:

\\

A literal backslash, ‘\’.

\a

The “alert” character, Ctrl-g, ASCII code 7 (BEL). (This often makes some sort of audible noise.)

\b

Backspace, Ctrl-h, ASCII code 8 (BS).

\f

Formfeed, Ctrl-l, ASCII code 12 (FF).

\n

Newline, Ctrl-j, ASCII code 10 (LF).

\r

Carriage return, Ctrl-m, ASCII code 13 (CR).

\t

Horizontal TAB, Ctrl-i, ASCII code 9 (HT).

\v

Vertical TAB, Ctrl-k, ASCII code 11 (VT).

\nnn

The octal value nnn, where nnn stands for 1 to 3 digits between ‘0’ and ‘7’. For example, the code for the ASCII ESC (escape) character is ‘\033’.

\xhh…

The hexadecimal value hh, where hh stands for a sequence of hexadecimal digits (‘0’–‘9’, and either ‘A’–‘F’ or ‘a’–‘f’). A maximum of two digits are allowed after the ‘\x’. Any further hexadecimal digits are treated as simple letters or numbers. (c.e.) (The ‘\x’ escape sequence is not allowed in POSIX awk.)

CAUTION: In ISO C, the escape sequence continues until the first nonhexadecimal digit is seen. For many years, gawk would continue incorporating hexadecimal digits into the value until a non-hexadecimal digit or the end of the string was encountered. However, using more than two hexadecimal digits produced undefined results. As of version 4.2, only two digits are processed.

\/

A literal slash (should be used for regexp constants only). This sequence is used when you want to write a regexp constant that contains a slash (such as /.*:\/home\/[[:alnum:]]+:.*/; the ‘[[:alnum:]]’ notation is discussed in Using Bracket Expressions). Because the regexp is delimited by slashes, you need to escape any slash that is part of the pattern, in order to tell awk to keep processing the rest of the regexp.

\"

A literal double quote (should be used for string constants only). This sequence is used when you want to write a string constant that contains a double quote (such as "He said \"hi!\" to her."). Because the string is delimited by double quotes, you need to escape any quote that is part of the string, in order to tell awk to keep processing the rest of the string.

In gawk, a number of additional two-character sequences that begin with a backslash have special meaning in regexps. See gawk-Specific Regexp Operators.

In a regexp, a backslash before any character that is not in the previous list and not listed in gawk-Specific Regexp Operators means that the next character should be taken literally, even if it would normally be a regexp operator. For example, /a\+b/ matches the three characters ‘a+b’.

For complete portability, do not use a backslash before any character not shown in the previous list or that is not an operator.

To summarize:

• The escape sequences in the preceding list are always processed first, for both string constants and regexp constants. This happens very early, as soon as awk reads your program.
• gawk processes both regexp constants and dynamic regexps (see Using Dynamic Regexps), for the special operators listed in gawk-Specific Regexp Operators.
• A backslash before any other character means to treat that character literally.

3.3 Regular Expression Operators

You can combine regular expressions with special characters, called regular expression operators or metacharacters, to increase the power and versatility of regular expressions.

• Regexp Operators in awk
• Some Notes On Interval Expressions

$ awk '
> BEGIN {
>     x = "ABC_CBA"
>     gsub(/B/, "bb", x)
>     print x
> }'
-| AbbC_CbbA

$ awk '
> BEGIN {
>     x = "ABC"
>     gsub(//, "x", x)
>     print x
> }'
-| xAxBxCx

3.4 Using Bracket Expressions

As mentioned earlier, a bracket expression matches any character among those listed between the opening and closing square brackets.

Within a bracket expression, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, based upon the system’s native character set. For example, ‘[0-9]’ is equivalent to ‘[0123456789]’. (See Regexp Ranges and Locales: A Long Sad Story for an explanation of how the POSIX standard and gawk have changed over time. This is mainly of historical interest.)

With the increasing popularity of the Unicode character standard, there is an additional wrinkle to consider. Octal and hexadecimal escape sequences inside bracket expressions are taken to represent only single-byte characters (characters whose values fit within the range 0–256). To match a range of characters where the endpoints of the range are larger than 256, enter the multibyte encodings of the characters directly.

To include one of the characters ‘\’, ‘]’, ‘-’, or ‘^’ in a bracket expression, put a ‘\’ in front of it. For example:

[d\]]

matches either ‘d’ or ‘]’. Additionally, if you place ‘]’ right after the opening ‘[’, the closing bracket is treated as one of the characters to be matched.

The treatment of ‘\’ in bracket expressions is compatible with other awk implementations and is also mandated by POSIX. The regular expressions in awk are a superset of the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by the traditional egrep utility.

Character classes are a feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but the actual characters can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs between the United States and France.

A character class is only valid in a regexp inside the brackets of a bracket expression. Character classes consist of ‘[:’, a keyword denoting the class, and ‘:]’. Table 3.1 lists the character classes defined by the POSIX standard.

For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.

Some utilities that match regular expressions provide a nonstandard ‘[:ascii:]’ character class; awk does not. However, you can simulate such a construct using ‘[\x00-\x7F]’. This matches all values numerically between zero and 127, which is the defined range of the ASCII character set. Use a complemented character list (‘[^\x00-\x7F]’) to match any single-byte characters that are not in the ASCII range.

NOTE: Some older versions of Unix awk treat [:blank:] like [:space:], incorrectly matching more characters than they should. Caveat Emptor.

Two additional special sequences can appear in bracket expressions. These apply to non-ASCII character sets, which can have single symbols (called collating elements) that are represented with more than one character. They can also have several characters that are equivalent for collating, or sorting, purposes. (For example, in French, a plain “e” and a grave-accented “è” are equivalent.) These sequences are:

Collating symbols

Multicharacter collating elements enclosed between ‘[.’ and ‘.]’. For example, if ‘ch’ is a collating element, then ‘[[.ch.]]’ is a regexp that matches this collating element, whereas ‘[ch]’ is a regexp that matches either ‘c’ or ‘h’.

Equivalence classes

Locale-specific names for a list of characters that are equal. The name is enclosed between ‘[=’ and ‘=]’. For example, the name ‘e’ might be used to represent all of “e,” “ê,” “è,” and “é.” In this case, ‘[[=e=]]’ is a regexp that matches any of ‘e’, ‘ê’, ‘é’, or ‘è’.

These features are very valuable in non-English-speaking locales.

CAUTION: The library functions that gawk uses for regular expression matching currently recognize only POSIX character classes; they do not recognize collating symbols or equivalence classes.

Inside a bracket expression, an opening bracket (‘[’) that does not start a character class, collating element or equivalence class is taken literally. This is also true of ‘.’ and ‘*’.

3.5 How Much Text Matches?

Consider the following:

echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'

This example uses the sub() function to make a change to the input record. (sub() replaces the first instance of any text matched by the first argument with the string provided as the second argument; see String-Manipulation Functions.) Here, the regexp /a+/ indicates “one or more ‘a’ characters,” and the replacement text is ‘<A>’.

The input contains four ‘a’ characters. awk (and POSIX) regular expressions always match the leftmost, longest sequence of input characters that can match. Thus, all four ‘a’ characters are replaced with ‘<A>’ in this example:

$ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'
-| <A>bcd

For simple match/no-match tests, this is not so important. But when doing text matching and substitutions with the match(), sub(), gsub(), and gensub() functions, it is very important. Understanding this principle is also important for regexp-based record and field splitting (see How Input Is Split into Records, and also see Specifying How Fields Are Separated).

3.6 Using Dynamic Regexps

The righthand side of a ‘~’ or ‘!~’ operator need not be a regexp constant (i.e., a string of characters between slashes). It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are then used as the regexp. A regexp computed in this way is called a dynamic regexp or a computed regexp:

BEGIN { digits_regexp = "[[:digit:]]+" }
$0 ~ digits_regexp    { print }

This sets digits_regexp to a regexp that describes one or more digits, and tests whether the input record matches this regexp.

NOTE: When using the ‘~’ and ‘!~’ operators, be aware that there is a difference between a regexp constant enclosed in slashes and a string constant enclosed in double quotes. If you are going to use a string constant, you have to understand that the string is, in essence, scanned twice: the first time when awk reads your program, and the second time when it goes to match the string on the lefthand side of the operator with the pattern on the right. This is true of any string-valued expression (such as digits_regexp, shown in the previous example), not just string constants.

What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes.

For example, /\*/ is a regexp constant for a literal ‘*’. Only one backslash is needed. To do the same thing with a string, you have to type "\\*". The first backslash escapes the second one so that the string actually contains the two characters ‘\’ and ‘*’.

Given that you can use both regexp and string constants to describe regular expressions, which should you use? The answer is “regexp constants,” for several reasons:

• String constants are more complicated to write and more difficult to read. Using regexp constants makes your programs less error-prone. Not understanding the difference between the two kinds of constants is a common source of errors.
• It is more efficient to use regexp constants. awk can note that you have supplied a regexp and store it internally in a form that makes pattern matching more efficient. When using a string constant, awk must first convert the string into this internal form and then perform the pattern matching.
• Using regexp constants is better form; it shows clearly that you intend a regexp match.

$ awk '$0 ~ "[ \t\n]"'
error→ awk: newline in character class [
error→ ]...
error→  source line number 1
error→  context is
error→        $0 ~ "[ >>>  \t\n]" <<<

$ awk '$0 ~ /[ \t\n]/'
here is a sample line
-| here is a sample line
Ctrl-d

3.7 gawk-Specific Regexp Operators

GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this section and are specific to gawk; they are not available in other awk implementations. Most of the additional operators deal with word matching. For our purposes, a word is a sequence of one or more letters, digits, or underscores (‘_’):

\s

Matches any space character as defined by the current locale. Think of it as shorthand for ‘[[:space:]]’.

\S

Matches any character that is not a space, as defined by the current locale. Think of it as shorthand for ‘[^[:space:]]’.

\w

Matches any word-constituent character—that is, it matches any letter, digit, or underscore. Think of it as shorthand for ‘[[:alnum:]_]’.

\W

Matches any character that is not word-constituent. Think of it as shorthand for ‘[^[:alnum:]_]’.

\<

Matches the empty string at the beginning of a word. For example, /\<away/ matches ‘away’ but not ‘stowaway’.

\>

Matches the empty string at the end of a word. For example, /stow\>/ matches ‘stow’ but not ‘stowaway’.

\y

Matches the empty string at either the beginning or the end of a word (i.e., the word boundary). For example, ‘\yballs?\y’ matches either ‘ball’ or ‘balls’, as a separate word.

\B

Matches the empty string that occurs between two word-constituent characters. For example, /\Brat\B/ matches ‘crate’, but it does not match ‘dirty rat’. ‘\B’ is essentially the opposite of ‘\y’. Another way to think of this is that ‘\B’ matches the empty string provided it’s not at the edge of a word.

There are two other operators that work on buffers. In Emacs, a buffer is, naturally, an Emacs buffer. Other GNU programs, including gawk, consider the entire string to match as the buffer. The operators are:

\`

Matches the empty string at the beginning of a buffer (string)

\'

Matches the empty string at the end of a buffer (string)

Because ‘^’ and ‘$’ always work in terms of the beginning and end of strings, these operators don’t add any new capabilities for awk. They are provided for compatibility with other GNU software.

In other GNU software, the word-boundary operator is ‘\b’. However, that conflicts with the awk language’s definition of ‘\b’ as backspace, so gawk uses a different letter. An alternative method would have been to require two backslashes in the GNU operators, but this was deemed too confusing. The current method of using ‘\y’ for the GNU ‘\b’ appears to be the lesser of two evils.

The various command-line options (see Command-Line Options) control how gawk interprets characters in regexps:

No options

In the default case, gawk provides all the facilities of POSIX regexps and the previously described GNU regexp operators. GNU regexp operators described in Regular Expression Operators.

--posix

Match only POSIX regexps; the GNU operators are not special (e.g., ‘\w’ matches a literal ‘w’). Interval expressions are allowed.

--traditional

Match traditional Unix awk regexps. The GNU operators are not special. Because BWK awk supports them, the POSIX character classes (‘[[:alnum:]]’, etc.) are available. So too, interval expressions are allowed. Characters described by octal and hexadecimal escape sequences are treated literally, even if they represent regexp metacharacters.

--re-interval

This option remains for backwards compatibility but no longer has any real effect.

3.8 Case Sensitivity in Matching

Case is normally significant in regular expressions, both when matching ordinary characters (i.e., not metacharacters) and inside bracket expressions. Thus, a ‘w’ in a regular expression matches only a lowercase ‘w’ and not an uppercase ‘W’.

The simplest way to do a case-independent match is to use a bracket expression—for example, ‘[Ww]’. However, this can be cumbersome if you need to use it often, and it can make the regular expressions harder to read. There are two alternatives that you might prefer.

One way to perform a case-insensitive match at a particular point in the program is to convert the data to a single case, using the tolower() or toupper() built-in string functions (which we haven’t discussed yet; see String-Manipulation Functions). For example:

tolower($1) ~ /foo/  { … }

converts the first field to lowercase before matching against it. This works in any POSIX-compliant awk.

Another method, specific to gawk, is to set the variable IGNORECASE to a nonzero value (see Predefined Variables). When IGNORECASE is not zero, all regexp and string operations ignore case.

Changing the value of IGNORECASE dynamically controls the case sensitivity of the program as it runs. Case is significant by default because IGNORECASE (like most variables) is initialized to zero:

x = "aB"
if (x ~ /ab/) …   # this test will fail
IGNORECASE = 1
if (x ~ /ab/) …   # now it will succeed

In general, you cannot use IGNORECASE to make certain rules case insensitive and other rules case sensitive, as there is no straightforward way to set IGNORECASE just for the pattern of a particular rule.¹⁸ To do this, use either bracket expressions or tolower(). However, one thing you can do with IGNORECASE only is dynamically turn case sensitivity on or off for all the rules at once.

IGNORECASE can be set on the command line or in a BEGIN rule (see Other Command-Line Arguments; also see Startup and Cleanup Actions). Setting IGNORECASE from the command line is a way to make a program case insensitive without having to edit it.

In multibyte locales, the equivalences between upper- and lowercase characters are tested based on the wide-character values of the locale’s character set. Prior to version 5.0, single-byte characters were tested based on the ISO-8859-1 (ISO Latin-1) character set. However, as of version 5.0, single-byte characters are also tested based on the values of the locale’s character set.¹⁹

The value of IGNORECASE has no effect if gawk is in compatibility mode (see Command-Line Options). Case is always significant in compatibility mode.

3.9 Summary

• Regular expressions describe sets of strings to be matched. In awk, regular expression constants are written enclosed between slashes: /…/.
• Regexp constants may be used standalone in patterns and in conditional expressions, or as part of matching expressions using the ‘~’ and ‘!~’ operators.
• Escape sequences let you represent nonprintable characters and also let you represent regexp metacharacters as literal characters to be matched.
• Regexp operators provide grouping, alternation, and repetition.
• Bracket expressions give you a shorthand for specifying sets of characters that can match at a particular point in a regexp. Within bracket expressions, POSIX character classes let you specify certain groups of characters in a locale-independent fashion.
• Regular expressions match the leftmost longest text in the string being matched. This matters for cases where you need to know the extent of the match, such as for text substitution and when the record separator is a regexp.
• Matching expressions may use dynamic regexps (i.e., string values treated as regular expressions).
• gawk’s IGNORECASE variable lets you control the case sensitivity of regexp matching. In other awk versions, use tolower() or toupper().

4.1 How Input Is Split into Records

awk divides the input for your program into records and fields. It keeps track of the number of records that have been read so far from the current input file. This value is stored in a predefined variable called FNR, which is reset to zero every time a new file is started. Another predefined variable, NR, records the total number of input records read so far from all data files. It starts at zero, but is never automatically reset to zero.

Normally, records are separated by newline characters. You can control how records are separated by assigning values to the built-in variable RS. If RS is any single character, that character separates records. Otherwise (in gawk), RS is treated as a regular expression. This mechanism is explained in greater detail shortly.

• Record Splitting with Standard awk
• Record Splitting with gawk

awk 'BEGIN { RS = "u" }
     { print $0 }' mail-list

$ awk 'BEGIN { RS = "u" }
>      { print $0 }' mail-list

-| Amelia       555-5553     amelia.zodiac
-| sq
-| [email protected]    F
-| Anthony      555-3412     anthony.assert
-| [email protected]   A
-| Becky        555-7685     becky.algebrar
-| [email protected]      A
-| Bill         555-1675     [email protected]       A
-| Broderick    555-0542     broderick.aliq
-| [email protected] R
-| Camilla      555-2912     camilla.inf
-| sar
-| [email protected]     R
-| Fabi
-| s       555-1234     fabi
-| s.
-| ndevicesim
-| s@
-| cb.ed
-|     F
-| J
-| lie        555-6699     j
-| lie.perscr
-| [email protected]   F
-| Martin       555-6480     martin.codicib
-| [email protected]    A
-| Sam
-| el       555-3430     sam
-| el.lanceolis@sh
-| .ed
-|         A
-| Jean-Pa
-| l    555-2127     jeanpa
-| l.campanor
-| m@ny
-| .ed
-|      R
-|

Bill         555-1675     [email protected]       A

awk '{ print $0 }' RS="u" mail-list

$ echo record 1 AAAA record 2 BBBB record 3 |
> gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" }
>             { print "Record =", $0,"and RT = [" RT "]" }'

-| Record = record 1 and RT = [ AAAA ]
-| Record = record 2 and RT = [ BBBB ]
-| Record = record 3 and RT = [
-| ]

BEGIN { RS = "\0" }  # whole file becomes one record?

4.2 Examining Fields

When awk reads an input record, the record is automatically parsed or separated by the awk utility into chunks called fields. By default, fields are separated by whitespace, like words in a line. Whitespace in awk means any string of one or more spaces, TABs, or newlines; other characters that are considered whitespace by other languages (such as formfeed, vertical tab, etc.) are not considered whitespace by awk.

The purpose of fields is to make it more convenient for you to refer to these pieces of the record. You don’t have to use them—you can operate on the whole record if you want—but fields are what make simple awk programs so powerful.

You use a dollar sign (‘$’) to refer to a field in an awk program, followed by the number of the field you want. Thus, $1 refers to the first field, $2 to the second, and so on. (Unlike in the Unix shells, the field numbers are not limited to single digits. $127 is the 127th field in the record.) For example, suppose the following is a line of input:

This seems like a pretty nice example.

Here the first field, or $1, is ‘This’, the second field, or $2, is ‘seems’, and so on. Note that the last field, $7, is ‘example.’. Because there is no space between the ‘e’ and the ‘.’, the period is considered part of the seventh field.

NF is a predefined variable whose value is the number of fields in the current record. awk automatically updates the value of NF each time it reads a record. No matter how many fields there are, the last field in a record can be represented by $NF. So, $NF is the same as $7, which is ‘example.’. If you try to reference a field beyond the last one (such as $8 when the record has only seven fields), you get the empty string. (If used in a numeric operation, you get zero.)

The use of $0, which looks like a reference to the “zeroth” field, is a special case: it represents the whole input record. Use it when you are not interested in specific fields. Here are some more examples:

$ awk '$1 ~ /li/ { print $0 }' mail-list
-| Amelia       555-5553     [email protected]    F
-| Julie        555-6699     [email protected]   F

This example prints each record in the file mail-list whose first field contains the string ‘li’.

By contrast, the following example looks for ‘li’ in the entire record and prints the first and last fields for each matching input record:

$ awk '/li/ { print $1, $NF }' mail-list
-| Amelia F
-| Broderick R
-| Julie F
-| Samuel A

4.3 Nonconstant Field Numbers

A field number need not be a constant. Any expression in the awk language can be used after a ‘$’ to refer to a field. The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example:

awk '{ print $NR }'

Recall that NR is the number of records read so far: one in the first record, two in the second, and so on. So this example prints the first field of the first record, the second field of the second record, and so on. For the twentieth record, field number 20 is printed; most likely, the record has fewer than 20 fields, so this prints a blank line. Here is another example of using expressions as field numbers:

awk '{ print $(2*2) }' mail-list

awk evaluates the expression ‘(2*2)’ and uses its value as the number of the field to print. The ‘*’ represents multiplication, so the expression ‘2*2’ evaluates to four. The parentheses are used so that the multiplication is done before the ‘$’ operation; they are necessary whenever there is a binary operator²¹ in the field-number expression. This example, then, prints the type of relationship (the fourth field) for every line of the file mail-list. (All of the awk operators are listed, in order of decreasing precedence, in Operator Precedence (How Operators Nest).)

If the field number you compute is zero, you get the entire record. Thus, ‘$(2-2)’ has the same value as $0. Negative field numbers are not allowed; trying to reference one usually terminates the program. (The POSIX standard does not define what happens when you reference a negative field number. gawk notices this and terminates your program. Other awk implementations may behave differently.)

As mentioned in Examining Fields, awk stores the current record’s number of fields in the built-in variable NF (also see Predefined Variables). Thus, the expression $NF is not a special feature—it is the direct consequence of evaluating NF and using its value as a field number.

4.4 Changing the Contents of a Field

The contents of a field, as seen by awk, can be changed within an awk program; this changes what awk perceives as the current input record. (The actual input is untouched; awk never modifies the input file.) Consider the following example and its output:

$ awk '{ nboxes = $3 ; $3 = $3 - 10
>        print nboxes, $3 }' inventory-shipped
-| 25 15
-| 32 22
-| 24 14
…

The program first saves the original value of field three in the variable nboxes. The ‘-’ sign represents subtraction, so this program reassigns field three, $3, as the original value of field three minus ten: ‘$3 - 10’. (See Arithmetic Operators.) Then it prints the original and new values for field three. (Someone in the warehouse made a consistent mistake while inventorying the red boxes.)

For this to work, the text in $3 must make sense as a number; the string of characters must be converted to a number for the computer to do arithmetic on it. The number resulting from the subtraction is converted back to a string of characters that then becomes field three. See Conversion of Strings and Numbers.

When the value of a field is changed (as perceived by awk), the text of the input record is recalculated to contain the new field where the old one was. In other words, $0 changes to reflect the altered field. Thus, this program prints a copy of the input file, with 10 subtracted from the second field of each line:

$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped
-| Jan 3 25 15 115
-| Feb 5 32 24 226
-| Mar 5 24 34 228
…

It is also possible to assign contents to fields that are out of range. For example:

$ awk '{ $6 = ($5 + $4 + $3 + $2)
>        print $6 }' inventory-shipped
-| 168
-| 297
-| 301
…

We’ve just created $6, whose value is the sum of fields $2, $3, $4, and $5. The ‘+’ sign represents addition. For the file inventory-shipped, $6 represents the total number of parcels shipped for a particular month.

Creating a new field changes awk’s internal copy of the current input record, which is the value of $0. Thus, if you do ‘print $0’ after adding a field, the record printed includes the new field, with the appropriate number of field separators between it and the previously existing fields.

This recomputation affects and is affected by NF (the number of fields; see Examining Fields). For example, the value of NF is set to the number of the highest field you create. The exact format of $0 is also affected by a feature that has not been discussed yet: the output field separator, OFS, used to separate the fields (see Output Separators).

Note, however, that merely referencing an out-of-range field does not change the value of either $0 or NF. Referencing an out-of-range field only produces an empty string. For example:

if ($(NF+1) != "")
    print "can't happen"
else
    print "everything is normal"

should print ‘everything is normal’, because NF+1 is certain to be out of range. (See The if-else Statement for more information about awk’s if-else statements. See Variable Typing and Comparison Expressions for more information about the ‘!=’ operator.)

It is important to note that making an assignment to an existing field changes the value of $0 but does not change the value of NF, even when you assign the empty string to a field. For example:

$ echo a b c d | awk '{ OFS = ":"; $2 = ""
>                       print $0; print NF }'
-| a::c:d
-| 4

The field is still there; it just has an empty value, delimited by the two colons between ‘a’ and ‘c’. This example shows what happens if you create a new field:

$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new"
>                       print $0; print NF }'
-| a::c:d::new
-| 6

The intervening field, $5, is created with an empty value (indicated by the second pair of adjacent colons), and NF is updated with the value six.

Decrementing NF throws away the values of the fields after the new value of NF and recomputes $0. (d.c.) Here is an example:

$ echo a b c d e f | awk '{ print "NF =", NF;
>                           NF = 3; print $0 }'
-| NF = 6
-| a b c

CAUTION: Some versions of awk don’t rebuild $0 when NF is decremented. Until August, 2018, this included BWK awk; fortunately his version now handles this correctly.

Finally, there are times when it is convenient to force awk to rebuild the entire record, using the current values of the fields and OFS. To do this, use the seemingly innocuous assignment:

$1 = $1   # force record to be reconstituted
print $0  # or whatever else with $0

This forces awk to rebuild the record. It does help to add a comment, as we’ve shown here.

There is a flip side to the relationship between $0 and the fields. Any assignment to $0 causes the record to be reparsed into fields using the current value of FS. This also applies to any built-in function that updates $0, such as sub() and gsub() (see String-Manipulation Functions).

4.5 Specifying How Fields Are Separated

The field separator, which is either a single character or a regular expression, controls the way awk splits an input record into fields. awk scans the input record for character sequences that match the separator; the fields themselves are the text between the matches.

In the examples that follow, we use the bullet symbol (•) to represent spaces in the output. If the field separator is ‘oo’, then the following line:

moo goo gai pan

is split into three fields: ‘m’, ‘•g’, and ‘•gai•pan’. Note the leading spaces in the values of the second and third fields.

The field separator is represented by the predefined variable FS. Shell programmers take note: awk does not use the name IFS that is used by the POSIX-compliant shells (such as the Unix Bourne shell, sh, or Bash).

The value of FS can be changed in the awk program with the assignment operator, ‘=’ (see Assignment Expressions). Often, the right time to do this is at the beginning of execution before any input has been processed, so that the very first record is read with the proper separator. To do this, use the special BEGIN pattern (see The BEGIN and END Special Patterns). For example, here we set the value of FS to the string ",":

awk 'BEGIN { FS = "," } ; { print $2 }'

Given the input line:

John Q. Smith, 29 Oak St., Walamazoo, MI 42139

this awk program extracts and prints the string ‘•29•Oak•St.’.

Sometimes the input data contains separator characters that don’t separate fields the way you thought they would. For instance, the person’s name in the example we just used might have a title or suffix attached, such as:

John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139

The same program would extract ‘•LXIX’ instead of ‘•29•Oak•St.’. If you were expecting the program to print the address, you would be surprised. The moral is to choose your data layout and separator characters carefully to prevent such problems. (If the data is not in a form that is easy to process, perhaps you can massage it first with a separate awk program.)

• Whitespace Normally Separates Fields
• Using Regular Expressions to Separate Fields
• Making Each Character a Separate Field
• Setting FS from the Command Line
• Making the Full Line Be a Single Field
• Field-Splitting Summary

FS = ", \t"

$ echo ' a b c d ' | awk '{ print $2 }'
-| b

$ echo ' a  b  c  d ' | awk 'BEGIN { FS = "[ \t\n]+" }
>                                  { print $2 }'
-| a

$ echo '   a b c d' | awk '{ print; $2 = $2; print }'
-|    a b c d
-| a b c d

$ echo 'xxAA  xxBxx  C' |
> gawk -F '(^x+)|( +)' '{ for (i = 1; i <= NF; i++)
>                             printf "-->%s<--\n", $i }'
-| --><--
-| -->AA<--
-| -->xxBxx<--
-| -->C<--

$ echo a b | gawk 'BEGIN { FS = "" }
>                  {
>                      for (i = 1; i <= NF; i = i + 1)
>                          print "Field", i, "is", $i
>                  }'
-| Field 1 is a
-| Field 2 is
-| Field 3 is b

awk -F, 'program' input-files

# same as FS = "\\"
awk -F\\\\ '…' files …

/edu/   { print $1 }

$ awk -F- -f edu.awk mail-list
-| Fabius       555
-| Samuel       555
-| Jean

Jean-Paul    555-2127     [email protected]     R

arnold:x:2076:10:Arnold Robbins:/home/arnold:/bin/bash

awk -F: '$5 == ""' /etc/passwd

awk -F'\n' 'program' files …

sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }'

root

root:x:0:0:Root:/:

FS = "c"
IGNORECASE = 1
$0 = "aCa"
print $1

4.6 Reading Fixed-Width Data

This section discusses an advanced feature of gawk. If you are a novice awk user, you might want to skip it on the first reading.

gawk provides a facility for dealing with fixed-width fields with no distinctive field separator. We discuss this feature in the following subsections.

• Processing Fixed-Width Data
• Skipping Intervening Fields
• Capturing Optional Trailing Data
• Field Values With Fixed-Width Data

 10:06pm  up 21 days, 14:04,  23 users
User     tty       login  idle   JCPU   PCPU  what
hzuo     ttyV0     8:58pm            9      5  vi p24.tex
hzang    ttyV3     6:37pm    50                -csh
eklye    ttyV5     9:53pm            7      1  em thes.tex
dportein ttyV6     8:17pm  1:47                -csh
gierd    ttyD3    10:00pm     1                elm
dave     ttyD4     9:47pm            4      4  w
brent    ttyp0    26Jun91  4:46  26:46   4:41  bash
dave     ttyq4    26Jun9115days     46     46  wnewmail

BEGIN  { FIELDWIDTHS = "9 6 10 6 7 7 35" }
NR > 2 {
    idle = $4
    sub(/^ +/, "", idle)   # strip leading spaces
    if (idle == "")
        idle = 0
    if (idle ~ /:/) {      # hh:mm
        split(idle, t, ":")
        idle = t[1] * 60 + t[2]
    }
    if (idle ~ /days/)
        idle *= 24 * 60 * 60
    print $1, $2, idle
}

hzuo      ttyV0  0
hzang     ttyV3  50
eklye     ttyV5  0
dportein  ttyV6  107
gierd     ttyD3  1
dave      ttyD4  0
brent     ttyp0  286
dave      ttyq4  1296000

BEGIN  { FIELDWIDTHS = "8 1:5 4:7 6 1:6 1:6 2:33" }

hzang    ttyV3 50
eklye    ttyV5 0
dportein ttyV6 107
gierd    ttyD3 1
dave     ttyD4 0
brent    ttyp0 286
dave     ttyq4 1296000

$ cat fw.awk                         Show the program
-| BEGIN { FIELDWIDTHS = "2 2 *" }
-| { print NF, $1, $2, $3 }
$ cat fw.in                          Show sample input
-| 1234abcdefghi
$ gawk -f fw.awk fw.in               Run the program
-| 3 12 34 abcdefghi

4.7 Defining Fields by Content

This section discusses an advanced feature of gawk. If you are a novice awk user, you might want to skip it on the first reading.

Normally, when using FS, gawk defines the fields as the parts of the record that occur in between each field separator. In other words, FS defines what a field is not, instead of what a field is. However, there are times when you really want to define the fields by what they are, and not by what they are not.

The most notorious such case is so-called comma-separated values (CSV) data. Many spreadsheet programs, for example, can export their data into text files, where each record is terminated with a newline, and fields are separated by commas. If commas only separated the data, there wouldn’t be an issue. The problem comes when one of the fields contains an embedded comma. In such cases, most programs embed the field in double quotes.²⁴ So, we might have data like this:

Robbins,Arnold,"1234 A Pretty Street, NE",MyTown,MyState,12345-6789,USA

The FPAT variable offers a solution for cases like this. The value of FPAT should be a string that provides a regular expression. This regular expression describes the contents of each field.

In the case of CSV data as presented here, each field is either “anything that is not a comma,” or “a double quote, anything that is not a double quote, and a closing double quote.” (There are more complicated definitions of CSV data, treated shortly.) If written as a regular expression constant (see Regular Expressions), we would have /([^,]+)|("[^"]+")/. Writing this as a string requires us to escape the double quotes, leading to:

FPAT = "([^,]+)|(\"[^\"]+\")"

Putting this to use, here is a simple program to parse the data:

BEGIN {
    FPAT = "([^,]+)|(\"[^\"]+\")"
}

{
    print "NF = ", NF
    for (i = 1; i <= NF; i++) {
        printf("$%d = <%s>\n", i, $i)
    }
}

When run, we get the following:

$ gawk -f simple-csv.awk addresses.csv
NF =  7
$1 = <Robbins>
$2 = <Arnold>
$3 = <"1234 A Pretty Street, NE">
$4 = <MyTown>
$5 = <MyState>
$6 = <12345-6789>
$7 = <USA>

Note the embedded comma in the value of $3.

A straightforward improvement when processing CSV data of this sort would be to remove the quotes when they occur, with something like this:

if (substr($i, 1, 1) == "\"") {
    len = length($i)
    $i = substr($i, 2, len - 2)    # Get text within the two quotes
}

NOTE: Some programs export CSV data that contains embedded newlines between the double quotes. gawk provides no way to deal with this. Even though a formal specification for CSV data exists, there isn’t much more to be done; the FPAT mechanism provides an elegant solution for the majority of cases, and the gawk developers are satisfied with that.

As written, the regexp used for FPAT requires that each field contain at least one character. A straightforward modification (changing the first ‘+’ to ‘*’) allows fields to be empty:

FPAT = "([^,]*)|(\"[^\"]+\")"

As with FS, the IGNORECASE variable (see Built-in Variables That Control awk) affects field splitting with FPAT.

Assigning a value to FPAT overrides field splitting with FS and with FIELDWIDTHS.

Finally, the patsplit() function makes the same functionality available for splitting regular strings (see String-Manipulation Functions).

• More on CSV Files
• FS Versus FPAT: A Subtle Difference

p,"q,r",s
p,"q""r",s
p,"q,""r",s
p,"",s
p,,s

BEGIN {
     fp[0] = "([^,]+)|(\"[^\"]+\")"
     fp[1] = "([^,]*)|(\"[^\"]+\")"
     fp[2] = "([^,]*)|(\"([^\"]|\"\")+\")"
     FPAT = fp[fpat+0]
}

{
     print "<" $0 ">"
     printf("NF = %s ", NF)
     for (i = 1; i <= NF; i++) {
         printf("<%s>", $i)
     }
     print ""
}

$ gawk -v fpat=2 -f test-csv.awk sample.csv
-| <p,"q,r",s>
-| NF = 3 <p><"q,r"><s>
-| <p,"q""r",s>
-| NF = 3 <p><"q""r"><s>
-| <p,"q,""r",s>
-| NF = 3 <p><"q,""r"><s>
-| <p,"",s>
-| NF = 3 <p><""><s>
-| <p,,s>
-| NF = 3 <p><><s>

4.8 Checking How gawk Is Splitting Records

As we’ve seen, gawk provides three independent methods to split input records into fields. The mechanism used is based on which of the three variables—FS, FIELDWIDTHS, or FPAT—was last assigned to. In addition, an API input parser may choose to override the record parsing mechanism; please refer to Customized Input Parsers for further information about this feature.

To restore normal field splitting after using FIELDWIDTHS and/or FPAT, simply assign a value to FS. You can use ‘FS = FS’ to do this, without having to know the current value of FS.

In order to tell which kind of field splitting is in effect, use PROCINFO["FS"] (see Built-in Variables That Convey Information). The value is "FS" if regular field splitting is being used, "FIELDWIDTHS" if fixed-width field splitting is being used, or "FPAT" if content-based field splitting is being used:

if (PROCINFO["FS"] == "FS")
    regular field splitting …
else if (PROCINFO["FS"] == "FIELDWIDTHS")
    fixed-width field splitting …
else if (PROCINFO["FS"] == "FPAT")
    content-based field splitting …
else
    API input parser field splitting … (advanced feature)

This information is useful when writing a function that needs to temporarily change FS or FIELDWIDTHS, read some records, and then restore the original settings (see Reading the User Database for an example of such a function).

4.9 Multiple-Line Records

In some databases, a single line cannot conveniently hold all the information in one entry. In such cases, you can use multiline records. The first step in doing this is to choose your data format.

One technique is to use an unusual character or string to separate records. For example, you could use the formfeed character (written ‘\f’ in awk, as in C) to separate them, making each record a page of the file. To do this, just set the variable RS to "\f" (a string containing the formfeed character). Any other character could equally well be used, as long as it won’t be part of the data in a record.

Another technique is to have blank lines separate records. By a special dispensation, an empty string as the value of RS indicates that records are separated by one or more blank lines. When RS is set to the empty string, each record always ends at the first blank line encountered. The next record doesn’t start until the first nonblank line that follows. No matter how many blank lines appear in a row, they all act as one record separator. (Blank lines must be completely empty; lines that contain only whitespace do not count.)

You can achieve the same effect as ‘RS = ""’ by assigning the string "\n\n+" to RS. This regexp matches the newline at the end of the record and one or more blank lines after the record. In addition, a regular expression always matches the longest possible sequence when there is a choice (see How Much Text Matches?). So, the next record doesn’t start until the first nonblank line that follows—no matter how many blank lines appear in a row, they are considered one record separator.

However, there is an important difference between ‘RS = ""’ and ‘RS = "\n\n+"’. In the first case, leading newlines in the input data file are ignored, and if a file ends without extra blank lines after the last record, the final newline is removed from the record. In the second case, this special processing is not done. (d.c.)

Now that the input is separated into records, the second step is to separate the fields in the records. One way to do this is to divide each of the lines into fields in the normal manner. This happens by default as the result of a special feature. When RS is set to the empty string and FS is set to a single character, the newline character always acts as a field separator. This is in addition to whatever field separations result from FS.

NOTE: When FS is the null string ("") or a regexp, this special feature of RS does not apply. It does apply to the default field separator of a single space: ‘FS = " "’.

Note that language in the POSIX specification implies that this special feature should apply when FS is a regexp. However, Unix awk has never behaved that way, nor has gawk. This is essentially a bug in POSIX.

The original motivation for this special exception was probably to provide useful behavior in the default case (i.e., FS is equal to " "). This feature can be a problem if you really don’t want the newline character to separate fields, because there is no way to prevent it. However, you can work around this by using the split() function to break up the record manually (see String-Manipulation Functions). If you have a single-character field separator, you can work around the special feature in a different way, by making FS into a regexp for that single character. For example, if the field separator is a percent character, instead of ‘FS = "%"’, use ‘FS = "[%]"’.

Another way to separate fields is to put each field on a separate line: to do this, just set the variable FS to the string "\n". (This single-character separator matches a single newline.) A practical example of a data file organized this way might be a mailing list, where blank lines separate the entries. Consider a mailing list in a file named addresses, which looks like this:

Jane Doe
123 Main Street
Anywhere, SE 12345-6789
John Smith
456 Tree-lined Avenue
Smallville, MW 98765-4321
…

A simple program to process this file is as follows:

# addrs.awk --- simple mailing list program
# Records are separated by blank lines.
# Each line is one field.
BEGIN { RS = "" ; FS = "\n" }
{
      print "Name is:", $1
      print "Address is:", $2
      print "City and State are:", $3
      print ""
}

Running the program produces the following output:

$ awk -f addrs.awk addresses
-| Name is: Jane Doe
-| Address is: 123 Main Street
-| City and State are: Anywhere, SE 12345-6789
-|
-| Name is: John Smith
-| Address is: 456 Tree-lined Avenue
-| City and State are: Smallville, MW 98765-4321
-|
…

See Printing Mailing Labels for a more realistic program dealing with address lists. The following list summarizes how records are split, based on the value of RS:

RS == "\n"

Records are separated by the newline character (‘\n’). In effect, every line in the data file is a separate record, including blank lines. This is the default.

RS == any single character

Records are separated by each occurrence of the character. Multiple successive occurrences delimit empty records.

RS == ""

Records are separated by runs of blank lines. When FS is a single character, then the newline character always serves as a field separator, in addition to whatever value FS may have. Leading and trailing newlines in a file are ignored.

RS == regexp

Records are separated by occurrences of characters that match regexp. Leading and trailing matches of regexp delimit empty records. (This is a gawk extension; it is not specified by the POSIX standard.)

If not in compatibility mode (see Command-Line Options), gawk sets RT to the input text that matched the value specified by RS. But if the input file ended without any text that matches RS, then gawk sets RT to the null string.

4.10 Explicit Input with getline

So far we have been getting our input data from awk’s main input stream—either the standard input (usually your keyboard, sometimes the output from another program) or the files specified on the command line. The awk language has a special built-in command called getline that can be used to read input under your explicit control.

The getline command is used in several different ways and should not be used by beginners. The examples that follow the explanation of the getline command include material that has not been covered yet. Therefore, come back and study the getline command after you have reviewed the rest of this Web page and have a good knowledge of how awk works.

The getline command returns 1 if it finds a record and 0 if it encounters the end of the file. If there is some error in getting a record, such as a file that cannot be opened, then getline returns −1. In this case, gawk sets the variable ERRNO to a string describing the error that occurred.

If ERRNO indicates that the I/O operation may be retried, and PROCINFO["input", "RETRY"] is set, then getline returns −2 instead of −1, and further calls to getline may be attempted. See Retrying Reads After Certain Input Errors for further information about this feature.

In the following examples, command stands for a string value that represents a shell command.

NOTE: When --sandbox is specified (see Command-Line Options), reading lines from files, pipes, and coprocesses is disabled.

• Using getline with No Arguments
• Using getline into a Variable
• Using getline from a File
• Using getline into a Variable from a File
• Using getline from a Pipe
• Using getline into a Variable from a Pipe
• Using getline from a Coprocess
• Using getline into a Variable from a Coprocess
• Points to Remember About getline
• Summary of getline Variants

# Remove text between /* and */, inclusive
{
    while ((start = index($0, "/*")) != 0) {
        out = substr($0, 1, start - 1)  # leading part of the string
        rest = substr($0, start + 2)    # ... */ ...
        while ((end = index(rest, "*/")) == 0) {  # is */ in trailing part?
            # get more text
            if (getline <= 0) {
                print("unexpected EOF or error:", ERRNO) > "/dev/stderr"
                exit
            }
            # build up the line using string concatenation
            rest = rest $0
        }
        rest = substr(rest, end + 2)  # remove comment
        # build up the output line using string concatenation
        $0 = out rest
    }
    print $0
}

mon/*comment*/key
rab/*commen
t*/bit
horse /*comment*/more text
part 1 /*comment*/part 2 /*comment*/part 3
no comment

$ awk -f strip_comments.awk example_text
-| monkey
-| rabbit
-| horse more text
-| part 1 part 2 part 3
-| no comment

{
     if ((getline tmp) > 0) {
          print tmp
          print $0
     } else
          print $0
}

wan
tew
free
phore

tew
wan
phore
free

{
    if ($1 == 10) {
         getline < "secondary.input"
         print
    } else
         print
}

{
     if (NF == 2 && $1 == "@include") {
          while ((getline line < $2) > 0)
               print line
          close($2)
     } else
          print
}

{
     if ($1 == "@execute") {
          tmp = substr($0, 10)        # Remove "@execute"
          while ((tmp | getline) > 0)
               print
          close(tmp)
     } else
          print
}

foo
bar
baz
@execute who
bletch

foo
bar
baz
arnold     ttyv0   Jul 13 14:22
miriam     ttyp0   Jul 13 14:23     (murphy:0)
bill       ttyp1   Jul 13 14:23     (murphy:0)
bletch

BEGIN {
     "date" | getline current_time
     close("date")
     print "Report printed on " current_time
}

print "some query" |& "db_server"
"db_server" |& getline

4.11 Reading Input with a Timeout

This section describes a feature that is specific to gawk.

You may specify a timeout in milliseconds for reading input from the keyboard, a pipe, or two-way communication, including TCP/IP sockets. This can be done on a per-input, per-command, or per-connection basis, by setting a special element in the PROCINFO array (see Built-in Variables That Convey Information):

PROCINFO["input_name", "READ_TIMEOUT"] = timeout in milliseconds

When set, this causes gawk to time out and return failure if no data is available to read within the specified timeout period. For example, a TCP client can decide to give up on receiving any response from the server after a certain amount of time:

Service = "/inet/tcp/0/localhost/daytime"
PROCINFO[Service, "READ_TIMEOUT"] = 100
if ((Service |& getline) > 0)
    print $0
else if (ERRNO != "")
    print ERRNO

Here is how to read interactively from the user²⁶ without waiting for more than five seconds:

PROCINFO["/dev/stdin", "READ_TIMEOUT"] = 5000
while ((getline < "/dev/stdin") > 0)
    print $0

gawk terminates the read operation if input does not arrive after waiting for the timeout period, returns failure, and sets ERRNO to an appropriate string value. A negative or zero value for the timeout is the same as specifying no timeout at all.

A timeout can also be set for reading from the keyboard in the implicit loop that reads input records and matches them against patterns, like so:

$ gawk 'BEGIN { PROCINFO["-", "READ_TIMEOUT"] = 5000 }
> { print "You entered: " $0 }'
gawk
-| You entered: gawk

In this case, failure to respond within five seconds results in the following error message:

error→ gawk: cmd. line:2: (FILENAME=- FNR=1) fatal: error reading input file `-': Connection timed out

The timeout can be set or changed at any time, and will take effect on the next attempt to read from the input device. In the following example, we start with a timeout value of one second, and progressively reduce it by one-tenth of a second until we wait indefinitely for the input to arrive:

PROCINFO[Service, "READ_TIMEOUT"] = 1000
while ((Service |& getline) > 0) {
    print $0
    PROCINFO[Service, "READ_TIMEOUT"] -= 100
}

NOTE: You should not assume that the read operation will block exactly after the tenth record has been printed. It is possible that gawk will read and buffer more than one record’s worth of data the first time. Because of this, changing the value of timeout like in the preceding example is not very useful.

If the PROCINFO element is not present and the GAWK_READ_TIMEOUT environment variable exists, gawk uses its value to initialize the timeout value. The exclusive use of the environment variable to specify timeout has the disadvantage of not being able to control it on a per-command or per-connection basis.

gawk considers a timeout event to be an error even though the attempt to read from the underlying device may succeed in a later attempt. This is a limitation, and it also means that you cannot use this to multiplex input from two or more sources. See Retrying Reads After Certain Input Errors for a way to enable later I/O attempts to succeed.

Assigning a timeout value prevents read operations from being blocked indefinitely. But bear in mind that there are other ways gawk can stall waiting for an input device to be ready. A network client can sometimes take a long time to establish a connection before it can start reading any data, or the attempt to open a FIFO special file for reading can be blocked indefinitely until some other process opens it for writing.