3.4 Defining Language Semantics

The grammar rules for a language determine only the syntax. The semantics are determined by the semantic values associated with various tokens and groupings, and by the actions taken when various groupings are recognized.

For example, the calculator calculates properly because the value associated with each expression is the proper number; it adds properly because the action for the grouping ‘x + y’ is to add the numbers associated with x and y.

3.4.1 Data Types of Semantic Values

In a simple program it may be sufficient to use the same data type for the semantic values of all language constructs. This was true in the RPN and infix calculator examples (see section Reverse Polish Notation Calculator).

Bison normally uses the type int for semantic values if your program uses the same data type for all language constructs. To specify some other type, define the %define variable api.value.type like this:

%define api.value.type {double}

or

%define api.value.type {struct semantic_type}

The value of api.value.type should be a type name that does not contain parentheses or square brackets.

Alternatively, instead of relying of Bison’s %define support, you may rely on the C/C++ preprocessor and define YYSTYPE as a macro, like this:

#define YYSTYPE double

This macro definition must go in the prologue of the grammar file (see section Outline of a Bison Grammar). If compatibility with POSIX Yacc matters to you, use this. Note however that Bison cannot know YYSTYPE’s value, not even whether it is defined, so there are services it cannot provide. Besides this works only for languages that have a preprocessor.

3.4.2 More Than One Value Type

In most programs, you will need different data types for different kinds of tokens and groupings. For example, a numeric constant may need type int or long int, while a string constant needs type char *, and an identifier might need a pointer to an entry in the symbol table.

To use more than one data type for semantic values in one parser, Bison requires you to do two things:

• Specify the entire collection of possible data types. There are several options:
  • let Bison compute the union type from the tags you assign to symbols;
  • use the %union Bison declaration (see section The Union Declaration);
  • define the %define variable api.value.type to be a union type whose members are the type tags (see section Providing a Structured Semantic Value Type);
  • use a typedef or a #define to define YYSTYPE to be a union type whose member names are the type tags.
• Choose one of those types for each symbol (terminal or nonterminal) for which semantic values are used. This is done for tokens with the %token Bison declaration (see section Token Type Names) and for groupings with the %type Bison declaration (see section Nonterminal Symbols).

3.4.3 Generating the Semantic Value Type

The special value union of the %define variable api.value.type instructs Bison that the tags used with the %token and %type directives are genuine types, not names of members of YYSTYPE.

For example:

%define api.value.type union
%token <int> INT "integer"
%token <int> 'n'
%type <int> expr
%token <char const *> ID "identifier"

generates an appropriate value of YYSTYPE to support each symbol type. The name of the member of YYSTYPE for tokens than have a declared identifier id (such as INT and ID above, but not 'n') is id. The other symbols have unspecified names on which you should not depend; instead, relying on C casts to access the semantic value with the appropriate type:

/* For an "integer".  */
yylval.INT = 42;
return INT;

/* For an 'n', also declared as int.  */
*((int*)&yylval) = 42;
return 'n';

/* For an "identifier".  */
yylval.ID = "42";
return ID;

If the %define variable api.token.prefix is defined (see section api.token.prefix), then it is also used to prefix the union member names. For instance, with ‘%define api.token.prefix {TOK_}’:

/* For an "integer".  */
yylval.TOK_INT = 42;
return TOK_INT;

This Bison extension cannot work if %yacc (or ‘-y’/‘--yacc’) is enabled, as POSIX mandates that Yacc generate tokens as macros (e.g., ‘#define INT 258’, or ‘#define TOK_INT 258’).

This feature is new, and user feedback would be most welcome.

A similar feature is provided for C++ that in addition overcomes C++ limitations (that forbid non-trivial objects to be part of a union): ‘%define api.value.type variant’, see C++ Variants.

3.4.4 The Union Declaration

The %union declaration specifies the entire collection of possible data types for semantic values. The keyword %union is followed by braced code containing the same thing that goes inside a union in C.

For example:

%union {
  double val;
  symrec *tptr;
}

This says that the two alternative types are double and symrec *. They are given names val and tptr; these names are used in the %token and %type declarations to pick one of the types for a terminal or nonterminal symbol (see section Nonterminal Symbols).

As an extension to POSIX, a tag is allowed after the %union. For example:

%union value {
  double val;
  symrec *tptr;
}

specifies the union tag value, so the corresponding C type is union value. If you do not specify a tag, it defaults to YYSTYPE.

As another extension to POSIX, you may specify multiple %union declarations; their contents are concatenated. However, only the first %union declaration can specify a tag.

Note that, unlike making a union declaration in C, you need not write a semicolon after the closing brace.

3.4.5 Providing a Structured Semantic Value Type

Instead of %union, you can define and use your own union type YYSTYPE if your grammar contains at least one ‘<type>’ tag. For example, you can put the following into a header file ‘parser.h’:

union YYSTYPE {
  double val;
  symrec *tptr;
};

and then your grammar can use the following instead of %union:

%{
#include "parser.h"
%}
%define api.value.type {union YYSTYPE}
%type <val> expr
%token <tptr> ID

Actually, you may also provide a struct rather that a union, which may be handy if you want to track information for every symbol (such as preceding comments).

The type you provide may even be structured and include pointers, in which case the type tags you provide may be composite, with ‘.’ and ‘->’ operators.

3.4.6 Actions

An action accompanies a syntactic rule and contains C code to be executed each time an instance of that rule is recognized. The task of most actions is to compute a semantic value for the grouping built by the rule from the semantic values associated with tokens or smaller groupings.

An action consists of braced code containing C statements, and can be placed at any position in the rule; it is executed at that position. Most rules have just one action at the end of the rule, following all the components. Actions in the middle of a rule are tricky and used only for special purposes (see section Actions in Mid-Rule).

The C code in an action can refer to the semantic values of the components matched by the rule with the construct $n, which stands for the value of the nth component. The semantic value for the grouping being constructed is $$. In addition, the semantic values of symbols can be accessed with the named references construct $name or $[name]. Bison translates both of these constructs into expressions of the appropriate type when it copies the actions into the parser implementation file. $$ (or $name, when it stands for the current grouping) is translated to a modifiable lvalue, so it can be assigned to.

Here is a typical example:

exp:
…
| exp '+' exp     { $$ = $1 + $3; }

Or, in terms of named references:

exp[result]:
…
| exp[left] '+' exp[right]  { $result = $left + $right; }

This rule constructs an exp from two smaller exp groupings connected by a plus-sign token. In the action, $1 and $3 ($left and $right) refer to the semantic values of the two component exp groupings, which are the first and third symbols on the right hand side of the rule. The sum is stored into $$ ($result) so that it becomes the semantic value of the addition-expression just recognized by the rule. If there were a useful semantic value associated with the ‘+’ token, it could be referred to as $2.

See section Named References, for more information about using the named references construct.

Note that the vertical-bar character ‘|’ is really a rule separator, and actions are attached to a single rule. This is a difference with tools like Flex, for which ‘|’ stands for either “or”, or “the same action as that of the next rule”. In the following example, the action is triggered only when ‘b’ is found:

a-or-b: 'a'|'b'   { a_or_b_found = 1; };

If you don’t specify an action for a rule, Bison supplies a default: $$ = $1. Thus, the value of the first symbol in the rule becomes the value of the whole rule. Of course, the default action is valid only if the two data types match. There is no meaningful default action for an empty rule; every empty rule must have an explicit action unless the rule’s value does not matter.

$n with n zero or negative is allowed for reference to tokens and groupings on the stack before those that match the current rule. This is a very risky practice, and to use it reliably you must be certain of the context in which the rule is applied. Here is a case in which you can use this reliably:

foo:
  expr bar '+' expr  { … }
| expr bar '-' expr  { … }
;

bar:
  %empty    { previous_expr = $0; }
;

As long as bar is used only in the fashion shown here, $0 always refers to the expr which precedes bar in the definition of foo.

It is also possible to access the semantic value of the lookahead token, if any, from a semantic action. This semantic value is stored in yylval. See section Special Features for Use in Actions.

3.4.7 Data Types of Values in Actions

If you have chosen a single data type for semantic values, the $$ and $n constructs always have that data type.

If you have used %union to specify a variety of data types, then you must declare a choice among these types for each terminal or nonterminal symbol that can have a semantic value. Then each time you use $$ or $n, its data type is determined by which symbol it refers to in the rule. In this example,

exp:
  …
| exp '+' exp    { $$ = $1 + $3; }

$1 and $3 refer to instances of exp, so they all have the data type declared for the nonterminal symbol exp. If $2 were used, it would have the data type declared for the terminal symbol '+', whatever that might be.

Alternatively, you can specify the data type when you refer to the value, by inserting ‘<type>’ after the ‘$’ at the beginning of the reference. For example, if you have defined types as shown here:

%union {
  int itype;
  double dtype;
}

then you can write $<itype>1 to refer to the first subunit of the rule as an integer, or $<dtype>1 to refer to it as a double.

3.4.8 Actions in Mid-Rule

Occasionally it is useful to put an action in the middle of a rule. These actions are written just like usual end-of-rule actions, but they are executed before the parser even recognizes the following components.

3.4.8.1 Using Mid-Rule Actions

A mid-rule action may refer to the components preceding it using $n, but it may not refer to subsequent components because it is run before they are parsed.

The mid-rule action itself counts as one of the components of the rule. This makes a difference when there is another action later in the same rule (and usually there is another at the end): you have to count the actions along with the symbols when working out which number n to use in $n.

The mid-rule action can also have a semantic value. The action can set its value with an assignment to $$, and actions later in the rule can refer to the value using $n. Since there is no symbol to name the action, there is no way to declare a data type for the value in advance, so you must use the ‘$<…>n’ construct to specify a data type each time you refer to this value.

There is no way to set the value of the entire rule with a mid-rule action, because assignments to $$ do not have that effect. The only way to set the value for the entire rule is with an ordinary action at the end of the rule.

Here is an example from a hypothetical compiler, handling a let statement that looks like ‘let (variable) statement’ and serves to create a variable named variable temporarily for the duration of statement. To parse this construct, we must put variable into the symbol table while statement is parsed, then remove it afterward. Here is how it is done:

stmt:
  "let" '(' var ')'
    {
      $<context>$ = push_context ();
      declare_variable ($3);
    }
  stmt
    {
      $$ = $6;
      pop_context ($<context>5);
    }

As soon as ‘let (variable)’ has been recognized, the first action is run. It saves a copy of the current semantic context (the list of accessible variables) as its semantic value, using alternative context in the data-type union. Then it calls declare_variable to add the new variable to that list. Once the first action is finished, the embedded statement stmt can be parsed.

Note that the mid-rule action is component number 5, so the ‘stmt’ is component number 6. Named references can be used to improve the readability and maintainability (see section Named References):

stmt:
  "let" '(' var ')'
    {
      $<context>let = push_context ();
      declare_variable ($3);
    }[let]
  stmt
    {
      $$ = $6;
      pop_context ($<context>let);
    }

After the embedded statement is parsed, its semantic value becomes the value of the entire let-statement. Then the semantic value from the earlier action is used to restore the prior list of variables. This removes the temporary let-variable from the list so that it won’t appear to exist while the rest of the program is parsed.

In the above example, if the parser initiates error recovery (see section Error Recovery) while parsing the tokens in the embedded statement stmt, it might discard the previous semantic context $<context>5 without restoring it. Thus, $<context>5 needs a destructor (see section Freeing Discarded Symbols). However, Bison currently provides no means to declare a destructor specific to a particular mid-rule action’s semantic value.

One solution is to bury the mid-rule action inside a nonterminal symbol and to declare a destructor for that symbol:

%type <context> let
%destructor { pop_context ($$); } let

%%

stmt:
  let stmt
    {
      $$ = $2;
      pop_context ($let);
    };

let:
  "let" '(' var ')'
    {
      $let = push_context ();
      declare_variable ($3);
    };

Note that the action is now at the end of its rule. Any mid-rule action can be converted to an end-of-rule action in this way, and this is what Bison actually does to implement mid-rule actions.

3.4.8.2 Mid-Rule Action Translation

As hinted earlier, mid-rule actions are actually transformed into regular rules and actions. The various reports generated by Bison (textual, graphical, etc., see Understanding Your Parser) reveal this translation, best explained by means of an example. The following rule:

exp: { a(); } "b" { c(); } { d(); } "e" { f(); };

is translated into:

$@1: %empty { a(); };
$@2: %empty { c(); };
$@3: %empty { d(); };
exp: $@1 "b" $@2 $@3 "e" { f(); };

with new nonterminal symbols $@n, where n is a number.

A mid-rule action is expected to generate a value if it uses $$, or the (final) action uses $n where n denote the mid-rule action. In that case its nonterminal is rather named @n:

exp: { a(); } "b" { $$ = c(); } { d(); } "e" { f = $1; };

is translated into

@1: %empty { a(); };
@2: %empty { $$ = c(); };
$@3: %empty { d(); };
exp: @1 "b" @2 $@3 "e" { f = $1; }

There are probably two errors in the above example: the first mid-rule action does not generate a value (it does not use $$ although the final action uses it), and the value of the second one is not used (the final action does not use $3). Bison reports these errors when the midrule-value warnings are enabled (see section Invoking Bison):

$ bison -fcaret -Wmidrule-value mid.y

mid.y:2.6-13: warning: unset value: $$
 exp: { a(); } "b" { $$ = c(); } { d(); } "e" { f = $1; };
      ^^^^^^^^

mid.y:2.19-31: warning: unused value: $3
 exp: { a(); } "b" { $$ = c(); } { d(); } "e" { f = $1; };
                   ^^^^^^^^^^^^^

3.4.8.3 Conflicts due to Mid-Rule Actions

Taking action before a rule is completely recognized often leads to conflicts since the parser must commit to a parse in order to execute the action. For example, the following two rules, without mid-rule actions, can coexist in a working parser because the parser can shift the open-brace token and look at what follows before deciding whether there is a declaration or not:

compound:
  '{' declarations statements '}'
| '{' statements '}'
;

But when we add a mid-rule action as follows, the rules become nonfunctional:

compound:
  { prepare_for_local_variables (); }
     '{' declarations statements '}'

|    '{' statements '}'
;

Now the parser is forced to decide whether to run the mid-rule action when it has read no farther than the open-brace. In other words, it must commit to using one rule or the other, without sufficient information to do it correctly. (The open-brace token is what is called the lookahead token at this time, since the parser is still deciding what to do about it. See section Lookahead Tokens.)

You might think that you could correct the problem by putting identical actions into the two rules, like this:

compound:
  { prepare_for_local_variables (); }
    '{' declarations statements '}'
| { prepare_for_local_variables (); }
    '{' statements '}'
;

But this does not help, because Bison does not realize that the two actions are identical. (Bison never tries to understand the C code in an action.)

If the grammar is such that a declaration can be distinguished from a statement by the first token (which is true in C), then one solution which does work is to put the action after the open-brace, like this:

compound:
  '{' { prepare_for_local_variables (); }
    declarations statements '}'
| '{' statements '}'
;

Now the first token of the following declaration or statement, which would in any case tell Bison which rule to use, can still do so.

Another solution is to bury the action inside a nonterminal symbol which serves as a subroutine:

subroutine:
  %empty  { prepare_for_local_variables (); }
;

compound:
  subroutine '{' declarations statements '}'
| subroutine '{' statements '}'
;

Now Bison can execute the action in the rule for subroutine without deciding which rule for compound it will eventually use.

This document was generated by Rick Perry on December 29, 2013 using texi2html 1.82.