# 高效的语法模糊化

> 原文： [https://www.fuzzingbook.org/html/GrammarFuzzer.html](https://www.fuzzingbook.org/html/GrammarFuzzer.html)

在语法的[一章中，我们已经看到了如何使用*语法*进行非常有效的测试。 在本章中，我们将以前的基于字符串的算法改进为基于树的算法，该算法速度更快，并且可以更好地控制模糊输入的产生。](Grammars.html)

本章中的算法是其他几种技术的基础。 因此，本章是书中的“枢纽”。

**前提条件**

*   您应该知道基于语法的模糊测试的工作原理，例如 摘自[语法](Grammars.html)一章。

## 内容提要

要使用本章中提供的代码来[，请编写](Importing.html)

```py
>>> from [fuzzingbook.GrammarFuzzer](GrammarFuzzer.html) import <identifier>

```

然后利用以下功能。

本章介绍`GrammarFuzzer`，这是一种高效的语法模糊器，它采用一种语法来产生语法上有效的输入字符串。 这是典型用法：

```py
>>> from [Grammars](Grammars.html) import US_PHONE_GRAMMAR
>>> phone_fuzzer = GrammarFuzzer(US_PHONE_GRAMMAR)
>>> phone_fuzzer.fuzz()
'(582)871-9500'

```

`GrammarFuzzer`构造函数采用许多关键字参数来控制其行为。 例如，`start_symbol`允许设置以扩展名开头的符号（而不是`<start>`）：

```py
>>> area_fuzzer = GrammarFuzzer(US_PHONE_GRAMMAR, start_symbol='<area>')
>>> area_fuzzer.fuzz()
'707'
>>> import [inspect](https://docs.python.org/3/library/inspect.html)
>>> print(inspect.getdoc(GrammarFuzzer.__init__))
Produce strings from `grammar`, starting with `start_symbol`.
If `min_nonterminals` or `max_nonterminals` is given, use them as limits 
for the number of nonterminals produced.  
If `disp` is set, display the intermediate derivation trees.
If `log` is set, show intermediate steps as text on standard output.

```

在内部，`GrammarFuzzer`使用[派生树](#Derivation-Trees)，并逐步扩展。 生成字符串后，可以在`derivation_tree`属性中访问生成的树。

```py
>>> display_tree(phone_fuzzer.derivation_tree)

```

!

在派生树的内部表示中，*节点*是一对（`symbol`，`children`）。 对于非终端，`symbol`是正在扩展的符号，`children`是其他节点的列表。 对于终端，`symbol`是终端字符串，`children`为空。

```py
>>> phone_fuzzer.derivation_tree
('<start>',
 [('<phone-number>',
   [('(', []),
    ('<area>',
     [('<lead-digit>', [('5', [])]),
      ('<digit>', [('8', [])]),
      ('<digit>', [('2', [])])]),
    (')', []),
    ('<exchange>',
     [('<lead-digit>', [('8', [])]),
      ('<digit>', [('7', [])]),
      ('<digit>', [('1', [])])]),
    ('-', []),
    ('<line>',
     [('<digit>', [('9', [])]),
      ('<digit>', [('5', [])]),
      ('<digit>', [('0', [])]),
      ('<digit>', [('0', [])])])])])

```

本章包含各种与派生树配合使用的助手，包括可视化工具。

## 算法

在[的前一章](Grammars.html)中，我们介绍了`simple_grammar_fuzzer()`函数，该函数采用语法并自动从中产生语法上有效的字符串。 但是，`simple_grammar_fuzzer()`的名称恰如其分-简单。 为了说明问题，让我们回到在语法的[一章中从`EXPR_GRAMMAR_BNF`创建的`expr_grammar`：](Grammars.html)

```py
import [fuzzingbook_utils](https://github.com/uds-se/fuzzingbook/tree/master/notebooks/fuzzingbook_utils)

```

```py
from [fuzzingbook_utils](https://github.com/uds-se/fuzzingbook/tree/master/notebooks/fuzzingbook_utils) import unicode_escape

```

```py
from [Grammars](Grammars.html) import EXPR_EBNF_GRAMMAR, convert_ebnf_grammar, simple_grammar_fuzzer, is_valid_grammar, exp_string, exp_opts

```

```py
expr_grammar = convert_ebnf_grammar(EXPR_EBNF_GRAMMAR)
expr_grammar

```

```py
{'<start>': ['<expr>'],
 '<expr>': ['<term> + <expr>', '<term> - <expr>', '<term>'],
 '<term>': ['<factor> * <term>', '<factor> / <term>', '<factor>'],
 '<factor>': ['<sign-1><factor>', '(<expr>)', '<integer><symbol-1>'],
 '<sign>': ['+', '-'],
 '<integer>': ['<digit-1>'],
 '<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],
 '<symbol>': ['.<integer>'],
 '<sign-1>': ['', '<sign>'],
 '<symbol-1>': ['', '<symbol>'],
 '<digit-1>': ['<digit>', '<digit><digit-1>']}

```

`expr_grammar`具有有趣的属性。 如果我们将其输入`simple_grammar_fuzzer()`，该函数将陷入无限扩展中：

```py
from [ExpectError](ExpectError.html) import ExpectTimeout

```

```py
with ExpectTimeout(1):
    simple_grammar_fuzzer(grammar=expr_grammar, max_nonterminals=3)

```

```py
Traceback (most recent call last):
  File "<ipython-input-6-fbcda5f486bb>", line 2, in <module>
    simple_grammar_fuzzer(grammar=expr_grammar, max_nonterminals=3)
  File "<string>", line 8, in simple_grammar_fuzzer
  File "<string>", line 7, in nonterminals
  File "/Users/zeller/anaconda3/lib/python3.6/re.py", line 222, in findall
    return _compile(pattern, flags).findall(string)
  File "/Users/zeller/anaconda3/lib/python3.6/re.py", line 288, in _compile
    try:
  File "/Users/zeller/anaconda3/lib/python3.6/re.py", line 288, in _compile
    try:
  File "<string>", line 16, in check_time
TimeoutError (expected)

```

为什么会这样？ 问题在于此规则：

```py
expr_grammar['<factor>']

```

```py
['<sign-1><factor>', '(<expr>)', '<integer><symbol-1>']

```

此处，除了`(expr)`以外的任何选择都会增加符号的数量，即使只是暂时的。 由于我们对要扩展的符号数设置了硬性限制，因此扩展`<factor>`剩下的唯一选择是`(<expr>)`，这将导致无限地添加括号。

潜在无限扩展的问题只是`simple_grammar_fuzzer()`的问题之一。 更多问题包括：

1.  *效率低下*。 每次迭代时，此模糊器将搜索到目前为止产生的字符串以扩展符号。 随着生产线的增长，这变得效率低下。

2.  *很难控制。* 如上所述，即使在限制符号数量的情况下，仍然可以获得非常长的字符串，甚至无限长的字符串。

让我们通过绘制不同长度的字符串所需的时间来说明这两个问题。

```py
from [Grammars](Grammars.html) import simple_grammar_fuzzer

```

```py
from [Grammars](Grammars.html) import START_SYMBOL, EXPR_GRAMMAR, URL_GRAMMAR, CGI_GRAMMAR

```

```py
from [Grammars](Grammars.html) import RE_NONTERMINAL, nonterminals, is_nonterminal

```

```py
from [Timer](Timer.html) import Timer

```

```py
trials = 50
xs = []
ys = []
for i in range(trials):
    with Timer() as t:
        s = simple_grammar_fuzzer(EXPR_GRAMMAR, max_nonterminals=15)
    xs.append(len(s))
    ys.append(t.elapsed_time())
    print(i, end=" ")
print()

```

```py
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 

```

```py
average_time = sum(ys) / trials
print("Average time:", average_time)

```

```py
Average time: 0.2103420232999997

```

```py
%matplotlib inline

import [matplotlib.pyplot](https://docs.python.org/3/library/matplotlib.pyplot.html) as [plt](https://docs.python.org/3/library/plt.html)
plt.scatter(xs, ys)
plt.title('Time required for generating an output')

```

```py
Text(0.5,1,'Time required for generating an output')

```

![](data:image/png;base64, 
)

我们看到（1）工作量随时间呈二次方增长，并且（2）我们可以轻松产生数万个字符长的输出。

为了解决这些问题，我们需要一种*更智能的算法*-一种效率更高的算法，可以更好地控制扩展，并且可以在`expr_grammar`中预见到`(expr)`替代方案会产生 与其他两个相反，无限扩展。

## 派生树

为了获得更有效的算法*和*更好地控制扩展，我们将对语法产生的字符串使用特殊表示形式。 总体思路是使用*树*结构，该结构随后将进行扩展–所谓的*派生树*。 这种表示形式使我们能够始终跟踪我们的扩展状态-回答问题，例如哪些元素已扩展为其他元素以及哪些符号仍需要扩展。 此外，向树中添加新元素比一次又一次地替换字符串要有效得多。

像在编程中使用的其他树一样，派生树（也称为*解析树*或*具体语法树*）由具有其他节点（称为*的*节点*）组成 ]子节点*）作为其*子节点*。 树开始于一个没有父节点的节点； 这称为*根节点*； 没有子节点的节点称为*叶*。

在以下步骤中，使用语法章节中的算术语法[来说明派生树的语法扩展过程。 我们从一个树的根节点开始，代表*起始符号* –在我们的例子中是`<start>`。](Grammars.html)

```py
tree

```

<svg height="23pt" viewBox="0.00 0.00 48.00 23.00" width="48pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 19)"><title>root</title> <g class="node" id="node1"><title>\<start\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-3.8"><start></text></g></g></svg>

为了扩展树，我们遍历它，搜索没有子元素的非终结符号$ S $。 因此，$ S $是仍然需要扩展的符号。 然后，我们从语法中选择$ S $的扩展。 然后，将扩展项添加为$ S $的新子项。 对于我们的起始符号`<start>`，唯一的扩展名是`<expr>`，因此我们将其添加为子代。

```py
tree

```

<svg height="74pt" viewBox="0.00 0.00 49.00 74.00" width="49pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 70)"><title>root</title> <g class="node" id="node1"><title>\<start\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20.5" y="-54.8"><start></text></g> <g class="node" id="node2"><title>\<expr\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20.5" y="-3.8"><expr></text></g> <g class="edge" id="edge1"><title>\<start\>->\<expr\></title></g></g></svg>

为了从派生树构造生成的字符串，我们按顺序遍历树，并在树的叶子处收集符号。 在上述情况下，我们获得字符串`"<expr>"`。

要进一步扩展树，我们选择另一个符号进行扩展，并将其扩展添加为新子代。 这将使我们获得`<expr>`符号，该符号被扩展为`<expr> + <term>`，并添加了三个子代。

```py
tree

```

<svg height="125pt" viewBox="0.00 0.00 139.50 125.00" width="140pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 121)"><title>root</title> <g class="node" id="node1"><title>\<start\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="67.5" y="-105.8"><start></text></g> <g class="node" id="node2"><title>\<expr\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="67.5" y="-54.8"><expr></text></g> <g class="edge" id="edge1"><title>\<start\>->\<expr\></title></g> <g class="node" id="node3"><title>\<expr\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="22.5" y="-3.8"><expr></text></g> <g class="edge" id="edge2"><title>\<expr\>->\<expr\></title></g> <g class="node" id="node4"><title>+</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="67.5" y="-3.8">+</text></g> <g class="edge" id="edge3"><title>\<expr\>->+</title></g> <g class="node" id="node5"><title>\<term\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="110.5" y="-3.8"><term></text></g> <g class="edge" id="edge4"><title>\<expr\>->\<term\></title></g></g></svg>

我们重复扩展，直到没有符号可扩展为止：

```py
tree

```

<svg height="380pt" viewBox="0.00 0.00 153.00 380.00" width="153pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 376)"><title>root</title> <g class="node" id="node1"><title>\<start\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="74.5" y="-360.8"><start></text></g> <g class="node" id="node2"><title>\<expr\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="74.5" y="-309.8"><expr></text></g> <g class="edge" id="edge1"><title>\<start\>->\<expr\></title></g> <g class="node" id="node3"><title>\<expr\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29.5" y="-258.8"><expr></text></g> <g class="edge" id="edge2"><title>\<expr\>->\<expr\></title></g> <g class="node" id="node4"><title>+</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="74.5" y="-258.8">+</text></g> <g class="edge" id="edge3"><title>\<expr\>->+</title></g> <g class="node" id="node5"><title>\<term\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="117.5" y="-258.8"><term></text></g> <g class="edge" id="edge4"><title>\<expr\>->\<term\></title></g> <g class="node" id="node6"><title>\<term\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29.5" y="-207.8"><term></text></g> <g class="edge" id="edge5"><title>\<expr\> ->\<term\></title></g> <g class="node" id="node11"><title>\<factor\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="117.5" y="-207.8"><factor></text></g> <g class="edge" id="edge10"><title>\<term\>->\<factor\></title></g> <g class="node" id="node7"><title>\<factor\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29.5" y="-156.8"><factor></text></g> <g class="edge" id="edge6"><title>\<term\> ->\<factor\></title></g> <g class="node" id="node8"><title>\<integer\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29.5" y="-105.8"><integer></text></g> <g class="edge" id="edge7"><title>\<factor\> ->\<integer\></title></g> <g class="node" id="node9"><title>\<digit\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29.5" y="-54.8"><digit></text></g> <g class="edge" id="edge8"><title>\<integer\> ->\<digit\></title></g> <g class="node" id="node10"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29.5" y="-3.8">2</text></g> <g class="edge" id="edge9"><title>\<digit\> ->2</title></g> <g class="node" id="node12"><title>\<integer\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="117.5" y="-156.8"><integer></text></g> <g class="edge" id="edge11"><title>\<factor\>->\<integer\></title></g> <g class="node" id="node13"><title>\<digit\></title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="117.5" y="-105.8"><digit></text></g> <g class="edge" id="edge12"><title>\<integer\>->\<digit\></title></g> <g class="node" id="node14"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="117.5" y="-54.8">2</text></g> <g class="edge" id="edge13"><title>\<digit\>->2</title></g></g></svg>

现在，我们有了字符串`2 + 2`的表示形式。 但是，与单独的字符串相反，派生树记录*所生成字符串的整个结构*（和生产历史，或*派生*历史）。 它还允许进行简单的比较和操作-例如，将一个子树（子结构）替换为另一子树。

## 表示派生树

为了表示Python中的派生树，我们使用以下格式。 一个节点是一对

```py
(SYMBOL_NAME, CHILDREN)

```

其中`SYMBOL_NAME`是代表节点的字符串（即`"<start>"`或`"+"`），`CHILDREN`是子节点的列表。

`CHILDREN`可以采用一些特殊值：

1.  `None`作为将来扩展的占位符。 这意味着该节点是*非终端符号*，应进一步扩展。
2.  `[]`（即空白列表），表示没有*个*子级。 这意味着该节点是无法再扩展的*终端符号*。

让我们采用一个非常简单的派生树，表示上面的中间步骤`<expr> + <term>`。

```py
derivation_tree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])

```

为了更好地理解该树的结构，让我们介绍一个可视化该树的函数。 我们通过算法使用`graphviz`包中的`dot`绘图程序，遍历上述结构。 （除非您对树的可视化非常感兴趣，可以直接跳到下面的示例。）

```py
from [graphviz](https://docs.python.org/3/library/graphviz.html) import Digraph

```

```py
from [IPython.display](https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html) import display

```

```py
import [re](https://docs.python.org/3/library/re.html)

```

```py
def dot_escape(s):
    """Return s in a form suitable for dot"""
    s = re.sub(r'([^a-zA-Z0-9" ])', r"\\\1", s)
    return s

```

```py
assert dot_escape("hello") == "hello"
assert dot_escape("<hello>, world") == "\\<hello\\>\\, world"
assert dot_escape("\\n") == "\\\\n"

```

尽管我们目前对可视化`derivation_tree`感兴趣，但对可视化过程进行一般化也符合我们的利益。 特别是，如果我们的方法`display_tree()`可以显示*任何*树之类的数据结构，则将很有帮助。 为此，我们定义了一个辅助方法`extract_node()`，该方法从给定的数据结构中提取当前符号和子代。 默认实现只是从任何`derivation_tree`节点中提取符号，子级和注释。

```py
def extract_node(node, id):
    symbol, children, *annotation = node
    return symbol, children, ''.join(str(a) for a in annotation)

```

在可视化树时，通常以不同的方式显示某些节点​​很有用。 例如，有时在未处理节点和已处理节点之间进行区分很有用。 我们定义了一个帮助程序`default_node_attr()`，它提供了基本显示，可以由用户自定义。

```py
def default_node_attr(dot, nid, symbol, ann):
    dot.node(repr(nid), dot_escape(unicode_escape(symbol)))

```

与节点相似，边缘也可能需要修改。 我们将`default_edge_attr()`定义为可以由用户自定义的帮助程序。

```py
def default_edge_attr(dot, start_node, stop_node):
    dot.edge(repr(start_node), repr(stop_node))

```

在可视化一棵树时，有时可能希望更改树的外观。 例如，如果树是从左到右而不是从上到下布置的，则有时更容易查看树。 为此，我们定义了另一个帮助程序`default_graph_attr()`。

```py
def default_graph_attr(dot):
    dot.attr('node', shape='plain')

```

最后，我们定义了一种方法`display_tree()`，该方法接受这四个函数`extract_node()`，`default_edge_attr()`，`default_node_attr()`和`default_graph_attr()`并使用它们显示树。

```py
def display_tree(derivation_tree,
                 log=False,
                 extract_node=extract_node,
                 node_attr=default_node_attr,
                 edge_attr=default_edge_attr,
                 graph_attr=default_graph_attr):

    # If we import display_tree, we also have to import its functions
    from [graphviz](https://docs.python.org/3/library/graphviz.html) import Digraph

    counter = 0

    def traverse_tree(dot, tree, id=0):
        (symbol, children, annotation) = extract_node(tree, id)
        node_attr(dot, id, symbol, annotation)

        if children:
            for child in children:
                nonlocal counter
                counter += 1
                child_id = counter
                edge_attr(dot, id, child_id)
                traverse_tree(dot, child, child_id)

    dot = Digraph(comment="Derivation Tree")
    graph_attr(dot)
    traverse_tree(dot, derivation_tree)
    if log:
        print(dot)
    return dot

```

这是我们的树可视化为：

```py
display_tree(derivation_tree)

```

<svg height="125pt" viewBox="0.00 0.00 142.50 125.00" width="143pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 121)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-105.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-54.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20.5" y="-3.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-3.8">+</text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="113.5" y="-3.8"><term></text></g> <g class="edge" id="edge4"><title>1->4</title></g></g></svg>

假设我们要自定义输出，并在其中注释某些节点和边。 这是一种方法`display_annotated_tree()`，它显示带注释的树结构，并从左到右放置图形。

```py
def display_annotated_tree(tree, a_nodes, a_edges, log=False):
    def graph_attr(dot):
        dot.attr('node', shape='plain')
        dot.graph_attr['rankdir'] = 'LR'

    def annotate_node(dot, nid, symbol, ann):
        if nid in a_nodes:
            dot.node(repr(nid), "%s (%s)" % (dot_escape(unicode_escape(symbol)), a_nodes[nid]))
        else:
            dot.node(repr(nid), dot_escape(unicode_escape(symbol)))

    def annotate_edge(dot, start_node, stop_node):
        if (start_node, stop_node) in a_edges:
            dot.edge(repr(start_node), repr(stop_node),
                     a_edges[(start_node, stop_node)])
        else:
            dot.edge(repr(start_node), repr(stop_node))

    return display_tree(tree, log=log,
                 node_attr=annotate_node,
                 edge_attr=annotate_edge,
                 graph_attr=graph_attr)

```

```py
display_annotated_tree(derivation_tree, {3: 'plus'}, {(1, 3): 'op'}, log=False)

```

<svg height="89pt" viewBox="0.00 0.00 229.00 89.00" width="229pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 85)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-36.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-36.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="195" y="-69.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="195" y="-36.8">+  (plus)</text></g> <g class="edge" id="edge3"><title>1->3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="143.5" y="-44.3">op</text></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="195" y="-3.8"><term></text></g> <g class="edge" id="edge4"><title>1->4</title></g></g></svg>

如果要查看树中的所有叶子节点，可以使用以下`all_terminals()`函数：

```py
def all_terminals(tree):
    (symbol, children) = tree
    if children is None:
        # This is a nonterminal symbol not expanded yet
        return symbol

    if len(children) == 0:
        # This is a terminal symbol
        return symbol

    # This is an expanded symbol:
    # Concatenate all terminal symbols from all children
    return ''.join([all_terminals(c) for c in children])

```

```py
all_terminals(derivation_tree)

```

```py
'<expr> + <term>'

```

`all_terminals()`函数返回所有叶节点的字符串表示形式。 但是，其中一些叶节点可能是由于非终结符派生空字符串所致。 对于这些，我们要返回空字符串。 因此，我们定义了一个新函数`tree_to_string()`，可从树状结构中检索出原始字符串。

```py
def tree_to_string(tree):
    symbol, children, *_ = tree
    if children:
        return ''.join(tree_to_string(c) for c in children)
    else:
        return '' if is_nonterminal(symbol) else symbol

```

```py
tree_to_string(derivation_tree)

```

```py
' + '

```

## 扩展节点

现在让我们开发一种算法，该算法采用带有未扩展符号的树（例如，上面的`derivation_tree`），然后依次扩展所有这些符号。 与早期的模糊测试类似，我们创建了`Fuzzer`的特殊子类-在本例中为`GrammarFuzzer`。 `GrammarFuzzer`得到一个语法和一个开始符号； 其他参数将在以后用于进一步控制创建并支持调试。

```py
from [Fuzzer](Fuzzer.html) import Fuzzer

```

```py
class GrammarFuzzer(Fuzzer):
    def __init__(self, grammar, start_symbol=START_SYMBOL,
                 min_nonterminals=0, max_nonterminals=10, disp=False, log=False):
        """Produce strings from `grammar`, starting with `start_symbol`.
 If `min_nonterminals` or `max_nonterminals` is given, use them as limits 
 for the number of nonterminals produced. 
 If `disp` is set, display the intermediate derivation trees.
 If `log` is set, show intermediate steps as text on standard output."""

        self.grammar = grammar
        self.start_symbol = start_symbol
        self.min_nonterminals = min_nonterminals
        self.max_nonterminals = max_nonterminals
        self.disp = disp
        self.log = log
        self.check_grammar()

```

在下文中，我们将使用已经为[引入`MutationFuzzer`类](MutationFuzzer.html)的技巧，向`GrammarFuzzer`添加更多方法。 构造

```py
class GrammarFuzzer(GrammarFuzzer):
    def new_method(self, args):
        pass

```

允许我们向`GrammarFuzzer`类添加新方法`new_method()`。 （实际上，我们得到了一个新的`GrammarFuzzer`类，该类扩展了旧的类，但是对于我们所有的目的，这都没有关系。）

使用此技巧，让我们定义一个辅助方法`check_grammar()`，该方法检查给定语法的一致性：

```py
class GrammarFuzzer(GrammarFuzzer):
    def check_grammar(self):
        assert self.start_symbol in self.grammar
        assert is_valid_grammar(
            self.grammar,
            start_symbol=self.start_symbol,
            supported_opts=self.supported_opts())

    def supported_opts(self):
        return set()

```

现在让我们定义一个辅助方法`init_tree()`，该方法仅使用开始符号来构建树：

```py
class GrammarFuzzer(GrammarFuzzer):
    def init_tree(self):
        return (self.start_symbol, None)

```

```py
f = GrammarFuzzer(EXPR_GRAMMAR)
display_tree(f.init_tree())

```

<svg height="23pt" viewBox="0.00 0.00 48.00 23.00" width="48pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 19)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-3.8"><start></text></g></g></svg>

接下来，我们将需要一个辅助函数`expansion_to_children()`，它将展开字符串并将其分解为派生树列表-字符串中的每个符号（末端或非末端）一个。 它使用`re.split()`方法将扩展字符串拆分为子节点列表：

```py
def expansion_to_children(expansion):
    # print("Converting " + repr(expansion))
    # strings contains all substrings -- both terminals and nonterminals such
    # that ''.join(strings) == expansion

    expansion = exp_string(expansion)
    assert isinstance(expansion, str)

    if expansion == "":  # Special case: epsilon expansion
        return [("", [])]

    strings = re.split(RE_NONTERMINAL, expansion)
    return [(s, None) if is_nonterminal(s) else (s, [])
            for s in strings if len(s) > 0]

```

```py
expansion_to_children("<term> + <expr>")

```

```py
[('<term>', None), (' + ', []), ('<expr>', None)]

```

*ε扩展*的情况，即像`<symbol> ::=`一样扩展为空字符串需要特殊处理：

```py
expansion_to_children("")

```

```py
[('', [])]

```

就像有关语法的[一章中的`nonterminals()`一样，我们提供了将来的扩展，使扩展成为具有额外数​​据的元组（将被忽略）。](Grammars.html)

```py
expansion_to_children(("+<term>", ["extra_data"]))

```

```py
[('+', []), ('<term>', None)]

```

我们在`GrammarFuzzer`中将这种帮助器实现为一种方法，以便它可以被子类重载：

```py
class GrammarFuzzer(GrammarFuzzer):
    def expansion_to_children(self, expansion):
        return expansion_to_children(expansion)

```

这样，我们现在可以在树中获取一些未扩展的节点，选择随机扩展，然后返回新树。 这就是方法`expand_node_randomly()`的工作，它使用辅助函数`choose_node_expansion()`从可能的子级数组中随机选择一个索引。 （`choose_node_expansion()`可以在子类中重载。）

```py
import [random](https://docs.python.org/3/library/random.html)

```

```py
class GrammarFuzzer(GrammarFuzzer):
    def choose_node_expansion(self, node, possible_children):
        """Return index of expansion in `possible_children` to be selected.  Defaults to random."""
        return random.randrange(0, len(possible_children))

    def expand_node_randomly(self, node):
        (symbol, children) = node
        assert children is None

        if self.log:
            print("Expanding", all_terminals(node), "randomly")

        # Fetch the possible expansions from grammar...
        expansions = self.grammar[symbol]
        possible_children = [self.expansion_to_children(
            expansion) for expansion in expansions]

        # ... and select a random expansion
        index = self.choose_node_expansion(node, possible_children)
        chosen_children = possible_children[index]

        # Process children (for subclasses)
        chosen_children = self.process_chosen_children(chosen_children,
                                                       expansions[index])

        # Return with new children
        return (symbol, chosen_children)

```

通用的`expand_node()`方法可以稍后用于选择不同的扩展策略； 截至目前，它仅使用`expand_node_randomly()`。

```py
class GrammarFuzzer(GrammarFuzzer):
    def expand_node(self, node):
        return self.expand_node_randomly(node)

```

辅助功能`process_chosen_children()`不执行任何操作。 子类可以重载它，以处理选定的子项。

```py
class GrammarFuzzer(GrammarFuzzer):
    def process_chosen_children(self, chosen_children, expansion):
        """Process children after selection.  By default, does nothing."""
        return chosen_children

```

`expand_node_randomly()`的工作方式如下：

```py
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)

print("Before:")
tree = ("<integer>", None)
display_tree(tree)

```

```py
Before:

```

<svg height="23pt" viewBox="0.00 0.00 63.00 23.00" width="63pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 19)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-3.8"><integer></text></g></g></svg>

```py
print("After:")
tree = f.expand_node_randomly(tree)
display_tree(tree)

```

```py
After:
Expanding <integer> randomly

```

<svg height="74pt" viewBox="0.00 0.00 123.50 74.00" width="124pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 70)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="54" y="-54.8"><integer></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="21" y="-3.8"><digit></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="88" y="-3.8"><integer></text></g> <g class="edge" id="edge2"><title>0->2</title></g></g></svg>

## 展开树

现在让我们将上述节点扩展应用于树中的某个节点。 为此，我们首先需要在树中搜索未扩展的节点。 `possible_expansions()`计算一棵树中有多少个未扩展符号：

```py
class GrammarFuzzer(GrammarFuzzer):
    def possible_expansions(self, node):
        (symbol, children) = node
        if children is None:
            return 1

        return sum(self.possible_expansions(c) for c in children)

```

```py
f = GrammarFuzzer(EXPR_GRAMMAR)
print(f.possible_expansions(derivation_tree))

```

```py
2

```

如果树有任何未扩展的节点，则方法`any_possible_expansions()`返回True。

```py
class GrammarFuzzer(GrammarFuzzer):
    def any_possible_expansions(self, node):
        (symbol, children) = node
        if children is None:
            return True

        return any(self.any_possible_expansions(c) for c in children)

```

```py
f = GrammarFuzzer(EXPR_GRAMMAR)
f.any_possible_expansions(derivation_tree)

```

```py
True

```

`expand_tree_once()`是我们的树扩展算法的核心方法。 它首先检查它当前是否正在不扩展的情况下应用于非终端符号。 如果是这样，它将如上所述调用`expand_node()`。

如果该节点已经展开（即具有子节点），则检查仍具有未扩展符号的子节点的子集，随机选择其中一个，然后将其自身递归应用于该子节点。

`expand_tree_once()`方法在位置处替换子级*，这意味着它实际上使作为参数传递的树发生了变异，而不是返回新树。 这种原位突变使此功能特别有效。 同样，我们使用辅助方法（`choose_tree_expansion()`）从可以扩展的子级列表中返回所选索引。*

```py
class GrammarFuzzer(GrammarFuzzer):
    def choose_tree_expansion(self, tree, children):
        """Return index of subtree in `children` to be selected for expansion.  Defaults to random."""
        return random.randrange(0, len(children))

    def expand_tree_once(self, tree):
        """Choose an unexpanded symbol in tree; expand it.  Can be overloaded in subclasses."""
        (symbol, children) = tree
        if children is None:
            # Expand this node
            return self.expand_node(tree)

        # Find all children with possible expansions
        expandable_children = [
            c for c in children if self.any_possible_expansions(c)]

        # `index_map` translates an index in `expandable_children`
        # back into the original index in `children`
        index_map = [i for (i, c) in enumerate(children)
                     if c in expandable_children]

        # Select a random child
        child_to_be_expanded = \
            self.choose_tree_expansion(tree, expandable_children)

        # Expand in place
        children[index_map[child_to_be_expanded]] = \
            self.expand_tree_once(expandable_children[child_to_be_expanded])

        return tree

```

让我们使用它，从上方扩展两次衍生树。

```py
derivation_tree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])
display_tree(derivation_tree)

```

<svg height="125pt" viewBox="0.00 0.00 142.50 125.00" width="143pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 121)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-105.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-54.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20.5" y="-3.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-3.8">+</text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="113.5" y="-3.8"><term></text></g> <g class="edge" id="edge4"><title>1->4</title></g></g></svg>

```py
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)
derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

```

```py
Expanding <term> randomly

```

<svg height="176pt" viewBox="0.00 0.00 187.50 176.00" width="188pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-105.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20.5" y="-54.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-54.8">+</text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="113.5" y="-54.8"><term></text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="65.5" y="-3.8"><factor></text></g> <g class="edge" id="edge5"><title>4->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="113.5" y="-3.8">/</text></g> <g class="edge" id="edge6"><title>4->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="158.5" y="-3.8"><term></text></g> <g class="edge" id="edge7"><title>4->7</title></g></g></svg>

```py
derivation_tree = f.expand_tree_once(derivation_tree)
display_tree(derivation_tree)

```

```py
Expanding <expr> randomly

```

<svg height="176pt" viewBox="0.00 0.00 299.00 176.00" width="299pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147" y="-105.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="93" y="-54.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147" y="-54.8">+</text></g> <g class="edge" id="edge6"><title>1->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="198" y="-54.8"><term></text></g> <g class="edge" id="edge7"><title>1->7</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="21" y="-3.8"><term></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="68" y="-3.8">+</text></g> <g class="edge" id="edge4"><title>2->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="114" y="-3.8"><expr></text></g> <g class="edge" id="edge5"><title>2->5</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="177" y="-3.8"><factor></text></g> <g class="edge" id="edge8"><title>7->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="225" y="-3.8">/</text></g> <g class="edge" id="edge9"><title>7->9</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="270" y="-3.8"><term></text></g> <g class="edge" id="edge10"><title>7->10</title></g></g></svg>

我们看到，每一步都会扩展一个符号。 现在所有要做的就是一次又一次地应用它，使树越来越远。

## 关闭扩展

使用`expand_tree_once()`，我们可以继续扩展树–但是我们如何实际上停止？ Luke在[ [Luke *等人*等，2000。](https://doi.org/10.1109/4235.873237)]中引入的关键思想是，在将派生树膨胀到某个最大大小之后，我们*只想应用 将树的大小增加最小的扩展*。 例如，对于`<factor>`，我们希望将其扩展为`<integer>`，因为这不会带来进一步的递归（和潜在的规模膨胀）。 对于`<integer>`，同样，最好将其扩展为`<digit>`，因为与`<digit><integer>`相比，树的大小增加较少。

为了确定扩展符号的*成本*，我们引入了两个相互依赖的函数：

*   `symbol_cost()`返回符号的所有扩展的最低成本，使用`expansion_cost()`计算每次扩展的成本。
*   `expansion_cost()`返回`expansions`中所有扩展的总和。 如果遍历过程中再次遇到非终结符，则扩展成本为$ \ infty $，表示（可能是无限的）递归。

```py
class GrammarFuzzer(GrammarFuzzer):
    def symbol_cost(self, symbol, seen=set()):
        expansions = self.grammar[symbol]
        return min(self.expansion_cost(e, seen | {symbol}) for e in expansions)

    def expansion_cost(self, expansion, seen=set()):
        symbols = nonterminals(expansion)
        if len(symbols) == 0:
            return 1  # no symbol

        if any(s in seen for s in symbols):
            return float('inf')

        # the value of a expansion is the sum of all expandable variables
        # inside + 1
        return sum(self.symbol_cost(s, seen) for s in symbols) + 1

```

这里有两个示例：扩展数字的最低成本为1，因为我们必须从其扩展之一中进行选择。

```py
f = GrammarFuzzer(EXPR_GRAMMAR)
assert f.symbol_cost("<digit>") == 1

```

但是，扩展`<expr>`的最小成本为5，因为这是所需的最小扩展数。 （`<expr>` $ \ rightarrow $ `<term>` $ \ rightarrow $ `<factor>` $ \ rightarrow $ `<integer>` $ \ rightarrow $ `<digit>` $ \ rightarrow $ 1）

```py
assert f.symbol_cost("<expr>") == 5

```

现在是`expand_node()`的变体，它考虑了上述成本。 它确定所有子项的最低成本`cost`，然后使用`choose`功能从列表中选择一个子项，默认情况下这是最低成本。 如果多个孩子都具有相同的最低费用，它将在这些孩子之间随机选择。

```py
class GrammarFuzzer(GrammarFuzzer):
    def expand_node_by_cost(self, node, choose=min):
        (symbol, children) = node
        assert children is None

        # Fetch the possible expansions from grammar...
        expansions = self.grammar[symbol]

        possible_children_with_cost = [(self.expansion_to_children(expansion),
                                        self.expansion_cost(
                                            expansion, {symbol}),
                                        expansion)
                                       for expansion in expansions]

        costs = [cost for (child, cost, expansion)
                 in possible_children_with_cost]
        chosen_cost = choose(costs)
        children_with_chosen_cost = [child for (child, child_cost, _) in possible_children_with_cost
                                     if child_cost == chosen_cost]
        expansion_with_chosen_cost = [expansion for (_, child_cost, expansion) in possible_children_with_cost
                                      if child_cost == chosen_cost]

        index = self.choose_node_expansion(node, children_with_chosen_cost)

        chosen_children = children_with_chosen_cost[index]
        chosen_expansion = expansion_with_chosen_cost[index]
        chosen_children = self.process_chosen_children(
            chosen_children, chosen_expansion)

        # Return with a new list
        return (symbol, chosen_children)

```

快捷方式`expand_node_min_cost()`通过`min()`作为`choose`功能，从而使其以最小的成本扩展了节点。

```py
class GrammarFuzzer(GrammarFuzzer):
    def expand_node_min_cost(self, node):
        if self.log:
            print("Expanding", all_terminals(node), "at minimum cost")

        return self.expand_node_by_cost(node, min)

```

现在，我们可以使用此函数来关闭派生树的扩展，将`expand_tree_once()`与上面的`expand_node_min_cost()`用作扩展函数。

```py
class GrammarFuzzer(GrammarFuzzer):
    def expand_node(self, node):
        return self.expand_node_min_cost(node)

```

```py
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)
display_tree(derivation_tree)

```

<svg height="176pt" viewBox="0.00 0.00 299.00 176.00" width="299pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147" y="-105.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="93" y="-54.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147" y="-54.8">+</text></g> <g class="edge" id="edge6"><title>1->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="198" y="-54.8"><term></text></g> <g class="edge" id="edge7"><title>1->7</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="21" y="-3.8"><term></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="68" y="-3.8">+</text></g> <g class="edge" id="edge4"><title>2->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="114" y="-3.8"><expr></text></g> <g class="edge" id="edge5"><title>2->5</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="177" y="-3.8"><factor></text></g> <g class="edge" id="edge8"><title>7->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="225" y="-3.8">/</text></g> <g class="edge" id="edge9"><title>7->9</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="270" y="-3.8"><term></text></g> <g class="edge" id="edge10"><title>7->10</title></g></g></svg>

```py
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
    display_tree(derivation_tree)

```

```py
Expanding <term> at minimum cost

```

```py
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
    display_tree(derivation_tree)

```

```py
Expanding <factor> at minimum cost

```

```py
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
    display_tree(derivation_tree)

```

```py
Expanding <factor> at minimum cost

```

我们一直在扩展，直到所有非终端都扩展了。

```py
while f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)    

```

```py
Expanding <expr> at minimum cost
Expanding <term> at minimum cost
Expanding <integer> at minimum cost
Expanding <factor> at minimum cost
Expanding <integer> at minimum cost
Expanding <digit> at minimum cost
Expanding <digit> at minimum cost
Expanding <term> at minimum cost
Expanding <integer> at minimum cost
Expanding <factor> at minimum cost
Expanding <digit> at minimum cost
Expanding <integer> at minimum cost
Expanding <digit> at minimum cost

```

这是最后一棵树：

```py
display_tree(derivation_tree)

```

<svg height="431pt" viewBox="0.00 0.00 314.00 431.00" width="314pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 427)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="154.5" y="-411.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="154.5" y="-360.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="100.5" y="-309.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="154.5" y="-309.8">+</text></g> <g class="edge" id="edge15"><title>1->15</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="205.5" y="-309.8"><term></text></g> <g class="edge" id="edge16"><title>1->16</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-258.8"><term></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="74.5" y="-258.8">+</text></g> <g class="edge" id="edge8"><title>2->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-258.8"><expr></text></g> <g class="edge" id="edge9"><title>2->9</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-207.8"><factor></text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-156.8"><integer></text></g> <g class="edge" id="edge5"><title>4->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-105.8"><digit></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-54.8">7</text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="119.5" y="-207.8"><term></text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="119.5" y="-156.8"><factor></text></g> <g class="edge" id="edge11"><title>10->11</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="119.5" y="-105.8"><integer></text></g> <g class="edge" id="edge12"><title>11->12</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="119.5" y="-54.8"><digit></text></g> <g class="edge" id="edge13"><title>12->13</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="119.5" y="-3.8">2</text></g> <g class="edge" id="edge14"><title>13->14</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="185.5" y="-258.8"><factor></text></g> <g class="edge" id="edge17"><title>16->17</title></g> <g class="node" id="node22"><title>21</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="233.5" y="-258.8">/</text></g> <g class="edge" id="edge21"><title>16->21</title></g> <g class="node" id="node23"><title>22</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="278.5" y="-258.8"><term></text></g> <g class="edge" id="edge22"><title>16->22</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="186.5" y="-207.8"><integer></text></g> <g class="edge" id="edge18"><title>17->18</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="186.5" y="-156.8"><digit></text></g> <g class="edge" id="edge19"><title>18->19</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="186.5" y="-105.8">8</text></g> <g class="edge" id="edge20"><title>19->20</title></g> <g class="node" id="node24"><title>23</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="278.5" y="-207.8"><factor></text></g> <g class="edge" id="edge23"><title>22->23</title></g> <g class="node" id="node25"><title>24</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="278.5" y="-156.8"><integer></text></g> <g class="edge" id="edge24"><title>23->24</title></g> <g class="node" id="node26"><title>25</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="278.5" y="-105.8"><digit></text></g> <g class="edge" id="edge25"><title>24->25</title></g> <g class="node" id="node27"><title>26</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="278.5" y="-54.8">1</text></g> <g class="edge" id="edge26"><title>25->26</title></g></g></svg>

我们看到，在每个步骤中，`expand_node_min_cost()`选择一个不增加符号数量的扩展，最终关闭所有打开的扩展。

## 节点膨胀

尤其是在扩展开始时，我们可能有兴趣让*尽可能多的节点* –也就是说，我们希望扩展使*的更多*非终端扩展。 实际上，这与`expand_node_min_cost()`所提供的完全相反，我们可以实现一种`expand_node_max_cost()`方法，该方法将始终在*成本最高*的节点之间进行选择：

```py
class GrammarFuzzer(GrammarFuzzer):
    def expand_node_max_cost(self, node):
        if self.log:
            print("Expanding", all_terminals(node), "at maximum cost")

        return self.expand_node_by_cost(node, max)

```

为了说明`expand_node_max_cost()`，我们可以再次重新定义`expand_node()`以使用它，然后使用`expand_tree_once()`显示一些扩展步骤：

```py
class GrammarFuzzer(GrammarFuzzer):
    def expand_node(self, node):
        return self.expand_node_max_cost(node)

```

```py
derivation_tree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])

```

```py
f = GrammarFuzzer(EXPR_GRAMMAR, log=True)
display_tree(derivation_tree)

```

<svg height="125pt" viewBox="0.00 0.00 142.50 125.00" width="143pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 121)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-105.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-54.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20.5" y="-3.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="66.5" y="-3.8">+</text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="113.5" y="-3.8"><term></text></g> <g class="edge" id="edge4"><title>1->4</title></g></g></svg>

```py
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
    display_tree(derivation_tree)

```

```py
Expanding <term> at maximum cost

```

```py
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
    display_tree(derivation_tree)

```

```py
Expanding <term> at maximum cost

```

```py
if f.any_possible_expansions(derivation_tree):
    derivation_tree = f.expand_tree_once(derivation_tree)
    display_tree(derivation_tree)

```

```py
Expanding <factor> at maximum cost

```

我们看到，每一步，非末端的数量都会增加。 显然，我们必须对此数字进行限制。

## 三个扩展阶段

现在，我们可以将所有三个阶段放到一个函数`expand_tree()`中，该函数的工作方式如下：

1.  **最大成本扩展。** 使用扩展以最大的代价扩展树，直到我们至少拥有`min_nonterminals`个非终结点。 通过将`min_nonterminals`设置为零，可以轻松跳过此阶段。
2.  **随机扩展。** 继续随机扩展树，直到我们到达`max_nonterminals`非终结点为止。
3.  **最低成本扩展。** 以最小的成本关闭扩展。

我们通过使`expand_node`引用要应用的扩展方法来实现这三个阶段。 通过先将`expand_node`（方法参考）设置为`expand_node_max_cost`（即，调用`expand_node()`会调用`expand_node_max_cost()`），然后是`expand_node_randomly`，最后是`expand_node_min_cost`来控制。 在前两个阶段，我们还分别设置了`min_nonterminals`和`max_nonterminals`的最大限制。

```py
class GrammarFuzzer(GrammarFuzzer):
    def log_tree(self, tree):
        """Output a tree if self.log is set; if self.display is also set, show the tree structure"""
        if self.log:
            print("Tree:", all_terminals(tree))
            if self.disp:
                display(display_tree(tree))
            # print(self.possible_expansions(tree), "possible expansion(s) left")

    def expand_tree_with_strategy(self, tree, expand_node_method, limit=None):
        """Expand tree using `expand_node_method` as node expansion function
 until the number of possible expansions reaches `limit`."""
        self.expand_node = expand_node_method
        while ((limit is None
                or self.possible_expansions(tree) < limit)
               and self.any_possible_expansions(tree)):
            tree = self.expand_tree_once(tree)
            self.log_tree(tree)
        return tree

    def expand_tree(self, tree):
        """Expand `tree` in a three-phase strategy until all expansions are complete."""
        self.log_tree(tree)
        tree = self.expand_tree_with_strategy(
            tree, self.expand_node_max_cost, self.min_nonterminals)
        tree = self.expand_tree_with_strategy(
            tree, self.expand_node_randomly, self.max_nonterminals)
        tree = self.expand_tree_with_strategy(
            tree, self.expand_node_min_cost)

        assert self.possible_expansions(tree) == 0

        return tree

```

让我们在我们的示例中尝试一下。

```py
derivation_tree = ("<start>",
                   [("<expr>",
                     [("<expr>", None),
                      (" + ", []),
                         ("<term>", None)]
                     )])

f = GrammarFuzzer(
    EXPR_GRAMMAR,
    min_nonterminals=3,
    max_nonterminals=5,
    log=True)
derivation_tree = f.expand_tree(derivation_tree)

```

```py
Tree: <expr> + <term>
Expanding <expr> at maximum cost
Tree: <term> + <expr> + <term>
Expanding <term> randomly
Tree: <factor> + <expr> + <term>
Expanding <term> randomly
Tree: <factor> + <expr> + <factor>
Expanding <expr> randomly
Tree: <factor> + <term> - <expr> + <factor>
Expanding <term> randomly
Tree: <factor> + <factor> * <term> - <expr> + <factor>
Expanding <factor> at minimum cost
Tree: <factor> + <factor> * <term> - <expr> + <integer>
Expanding <term> at minimum cost
Tree: <factor> + <factor> * <factor> - <expr> + <integer>
Expanding <factor> at minimum cost
Tree: <integer> + <factor> * <factor> - <expr> + <integer>
Expanding <integer> at minimum cost
Tree: <integer> + <factor> * <factor> - <expr> + <digit>
Expanding <digit> at minimum cost
Tree: <integer> + <factor> * <factor> - <expr> + 1
Expanding <expr> at minimum cost
Tree: <integer> + <factor> * <factor> - <term> + 1
Expanding <integer> at minimum cost
Tree: <digit> + <factor> * <factor> - <term> + 1
Expanding <digit> at minimum cost
Tree: 8 + <factor> * <factor> - <term> + 1
Expanding <term> at minimum cost
Tree: 8 + <factor> * <factor> - <factor> + 1
Expanding <factor> at minimum cost
Tree: 8 + <factor> * <integer> - <factor> + 1
Expanding <factor> at minimum cost
Tree: 8 + <factor> * <integer> - <integer> + 1
Expanding <factor> at minimum cost
Tree: 8 + <integer> * <integer> - <integer> + 1
Expanding <integer> at minimum cost
Tree: 8 + <digit> * <integer> - <integer> + 1
Expanding <integer> at minimum cost
Tree: 8 + <digit> * <digit> - <integer> + 1
Expanding <integer> at minimum cost
Tree: 8 + <digit> * <digit> - <digit> + 1
Expanding <digit> at minimum cost
Tree: 8 + 8 * <digit> - <digit> + 1
Expanding <digit> at minimum cost
Tree: 8 + 8 * <digit> - 1 + 1
Expanding <digit> at minimum cost
Tree: 8 + 8 * 2 - 1 + 1

```

```py
display_tree(derivation_tree)

```

<svg height="482pt" viewBox="0.00 0.00 349.00 482.00" width="349pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 478)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="210.5" y="-462.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="210.5" y="-411.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="160.5" y="-360.8"><expr></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node29"><title>28</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="210.5" y="-360.8">+</text></g> <g class="edge" id="edge28"><title>1->28</title></g> <g class="node" id="node30"><title>29</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="259.5" y="-360.8"><term></text></g> <g class="edge" id="edge29"><title>1->29</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-309.8"><term></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="152.5" y="-309.8">+</text></g> <g class="edge" id="edge8"><title>2->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="198.5" y="-309.8"><expr></text></g> <g class="edge" id="edge9"><title>2->9</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="56.5" y="-258.8"><factor></text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-207.8"><integer></text></g> <g class="edge" id="edge5"><title>4->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-156.8"><digit></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-105.8">8</text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="148.5" y="-258.8"><term></text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node22"><title>21</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="198.5" y="-258.8">-</text></g> <g class="edge" id="edge21"><title>9->21</title></g> <g class="node" id="node23"><title>22</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="246.5" y="-258.8"><expr></text></g> <g class="edge" id="edge22"><title>9->22</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-207.8"><factor></text></g> <g class="edge" id="edge11"><title>10->11</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="147.5" y="-207.8">*</text></g> <g class="edge" id="edge15"><title>10->15</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="194.5" y="-207.8"><term></text></g> <g class="edge" id="edge16"><title>10->16</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-156.8"><integer></text></g> <g class="edge" id="edge12"><title>11->12</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-105.8"><digit></text></g> <g class="edge" id="edge13"><title>12->13</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-54.8">8</text></g> <g class="edge" id="edge14"><title>13->14</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="189.5" y="-156.8"><factor></text></g> <g class="edge" id="edge17"><title>16->17</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="187.5" y="-105.8"><integer></text></g> <g class="edge" id="edge18"><title>17->18</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="187.5" y="-54.8"><digit></text></g> <g class="edge" id="edge19"><title>18->19</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="187.5" y="-3.8">2</text></g> <g class="edge" id="edge20"><title>19->20</title></g> <g class="node" id="node24"><title>23</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="254.5" y="-207.8"><term></text></g> <g class="edge" id="edge23"><title>22->23</title></g> <g class="node" id="node25"><title>24</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="259.5" y="-156.8"><factor></text></g> <g class="edge" id="edge24"><title>23->24</title></g> <g class="node" id="node26"><title>25</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="260.5" y="-105.8"><integer></text></g> <g class="edge" id="edge25"><title>24->25</title></g> <g class="node" id="node27"><title>26</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="260.5" y="-54.8"><digit></text></g> <g class="edge" id="edge26"><title>25->26</title></g> <g class="node" id="node28"><title>27</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="260.5" y="-3.8">1</text></g> <g class="edge" id="edge27"><title>26->27</title></g> <g class="node" id="node31"><title>30</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="285.5" y="-309.8"><factor></text></g> <g class="edge" id="edge30"><title>29->30</title></g> <g class="node" id="node32"><title>31</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="313.5" y="-258.8"><integer></text></g> <g class="edge" id="edge31"><title>30->31</title></g> <g class="node" id="node33"><title>32</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="314.5" y="-207.8"><digit></text></g> <g class="edge" id="edge32"><title>31->32</title></g> <g class="node" id="node34"><title>33</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="314.5" y="-156.8">1</text></g> <g class="edge" id="edge33"><title>32->33</title></g></g></svg>

```py
all_terminals(derivation_tree)

```

```py
'8 + 8 * 2 - 1 + 1'

```

## 全部放在一起

基于此，我们现在可以定义一个函数`fuzz()`，它像`simple_grammar_fuzzer()`一样简单地采用语法并从中产生一个字符串。 因此，它不再暴露出派生树的复杂性。

```py
class GrammarFuzzer(GrammarFuzzer):
    def fuzz_tree(self):
        # Create an initial derivation tree
        tree = self.init_tree()
        # print(tree)

        # Expand all nonterminals
        tree = self.expand_tree(tree)
        if self.log:
            print(repr(all_terminals(tree)))
        if self.disp:
            display(display_tree(tree))
        return tree

    def fuzz(self):
        self.derivation_tree = self.fuzz_tree()
        return all_terminals(self.derivation_tree)

```

现在，我们可以将其应用到所有定义的语法中（并可视化派生树）

```py
f = GrammarFuzzer(EXPR_GRAMMAR)
f.fuzz()

```

```py
'(4 + 8) + -4 / 6 * +3 / 3 / 0 + 8 * 5 + 8'

```

调用`fuzz()`之后，可以在`derivation_tree`属性中访问生成的派生树：

```py
display_tree(f.derivation_tree)

```

<svg height="584pt" viewBox="0.00 0.00 740.00 584.00" width="740pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 580)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="245.5" y="-564.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="245.5" y="-513.8"><expr></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="186.5" y="-462.8"><term></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="245.5" y="-462.8">+</text></g> <g class="edge" id="edge19"><title>1->19</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="297.5" y="-462.8"><expr></text></g> <g class="edge" id="edge20"><title>1->20</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="118.5" y="-411.8"><factor></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="56.5" y="-360.8">(</text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="97.5" y="-360.8"><expr></text></g> <g class="edge" id="edge5"><title>3->5</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="138.5" y="-360.8">)</text></g> <g class="edge" id="edge18"><title>3->18</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-309.8"><term></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="74.5" y="-309.8">+</text></g> <g class="edge" id="edge11"><title>5->11</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-309.8"><expr></text></g> <g class="edge" id="edge12"><title>5->12</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-258.8"><factor></text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-207.8"><integer></text></g> <g class="edge" id="edge8"><title>7->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-156.8"><digit></text></g> <g class="edge" id="edge9"><title>8->9</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="27.5" y="-105.8">4</text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-258.8"><term></text></g> <g class="edge" id="edge13"><title>12->13</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-207.8"><factor></text></g> <g class="edge" id="edge14"><title>13->14</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-156.8"><integer></text></g> <g class="edge" id="edge15"><title>14->15</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-105.8"><digit></text></g> <g class="edge" id="edge16"><title>15->16</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="120.5" y="-54.8">8</text></g> <g class="edge" id="edge17"><title>16->17</title></g> <g class="node" id="node22"><title>21</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="274.5" y="-411.8"><term></text></g> <g class="edge" id="edge21"><title>20->21</title></g> <g class="node" id="node55"><title>54</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="321.5" y="-411.8">+</text></g> <g class="edge" id="edge54"><title>20->54</title></g> <g class="node" id="node56"><title>55</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="611.5" y="-411.8"><expr></text></g> <g class="edge" id="edge55"><title>20->55</title></g> <g class="node" id="node23"><title>22</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="206.5" y="-360.8"><factor></text></g> <g class="edge" id="edge22"><title>21->22</title></g> <g class="node" id="node29"><title>28</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="274.5" y="-360.8">/</text></g> <g class="edge" id="edge28"><title>21->28</title></g> <g class="node" id="node30"><title>29</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="320.5" y="-360.8"><term></text></g> <g class="edge" id="edge29"><title>21->29</title></g> <g class="node" id="node24"><title>23</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="161.5" y="-309.8">-</text></g> <g class="edge" id="edge23"><title>22->23</title></g> <g class="node" id="node25"><title>24</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="206.5" y="-309.8"><factor></text></g> <g class="edge" id="edge24"><title>22->24</title></g> <g class="node" id="node26"><title>25</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="189.5" y="-258.8"><integer></text></g> <g class="edge" id="edge25"><title>24->25</title></g> <g class="node" id="node27"><title>26</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="186.5" y="-207.8"><digit></text></g> <g class="edge" id="edge26"><title>25->26</title></g> <g class="node" id="node28"><title>27</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="186.5" y="-156.8">4</text></g> <g class="edge" id="edge27"><title>26->27</title></g> <g class="node" id="node31"><title>30</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="273.5" y="-309.8"><factor></text></g> <g class="edge" id="edge30"><title>29->30</title></g> <g class="node" id="node35"><title>34</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="323.5" y="-309.8">*</text></g> <g class="edge" id="edge34"><title>29->34</title></g> <g class="node" id="node36"><title>35</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="374.5" y="-309.8"><term></text></g> <g class="edge" id="edge35"><title>29->35</title></g> <g class="node" id="node32"><title>31</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="263.5" y="-258.8"><integer></text></g> <g class="edge" id="edge31"><title>30->31</title></g> <g class="node" id="node33"><title>32</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="246.5" y="-207.8"><digit></text></g> <g class="edge" id="edge32"><title>31->32</title></g> <g class="node" id="node34"><title>33</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="246.5" y="-156.8">6</text></g> <g class="edge" id="edge33"><title>32->33</title></g> <g class="node" id="node37"><title>36</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="333.5" y="-258.8"><factor></text></g> <g class="edge" id="edge36"><title>35->36</title></g> <g class="node" id="node43"><title>42</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="381.5" y="-258.8">/</text></g> <g class="edge" id="edge42"><title>35->42</title></g> <g class="node" id="node44"><title>43</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="440.5" y="-258.8"><term></text></g> <g class="edge" id="edge43"><title>35->43</title></g> <g class="node" id="node38"><title>37</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="289.5" y="-207.8">+</text></g> <g class="edge" id="edge37"><title>36->37</title></g> <g class="node" id="node39"><title>38</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="336.5" y="-207.8"><factor></text></g> <g class="edge" id="edge38"><title>36->38</title></g> <g class="node" id="node40"><title>39</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="333.5" y="-156.8"><integer></text></g> <g class="edge" id="edge39"><title>38->39</title></g> <g class="node" id="node41"><title>40</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="333.5" y="-105.8"><digit></text></g> <g class="edge" id="edge40"><title>39->40</title></g> <g class="node" id="node42"><title>41</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="333.5" y="-54.8">3</text></g> <g class="edge" id="edge41"><title>40->41</title></g> <g class="node" id="node45"><title>44</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="403.5" y="-207.8"><factor></text></g> <g class="edge" id="edge44"><title>43->44</title></g> <g class="node" id="node49"><title>48</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="451.5" y="-207.8">/</text></g> <g class="edge" id="edge48"><title>43->48</title></g> <g class="node" id="node50"><title>49</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="496.5" y="-207.8"><term></text></g> <g class="edge" id="edge49"><title>43->49</title></g> <g class="node" id="node46"><title>45</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="406.5" y="-156.8"><integer></text></g> <g class="edge" id="edge45"><title>44->45</title></g> <g class="node" id="node47"><title>46</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="406.5" y="-105.8"><digit></text></g> <g class="edge" id="edge46"><title>45->46</title></g> <g class="node" id="node48"><title>47</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="406.5" y="-54.8">3</text></g> <g class="edge" id="edge47"><title>46->47</title></g> <g class="node" id="node51"><title>50</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="496.5" y="-156.8"><factor></text></g> <g class="edge" id="edge50"><title>49->50</title></g> <g class="node" id="node52"><title>51</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="496.5" y="-105.8"><integer></text></g> <g class="edge" id="edge51"><title>50->51</title></g> <g class="node" id="node53"><title>52</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="496.5" y="-54.8"><digit></text></g> <g class="edge" id="edge52"><title>51->52</title></g> <g class="node" id="node54"><title>53</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="496.5" y="-3.8">0</text></g> <g class="edge" id="edge53"><title>52->53</title></g> <g class="node" id="node57"><title>56</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="588.5" y="-360.8"><term></text></g> <g class="edge" id="edge56"><title>55->56</title></g> <g class="node" id="node68"><title>67</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="635.5" y="-360.8">+</text></g> <g class="edge" id="edge67"><title>55->67</title></g> <g class="node" id="node69"><title>68</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="689.5" y="-360.8"><expr></text></g> <g class="edge" id="edge68"><title>55->68</title></g> <g class="node" id="node58"><title>57</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="538.5" y="-309.8"><factor></text></g> <g class="edge" id="edge57"><title>56->57</title></g> <g class="node" id="node62"><title>61</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="588.5" y="-309.8">*</text></g> <g class="edge" id="edge61"><title>56->61</title></g> <g class="node" id="node63"><title>62</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="635.5" y="-309.8"><term></text></g> <g class="edge" id="edge62"><title>56->62</title></g> <g class="node" id="node59"><title>58</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="549.5" y="-258.8"><integer></text></g> <g class="edge" id="edge58"><title>57->58</title></g> <g class="node" id="node60"><title>59</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="556.5" y="-207.8"><digit></text></g> <g class="edge" id="edge59"><title>58->59</title></g> <g class="node" id="node61"><title>60</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="556.5" y="-156.8">8</text></g> <g class="edge" id="edge60"><title>59->60</title></g> <g class="node" id="node64"><title>63</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="632.5" y="-258.8"><factor></text></g> <g class="edge" id="edge63"><title>62->63</title></g> <g class="node" id="node65"><title>64</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="631.5" y="-207.8"><integer></text></g> <g class="edge" id="edge64"><title>63->64</title></g> <g class="node" id="node66"><title>65</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="631.5" y="-156.8"><digit></text></g> <g class="edge" id="edge65"><title>64->65</title></g> <g class="node" id="node67"><title>66</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="631.5" y="-105.8">5</text></g> <g class="edge" id="edge66"><title>65->66</title></g> <g class="node" id="node70"><title>69</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="698.5" y="-309.8"><term></text></g> <g class="edge" id="edge69"><title>68->69</title></g> <g class="node" id="node71"><title>70</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="702.5" y="-258.8"><factor></text></g> <g class="edge" id="edge70"><title>69->70</title></g> <g class="node" id="node72"><title>71</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="704.5" y="-207.8"><integer></text></g> <g class="edge" id="edge71"><title>70->71</title></g> <g class="node" id="node73"><title>72</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="704.5" y="-156.8"><digit></text></g> <g class="edge" id="edge72"><title>71->72</title></g> <g class="node" id="node74"><title>73</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="704.5" y="-105.8">8</text></g> <g class="edge" id="edge73"><title>72->73</title></g></g></svg>

让我们尝试其他语法格式的语法模糊器（及其树）。

```py
f = GrammarFuzzer(URL_GRAMMAR)
f.fuzz()

```

```py
'https://user:password@www.google.com?def=6'

```

```py
display_tree(f.derivation_tree)

```

<svg height="380pt" viewBox="0.00 0.00 381.00 380.00" width="381pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 376)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="144" y="-360.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="144" y="-309.8"><url></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="29" y="-258.8"><scheme></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="86" y="-258.8">://</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="144" y="-258.8"><authority></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="216" y="-258.8"><path></text></g> <g class="edge" id="edge11"><title>1->11</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="278" y="-258.8"><query></text></g> <g class="edge" id="edge13"><title>1->13</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25" y="-207.8">https</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="88" y="-207.8"><userinfo></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="144" y="-207.8">@</text></g> <g class="edge" id="edge8"><title>5->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="189" y="-207.8"><host></text></g> <g class="edge" id="edge9"><title>5->9</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="84" y="-156.8">user:password</text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="193" y="-156.8">www.google.com</text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node13"><title>12</title></g> <g class="edge" id="edge12"><title>11->12</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="263" y="-207.8">?</text></g> <g class="edge" id="edge14"><title>13->14</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="313" y="-207.8"><params></text></g> <g class="edge" id="edge15"><title>13->15</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="313" y="-156.8"><param></text></g> <g class="edge" id="edge16"><title>15->16</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="277" y="-105.8"><id></text></g> <g class="edge" id="edge17"><title>16->17</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="313" y="-105.8">=</text></g> <g class="edge" id="edge19"><title>16->19</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="352" y="-105.8"><nat></text></g> <g class="edge" id="edge20"><title>16->20</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="277" y="-54.8">def</text></g> <g class="edge" id="edge18"><title>17->18</title></g> <g class="node" id="node22"><title>21</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="352" y="-54.8"><digit></text></g> <g class="edge" id="edge21"><title>20->21</title></g> <g class="node" id="node23"><title>22</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="352" y="-3.8">6</text></g> <g class="edge" id="edge22"><title>21->22</title></g></g></svg>

```py
f = GrammarFuzzer(CGI_GRAMMAR, min_nonterminals=3, max_nonterminals=5)
f.fuzz()

```

```py
'c%2b%f2'

```

```py
display_tree(f.derivation_tree)

```

<svg height="380pt" viewBox="0.00 0.00 341.00 380.00" width="341pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 376)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="73" y="-360.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="73" y="-309.8"><string></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="41" y="-258.8"><letter></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="106" y="-258.8"><string></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="41" y="-207.8"><other></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="41" y="-156.8">c</text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="105" y="-207.8"><letter></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="172" y="-207.8"><string></text></g> <g class="edge" id="edge13"><title>5->13</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="101" y="-156.8"><percent></text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="6" y="-105.8">%</text></g> <g class="edge" id="edge8"><title>7->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="61" y="-105.8"><hexdigit></text></g> <g class="edge" id="edge9"><title>7->9</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="141" y="-105.8"><hexdigit></text></g> <g class="edge" id="edge11"><title>7->11</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="61" y="-54.8">2</text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="139" y="-54.8">b</text></g> <g class="edge" id="edge12"><title>11->12</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="176" y="-156.8"><letter></text></g> <g class="edge" id="edge14"><title>13->14</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="221" y="-105.8"><percent></text></g> <g class="edge" id="edge15"><title>14->15</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="167" y="-54.8">%</text></g> <g class="edge" id="edge16"><title>15->16</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="222" y="-54.8"><hexdigit></text></g> <g class="edge" id="edge17"><title>15->17</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="302" y="-54.8"><hexdigit></text></g> <g class="edge" id="edge19"><title>15->19</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="222" y="-3.8">f</text></g> <g class="edge" id="edge18"><title>17->18</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="302" y="-3.8">2</text></g> <g class="edge" id="edge20"><title>19->20</title></g></g></svg>

我们如何对抗`simple_grammar_fuzzer()`？

```py
trials = 50
xs = []
ys = []
f = GrammarFuzzer(EXPR_GRAMMAR, max_nonterminals=20)
for i in range(trials):
    with Timer() as t:
        s = f.fuzz()
    xs.append(len(s))
    ys.append(t.elapsed_time())
    print(i, end=" ")
print()

```

```py
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 

```

```py
average_time = sum(ys) / trials
print("Average time:", average_time)

```

```py
Average time: 0.05556378066001343

```

```py
%matplotlib inline

import [matplotlib.pyplot](https://docs.python.org/3/library/matplotlib.pyplot.html) as [plt](https://docs.python.org/3/library/plt.html)
plt.scatter(xs, ys)
plt.title('Time required for generating an output')

```

```py
Text(0.5,1,'Time required for generating an output')

```

![](data:image/png;base64, 
)

我们的测试生成速度要快得多，但输入的内容也要小得多。 我们看到，使用派生树，我们可以更好地控制语法产生。

最后，在`simple_grammar_fuzzer()`失败的情况下`GrammarFuzzer`如何与`expr_grammar`一起工作？ 它可以正常工作：

```py
f = GrammarFuzzer(expr_grammar, max_nonterminals=10)
f.fuzz()

```

```py
'((1 + 9) / 2 * 2 / 7 * 3 - 3.1 / 4) * -0'

```

有了`GrammarFuzzer`，我们现在有了坚实的基础，可以在此基础上构建更多的模糊测试器，并从生成软件测试的世界中说明更多令人兴奋的概念。 其中许多甚至不需要编写语法-而是*从当前域中推断*语法，因此即使不编写语法也可以使用基于语法的模糊测试。 敬请关注！

## 经验教训

*   *派生树*对于表达输入结构很重要
*   *基于派生树的语法模糊处理*
    1.  比基于字符串的语法模糊测试效率更高，
    2.  更好地控制输入生成，并且
    3.  有效避免遇到无限扩展。

## 后续步骤

恭喜你！ 您已经达到本书的中心“枢纽”之一。 从这里开始，有大量基于语法模糊的技术。

### 扩展语法

首先，我们有许多技术可以使所有*以某种形式扩展*语法：

*   [解析和重新组合输入](Parser.html)允许使用现有输入，再次使用派生树
*   [覆盖语法扩展](GrammarCoverageFuzzer.html)允许*组合*覆盖
*   [将*概率*分配给各个扩展](ProbabilisticGrammarFuzzer.html)可对扩展进行额外控制
*   [将*约束*分配给单个扩展](GeneratorGrammarFuzzer.html)允许在单个规则上表达*语义约束*。

### 应用语法

其次，我们可以*在各种涉及自动学习某种形式的上下文中应用*语法：

*   [模糊化API](APIFuzzer.html) ，从API学习语法
*   [模糊化图形用户界面](WebFuzzer.html)，从用户界面中学习语法以进行后续模糊测试
*   [挖掘语法](GrammarMiner.html)，学习任意输入格式的语法

继续扩大！

## 背景

派生树（通常称为*解析树*）是一种标准数据结构，*解析器*将其分解为输入。 *龙书*（也称为*编译器：原理，技术和工具*）[ [Aho *等人*，2006。](https://www.pearson.com/us/higher-education/program/Aho-Compilers-Principles-Techniques-and-Tools-2nd-Edition/PGM167067.html)讨论了解析。 进入派生树作为编译程序的一部分。 在解析和重组输入时，我们也使用派生树[。](Parser.html)

本章的主要思想，即扩展到达到符号的限制，然后总是选择最短的路径，源于Luke [ [Luke *等人*，2000。](https://doi.org/10.1109/4235.873237)]。

## 练习

### 练习1：缓存方法结果

跟踪`GrammarFuzzer`表明，某些方法总是以相同的值反复调用。

用*缓存*设置类`FasterGrammarFuzzer`，该类检查该方法是否之前已被调用，如果已调用过，则返回先前计算的“记忆”值。 对`expansion_to_children()`执行此操作。 比较优化前后的调用次数。

**重要**：对于`expansion_to_children()`，请确保返回的每个列表都是一个单独的副本。 如果返回相同（缓存的）列表，则将干扰`GrammarFuzzer`的就地修改。 为此，请使用Python `copy.deepcopy()`函数。

[Use the notebook](https://mybinder.org/v2/gh/uds-se/fuzzingbook/master?filepath=docs/notebooks/GrammarFuzzer.ipynb#Exercises) to work on the exercises and see solutions.

### 练习2：语法预编译

某些方法（例如`symbol_cost()`或`expansion_cost()`）返回的值仅取决于语法。 设置一个`EvenFasterGrammarFuzzer()`类，该类在初始化时会预先计算一次这些值，以便以后`symbol_cost()`或`expansion_cost()`的调用仅需要查找这些值。

[Use the notebook](https://mybinder.org/v2/gh/uds-se/fuzzingbook/master?filepath=docs/notebooks/GrammarFuzzer.ipynb#Exercises) to work on the exercises and see solutions.

### 练习3：维护要扩展的树

在`expand_tree_once()`中，该算法一次又一次遍历树以查找仍可以扩展的非终结点。 通过在树中保留一系列仍可以扩展的非终止符号来加快处理速度。

[Use the notebook](https://mybinder.org/v2/gh/uds-se/fuzzingbook/master?filepath=docs/notebooks/GrammarFuzzer.ipynb#Exercises) to work on the exercises and see solutions.

### 练习4：备用随机扩展

我们可以定义`expand_node_randomly()`使其仅调用`expand_node_by_cost(node, random.choice)`：

```py
class ExerciseGrammarFuzzer(GrammarFuzzer):
    def expand_node_randomly(self, node):
        if self.log:
            print("Expanding", all_terminals(node), "randomly by cost")

        return self.expand_node_by_cost(node, random.choice)

```

原始实施和此替代之间有什么区别？

[Use the notebook](https://mybinder.org/v2/gh/uds-se/fuzzingbook/master?filepath=docs/notebooks/GrammarFuzzer.ipynb#Exercises) to work on the exercises and see solutions.