# 挖掘输入文法

> 原文： [https://www.fuzzingbook.org/html/GrammarMiner.html](https://www.fuzzingbook.org/html/GrammarMiner.html)

到目前为止，我们所看到的语法大多是手动指定的-也就是说，您（或知道输入格式的人）必须首先设计和编写语法。 尽管到目前为止我们所看到的语法非常简单，但是为复杂的输入创建语法可能需要付出很多努力。 因此，在本章中，我们将介绍*自动从程序*中挖掘语法的技术-通过执行程序并观察它们如何处理输入的哪些部分。 结合语法模糊测试，我们可以

1.  参加节目
2.  提取其输入语法，然后
3.  使用本书中的概念，以高效率和有效性对其进行模糊处理。

**前提条件**

*   您应该阅读语法的[一章。](Grammars.html)
*   关于配置模糊检测的[一章介绍了配置选项的语法挖掘，以及在执行过程中观察变量和值。](ConfigurationFuzzer.html)
*   我们使用覆盖的[一章中的跟踪器。](Coverage.html)
*   解析器的[章中的解析概念也很有用。](Parser.html)

## 内容提要

要使用本章中提供的代码来[，请编写](Importing.html)

```py
>>> from [fuzzingbook.GrammarMiner](GrammarMiner.html) import <identifier>

```

然后利用以下功能。

本章提供了许多类来从现有程序中挖掘输入语法。 函数`recover_grammar()`可能最容易使用。 它接受一个函数和一组输入，并返回描述其输入语言的语法。

我们在`url_parse()`函数上应用`recover_grammar()`，该函数采用并分解URL：

```py
>>> url_parse('https://www.fuzzingbook.org/')
>>> URLS
['http://user:pass@www.google.com:80/?q=path#ref',
 'https://www.cispa.saarland:80/',
 'http://www.fuzzingbook.org/#News']

```

我们使用`recover_grammar()`提取`url_parse()`的输入语法：

```py
>>> grammar = recover_grammar(url_parse, URLS)
>>> grammar
{'<start>': ['<urlsplit@394:url>'],
 '<urlsplit@394:url>': ['<__new__@12:scheme>:<_splitnetloc@386:url>'],
 '<__new__@12:scheme>': ['https', '<__new__@12:scheme>', 'http'],
 '<_splitnetloc@386:url>': ['//<__new__@12:netloc><urlsplit@415:url>',
  '//<__new__@12:netloc>/'],
 '<__new__@12:netloc>': ['www.cispa.saarland:80',
  '<__new__@12:netloc>',
  'user:pass@www.google.com:80',
  'www.fuzzingbook.org'],
 '<urlsplit@415:url>': ['/#<__new__@12:fragment>',
  '<urlsplit@420:url>#<__new__@12:fragment>'],
 '<urlsplit@420:url>': ['/?<__new__@12:query>'],
 '<__new__@12:query>': ['q=path', '<__new__@12:query>'],
 '<__new__@12:fragment>': ['News', '<__new__@12:fragment>', 'ref']}

```

非终端的名称有点技术性。 但是语法很好地表示了输入的结构； 例如，可以识别不同的方案（`"http"`和`"https"`）。 语法可立即用于模糊测试，产生输入元素的任意组合，这些组合在语法上均有效。

```py
>>> from [GrammarCoverageFuzzer](GrammarCoverageFuzzer.html) import GrammarCoverageFuzzer
>>> fuzzer = GrammarCoverageFuzzer(grammar)
>>> [fuzzer.fuzz() for i in range(5)]
['https://www.cispa.saarland:80/?q=path#News',
 'http://www.fuzzingbook.org/',
 'http://user:pass@www.google.com:80/#ref',
 'http://www.fuzzingbook.org/#News',
 'https://user:pass@www.google.com:80/?q=path#ref']

```

能够自动提取语法并将此语法用于模糊测试，可以用最少的人工就非常有效地生成测试。

## 语法挑战

考虑解析器的[章中的`process_inventory()`方法：](Parser.html)

```py
import [fuzzingbook_utils](https://github.com/uds-se/fuzzingbook/tree/master/notebooks/fuzzingbook_utils)

```

```py
from [Parser](Parser.html) import process_inventory, process_vehicle, process_car, process_van, lr_graph  # minor dependency

```

它采用以下形式的输入。

```py
INVENTORY = """\
1997,van,Ford,E350
2000,car,Mercury,Cougar
1999,car,Chevy,Venture\
"""

```

```py
print(process_inventory(INVENTORY))

```

```py
We have a Ford E350 van from 1997 vintage.
It is an old but reliable model!
We have a Mercury Cougar car from 2000 vintage.
It is an old but reliable model!
We have a Chevy Venture car from 1999 vintage.
It is an old but reliable model!

```

从解析器的[一章中我们发现，当输入格式仅包含用代码表示的细节时，粗略的语法就不能很好地用于模糊测试。 也就是说，即使我们已经有了CSV文件的正式规范（](Parser.html) [RFC 4180](https://tools.ietf.org/html/rfc4180) ），库存系统仍包含有关CSV文件每个索引的预期规则。 仅合并现有输入的解决方案虽然可行，但并不完整。 特别是，它首先依赖于正式的输入规范。 但是，我们不能保证程序遵守给定的输入规范。

摆脱这种困境的方法之一就是询问被测程序的输入规范。 也就是说，如果以某种方式编写被测程序，使得特定方法负责处理输入的特定部分，则可以通过观察解析过程来恢复解析树。 此外，可以通过从多个输入树中提取来恢复语法的合理近似。

*我们从假设（1）开始，该程序是以这样的方式编写的：特定的方法负责解析程序的特定片段-这几乎包括所有临时解析器。*

这个想法如下：

*   深入了解Python执行过程，观察输入字符串的片段，这些片段是用不同的方法生成和命名的。
*   将输入片段以树形结构拼接在一起，以检索**解析树**。
*   从多个解析树中提取公共元素，以产生输入的**上下文无关语法**。

## 一个简单的语法矿工

假设我们要获取函数`process_vehicle()`的输入语法。 我们首先收集此功能的样本输入。

```py
VEHICLES = INVENTORY.split('\n')

```

负责处理库存的方法集如下。

```py
INVENTORY_METHODS = {
    'process_inventory',
    'process_vehicle',
    'process_van',
    'process_car'}

```

从[配置模糊化](ConfigurationFuzzer.html)的一章中可以看出，人们可以加入Python运行时以观察函数的参数和创建的任何局部变量。 我们还看到，可以通过检查`frame`参数来获取执行的上下文。 这是一个简单的跟踪器，可以在被跟踪的函数中返回局部变量和其他上下文信息。 我们重用`Coverage`跟踪类。

### 追踪器

```py
from [Coverage](Coverage.html) import Coverage

```

```py
import [inspect](https://docs.python.org/3/library/inspect.html)

```

```py
class Tracer(Coverage):
    def traceit(self, frame, event, arg):
        method_name = inspect.getframeinfo(frame).function
        if method_name not in INVENTORY_METHODS:
            return
        file_name = inspect.getframeinfo(frame).filename

        param_names = inspect.getargvalues(frame).args
        lineno = inspect.getframeinfo(frame).lineno
        local_vars = inspect.getargvalues(frame).locals
        print(event, file_name, lineno, method_name, param_names, local_vars)
        return self.traceit

```

我们在跟踪上下文下运行代码。

```py
with Tracer() as tracer:
    process_vehicle(VEHICLES[0])

```

```py
call <string> 1 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350'}
line <string> 2 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350'}
line <string> 3 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', '_': [], 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
line <string> 4 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', '_': [], 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
call <string> 1 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line <string> 2 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line <string> 3 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.']}
line <string> 4 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.'], 'iyear': 1997}
line <string> 7 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.'], 'iyear': 1997}
line <string> 8 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], 'iyear': 1997}
return <string> 8 process_van ['year', 'company', 'model'] {'model': 'E350', 'company': 'Ford', 'year': '1997', 'res': ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], 'iyear': 1997}
return <string> 4 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', '_': [], 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}

```

我们希望从跟踪中获得的主要内容是将输入片段分配给不同变量的列表。 如上所示，我们可以使用跟踪工具`settrace()`来获取它。

但是，`settrace()`函数挂接到Python调试工具中。 当它运行时，没有调试器可以挂接到程序中。 也就是说，如果我们的语法挖掘器存在问题，我们将无法在其上附加调试器以了解正在发生的情况。 这是不理想的。 因此，我们将跟踪器限制为最简单的实现方式，并在以后的阶段中实现语法挖掘的核心。

`traceit()`函数依赖于`frame`变量的信息，该变量公开了Python内部。 我们定义了一个`context`类，该类封装了`frame`中需要的信息。

### 上下文

通过`Context`类，可以轻松访问信息，例如当前模块和参数名称。

```py
class Context:
    def __init__(self, frame, track_caller=True):
        self.method = inspect.getframeinfo(frame).function
        self.parameter_names = inspect.getargvalues(frame).args
        self.file_name = inspect.getframeinfo(frame).filename
        self.line_no = inspect.getframeinfo(frame).lineno

    def _t(self):
        return (self.file_name, self.line_no, self.method,
                ','.join(self.parameter_names))

    def __repr__(self):
        return "%s:%d:%s(%s)" % self._t()

```

以下是一些在`frame`至`Context`上操作的便捷方法。

```py
class Context(Context):
    def extract_vars(self, frame):
        return inspect.getargvalues(frame).locals

    def parameters(self, all_vars):
        return {k: v for k, v in all_vars.items() if k in self.parameter_names}

    def qualified(self, all_vars):
        return {"%s:%s" % (self.method, k): v for k, v in all_vars.items()}

```

我们将上下文打印到我们的`traceit()`上以查看其实际效果。 首先，我们定义一个用于显示事件的`log_event()`。

```py
def log_event(event, var):
    print({'call': '->', 'return': '<-'}.get(event, '  '), var)

```

并在`traceit()`功能中使用`log_event()`。

```py
class Tracer(Tracer):
    def traceit(self, frame, event, arg):
        log_event(event, Context(frame))
        return self.traceit

```

在跟踪下运行`process_vehicle()`将打印遇到的上下文。

```py
with Tracer() as tracer:
    process_vehicle(VEHICLES[0])

```

```py
-> <string>:1:process_vehicle(vehicle)
   <string>:2:process_vehicle(vehicle)
   <string>:3:process_vehicle(vehicle)
   <string>:4:process_vehicle(vehicle)
-> <string>:1:process_van(year,company,model)
   <string>:2:process_van(year,company,model)
   <string>:3:process_van(year,company,model)
   <string>:4:process_van(year,company,model)
   <string>:7:process_van(year,company,model)
   <string>:8:process_van(year,company,model)
<- <string>:8:process_van(year,company,model)
<- <string>:4:process_vehicle(vehicle)
-> <string>:24:__exit__(self,exc_type,exc_value,tb)
   <string>:25:__exit__(self,exc_type,exc_value,tb)

```

通过执行任何功能产生的跟踪可能会变得非常庞大。 因此，我们需要将注意力集中在特定的模块上。 此外，由于这些变量更可能包含输入片段，因此我们也仅将注意力集中在`str`变量上。 （我们将在后面的练习中展示如何处理复杂的对象。）

我们之前开发的`Context`类用于确定要监视的模块以及要跟踪的变量。

我们存储当前的*输入字符串*，以便可以用来确定是否有任何特定的字符串片段来自当前的输入字符串。 任何可选参数将单独处理。

```py
class Tracer(Tracer):
    def __init__(self, my_input, **kwargs):
        self.options(kwargs)
        self.my_input, self.trace = my_input, []

```

我们使用可选参数`files`来指示我们感兴趣的特定源文件，并使用`methods`来指示感兴趣的特定方法。 此外，我们还使用`log`指定在跟踪过程中是否应启用详细日志记录。 我们使用先前定义的`log_event()`方法进行记录。

选项处理如下。

```py
class Tracer(Tracer):
    def options(self, kwargs):
        self.files = kwargs.get('files', [])
        self.methods = kwargs.get('methods', [])
        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None

```

检查`files`和`methods`，以确定是否应跟踪特定事件

```py
class Tracer(Tracer):
    def tracing_context(self, cxt, event, arg):
        fres = not self.files or any(
            cxt.file_name.endswith(f) for f in self.files)
        mres = not self.methods or any(cxt.method == m for m in self.methods)
        return fres and mres

```

类似于事件的上下文，我们还希望将注意力集中在特定变量上。 现在，我们只想关注字符串。 （有关如何将其扩展到其他类型的对象，请参见本章末尾的练习）。

```py
class Tracer(Tracer):
    def tracing_var(self, k, v):
        return isinstance(v, str)

```

我们修改`traceit()`以仅使用我们感兴趣的特定事件的上下文信息来调用`on_event()`函数。

```py
class Tracer(Tracer):
    def on_event(self, event, arg, cxt, my_vars):
        self.trace.append((event, arg, cxt, my_vars))

    def traceit(self, frame, event, arg):
        cxt = Context(frame)
        if not self.tracing_context(cxt, event, arg):
            return self.traceit
        self.log(event, cxt)

        my_vars = {
            k: v
            for k, v in cxt.extract_vars(frame).items()
            if self.tracing_var(k, v)
        }
        self.on_event(event, arg, cxt, my_vars)
        return self.traceit

```

`Tracer`类现在可以专注于特定文件上的特定类型的事件。 此外，它为我们发现有趣的变量提供了一级过滤器。 例如，我们要特别关注`process_*`方法中包含输入片段的变量。 这是我们更新后的`Tracer`可以使用的方式

```py
with Tracer(VEHICLES[0], methods=INVENTORY_METHODS, log=True) as tracer:
    process_vehicle(VEHICLES[0])

```

```py
-> <string>:1:process_vehicle(vehicle)
   <string>:2:process_vehicle(vehicle)
   <string>:3:process_vehicle(vehicle)
   <string>:4:process_vehicle(vehicle)
-> <string>:1:process_van(year,company,model)
   <string>:2:process_van(year,company,model)
   <string>:3:process_van(year,company,model)
   <string>:4:process_van(year,company,model)
   <string>:7:process_van(year,company,model)
   <string>:8:process_van(year,company,model)
<- <string>:8:process_van(year,company,model)
<- <string>:4:process_vehicle(vehicle)

```

执行产生了以下跟踪。

```py
for t in tracer.trace:
    print(t[0], t[2].method, dict(t[3]))

```

```py
call process_vehicle {'vehicle': '1997,van,Ford,E350'}
line process_vehicle {'vehicle': '1997,van,Ford,E350'}
line process_vehicle {'vehicle': '1997,van,Ford,E350', 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
line process_vehicle {'vehicle': '1997,van,Ford,E350', 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}
call process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
line process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
return process_van {'model': 'E350', 'company': 'Ford', 'year': '1997'}
return process_vehicle {'vehicle': '1997,van,Ford,E350', 'model': 'E350', 'company': 'Ford', 'kind': 'van', 'year': '1997'}

```

由于我们已经将输入保存在Tracer中，因此再次单独将其指定为参数是多余的。

```py
with Tracer(VEHICLES[0], methods=INVENTORY_METHODS, log=True) as tracer:
    process_vehicle(tracer.my_input)

```

```py
-> <string>:1:process_vehicle(vehicle)
   <string>:2:process_vehicle(vehicle)
   <string>:3:process_vehicle(vehicle)
   <string>:4:process_vehicle(vehicle)
-> <string>:1:process_van(year,company,model)
   <string>:2:process_van(year,company,model)
   <string>:3:process_van(year,company,model)
   <string>:4:process_van(year,company,model)
   <string>:7:process_van(year,company,model)
   <string>:8:process_van(year,company,model)
<- <string>:8:process_van(year,company,model)
<- <string>:4:process_vehicle(vehicle)

```

### DefineTracker

我们定义一个`DefineTracker`类，该类处理来自`Tracer`的跟踪。 这个想法是存储不同的变量定义，它们是输入片段。

跟踪器识别属于输入字符串的字符串片段，并将其存储在字典`my_assignments`中。 它保存跟踪以及相应的输入以进行处理。 最后，它调用`process()`处理给定的`trace`。 我们将从一个简单的跟踪器开始，该跟踪器依赖于某些假设，然后再看如何放松这些假设。

```py
class DefineTracker:
    def __init__(self, my_input, trace, **kwargs):
        self.options(kwargs)
        self.my_input = my_input
        self.trace = trace
        self.my_assignments = {}
        self.process()

```

使用子字符串搜索的问题之一是，短字符串序列往往会包含在其他字符串序列中，即使它们可能并非来自原始字符串也是如此。 也就是说，输入片段是`v`。 它可能同样来自`van`或`chevy`。 我们依靠能够预测给定片段发生的确切位置输入。 因此，我们定义一个常量`FRAGMENT_LEN`，以便我们忽略该长度的字符串。 我们还像以前一样合并了一个日志记录工具。

```py
FRAGMENT_LEN = 3

```

```py
class DefineTracker(DefineTracker):
    def options(self, kwargs):
        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None
        self.fragment_len = kwargs.get('fragment_len', FRAGMENT_LEN)

```

我们的跟踪器仅记录变量值出现的时间。 接下来，我们需要检查变量是否包含**输入字符串**中的值。 常用的方法是依靠符号执行或至少动态的污点处理，它们既强大又复杂。 但是，仅依靠子字符串搜索就可以得到一个合理的近似值。 也就是说，我们认为所产生的作为原始输入字符串的子字符串的任何值都来自原始输入。

我们定义`is_input_fragment()`方法，该方法依赖于字符串包含来检测字符串是否来自输入。

```py
class DefineTracker(DefineTracker):
    def is_input_fragment(self, var, value):
        return len(value) >= self.fragment_len and value in self.my_input

```

我们可以使用`is_input_fragment()`仅选择定义的变量的子集，如下面`fragments()`中所实现的。

```py
class DefineTracker(DefineTracker):
    def fragments(self, variables):
        return {k: v for k, v in variables.items(
        ) if self.is_input_fragment(k, v)}

```

跟踪器处理每个事件，并在每个事件处使用当前局部变量更新字典`my_assignments`，该局部变量包含作为输入一部分的字符串。 请注意，这里有一个关于重新分配期间发生情况的选择。 我们可以丢弃所有重新分配，也可以仅保留最后一个分配。 在这里，我们选择后者。 如果要使用以前的行为，请在存储片段之前检查`my_assignments`中是否存在该值。

```py
class DefineTracker(DefineTracker):
    def track_event(self, event, arg, cxt, my_vars):
        self.log(event, (cxt.method, my_vars))
        self.my_assignments.update(self.fragments(my_vars))

    def process(self):
        for event, arg, cxt, my_vars in self.trace:
            self.track_event(event, arg, cxt, my_vars)

```

使用跟踪器，我们可以获得输入片段。 例如，假设我们只对至少`5`个字符长的字符串感兴趣。

```py
tracker = DefineTracker(tracer.my_input, tracer.trace, fragment_len=5)
for k, v in tracker.my_assignments.items():
    print(k, '=', repr(v))

```

```py
vehicle = '1997,van,Ford,E350'

```

或长度为`2`个字符的字符串（默认值）。

```py
tracker = DefineTracker(tracer.my_input, tracer.trace)
for k, v in tracker.my_assignments.items():
    print(k, '=', repr(v))

```

```py
vehicle = '1997,van,Ford,E350'
model = 'E350'
company = 'Ford'
kind = 'van'
year = '1997'

```

```py
class DefineTracker(DefineTracker):
    def assignments(self):
        return self.my_assignments.items()

```

### 组装派生树

```py
from [Grammars](Grammars.html) import START_SYMBOL, syntax_diagram, is_nonterminal

```

```py
from [GrammarFuzzer](GrammarFuzzer.html) import GrammarFuzzer, FasterGrammarFuzzer, display_tree, tree_to_string

```

`DefineTracker`的输入片段只说明了一半。 片段可以在解析的不同阶段创建。 因此，我们需要将片段组装为输入的派生树。 基本思想如下：

上一步的输入是：

```py
"1997,van,Ford,E350"

```

我们开始一个派生树，并将其与语法中的开始符号相关联。

```py
derivation_tree = (START_SYMBOL, [("1997,van,Ford,E350", [])])

```

```py
display_tree(derivation_tree)

```

<svg height="74pt" viewBox="0.00 0.00 123.00 74.00" width="123pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 70)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-54.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-3.8">1997,van,Ford,E350</text></g> <g class="edge" id="edge1"><title>0->1</title></g></g></svg>

下一个输入是：

```py
vehicle = "1997,van,Ford,E350"

```

由于车辆完全覆盖了`<start>`节点的值，因此我们将其替换为车辆节点。

```py
derivation_tree = (START_SYMBOL, [('<vehicle>', [("1997,van,Ford,E350", [])],
                                   [])])

```

```py
display_tree(derivation_tree)

```

<svg height="125pt" viewBox="0.00 0.00 123.00 125.00" width="123pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 121)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-105.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-54.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-3.8">1997,van,Ford,E350</text></g> <g class="edge" id="edge2"><title>1->2</title></g></g></svg>

The next input was:

```py
model = 'E350'

```

遍历`<start>`的派生树，我们看到它替换了`<vehicle>`节点值的一部分。 因此，我们将`<vehicle>`节点的值划分为两个子节点，其中一个对应于`"1997"`的值，另一个对应于`",van,Ford,E350"`的值，并将第一个替换为节点`<model>`。

```py
derivation_tree = (START_SYMBOL, [('<vehicle>', [('<model>', [('1997', [])]),
                                                 (",van,Ford,E350", [])], [])])

```

```py
display_tree(derivation_tree)

```

<svg height="176pt" viewBox="0.00 0.00 164.00 176.00" width="164pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="68.5" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="68.5" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25.5" y="-54.8"><model></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112.5" y="-54.8">,van,Ford,E350</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25.5" y="-3.8">1997</text></g> <g class="edge" id="edge3"><title>2->3</title></g></g></svg>

我们为执行类似的操作

```py
company = 'Ford'

```

```py
derivation_tree = (START_SYMBOL, [('<vehicle>', [('<model>', [('1997', [])]),
                                                 (",van,", []),
                                                 ('<company>', [('Ford', [])]),
                                                 (",E350", [])], [])])

```

```py
display_tree(derivation_tree)

```

<svg height="176pt" viewBox="0.00 0.00 243.50 176.00" width="244pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="116.5" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="116.5" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25.5" y="-54.8"><model></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="83.5" y="-54.8">,van,</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="149.5" y="-54.8"><company></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="218.5" y="-54.8">,E350</text></g> <g class="edge" id="edge7"><title>1->7</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25.5" y="-3.8">1997</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="149.5" y="-3.8">Ford</text></g> <g class="edge" id="edge6"><title>5->6</title></g></g></svg>

同样的

```py
kind = 'van'

```

和

```py
model = 'E350'

```

```py
derivation_tree = (START_SYMBOL, [('<vehicle>', [('<model>', [('1997', [])]),
                                                 (",", []),
                                                 ("<kind>", [('van', [])]),
                                                 (",", []),
                                                 ('<company>', [('Ford', [])]),
                                                 (",", []),
                                                 ("<model>", [('E350', [])])
                                                 ], [])])

```

```py
display_tree(derivation_tree)

```

<svg height="176pt" viewBox="0.00 0.00 341.00 176.00" width="341pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="153.5" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="153.5" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25.5" y="-54.8"><model></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="71.5" y="-54.8">,</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112.5" y="-54.8"><kind></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="153.5" y="-54.8">,</text></g> <g class="edge" id="edge7"><title>1->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="207.5" y="-54.8"><company></text></g> <g class="edge" id="edge8"><title>1->8</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="261.5" y="-54.8">,</text></g> <g class="edge" id="edge10"><title>1->10</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="307.5" y="-54.8"><model></text></g> <g class="edge" id="edge11"><title>1->11</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="25.5" y="-3.8">1997</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112.5" y="-3.8">van</text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="207.5" y="-3.8">Ford</text></g> <g class="edge" id="edge9"><title>8->9</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="307.5" y="-3.8">E350</text></g> <g class="edge" id="edge12"><title>11->12</title></g></g></svg>

现在，我们使用上述步骤开发完整的算法。 派生树`TreeMiner`使用输入字符串和变量分配进行初始化，并将派生树转换为相应的派生树。

```py
class TreeMiner:
    def __init__(self, my_input, my_assignments, **kwargs):
        self.options(kwargs)
        self.my_input = my_input
        self.my_assignments = my_assignments
        self.tree = self.get_derivation_tree()

    def options(self, kwargs):
        self.log = log_call if kwargs.get('log') else lambda _i, _v: None

    def get_derivation_tree(self):
        return (START_SYMBOL, [])

```

`log_call()`如下。

```py
def log_call(indent, var):
    print('\t' * indent, var)

```

基本思想如下：

*   **现在，我们假设分配给变量的值是稳定的。 也就是说，它永远不会重新分配。 特别是，没有递归调用，也没有从不同部分多次调用同一函数。** （稍后我们将展示如何克服此限制）。
*   对于每对 *var* ，在`my_assignments`中发现*值*：
    1.  我们在派生树中递归搜索*值* `val`的出现。
    2.  如果发现某个事件是节点`P1`的值`V1`，则将节点`P1`的值分为三部分，中间部分与*值* `val`匹配，并且 第一部分和最后一部分是`V1`中相应的前缀和后缀。
    3.  用三个子节点重构节点`P1`，其中前面提到的前缀和后缀是字符串值，并且匹配值`val`被具有单个值`val`的节点`var`代替。

首先，我们定义一个包装器，以根据变量名生成非终结符。

```py
def to_nonterminal(var):
    return "<" + var.lower() + ">"

```

`insert_into_tree`方法接受给定的树`tree`和`(k,v)`对。 它递归检查给定对是否可以应用。 如果可以应用该对，则应用该对并返回`True`。

```py
class TreeMiner(TreeMiner):
    def insert_into_tree(self, my_tree, pair):
        var, values = my_tree
        k, v = pair
        self.log(1, "- Node: %s\t\t? (%s:%s)" % (var, k, repr(v)))
        applied = False
        for i, value_ in enumerate(values):
            value, arr = value_
            self.log(2, "-> [%d] %s" % (i, repr(value)))
            if is_nonterminal(value):
                applied = self.insert_into_tree(value_, pair)
                if applied:
                    break
            elif v in value:
                prefix_k_suffix = [
                    (k, [[v, []]]) if i == 1 else (e, [])
                    for i, e in enumerate(value.partition(v))
                    if e
                ]
                del values[i]
                for j, rep in enumerate(prefix_k_suffix):
                    values.insert(j + i, rep)
                applied = True

                self.log(2, " > %s" % (repr([i[0] for i in prefix_k_suffix])))
                break
            else:
                continue
        return applied

```

`insert_into_tree()`的使用方法如下。

```py
tree = (START_SYMBOL, [("1997,van,Ford,E350", [])])
m = TreeMiner('', {}, log=True)

```

首先，我们将输入字符串作为唯一节点。

```py
display_tree(tree)

```

<svg height="74pt" viewBox="0.00 0.00 123.00 74.00" width="123pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 70)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-54.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-3.8">1997,van,Ford,E350</text></g> <g class="edge" id="edge1"><title>0->1</title></g></g></svg>

插入`<vehicle>`节点。

```py
v = m.insert_into_tree(tree, ('<vehicle>', "1997,van,Ford,E350"))

```

```py
	 - Node: <start>		? (<vehicle>:'1997,van,Ford,E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['<vehicle>']

```

```py
display_tree(tree)

```

<svg height="125pt" viewBox="0.00 0.00 123.00 125.00" width="123pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 121)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-105.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-54.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="57.5" y="-3.8">1997,van,Ford,E350</text></g> <g class="edge" id="edge2"><title>1->2</title></g></g></svg>

插入`<model>`节点。

```py
v = m.insert_into_tree(tree, ('<model>', 'E350'))

```

```py
	 - Node: <start>		? (<model>:'E350')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<model>:'E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['1997,van,Ford,', '<model>']

```

```py
display_tree((tree))

```

<svg height="176pt" viewBox="0.00 0.00 163.50 176.00" width="164pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="86" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="86" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="43" y="-54.8">1997,van,Ford,</text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="130" y="-54.8"><model></text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="130" y="-3.8">E350</text></g> <g class="edge" id="edge4"><title>3->4</title></g></g></svg>

插入`<company>`。

```py
v = m.insert_into_tree(tree, ('<company>', 'Ford'))

```

```py
	 - Node: <start>		? (<company>:'Ford')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<company>:'Ford')
		 -> [0] '1997,van,Ford,'
		  > ['1997,van,', '<company>', ',']

```

```py
display_tree(tree)

```

<svg height="176pt" viewBox="0.00 0.00 241.50 176.00" width="242pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="135" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="135" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="28" y="-54.8">1997,van,</text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="108" y="-54.8"><company></text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="162" y="-54.8">,</text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="208" y="-54.8"><model></text></g> <g class="edge" id="edge6"><title>1->6</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="108" y="-3.8">Ford</text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="208" y="-3.8">E350</text></g> <g class="edge" id="edge7"><title>6->7</title></g></g></svg>

插入`<kind>`。

```py
v = m.insert_into_tree(tree, ('<kind>', 'van'))

```

```py
	 - Node: <start>		? (<kind>:'van')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<kind>:'van')
		 -> [0] '1997,van,'
		  > ['1997,', '<kind>', ',']

```

```py
display_tree(tree)

```

<svg height="176pt" viewBox="0.00 0.00 299.50 176.00" width="300pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="139" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="139" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="16" y="-54.8">1997,</text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="71" y="-54.8"><kind></text></g> <g class="edge" id="edge3"><title>1->3</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-54.8">,</text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="166" y="-54.8"><company></text></g> <g class="edge" id="edge6"><title>1->6</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="220" y="-54.8">,</text></g> <g class="edge" id="edge8"><title>1->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="266" y="-54.8"><model></text></g> <g class="edge" id="edge9"><title>1->9</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="71" y="-3.8">van</text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="166" y="-3.8">Ford</text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="266" y="-3.8">E350</text></g> <g class="edge" id="edge10"><title>9->10</title></g></g></svg>

插入`<year>`。

```py
v = m.insert_into_tree(tree, ('<year>', '1997'))

```

```py
	 - Node: <start>		? (<year>:'1997')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<year>:'1997')
		 -> [0] '1997,'
		  > ['<year>', ',']

```

```py
display_tree(tree)

```

<svg height="176pt" viewBox="0.00 0.00 329.50 176.00" width="330pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="142" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="142" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-54.8"><year></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="60" y="-54.8">,</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="101" y="-54.8"><kind></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="142" y="-54.8">,</text></g> <g class="edge" id="edge7"><title>1->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="196" y="-54.8"><company></text></g> <g class="edge" id="edge8"><title>1->8</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="250" y="-54.8">,</text></g> <g class="edge" id="edge10"><title>1->10</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="296" y="-54.8"><model></text></g> <g class="edge" id="edge11"><title>1->11</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-3.8">1997</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="101" y="-3.8">van</text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="196" y="-3.8">Ford</text></g> <g class="edge" id="edge9"><title>8->9</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="296" y="-3.8">E350</text></g> <g class="edge" id="edge12"><title>11->12</title></g></g></svg>

为了简化生活，我们定义了包装器函数`nt_var()`，该函数将令牌转换为相应的非终结符。

```py
class TreeMiner(TreeMiner):
    def nt_var(self, var):
        return var if is_nonterminal(var) else to_nonterminal(var)

```

现在，我们需要对整个语法应用新的定义。

```py
class TreeMiner(TreeMiner):
    def apply_new_definition(self, tree, var, value):
        nt_var = self.nt_var(var)
        return self.insert_into_tree(tree, (nt_var, value))

```

该算法实现为`get_derivation_tree()`。

```py
class TreeMiner(TreeMiner):
    def get_derivation_tree(self):
        tree = (START_SYMBOL, [(self.my_input, [])])

        for var, value in self.my_assignments:
            self.log(0, "%s=%s" % (var, repr(value)))
            self.apply_new_definition(tree, var, value)
        return tree

```

`TreeMiner`的用法如下：

```py
with Tracer(VEHICLES[0]) as tracer:
    process_vehicle(tracer.my_input)
assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()
dt = TreeMiner(tracer.my_input, assignments, log=True)
dt.tree

```

```py
 vehicle='1997,van,Ford,E350'
	 - Node: <start>		? (<vehicle>:'1997,van,Ford,E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['<vehicle>']
 model='E350'
	 - Node: <start>		? (<model>:'E350')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<model>:'E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['1997,van,Ford,', '<model>']
 company='Ford'
	 - Node: <start>		? (<company>:'Ford')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<company>:'Ford')
		 -> [0] '1997,van,Ford,'
		  > ['1997,van,', '<company>', ',']
 kind='van'
	 - Node: <start>		? (<kind>:'van')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<kind>:'van')
		 -> [0] '1997,van,'
		  > ['1997,', '<kind>', ',']
 year='1997'
	 - Node: <start>		? (<year>:'1997')
		 -> [0] '<vehicle>'
	 - Node: <vehicle>		? (<year>:'1997')
		 -> [0] '1997,'
		  > ['<year>', ',']

```

```py
('<start>',
 [('<vehicle>',
   [('<year>', [['1997', []]]),
    (',', []),
    ('<kind>', [['van', []]]),
    (',', []),
    ('<company>', [['Ford', []]]),
    (',', []),
    ('<model>', [['E350', []]])])])

```

所获得的派生树如下。

```py
display_tree(TreeMiner(tracer.my_input, assignments).tree)

```

<svg height="176pt" viewBox="0.00 0.00 329.50 176.00" width="330pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 172)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="142" y="-156.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="142" y="-105.8"><vehicle></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-54.8"><year></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="60" y="-54.8">,</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="101" y="-54.8"><kind></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="142" y="-54.8">,</text></g> <g class="edge" id="edge7"><title>1->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="196" y="-54.8"><company></text></g> <g class="edge" id="edge8"><title>1->8</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="250" y="-54.8">,</text></g> <g class="edge" id="edge10"><title>1->10</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="296" y="-54.8"><model></text></g> <g class="edge" id="edge11"><title>1->11</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-3.8">1997</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="101" y="-3.8">van</text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="196" y="-3.8">Ford</text></g> <g class="edge" id="edge9"><title>8->9</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="296" y="-3.8">E350</text></g> <g class="edge" id="edge12"><title>11->12</title></g></g></svg>

结合所有的部分：

```py
trees = []
for vehicle in VEHICLES:
    print(vehicle)
    with Tracer(vehicle) as tracer:
        process_vehicle(tracer.my_input)
    assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()
    trees.append((tracer.my_input, assignments))
    for var, val in assignments:
        print(var + " = " + repr(val))
    print()

```

```py
1997,van,Ford,E350
vehicle = '1997,van,Ford,E350'
model = 'E350'
company = 'Ford'
kind = 'van'
year = '1997'

2000,car,Mercury,Cougar
vehicle = '2000,car,Mercury,Cougar'
model = 'Cougar'
company = 'Mercury'
kind = 'car'
year = '2000'

1999,car,Chevy,Venture
vehicle = '1999,car,Chevy,Venture'
model = 'Venture'
company = 'Chevy'
kind = 'car'
year = '1999'

```

相应的派生树如下。

```py
csv_dt = []
for inputstr, assignments in trees:
    print(inputstr)
    dt = TreeMiner(inputstr, assignments)
    csv_dt.append(dt)
    display_tree(dt.tree)

```

```py
1997,van,Ford,E350
2000,car,Mercury,Cougar
1999,car,Chevy,Venture

```

### 从派生树中恢复语法

我们定义了一个类`Miner`，该类可以组合多个派生树以生成语法。 初始语法为空。

```py
class GrammarMiner:
    def __init__(self):
        self.grammar = {}

```

`tree_to_grammar()`方法通过一次选择一个节点并将其添加到语法，将我们的派生树转换为语法。 节点名称成为密钥，并且它拥有的所有子代列表都成为该密钥的另一种选择。

```py
class GrammarMiner(GrammarMiner):
    def tree_to_grammar(self, tree):
        node, children = tree
        one_alt = [ck for ck, gc in children]
        hsh = {node: [one_alt] if one_alt else []}
        for child in children:
            if not is_nonterminal(child[0]):
                continue
            chsh = self.tree_to_grammar(child)
            for k in chsh:
                if k not in hsh:
                    hsh[k] = chsh[k]
                else:
                    hsh[k].extend(chsh[k])
        return hsh

```

```py
gm = GrammarMiner()
gm.tree_to_grammar(csv_dt[0].tree)

```

```py
{'<start>': [['<vehicle>']],
 '<vehicle>': [['<year>', ',', '<kind>', ',', '<company>', ',', '<model>']],
 '<year>': [['1997']],
 '<kind>': [['van']],
 '<company>': [['Ford']],
 '<model>': [['E350']]}

```

此处生成的语法为`canonical`。 我们定义一个函数`readable()`，该函数采用规范语法并以可读形式返回。

```py
def readable(grammar):
    def readable_rule(rule):
        return ''.join(rule)

    return {k: list(set(readable_rule(a) for a in grammar[k]))
            for k in grammar}

```

```py
syntax_diagram(readable(gm.tree_to_grammar(csv_dt[0].tree)))

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 199.5 62" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="99.75" y="35">vehicle</text></g></g></g></g></svg>

```py
vehicle

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 575.5 62" width="575.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="87.0" y="35">year</text></g> <g class="terminal"><text x="148.25" y="35">,</text></g> <g class="non-terminal"><text x="209.5" y="35">kind</text></g> <g class="terminal"><text x="270.75" y="35">,</text></g> <g class="non-terminal"><text x="344.75" y="35">company</text></g> <g class="terminal"><text x="418.75" y="35">,</text></g> <g class="non-terminal"><text x="484.25" y="35">model</text></g></g></g></g></svg>

```py
year

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">1997</text></g></g></g></g></svg>

```py
kind

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 165.5 62" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="82.75" y="35">van</text></g></g></g></g></svg>

```py
company

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">Ford</text></g></g></g></g></svg>

```py
model

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">E350</text></g></g></g></g></svg>

`add_tree()`方法从当前语法中获取非终结点的组合列表，并添加要添加到语法中的树，并更新每个非终结点的定义。

```py
import [itertools](https://docs.python.org/3/library/itertools.html)

```

```py
class GrammarMiner(GrammarMiner):
    def add_tree(self, t):
        t_grammar = self.tree_to_grammar(t.tree)
        self.grammar = {
            key: self.grammar.get(key, []) + t_grammar.get(key, [])
            for key in itertools.chain(self.grammar.keys(), t_grammar.keys())
        }

```

`add_tree()`的用法如下：

```py
inventory_grammar = GrammarMiner()
for dt in csv_dt:
    inventory_grammar.add_tree(dt)

```

```py
syntax_diagram(readable(inventory_grammar.grammar))

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 199.5 62" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="99.75" y="35">vehicle</text></g></g></g></g></svg>

```py
vehicle

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 575.5 62" width="575.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="87.0" y="35">year</text></g> <g class="terminal"><text x="148.25" y="35">,</text></g> <g class="non-terminal"><text x="209.5" y="35">kind</text></g> <g class="terminal"><text x="270.75" y="35">,</text></g> <g class="non-terminal"><text x="344.75" y="35">company</text></g> <g class="terminal"><text x="418.75" y="35">,</text></g> <g class="non-terminal"><text x="484.25" y="35">model</text></g></g></g></g></svg>

```py
year

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 174.0 122" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">2000</text></g></g> <g><g class="terminal"><text x="87.0" y="65">1999</text></g></g> <g><g class="terminal"><text x="87.0" y="95">1997</text></g></g></g></g></svg>

```py
kind

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 165.5 92" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="82.75" y="35">van</text></g></g> <g><g class="terminal"><text x="82.75" y="65">car</text></g></g></g></g></svg>

```py
company

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 199.5 122" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Ford</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Chevy</text></g></g> <g><g class="terminal"><text x="99.75" y="95">Mercury</text></g></g></g></g></svg>

```py
model

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 199.5 122" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Venture</text></g></g> <g><g class="terminal"><text x="99.75" y="65">E350</text></g></g> <g><g class="terminal"><text x="99.75" y="95">Cougar</text></g></g></g></g></svg>

给定来自各种输入的执行跟踪，可以定义`update_grammar()`以从跟踪中获取完整的语法。

```py
class GrammarMiner(GrammarMiner):
    def update_grammar(self, inputstr, trace):
        at = self.create_tracker(inputstr, trace)
        dt = self.create_tree_miner(inputstr, at.assignments())
        self.add_tree(dt)
        return self.grammar

    def create_tracker(self, *args):
        return DefineTracker(*args)

    def create_tree_miner(self, *args):
        return TreeMiner(*args)

```

完整的语法恢复在`recover_grammar()`中实现。

```py
def recover_grammar(fn, inputs, **kwargs):
    miner = GrammarMiner()
    for inputstr in inputs:
        with Tracer(inputstr, **kwargs) as tracer:
            fn(tracer.my_input)
        miner.update_grammar(tracer.my_input, tracer.trace)
    return readable(miner.grammar)

```

请注意，语法可以直接从跟踪器检索，而无需中间派生树阶段。 但是，遍历派生树可以使人们检查被碎片化的输入并验证其正确执行。

#### 示例1.恢复清单语法

```py
inventory_grammar = recover_grammar(process_vehicle, VEHICLES)

```

```py
inventory_grammar

```

```py
{'<start>': ['<vehicle>'],
 '<vehicle>': ['<year>,<kind>,<company>,<model>'],
 '<year>': ['2000', '1999', '1997'],
 '<kind>': ['van', 'car'],
 '<company>': ['Ford', 'Chevy', 'Mercury'],
 '<model>': ['Venture', 'E350', 'Cougar']}

```

#### 例子2.恢复URL语法

我们的算法足够强大，可以从现实世界的程序中恢复语法。 例如，Python `urlib`模块中的`urlparse`函数接受以下示例URL。

```py
URLS = [
    'http://user:pass@www.google.com:80/?q=path#ref',
    'https://www.cispa.saarland:80/',
    'http://www.fuzzingbook.org/#News',
]

```

urllib缓存其中间结果以加快访问速度。 因此，我们需要在每次调用后使用`clear_cache()`将其禁用。

```py
from [urllib.parse](https://docs.python.org/3/library/urllib.parse.html) import urlparse, clear_cache

```

我们使用示例URL来恢复语法，如下所示。 `urlparse`函数倾向于缓存其先前的解析结果。 因此，我们定义了一个新方法`url_parse()`，该方法在每次调用之前清除缓存。

```py
def url_parse(url):
    clear_cache()
    urlparse(url)

```

```py
trees = []
for url in URLS:
    print(url)
    with Tracer(url) as tracer:
        url_parse(tracer.my_input)
    assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()
    trees.append((tracer.my_input, assignments))
    for var, val in assignments:
        print(var + " = " + repr(val))
    print()

url_dt = []
for inputstr, assignments in trees:
    print(inputstr)
    dt = TreeMiner(inputstr, assignments)
    url_dt.append(dt)
    display_tree(dt.tree)

```

```py
http://user:pass@www.google.com:80/?q=path#ref
url = 'http://user:pass@www.google.com:80/?q=path#ref'
scheme = 'http'
netloc = 'user:pass@www.google.com:80'
fragment = 'ref'
query = 'q=path'

https://www.cispa.saarland:80/
url = 'https://www.cispa.saarland:80/'
rest = '//www.cispa.saarland:80/'
scheme = 'https'
netloc = 'www.cispa.saarland:80'

http://www.fuzzingbook.org/#News
url = 'http://www.fuzzingbook.org/#News'
scheme = 'http'
netloc = 'www.fuzzingbook.org'
fragment = 'News'

http://user:pass@www.google.com:80/?q=path#ref
https://www.cispa.saarland:80/
http://www.fuzzingbook.org/#News

```

使用`url_parse()`恢复语法。

```py
url_grammar = recover_grammar(url_parse, URLS, files=['urllib/parse.py'])

```

```py
syntax_diagram(url_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 165.5 62" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="82.75" y="35">url</text></g></g></g></g></svg>

```py
url

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 643.5 122" width="643.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="260.5" y="35">scheme</text></g> <g class="terminal"><text x="330.25" y="35">:</text></g> <g class="non-terminal"><text x="391.5" y="35">rest</text></g></g> <g><g class="non-terminal"><text x="161.0" y="65">scheme</text></g> <g class="terminal"><text x="239.25" y="65">://</text></g> <g class="non-terminal"><text x="317.5" y="65">netloc</text></g> <g class="terminal"><text x="391.5" y="65">/#</text></g> <g class="non-terminal"><text x="474.0" y="65">fragment</text></g></g> <g><g class="non-terminal"><text x="95.5" y="95">scheme</text></g> <g class="terminal"><text x="173.75" y="95">://</text></g> <g class="non-terminal"><text x="252.0" y="95">netloc</text></g> <g class="terminal"><text x="326.0" y="95">/?</text></g> <g class="non-terminal"><text x="395.75" y="95">query</text></g> <g class="terminal"><text x="461.25" y="95">#</text></g> <g class="non-terminal"><text x="539.5" y="95">fragment</text></g></g></g></g></svg>

```py
scheme

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 182.5 92" width="182.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="91.25" y="35">https</text></g></g> <g><g class="terminal"><text x="91.25" y="65">http</text></g></g></g></g></svg>

```py
netloc

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 369.5 122" width="369.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="184.75" y="35">www.cispa.saarland:80</text></g></g> <g><g class="terminal"><text x="184.75" y="65">user:pass@www.google.com:80</text></g></g> <g><g class="terminal"><text x="184.75" y="95">www.fuzzingbook.org</text></g></g></g></g></svg>

```py
query

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 191.0 62" width="191.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="95.5" y="35">q=path</text></g></g></g></g></svg>

```py
fragment

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">News</text></g></g> <g><g class="terminal"><text x="87.0" y="65">ref</text></g></g></g></g></svg>

```py
rest

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 296.5 62" width="296.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">//</text></g> <g class="non-terminal"><text x="152.5" y="35">netloc</text></g> <g class="terminal"><text x="222.25" y="35">/</text></g></g></g></g></svg>

恢复的语法相当合理地描述了URL格式。

### 模糊

现在，我们可以使用恢复的语法进行模糊测试，如下所示。

首先，清单语法。

```py
f = GrammarFuzzer(inventory_grammar)
for _ in range(10):
    print(f.fuzz())

```

```py
1997,van,Ford,Venture
1999,van,Chevy,E350
1997,car,Ford,Cougar
2000,van,Mercury,Cougar
1999,van,Ford,Venture
1999,van,Ford,E350
1997,van,Chevy,Venture
1997,van,Chevy,Venture
1997,van,Ford,E350
2000,van,Chevy,Venture

```

接下来，URL语法。

```py
f = GrammarFuzzer(url_grammar)
for _ in range(10):
    print(f.fuzz())

```

```py
http://www.cispa.saarland:80/#ref
http://www.fuzzingbook.org/?q=path#ref
http://user:pass@www.google.com:80/?q=path#ref
https://www.fuzzingbook.org/?q=path#News
https://www.fuzzingbook.org/#ref
http://www.cispa.saarland:80/
https://www.cispa.saarland:80/
http://www.cispa.saarland:80/
https://www.cispa.saarland:80/#ref
http://user:pass@www.google.com:80/

```

这意味着我们现在可以获取一个程序和一些示例，提取其语法，然后将这种语法用于模糊测试。 现在，这是一个很大的机会！

### 简单矿工的问题

我们简单的语法挖掘器的问题之一是假设分配给变量的值是稳定的。 不幸的是，并非在所有情况下都适用。 例如，这里的URL格式略有不同。

```py
URLS_X = URLS + ['ftp://freebsd.org/releases/5.8']

```

从这组样本生成的语法不如我们先前获得的好

```py
url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])

```

```py
syntax_diagram(url_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 413.0 92" width="413.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="206.5" y="35">url</text></g></g> <g><g class="non-terminal"><text x="95.5" y="65">scheme</text></g> <g class="terminal"><text x="173.75" y="65">://</text></g> <g class="non-terminal"><text x="252.0" y="65">netloc</text></g> <g class="non-terminal"><text x="330.25" y="65">url</text></g></g></g></g></svg>

```py
url

```

 <svg class="railroad-diagram" height="152" viewBox="0 0 643.5 152" width="643.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="161.0" y="65">scheme</text></g> <g class="terminal"><text x="239.25" y="65">://</text></g> <g class="non-terminal"><text x="317.5" y="65">netloc</text></g> <g class="terminal"><text x="391.5" y="65">/#</text></g> <g class="non-terminal"><text x="474.0" y="65">fragment</text></g></g> <g><g class="non-terminal"><text x="260.5" y="35">scheme</text></g> <g class="terminal"><text x="330.25" y="35">:</text></g> <g class="non-terminal"><text x="391.5" y="35">rest</text></g></g> <g><g class="non-terminal"><text x="95.5" y="95">scheme</text></g> <g class="terminal"><text x="173.75" y="95">://</text></g> <g class="non-terminal"><text x="252.0" y="95">netloc</text></g> <g class="terminal"><text x="326.0" y="95">/?</text></g> <g class="non-terminal"><text x="395.75" y="95">query</text></g> <g class="terminal"><text x="461.25" y="95">#</text></g> <g class="non-terminal"><text x="539.5" y="95">fragment</text></g></g> <g><g class="terminal"><text x="321.75" y="125">/releases/5.8</text></g></g></g></g></svg>

```py
scheme

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 182.5 122" width="182.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="91.25" y="35">ftp</text></g></g> <g><g class="terminal"><text x="91.25" y="65">https</text></g></g> <g><g class="terminal"><text x="91.25" y="95">http</text></g></g></g></g></svg>

```py
netloc

```

 <svg class="railroad-diagram" height="152" viewBox="0 0 369.5 152" width="369.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="184.75" y="65">www.cispa.saarland:80</text></g></g> <g><g class="terminal"><text x="184.75" y="35">freebsd.org</text></g></g> <g><g class="terminal"><text x="184.75" y="95">user:pass@www.google.com:80</text></g></g> <g><g class="terminal"><text x="184.75" y="125">www.fuzzingbook.org</text></g></g></g></g></svg>

```py
query

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 191.0 62" width="191.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="95.5" y="35">q=path</text></g></g></g></g></svg>

```py
fragment

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">News</text></g></g> <g><g class="terminal"><text x="87.0" y="65">ref</text></g></g></g></g></svg>

```py
rest

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 296.5 62" width="296.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">//</text></g> <g class="non-terminal"><text x="152.5" y="35">netloc</text></g> <g class="terminal"><text x="222.25" y="35">/</text></g></g></g></g></svg>

显然，出了点问题。

要调查`url`定义出错的原因，让我们检查URL的跟踪。

```py
clear_cache()
with Tracer(URLS_X[0]) as tracer:
    urlparse(tracer.my_input)
for i, t in enumerate(tracer.trace):
    if t[0] in {'call', 'line'} and 'parse.py' in str(t[2]) and t[3]:
        print(i, t[2]._t()[1], t[3:])

```

```py
0 361 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
1 367 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
5 119 ({'arg': ''},)
6 116 ({'arg': ''},)
7 121 ({'arg': ''},)
8 122 ({'arg': ''},)
10 368 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
11 394 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
12 400 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
16 119 ({'arg': ''},)
17 116 ({'arg': ''},)
18 121 ({'arg': ''},)
19 122 ({'arg': ''},)
21 401 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
22 402 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
23 403 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
24 404 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
25 406 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
26 408 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
27 409 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
28 410 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
29 411 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
30 412 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
31 413 ({'scheme': 'http', 'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
32 414 ({'scheme': 'http', 'url': '//user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
33 415 ({'scheme': 'http', 'url': '//user:pass@www.google.com:80/?q=path#ref', 'fragment': '', 'query': '', 'netloc': ''},)
34 386 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)
35 387 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)
36 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)
37 389 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
38 390 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
39 391 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
40 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)
41 389 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
42 390 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
43 391 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
44 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)
45 389 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
46 390 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
47 391 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
48 388 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
49 392 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)
51 416 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
52 417 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
53 419 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
54 420 ({'scheme': 'http', 'url': '/?q=path#ref', 'fragment': '', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
55 421 ({'scheme': 'http', 'url': '/?q=path', 'fragment': 'ref', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
56 422 ({'scheme': 'http', 'url': '/?q=path', 'fragment': 'ref', 'query': '', 'netloc': 'user:pass@www.google.com:80'},)
57 423 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
61 424 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
62 425 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
67 369 ({'scheme': '', 'url': 'http://user:pass@www.google.com:80/?q=path#ref'},)
68 370 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
69 373 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80'},)
70 374 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80', 'params': ''},)
74 375 ({'scheme': 'http', 'url': '/', 'fragment': 'ref', 'query': 'q=path', 'netloc': 'user:pass@www.google.com:80', 'params': ''},)

```

注意`url`的值如何随着解析的进行而变化？ 这违反了我们的假设，即分配给变量的值是稳定的。 接下来，我们将探讨如何消除此限制。

## 重新分配语法的矿工

唯一标识不同变量的一种方法是，在定义变量以及变量值更改时，均用*行号*对其进行注释。 考虑下面的代码片段

### 跟踪变量分配位置

```py
def C(cp_1):
    c_2 = cp_1 + '@2'
    c_3 = c_2 + '@3'
    return c_3

def B(bp_7):
    b_8 = bp_7 + '@8'
    return C(b_8)

def A(ap_12):
    a_13 = ap_12 + '@13'
    a_14 = B(a_13) + '@14'
    a_14 = a_14 + '@15'
    a_13 = a_14 + '@16'
    a_14 = B(a_13) + '@17'
    a_14 = B(a_13) + '@18'

```

请注意，如何按照定义它们的位置来命名所有变量，或者对值进行注释以指示其已更改。

让我们在跟踪下运行它。

```py
with Tracer('____') as tracer:
    A(tracer.my_input)

for t in tracer.trace:
    print(t[0], "%d:%s" % (t[2].line_no, t[2].method), t[3])

```

```py
call 12:A {'ap_12': '____'}
line 13:A {'ap_12': '____'}
line 14:A {'ap_12': '____', 'a_13': '____@13'}
call 7:B {'bp_7': '____@13'}
line 8:B {'bp_7': '____@13'}
line 9:B {'bp_7': '____@13', 'b_8': '____@13@8'}
call 1:C {'cp_1': '____@13@8'}
line 2:C {'cp_1': '____@13@8'}
line 3:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2'}
line 4:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2', 'c_3': '____@13@8@2@3'}
return 4:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2', 'c_3': '____@13@8@2@3'}
return 9:B {'bp_7': '____@13', 'b_8': '____@13@8'}
line 15:A {'ap_12': '____', 'a_13': '____@13', 'a_14': '____@13@8@2@3@14'}
line 16:A {'ap_12': '____', 'a_13': '____@13', 'a_14': '____@13@8@2@3@14@15'}
line 17:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15'}
call 7:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 8:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
call 1:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 2:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 3:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2'}
line 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
line 18:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15@16@8@2@3@17'}
call 7:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 8:B {'bp_7': '____@13@8@2@3@14@15@16'}
line 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
call 1:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 2:C {'cp_1': '____@13@8@2@3@14@15@16@8'}
line 3:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2'}
line 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}
return 9:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}
return 18:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15@16@8@2@3@18'}
call 24:__exit__ {}
line 25:__exit__ {}

```

每个变量首先被引用如下：

*   `cp_1`-*呼叫* `1:C`
*   `c_2`-*行* `3:C`（但上一个事件是*行* `2:C`）
*   `c_3`-*行* `4:C`（但上一个事件是*行* `3:C`）
*   `bp_7`-*呼叫* `7:B`
*   `b_8`-*行* `9:B`（但上一个事件是*行* `8:B`）
*   `ap_12`-*呼叫* `12:A`
*   `a_13`-*行* `14:A`（但上一个事件是*行* `13:A`）
*   `a_14`-*行* `15:A`（先前的事件是*返回* `9:B`。但是，A中的先前事件是*行* `14:A`）
*   在 *15* -*行* `16:A`上重新分配`a_14`（上一个事件是*行* `15:A`）
*   在 *16* -*行* `17:A`中重新分配`a_13`（上一个事件是*行* `16:A`）
*   在 *17* 上重新分配`a_14`-*返回* `17:A`（A中的上一个事件是*行* `17:A`）
*   在 *18* -*返回* `18:A`时重新分配`a_14`（A中的上一个事件是*行* `18:A`）

因此，我们的观察结果是，如果这是一个调用，则当前位置对于定义的任何新变量都是正确的位置。 另一方面，如果是首次引用变量（或重新分配了新值），则要考虑的正确位置是同一方法调用中的先前位置*。 接下来，让我们看看如何将这些信息整合到变量命名中。*

接下来，我们需要一种方法来跟踪各个方法的调用。 为此，我们定义了`CallStack`类。 每个方法调用都获得一个单独的标识符，并且在方法调用结束后，将重置该标识符。

### CallStack

```py
class CallStack:
    def __init__(self, **kwargs):
        self.options(kwargs)
        self.method_id = (START_SYMBOL, 0)
        self.method_register = 0
        self.mstack = [self.method_id]

    def enter(self, method):
        self.method_register += 1
        self.method_id = (method, self.method_register)
        self.log('call', "%s%s" % (self.indent(), str(self)))
        self.mstack.append(self.method_id)

    def leave(self):
        self.mstack.pop()
        self.log('return', "%s%s" % (self.indent(), str(self)))
        self.method_id = self.mstack[-1]

```

一些额外的功能可简化生活。

```py
class CallStack(CallStack):
    def options(self, kwargs):
        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None

    def indent(self):
        return len(self.mstack) * "\t"

    def at(self, n):
        return self.mstack[n]

    def __len__(self):
        return len(mstack) - 1

    def __str__(self):
        return "%s:%d" % self.method_id

    def __repr__(self):
        return repr(self.method_id)

```

我们还定义了一种方便的方法来显示给定的堆栈。

```py
def display_stack(istack):
    def stack_to_tree(stack):
        current, *rest = stack
        if not rest:
            return (repr(current), [])
        return (repr(current), [stack_to_tree(rest)])
    display_tree(stack_to_tree(istack.mstack), graph_attr=lr_graph)

```

这是我们如何使用`CallStack`的方法。

```py
cs = CallStack()
display_stack(cs)
cs

```

```py
('<start>', 0)

```

```py
cs.enter('hello')
display_stack(cs)
cs

```

```py
('hello', 1)

```

```py
cs.enter('world')
display_stack(cs)
cs

```

```py
('world', 2)

```

```py
cs.leave()
display_stack(cs)
cs

```

```py
('hello', 1)

```

```py
cs.enter('world')
display_stack(cs)
cs

```

```py
('world', 3)

```

```py
cs.leave()
display_stack(cs)
cs

```

```py
('hello', 1)

```

为了考虑变量的重新分配，我们需要一种比用于存储变量的字典更智能的数据结构。 我们首先定义一个简单的接口`Vars`。 它充当变量的容器，并在`my_assignments`处实例化。

### 新鲜的

`Vars`在对其内部字典`defs`进行解析时会存储对变量的引用。 我们用原始字符串初始化字典。

```py
class Vars:
    def __init__(self, original):
        self.defs = {}
        self.my_input = original

```

该词典需要两种方法：`update()`需要一组键值对来更新自身，而`_set_kv()`需要更新特定的键值对。

```py
class Vars(Vars):
    def _set_kv(self, k, v):
        self.defs[k] = v

    def __setitem__(self, k, v):
        self._set_kv(k, v)

    def update(self, v):
        for k, v in v.items():
            self._set_kv(k, v)

```

vars是内部字典的代理。 例如，这是一种使用方式。

```py
v = Vars('')
v.defs

```

```py
{}

```

```py
v['x'] = 'X'
v.defs

```

```py
{'x': 'X'}

```

```py
v.update({'x': 'x', 'y': 'y'})
v.defs

```

```py
{'x': 'x', 'y': 'y'}

```

### AssignmentVars

现在，我们将简单的`Vars`扩展为考虑变量重新分配。 为此，我们定义`AssignmentVars`。

检测重新分配和重命名变量的想法如下：我们使用`accessed_seq_var`跟踪先前对特定变量的重新分配。 它包含任何特定变量的最后重命名作为其相应的值。 `new_vars`包含此迭代中添加的所有新变量的列表。

```py
class AssignmentVars(Vars):
    def __init__(self, original):
        super().__init__(original)
        self.accessed_seq_var = {}
        self.var_def_lines = {}
        self.current_event = None
        self.new_vars = set()
        self.method_init()

```

`method_init()`方法使用保存在`call_stack`中的记录来跟踪方法调用。 `event_locations`用于跟踪此方法中访问的位置*。 这用于跟踪变量定义的行号。*

```py
class AssignmentVars(AssignmentVars):
    def method_init(self):
        self.call_stack = CallStack()
        self.event_locations = {self.call_stack.method_id: []}

```

现在，可以使用`var_location_register()`修改`update()`以跟踪更改的行号。 在用于下一个事件后，我们将重新初始化`new_vars`。

```py
class AssignmentVars(AssignmentVars):
    def update(self, v):
        for k, v in v.items():
            self._set_kv(k, v)
        self.var_location_register(self.new_vars)
        self.new_vars = set()

```

变量名称现在包含一个索引，该索引显示了已完成的重新分配次数，有效地使每个重新分配成为唯一变量。

```py
class AssignmentVars(AssignmentVars):
    def var_name(self, var):
        return (var, self.accessed_seq_var[var])

```

在存储变量时，我们需要首先检查它是否先前已知。 如果不是，则需要初始化重命名计数。 这是通过`var_access`完成的。

```py
class AssignmentVars(AssignmentVars):
    def var_access(self, var):
        if var not in self.accessed_seq_var:
            self.accessed_seq_var[var] = 0
        return self.var_name(var)

```

在变量重新分配期间，我们更新`accessed_seq_var`以反映新的计数。

```py
class AssignmentVars(AssignmentVars):
    def var_assign(self, var):
        self.accessed_seq_var[var] += 1
        self.new_vars.add(self.var_name(var))
        return self.var_name(var)

```

这些方法可以如下使用

```py
sav = AssignmentVars('')
sav.defs

```

```py
{}

```

```py
sav.var_access('v1')

```

```py
('v1', 0)

```

```py
sav.var_assign('v1')

```

```py
('v1', 1)

```

再次为其分配计数器将递增。

```py
sav.var_assign('v1')

```

```py
('v1', 2)

```

逻辑的核心在`_set_kv()`中。 当分配一个变量时，我们得到序列变量名`s_var`。 如果在`defs`中以前未知序列变量名称，那么我们就没有其他问题了。 我们将排序后的变量添加到`defs`中。

如果该变量是先前已知的，则表明可能重新分配。 在这种情况下，我们查看变量持有的值。 我们检查值是否更改。 如果没有，则没有。

如果值已更改，则为重新分配。 我们首先使用`var_assign`递增变量使用顺序，检索新名称，然后在`defs`中更新新名称。

```py
class AssignmentVars(AssignmentVars):
    def _set_kv(self, var, val):
        s_var = self.var_access(var)
        if s_var in self.defs and self.defs[s_var] == val:
            return
        self.defs[self.var_assign(var)] = val

```

这是如何使用它。 第一次分配变量将初始化其计数器。

```py
sav = AssignmentVars('')
sav['x'] = 'X'
sav.defs

```

```py
{('x', 1): 'X'}

```

如果再次给变量​​分配相同的值，则可能不是重新分配。

```py
sav['x'] = 'X'
sav.defs

```

```py
{('x', 1): 'X'}

```

但是，如果值更改，则是重新分配。

```py
sav['x'] = 'Y'
sav.defs

```

```py
{('x', 1): 'X', ('x', 2): 'Y'}

```

这里有一个微妙之处。 可以从父方法的中间调用子方法，并且两个方法都可以使用具有不同值的相同变量名。 在这种情况下，当子级返回时，父级将具有上下文中具有旧值的旧变量。 通过我们的实施，我们认为这是重新分配。 但是，这是可以的，因为添加新的重新分配是无害的，但是丢失新分配不会。 此外，我们稍后将讨论如何避免这种情况。

我们还为`register_event()` `method_enter()`和`method_exit()`定义了记账代码，它们是负责跟踪方法堆栈的方法。 基本思想是，每个`method_enter()`代表一个新的方法调用。 因此，它具有一个新的方法id，该方法id由`method_register`生成并保存在`method_id`中。 由于这是一种新方法，因此该方法堆栈将使用具有此id的一个元素扩展。 在`method_exit()`的情况下，我们弹出方法堆栈，并将当前的`method_id`重置为低于当前的值。

```py
class AssignmentVars(AssignmentVars):
    def method_enter(self, cxt, my_vars):
        self.current_event = 'call'
        self.call_stack.enter(cxt.method)
        self.event_locations[self.call_stack.method_id] = []
        self.register_event(cxt)
        self.update(my_vars)

    def method_exit(self, cxt, my_vars):
        self.current_event = 'return'
        self.register_event(cxt)
        self.update(my_vars)
        self.call_stack.leave()

    def method_statement(self, cxt, my_vars):
        self.current_event = 'line'
        self.register_event(cxt)
        self.update(my_vars)

```

对于每个方法事件，我们还使用`register_event()`注册事件，该事件跟踪*此*方法中引用的行号。

```py
class AssignmentVars(AssignmentVars):
    def register_event(self, cxt):
        self.event_locations[self.call_stack.method_id].append(cxt.line_no)

```

`var_location_register()`保留新添加变量的位置。 `call`中变量的定义位置是*当前*位置。 但是，对于`line`，它将是当前方法中的上一个事件。

```py
class AssignmentVars(AssignmentVars):
    def var_location_register(self, my_vars):
        def loc(mid):
            if self.current_event == 'call':
                return self.event_locations[mid][-1]
            elif self.current_event == 'line':
                return self.event_locations[mid][-2]
            elif self.current_event == 'return':
                return self.event_locations[mid][-2]
            else:
                assert False

        my_loc = loc(self.call_stack.method_id)
        for var in my_vars:
            self.var_def_lines[var] = my_loc

```

我们定义`defined_vars()`，它返回带有行号的变量名称，如下所示。

```py
class AssignmentVars(AssignmentVars):
    def defined_vars(self, formatted=True):
        def fmt(k):
            v = (k[0], self.var_def_lines[k])
            return "%s@%s" % v if formatted else v

        return [(fmt(k), v) for k, v in self.defs.items()]

```

与`defined_vars()`相似，我们定义了`seq_vars()`，它以使用次数来注释不同的变量。

```py
class AssignmentVars(AssignmentVars):
    def seq_vars(self, formatted=True):
        def fmt(k):
            v = (k[0], self.var_def_lines[k], k[1])
            return "%s@%s:%s" % v if formatted else v

        return {fmt(k): v for k, v in self.defs.items()}

```

### AssignmentTracker

`AssignmentTracker`使用我们先前定义的`AssignmentVars`保留分配定义。

```py
class AssignmentTracker(DefineTracker):
    def __init__(self, my_input, trace, **kwargs):
        self.options(kwargs)
        self.my_input = my_input

        self.my_assignments = self.create_assignments(my_input)

        self.trace = trace
        self.process()

    def create_assignments(self, *args):
        return AssignmentVars(*args)

```

为了对过程进行微调，我们定义了一个可选参数`track_return`。 在跟踪方法返回的过程中，Python会生成一个虚拟变量，其中包含返回值的结果。 如果设置了`track_return`，我们将捕获此值作为变量。

*   `track_return`-如果为true，则将*虚拟变量*添加到表示返回值的Var中

```py
class AssignmentTracker(AssignmentTracker):
    def options(self, kwargs):
        self.track_return = kwargs.get('track_return', False)
        super().options(kwargs)

```

跟踪期间可能会有不同类型的事件，包括进入函数时的`call`，返回函数时的`return`，引发异常时的`exception`和执行语句时的`line`。

先前的`Tracker`过于简单，因为它无法区分不同的事件。 我们对此进行纠正并定义分别在其相应事件上调用的`on_call()`，`on_return()`和`on_line()`。

注意，`on_return()`也被调用`on_line()`。 原因是，Python在执行相应行之前调用了跟踪函数*。 因此，有效地，通过在环境中执行前一个语句所产生的绑定来调用`on_return()`。 实际上，我们的处理是对上一条语句所绑定的值进行的。 因此，在此处调用`on_line()`是适当的，因为它为事件处理程序提供了处理先前绑定的机会。*

```py
class AssignmentTracker(AssignmentTracker):
    def on_call(self, arg, cxt, my_vars):
        my_vars = cxt.parameters(my_vars)
        self.my_assignments.method_enter(cxt, self.fragments(my_vars))

    def on_line(self, arg, cxt, my_vars):
        self.my_assignments.method_statement(cxt, self.fragments(my_vars))

    def on_return(self, arg, cxt, my_vars):
        self.on_line(arg, cxt, my_vars)
        my_vars = {'<-%s' % cxt.method: arg} if self.track_return else {}
        self.my_assignments.method_exit(cxt, my_vars)

    def on_exception(self, arg, cxt, my_vara):
        return

    def track_event(self, event, arg, cxt, my_vars):
        self.current_event = event
        dispatch = {
            'call': self.on_call,
            'return': self.on_return,
            'line': self.on_line,
            'exception': self.on_exception
        }
        dispatch[event](arg, cxt, my_vars)

```

现在，我们可以使用`AssignmentTracker`来跟踪不同的变量。 为了验证可变行号推断是否有效，我们从函数A，B和C恢复定义（删除了数据注释，以便正确识别输入片段）。

```py
def C(cp_1):
    c_2 = cp_1
    c_3 = c_2
    return c_3

def B(bp_7):
    b_8 = bp_7
    return C(b_8)

def A(ap_12):
    a_13 = ap_12
    a_14 = B(a_13)
    a_14 = a_14
    a_13 = a_14
    a_14 = B(a_13)
    a_14 = B(a_14)[3:]

```

在有足够输入的情况下运行`A()`。

```py
with Tracer('---xxx') as tracer:
    A(tracer.my_input)
tracker = AssignmentTracker(tracer.my_input, tracer.trace, log=True)
for k, v in tracker.my_assignments.seq_vars().items():
    print(k, '=', repr(v))
print()
for k, v in tracker.my_assignments.defined_vars(formatted=True):
    print(k, '=', repr(v))

```

```py
ap_12@12:1 = '---xxx'
a_13@13:1 = '---xxx'
bp_7@7:1 = '---xxx'
b_8@8:1 = '---xxx'
cp_1@1:1 = '---xxx'
c_2@2:1 = '---xxx'
c_3@3:1 = '---xxx'
a_14@14:1 = '---xxx'
a_14@18:2 = 'xxx'

ap_12@12 = '---xxx'
a_13@13 = '---xxx'
bp_7@7 = '---xxx'
b_8@8 = '---xxx'
cp_1@1 = '---xxx'
c_2@2 = '---xxx'
c_3@3 = '---xxx'
a_14@14 = '---xxx'
a_14@18 = 'xxx'

```

可以看出，现在已经为每个变量正确标识了行号。

让我们尝试检索一个实际示例的分配。

```py
traces = []
for inputstr in URLS_X:
    clear_cache()
    with Tracer(inputstr, files=['urllib/parse.py']) as tracer:
        urlparse(tracer.my_input)
    traces.append((tracer.my_input, tracer.trace))

    tracker = AssignmentTracker(tracer.my_input, tracer.trace, log=True)
    for k, v in tracker.my_assignments.defined_vars():
        print(k, '=', repr(v))
    print()

```

```py
url@361 = 'http://user:pass@www.google.com:80/?q=path#ref'
scheme@412 = 'http'
url@413 = '//user:pass@www.google.com:80/?q=path#ref'
url@415 = '/?q=path#ref'
netloc@415 = 'user:pass@www.google.com:80'
url@420 = '/?q=path'
fragment@420 = 'ref'
query@422 = 'q=path'
url@368 = 'http://user:pass@www.google.com:80/?q=path#ref'

url@361 = 'https://www.cispa.saarland:80/'
rest@432 = '//www.cispa.saarland:80/'
scheme@435 = 'https'
url@435 = '//www.cispa.saarland:80/'
netloc@438 = 'www.cispa.saarland:80'
url@368 = 'https://www.cispa.saarland:80/'

url@361 = 'http://www.fuzzingbook.org/#News'
scheme@412 = 'http'
url@413 = '//www.fuzzingbook.org/#News'
url@415 = '/#News'
netloc@415 = 'www.fuzzingbook.org'
fragment@420 = 'News'
url@368 = 'http://www.fuzzingbook.org/#News'

url@361 = 'ftp://freebsd.org/releases/5.8'
rest@432 = '//freebsd.org/releases/5.8'
scheme@435 = 'ftp'
url@435 = '//freebsd.org/releases/5.8'
url@438 = '/releases/5.8'
netloc@438 = 'freebsd.org'
url@368 = 'ftp://freebsd.org/releases/5.8'
url@369 = '/releases/5.8'

```

可以从 [urllib / parse.py](https://github.com/python/cpython/blob/3.6/Lib/urllib/parse.py) 的源代码验证变量的行号。

### 恢复派生树

处理变量重新分配对我们的URL示例有帮助吗？ 接下来我们看这些。

```py
class TreeMiner(TreeMiner):
    def get_derivation_tree(self):
        tree = (START_SYMBOL, [(self.my_input, [])])
        for var, value in self.my_assignments:
            self.log(0, "%s=%s" % (var, repr(value)))
            self.apply_new_definition(tree, var, value)
        return tree

```

#### 示例1：恢复URL派生树

首先，我们获得URL 1的派生树。

##### URL 1派生树

```py
clear_cache()
with Tracer(URLS_X[0], files=['urllib/parse.py']) as tracer:
    urlparse(tracer.my_input)
sm = AssignmentTracker(tracer.my_input, tracer.trace)
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())
display_tree(dt.tree)

```

<svg height="329pt" viewBox="0.00 0.00 469.50 329.00" width="470pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 325)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-309.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-258.8"><url@361></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="46" y="-207.8"><scheme@412></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-207.8">:</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="165" y="-207.8"><url@413></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="46" y="-156.8">http</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="101" y="-156.8">//</text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="165" y="-156.8"><netloc@415></text></g> <g class="edge" id="edge7"><title>5->7</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="310" y="-156.8"><url@415></text></g> <g class="edge" id="edge9"><title>5->9</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="141" y="-105.8">user:pass@www.google.com:80</text></g> <g class="edge" id="edge8"><title>7->8</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="283" y="-105.8"><url@420></text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="338" y="-105.8">#</text></g> <g class="edge" id="edge14"><title>9->14</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="411" y="-105.8"><fragment@420></text></g> <g class="edge" id="edge15"><title>9->15</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="248" y="-54.8">/?</text></g> <g class="edge" id="edge11"><title>10->11</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="313" y="-54.8"><query@422></text></g> <g class="edge" id="edge12"><title>10->12</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="313" y="-3.8">q=path</text></g> <g class="edge" id="edge13"><title>12->13</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="411" y="-54.8">ref</text></g> <g class="edge" id="edge16"><title>15->16</title></g></g></svg>

接下来，我们获得URL 4的派生树

##### URL 4派生树

```py
clear_cache()
with Tracer(URLS_X[-1], files=['urllib/parse.py']) as tracer:
    urlparse(tracer.my_input)
sm = AssignmentTracker(tracer.my_input, tracer.trace)
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())
display_tree(dt.tree)

```

<svg height="329pt" viewBox="0.00 0.00 303.00 329.00" width="303pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 325)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-309.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-258.8"><url@361></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="46" y="-207.8"><scheme@435></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="112" y="-207.8">:</text></g> <g class="edge" id="edge4"><title>1->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="167" y="-207.8"><rest@432></text></g> <g class="edge" id="edge5"><title>1->5</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="46" y="-156.8">ftp</text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="167" y="-156.8"><url@435></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="103" y="-105.8">//</text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="167" y="-105.8"><netloc@438></text></g> <g class="edge" id="edge8"><title>6->8</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="260" y="-105.8"><url@438></text></g> <g class="edge" id="edge10"><title>6->10</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="167" y="-54.8">freebsd.org</text></g> <g class="edge" id="edge9"><title>8->9</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="260" y="-54.8"><url@369></text></g> <g class="edge" id="edge11"><title>10->11</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="260" y="-3.8">/releases/5.8</text></g> <g class="edge" id="edge12"><title>11->12</title></g></g></svg>

派生树似乎属于同一语法。 因此，我们获得了完整的语法集。 首先，我们将`recover_grammar()`更新为使用`AssignTracker`。

### 恢复语法

```py
class GrammarMiner(GrammarMiner):
    def update_grammar(self, inputstr, trace):
        at = self.create_tracker(inputstr, trace)
        dt = self.create_tree_miner(inputstr, at.my_assignments.defined_vars())
        self.add_tree(dt)
        return self.grammar

    def create_tracker(self, *args):
        return AssignmentTracker(*args)

    def create_tree_miner(self, *args):
        return TreeMiner(*args)

```

接下来，我们对从URL获得的派生树使用修改后的`recover_grammar()`。

```py
url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])

```

恢复的语法如下。

```py
syntax_diagram(url_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 199.5 62" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="99.75" y="35">url@361</text></g></g></g></g></svg>

```py
url@361

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 381.5 92" width="381.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="116.75" y="35">scheme@412</text></g> <g class="terminal"><text x="203.5" y="35">:</text></g> <g class="non-terminal"><text x="277.5" y="35">url@413</text></g></g> <g><g class="non-terminal"><text x="112.5" y="65">scheme@435</text></g> <g class="terminal"><text x="199.25" y="65">:</text></g> <g class="non-terminal"><text x="277.5" y="65">rest@432</text></g></g></g></g></svg>

```py
scheme@412

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">http</text></g></g></g></g></svg>

```py
url@413

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 381.5 62" width="381.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">//</text></g> <g class="non-terminal"><text x="169.5" y="35">netloc@415</text></g> <g class="non-terminal"><text x="281.75" y="35">url@415</text></g></g></g></g></svg>

```py
netloc@415

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 369.5 92" width="369.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="184.75" y="35">www.fuzzingbook.org</text></g></g> <g><g class="terminal"><text x="184.75" y="65">user:pass@www.google.com:80</text></g></g></g></g></svg>

```py
url@415

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 390.0 92" width="390.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="124.0" y="35">/#</text></g> <g class="non-terminal"><text x="223.5" y="35">fragment@420</text></g></g> <g><g class="non-terminal"><text x="99.75" y="65">url@420</text></g> <g class="terminal"><text x="173.75" y="65">#</text></g> <g class="non-terminal"><text x="269.0" y="65">fragment@420</text></g></g></g></g></svg>

```py
url@420

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 273.5 62" width="273.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">/?</text></g> <g class="non-terminal"><text x="165.25" y="35">query@422</text></g></g></g></g></svg>

```py
query@422

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 191.0 62" width="191.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="95.5" y="35">q=path</text></g></g></g></g></svg>

```py
fragment@420

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">News</text></g></g> <g><g class="terminal"><text x="87.0" y="65">ref</text></g></g></g></g></svg>

```py
scheme@435

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 182.5 92" width="182.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="91.25" y="35">ftp</text></g></g> <g><g class="terminal"><text x="91.25" y="65">https</text></g></g></g></g></svg>

```py
rest@432

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 199.5 62" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="99.75" y="35">url@435</text></g></g></g></g></svg>

```py
url@435

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 381.5 92" width="381.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">//</text></g> <g class="non-terminal"><text x="169.5" y="35">netloc@438</text></g> <g class="non-terminal"><text x="281.75" y="35">url@438</text></g></g> <g><g class="terminal"><text x="104.0" y="65">//</text></g> <g class="non-terminal"><text x="195.0" y="65">netloc@438</text></g> <g class="terminal"><text x="281.75" y="65">/</text></g></g></g></g></svg>

```py
netloc@438

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 318.5 92" width="318.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="159.25" y="35">freebsd.org</text></g></g> <g><g class="terminal"><text x="159.25" y="65">www.cispa.saarland:80</text></g></g></g></g></svg>

```py
url@438

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 199.5 62" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="99.75" y="35">url@369</text></g></g></g></g></svg>

```py
url@369

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 250.5 62" width="250.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="125.25" y="35">/releases/5.8</text></g></g></g></g></svg>

让我们稍微模糊一下，看看产生的值是否合理。

```py
f = GrammarFuzzer(url_grammar)
for _ in range(10):
    print(f.fuzz())

```

```py
https://freebsd.org/releases/5.8
http://www.fuzzingbook.org/#News
http://user:pass@www.google.com:80/#News
ftp://www.cispa.saarland:80/
http://user:pass@www.google.com:80/#News
http://user:pass@www.google.com:80/#News
https://www.cispa.saarland:80/releases/5.8
http://user:pass@www.google.com:80/?q=path#ref
https://www.cispa.saarland:80/releases/5.8
http://www.fuzzingbook.org/#ref

```

我们的修改似乎确实有帮助。 接下来，我们检查是否仍然可以检索清单的语法。

#### 示例2：恢复库存语法

```py
inventory_grammar = recover_grammar(process_vehicle, VEHICLES)

```

```py
syntax_diagram(inventory_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 216.5 62" width="216.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="108.25" y="35">vehicle@1</text></g></g></g></g></svg>

```py
vehicle@1

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 643.5 62" width="643.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="95.5" y="35">year@2</text></g> <g class="terminal"><text x="165.25" y="35">,</text></g> <g class="non-terminal"><text x="235.0" y="35">kind@2</text></g> <g class="terminal"><text x="304.75" y="35">,</text></g> <g class="non-terminal"><text x="387.25" y="35">company@2</text></g> <g class="terminal"><text x="469.75" y="35">,</text></g> <g class="non-terminal"><text x="543.75" y="35">model@2</text></g></g></g></g></svg>

```py
year@2

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 174.0 122" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">2000</text></g></g> <g><g class="terminal"><text x="87.0" y="65">1999</text></g></g> <g><g class="terminal"><text x="87.0" y="95">1997</text></g></g></g></g></svg>

```py
kind@2

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 165.5 92" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="82.75" y="35">van</text></g></g> <g><g class="terminal"><text x="82.75" y="65">car</text></g></g></g></g></svg>

```py
company@2

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 199.5 122" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Ford</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Chevy</text></g></g> <g><g class="terminal"><text x="99.75" y="95">Mercury</text></g></g></g></g></svg>

```py
model@2

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 199.5 122" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Venture</text></g></g> <g><g class="terminal"><text x="99.75" y="65">E350</text></g></g> <g><g class="terminal"><text x="99.75" y="95">Cougar</text></g></g></g></g></svg>

使用模糊测试从语法中产生值。

```py
f = GrammarFuzzer(inventory_grammar)
for _ in range(10):
    print(f.fuzz())

```

```py
1997,van,Mercury,Cougar
2000,van,Mercury,E350
2000,van,Chevy,E350
1997,car,Mercury,Venture
1999,van,Mercury,Venture
1999,van,Mercury,Venture
1999,car,Mercury,Cougar
2000,van,Mercury,E350
1999,car,Chevy,Cougar
1999,van,Mercury,Venture

```

### 重新分配[的语法矿工存在的问题](#Problems-with-the-Grammar-Miner-with-Reassignment)

我们的语法挖掘器的问题之一是它尚未考虑当前上下文。 也就是说，在替换时，变量可以替换其无权访问的令牌（因此，不是令牌的一部分）。 考虑这个例子。

```py
with Tracer(INVENTORY) as tracer:
    process_inventory(tracer.my_input)
sm = AssignmentTracker(tracer.my_input, tracer.trace)
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())
display_tree(dt.tree, graph_attr=lr_graph)

```

<svg height="617pt" viewBox="0.00 0.00 496.00 617.00" width="496pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 613)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-267.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="121" y="-267.8"><inventory@1></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="240.5" y="-481.8"><vehicle@3></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node15"><title>14</title></g> <g class="edge" id="edge14"><title>1->14</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="240.5" y="-267.8"><vehicle@3></text></g> <g class="edge" id="edge15"><title>1->15</title></g> <g class="node" id="node28"><title>27</title></g> <g class="edge" id="edge27"><title>1->27</title></g> <g class="node" id="node29"><title>28</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="240.5" y="-86.8"><vehicle@3></text></g> <g class="edge" id="edge28"><title>1->28</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-597.8"><year@2></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-564.8">,</text></g> <g class="edge" id="edge5"><title>2->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-531.8"><kind@2></text></g> <g class="edge" id="edge6"><title>2->6</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-498.8">,</text></g> <g class="edge" id="edge8"><title>2->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-465.8"><company@2></text></g> <g class="edge" id="edge9"><title>2->9</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-432.8">,</text></g> <g class="edge" id="edge11"><title>2->11</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-399.8"><model@2></text></g> <g class="edge" id="edge12"><title>2->12</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-597.8">1997</text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-531.8">van</text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-465.8">Ford</text></g> <g class="edge" id="edge10"><title>9->10</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-399.8">E350</text></g> <g class="edge" id="edge13"><title>12->13</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-366.8"><year@2></text></g> <g class="edge" id="edge16"><title>15->16</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-333.8">,</text></g> <g class="edge" id="edge18"><title>15->18</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-300.8"><kind@2></text></g> <g class="edge" id="edge19"><title>15->19</title></g> <g class="node" id="node22"><title>21</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-267.8">,</text></g> <g class="edge" id="edge21"><title>15->21</title></g> <g class="node" id="node23"><title>22</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-234.8"><company@2></text></g> <g class="edge" id="edge22"><title>15->22</title></g> <g class="node" id="node25"><title>24</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-201.8">,</text></g> <g class="edge" id="edge24"><title>15->24</title></g> <g class="node" id="node26"><title>25</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-168.8"><model@2></text></g> <g class="edge" id="edge25"><title>15->25</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-366.8">2000</text></g> <g class="edge" id="edge17"><title>16->17</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-300.8">car</text></g> <g class="edge" id="edge20"><title>19->20</title></g> <g class="node" id="node24"><title>23</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-234.8">Mercury</text></g> <g class="edge" id="edge23"><title>22->23</title></g> <g class="node" id="node27"><title>26</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-168.8">Cougar</text></g> <g class="edge" id="edge26"><title>25->26</title></g> <g class="node" id="node30"><title>29</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-135.8"><year@2></text></g> <g class="edge" id="edge29"><title>28->29</title></g> <g class="node" id="node32"><title>31</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-102.8">,car,</text></g> <g class="edge" id="edge31"><title>28->31</title></g> <g class="node" id="node33"><title>32</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-69.8"><company@2></text></g> <g class="edge" id="edge32"><title>28->32</title></g> <g class="node" id="node35"><title>34</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-36.8">,</text></g> <g class="edge" id="edge34"><title>28->34</title></g> <g class="node" id="node36"><title>35</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="359" y="-3.8"><model@2></text></g> <g class="edge" id="edge35"><title>28->35</title></g> <g class="node" id="node31"><title>30</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-135.8">1999</text></g> <g class="edge" id="edge30"><title>29->30</title></g> <g class="node" id="node34"><title>33</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-69.8">Chevy</text></g> <g class="edge" id="edge33"><title>32->33</title></g> <g class="node" id="node37"><title>36</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="463.5" y="-3.8">Venture</text></g> <g class="edge" id="edge36"><title>35->36</title></g></g></svg>

可以看出，获得的派生树与我们的预期并不完全相同。 如果我们启用登录`TreeMiner`，则很容易看到该问题。

```py
dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars(), log=True)

```

```py
 inventory@1='1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
	 - Node: <start>		? (<inventory@1>:'1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture')
		 -> [0] '1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
		  > ['<inventory@1>']
 vehicle@3='1997,van,Ford,E350'
	 - Node: <start>		? (<vehicle@3>:'1997,van,Ford,E350')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<vehicle@3>:'1997,van,Ford,E350')
		 -> [0] '1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
		  > ['<vehicle@3>', '\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture']
 model@2='E350'
	 - Node: <start>		? (<model@2>:'E350')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<model@2>:'E350')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'E350')
		 -> [0] '1997,van,Ford,E350'
		  > ['1997,van,Ford,', '<model@2>']
 company@2='Ford'
	 - Node: <start>		? (<company@2>:'Ford')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<company@2>:'Ford')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Ford')
		 -> [0] '1997,van,Ford,'
		  > ['1997,van,', '<company@2>', ',']
 kind@2='van'
	 - Node: <start>		? (<kind@2>:'van')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<kind@2>:'van')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<kind@2>:'van')
		 -> [0] '1997,van,'
		  > ['1997,', '<kind@2>', ',']
 year@2='1997'
	 - Node: <start>		? (<year@2>:'1997')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<year@2>:'1997')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1997')
		 -> [0] '1997,'
		  > ['<year@2>', ',']
 vehicle@3='2000,car,Mercury,Cougar'
	 - Node: <start>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<vehicle@3>:'2000,car,Mercury,Cougar')
		 -> [0] 'E350'
		 -> [1] '\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture'
		  > ['\n', '<vehicle@3>', '\n1999,car,Chevy,Venture']
 model@2='Cougar'
	 - Node: <start>		? (<model@2>:'Cougar')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<model@2>:'Cougar')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Cougar')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<model@2>:'Cougar')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<model@2>:'Cougar')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<model@2>:'Cougar')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<model@2>:'Cougar')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Cougar')
		 -> [0] '2000,car,Mercury,Cougar'
		  > ['2000,car,Mercury,', '<model@2>']
 company@2='Mercury'
	 - Node: <start>		? (<company@2>:'Mercury')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<company@2>:'Mercury')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Mercury')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<company@2>:'Mercury')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<company@2>:'Mercury')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<company@2>:'Mercury')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<company@2>:'Mercury')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Mercury')
		 -> [0] '2000,car,Mercury,'
		  > ['2000,car,', '<company@2>', ',']
 kind@2='car'
	 - Node: <start>		? (<kind@2>:'car')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<kind@2>:'car')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<kind@2>:'car')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<kind@2>:'car')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<kind@2>:'car')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<kind@2>:'car')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<kind@2>:'car')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<kind@2>:'car')
		 -> [0] '2000,car,'
		  > ['2000,', '<kind@2>', ',']
 year@2='2000'
	 - Node: <start>		? (<year@2>:'2000')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<year@2>:'2000')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'2000')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<year@2>:'2000')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<year@2>:'2000')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<year@2>:'2000')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<year@2>:'2000')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'2000')
		 -> [0] '2000,'
		  > ['<year@2>', ',']
 vehicle@3='1999,car,Chevy,Venture'
	 - Node: <start>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<vehicle@3>:'1999,car,Chevy,Venture')
		 -> [0] 'Cougar'
		 -> [3] '\n1999,car,Chevy,Venture'
		  > ['\n', '<vehicle@3>']
 model@2='Venture'
	 - Node: <start>		? (<model@2>:'Venture')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<model@2>:'Venture')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<model@2>:'Venture')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<model@2>:'Venture')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<model@2>:'Venture')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<model@2>:'Venture')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Venture')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<model@2>:'Venture')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<model@2>:'Venture')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<model@2>:'Venture')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<model@2>:'Venture')
		 -> [0] 'Cougar'
		 -> [3] '\n'
		 -> [4] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<model@2>:'Venture')
		 -> [0] '1999,car,Chevy,Venture'
		  > ['1999,car,Chevy,', '<model@2>']
 company@2='Chevy'
	 - Node: <start>		? (<company@2>:'Chevy')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<company@2>:'Chevy')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Chevy')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<company@2>:'Chevy')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<company@2>:'Chevy')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<company@2>:'Chevy')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<company@2>:'Chevy')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Chevy')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<company@2>:'Chevy')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<company@2>:'Chevy')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<company@2>:'Chevy')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<company@2>:'Chevy')
		 -> [0] 'Cougar'
		 -> [3] '\n'
		 -> [4] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<company@2>:'Chevy')
		 -> [0] '1999,car,Chevy,'
		  > ['1999,car,', '<company@2>', ',']
 year@2='1999'
	 - Node: <start>		? (<year@2>:'1999')
		 -> [0] '<inventory@1>'
	 - Node: <inventory@1>		? (<year@2>:'1999')
		 -> [0] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1999')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<year@2>:'1999')
		 -> [0] '1997'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<year@2>:'1999')
		 -> [0] 'van'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<year@2>:'1999')
		 -> [0] 'Ford'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<year@2>:'1999')
		 -> [0] 'E350'
		 -> [1] '\n'
		 -> [2] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1999')
		 -> [0] '<year@2>'
	 - Node: <year@2>		? (<year@2>:'1999')
		 -> [0] '2000'
		 -> [1] ','
		 -> [2] '<kind@2>'
	 - Node: <kind@2>		? (<year@2>:'1999')
		 -> [0] 'car'
		 -> [3] ','
		 -> [4] '<company@2>'
	 - Node: <company@2>		? (<year@2>:'1999')
		 -> [0] 'Mercury'
		 -> [5] ','
		 -> [6] '<model@2>'
	 - Node: <model@2>		? (<year@2>:'1999')
		 -> [0] 'Cougar'
		 -> [3] '\n'
		 -> [4] '<vehicle@3>'
	 - Node: <vehicle@3>		? (<year@2>:'1999')
		 -> [0] '1999,car,'
		  > ['<year@2>', ',car,']

```

看最后一条语句。 我们有一个值`1999,car,`，其中仅`year`被替换了。 我们不再具有`'car'`变量，可以在此处继续替换。 发生这种情况是因为`'1999,car,Chevy,Venture'`中的`'car'`值未被视为新值，因为对于*不同的*方法调用（对于 `'2000,car,Mercury,Cougar'`）。

## 范围为[的语法矿工](#A-Grammar-Miner-with-Scope)

我们需要结合检查当前环境中的变量。 我们已经有很多方法调用，以便我们可以随时获取当前方法。 我们需要对变量做同样的事情。

为此，我们将`CallStack`扩展到一个新类`InputStack`，该类包含调用的方法以及观察到的参数。 它实质上是方法激活的记录。 我们从堆栈底部的原始输入开始，对于每个新的方法调用，我们将该调用的参数作为新记录推入堆栈。

### 输入堆栈

```py
class InputStack(CallStack):
    def __init__(self, i, fragment_len=FRAGMENT_LEN):
        self.inputs = [{START_SYMBOL: i}]
        self.fragment_len = fragment_len
        super().__init__()

```

为了检查是否保存了特定的变量，我们定义了`in_current_record()`，它仅检查当前作用域中的变量是否包含（而不是原始输入字符串）。

```py
class InputStack(InputStack):
    def in_current_record(self, val):
        return any(val in var for var in self.inputs[-1].values())

```

```py
my_istack = InputStack('hello my world')

```

```py
my_istack.in_current_record('hello')

```

```py
True

```

```py
my_istack.in_current_record('bye')

```

```py
False

```

```py
my_istack.inputs.append({'greeting': 'hello', 'location': 'world'})

```

```py
my_istack.in_current_record('hello')

```

```py
True

```

```py
my_istack.in_current_record('my')

```

```py
False

```

我们定义了`ignored()`方法，如果变量不是字符串，或者变量长度小于定义的`fragment_len`，则返回true。

```py
class InputStack(InputStack):
    def ignored(self, val):
        return not (isinstance(val, str) and len(val) >= self.fragment_len)

```

```py
my_istack = InputStack('hello world')
my_istack.ignored(1)

```

```py
True

```

```py
my_istack.ignored('a')

```

```py
True

```

```py
my_istack.ignored('help')

```

```py
False

```

现在，我们可以定义`in_scope()`方法，该方法检查是否需要忽略变量，以及是否不忽略变量，该变量值是否存在于当前作用域中。

```py
class InputStack(InputStack):
    def in_scope(self, k, val):
        if self.ignored(val):
            return False
        return self.in_current_record(val)

```

最后，我们更新`enter()`，它将当前上下文中的相关变量推入堆栈。

```py
class InputStack(InputStack):
    def enter(self, method, inputs):
        my_inputs = {k: v for k, v in inputs.items() if self.in_scope(k, v)}
        self.inputs.append(my_inputs)
        super().enter(method)

```

当方法返回时，我们还需要相应的`leave()`来弹出输入并展开堆栈。

```py
class InputStack(InputStack):
    def leave(self):
        self.inputs.pop()
        super().leave()

```

### ScopedVars

我们需要更新`AssignmentVars`，以包含有关定义变量的范围的信息。 我们从更新`method_init()`开始。

```py
class ScopedVars(AssignmentVars):
    def method_init(self):
        self.call_stack = self.create_call_stack(self.my_input)
        self.event_locations = {self.call_stack.method_id: []}

    def create_call_stack(self, i):
        return InputStack(i)

```

同样，`method_enter()`现在为当前​​方法调用初始化`accessed_seq_var`。

```py
class ScopedVars(ScopedVars):
    def method_enter(self, cxt, my_vars):
        self.current_event = 'call'
        self.call_stack.enter(cxt.method, my_vars)
        self.accessed_seq_var[self.call_stack.method_id] = {}
        self.event_locations[self.call_stack.method_id] = []
        self.register_event(cxt)
        self.update(my_vars)

```

`update()`方法现在保存定义该值的上下文。 对于函数的参数，上下文应为调用该函数的上下文。 另一方面，在语句执行期间定义的值将具有当前上下文。

此外，我们在值上注释而不是在键上注释，因为当下一行中的参数位于上下文中时，我们不想重复变量。 它们将具有相同的值，但上下文不同，因为它们存在于语句执行中。

```py
class ScopedVars(ScopedVars):
    def update(self, v):
        if self.current_event == 'call':
            context = -2
        elif self.current_event == 'line':
            context = -1
        else:
            context = -1
        for k, v in v.items():
            self._set_kv(k, (v, self.call_stack.at(context)))
        self.var_location_register(self.new_vars)
        self.new_vars = set()

```

我们还需要保存当前的方法调用，以确定哪些变量在范围内。 现在，此信息以`accessed_seq_var[method_id][var]`的形式包含在变量名称中。

```py
class ScopedVars(ScopedVars):
    def var_name(self, var):
        return (var, self.call_stack.method_id,
                self.accessed_seq_var[self.call_stack.method_id][var])

```

和以前一样，这次`var_access`只是在`method_id`的上下文中初始化相应的计数器。

```py
class ScopedVars(ScopedVars):
    def var_access(self, var):
        if var not in self.accessed_seq_var[self.call_stack.method_id]:
            self.accessed_seq_var[self.call_stack.method_id][var] = 0
        return self.var_name(var)

```

During a variable reassignment, we update the `accessed_seq_var` to reflect the new count.

```py
class ScopedVars(ScopedVars):
    def var_assign(self, var):
        self.accessed_seq_var[self.call_stack.method_id][var] += 1
        self.new_vars.add(self.var_name(var))
        return self.var_name(var)

```

现在，我们更新`defined_vars()`以说明新信息。

```py
class ScopedVars(ScopedVars):
    def defined_vars(self, formatted=True):
        def fmt(k):
            method, i = k[1]
            v = (method, i, k[0], self.var_def_lines[k])
            return "%s[%d]:%s@%s" % v if formatted else v

        return [(fmt(k), v) for k, v in self.defs.items()]

```

更新`seq_vars()`以考虑新信息。

```py
class ScopedVars(ScopedVars):
    def seq_vars(self, formatted=True):
        def fmt(k):
            method, i = k[1]
            v = (method, i, k[0], self.var_def_lines[k], k[2])
            return "%s[%d]:%s@%s:%s" % v if formatted else v

        return {fmt(k): v for k, v in self.defs.items()}

```

### 范围跟踪器

定义`InputStack`和`Vars`之后，我们现在可以定义`ScopeTracker`了。 `ScopeTracker`仅在当前作用域中存在该值时才保存变量。

```py
class ScopeTracker(AssignmentTracker):
    def __init__(self, my_input, trace, **kwargs):
        self.current_event = None
        super().__init__(my_input, trace, **kwargs)

    def create_assignments(self, *args):
        return ScopedVars(*args)

```

我们定义了一个包装器，用于检查作用域中是否存在变量。

```py
class ScopeTracker(ScopeTracker):
    def is_input_fragment(self, var, value):
        return self.my_assignments.call_stack.in_scope(var, value)

```

我们可以如下使用`ScopeTracker`

```py
vehicle_traces = []
with Tracer(INVENTORY) as tracer:
    process_inventory(tracer.my_input)
sm = ScopeTracker(tracer.my_input, tracer.trace)
vehicle_traces.append((tracer.my_input, sm))
for k, v in sm.my_assignments.seq_vars().items():
    print(k, '=', repr(v))

```

```py
process_inventory[1]:inventory@1:1 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('<start>', 0))
process_inventory[1]:inventory@1:2 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('process_inventory', 1))
process_inventory[1]:vehicle@3:1 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1:1 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1:2 = ('1997,van,Ford,E350', ('process_vehicle', 2))
process_vehicle[2]:model@2:1 = ('E350', ('process_vehicle', 2))
process_vehicle[2]:company@2:1 = ('Ford', ('process_vehicle', 2))
process_vehicle[2]:kind@2:1 = ('van', ('process_vehicle', 2))
process_vehicle[2]:year@2:1 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1:1 = ('E350', ('process_vehicle', 2))
process_van[3]:company@1:1 = ('Ford', ('process_vehicle', 2))
process_van[3]:year@1:1 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1:2 = ('E350', ('process_van', 3))
process_van[3]:company@1:2 = ('Ford', ('process_van', 3))
process_van[3]:year@1:2 = ('1997', ('process_van', 3))
process_inventory[1]:vehicle@3:2 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1:1 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1:2 = ('2000,car,Mercury,Cougar', ('process_vehicle', 4))
process_vehicle[4]:model@2:1 = ('Cougar', ('process_vehicle', 4))
process_vehicle[4]:company@2:1 = ('Mercury', ('process_vehicle', 4))
process_vehicle[4]:kind@2:1 = ('car', ('process_vehicle', 4))
process_vehicle[4]:year@2:1 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1:1 = ('Cougar', ('process_vehicle', 4))
process_car[5]:company@1:1 = ('Mercury', ('process_vehicle', 4))
process_car[5]:year@1:1 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1:2 = ('Cougar', ('process_car', 5))
process_car[5]:company@1:2 = ('Mercury', ('process_car', 5))
process_car[5]:year@1:2 = ('2000', ('process_car', 5))
process_inventory[1]:vehicle@3:3 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1:1 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1:2 = ('1999,car,Chevy,Venture', ('process_vehicle', 6))
process_vehicle[6]:model@2:1 = ('Venture', ('process_vehicle', 6))
process_vehicle[6]:company@2:1 = ('Chevy', ('process_vehicle', 6))
process_vehicle[6]:kind@2:1 = ('car', ('process_vehicle', 6))
process_vehicle[6]:year@2:1 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1:1 = ('Venture', ('process_vehicle', 6))
process_car[7]:company@1:1 = ('Chevy', ('process_vehicle', 6))
process_car[7]:year@1:1 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1:2 = ('Venture', ('process_car', 7))
process_car[7]:company@1:2 = ('Chevy', ('process_car', 7))
process_car[7]:year@1:2 = ('1999', ('process_car', 7))

```

### Recovering a Derivation Tree

`apply_new_definition()`的主要区别在于，我们添加了第二个条件来检查范围。 特别是，仅允许变量替换范围内的字符串片段部分。 可变范围由`scope`指示。 但是，仅考虑范围是不够的。 例如，考虑下面的片段。

```py
def my_fn(stringval):
    partA, partB = stringval.split('/')
    return partA, partB

svalue = ...
v1, v2 = my_fn(svalue)

```

在这里，`v1`和`v2`从上一个函数调用中获取它们的值。 不是从他们当前的背景出发。 也就是说，我们必须为内部子方法调用可能生成了如上所述的大片段的情况提供例外。 为了解决这个问题，我们定义了`mseq()`来检索方法调用序列。 在上述情况下，内部子方法调用的`mseq()`将大于当前的`mseq()`。 如果是这样，我们允许进行更换。

```py
class ScopeTreeMiner(TreeMiner):
    def mseq(self, key):
        method, seq, var, lno = key
        return seq

```

`nt_var()`方法需要提取元组并从中生成一个非终止符号。 我们跳过方法序列，因为它与语法无关。

```py
class ScopeTreeMiner(ScopeTreeMiner):
    def nt_var(self, key):
        method, seq, var, lno = key
        return to_nonterminal("%s@%d:%s" % (method, lno, var))

```

现在，我们重新定义`apply_new_definition()`以说明上下文和范围。 特别是，仅当变量在*范围*的范围内时才允许替换该值的一部分，也就是说，它是范围（如果是参数或参数，则为调用上下文的方法序列号） （如果是语句，则为当前上下文）与值的方法序列号的上下文相同。 当值的方法序列号大于变量的方法序列号时，将发生异常。 在这种情况下，该值可能来自内部调用。 在这种情况下，我们允许进行替换。

```py
class ScopeTreeMiner(ScopeTreeMiner):
    def insert_into_tree(self, my_tree, pair):
        var, values, my_scope = my_tree
        (nt_var, nt_seq), (v, v_scope) = pair
        applied = False
        for i, value_ in enumerate(values):
            key, arr, scope = value_
            self.log(2, "-> [%d] %s" % (i, repr(value_)))
            if is_nonterminal(key):
                applied = self.insert_into_tree(value_, pair)
                if applied:
                    break
            else:
                if v_scope != scope:
                    if nt_seq > scope:
                        continue
                if not v or v not in key:
                    continue
                prefix_k_suffix = [
                    (nt_var, [(v, [], nt_seq)],
                     scope) if i == 1 else (e, [], scope)
                    for i, e in enumerate(key.partition(v))
                    if e
                ]
                del values[i]
                for j, rep in enumerate(prefix_k_suffix):
                    values.insert(j + i, rep)

                applied = True
                self.log(2, " > %s" % (repr([i[0] for i in prefix_k_suffix])))
                break
        return applied

```

现在，将`apply_new_definition()`修改为携带其他上下文信息`mseq`。

```py
class ScopeTreeMiner(ScopeTreeMiner):
    def apply_new_definition(self, tree, var, value_):
        nt_var = self.nt_var(var)
        seq = self.mseq(var)
        val, (smethod, mseq) = value_
        return self.insert_into_tree(tree, ((nt_var, seq), (val, mseq)))

```

我们还修改了`get_derivation_tree()`，以便初始节点携带上下文。

```py
class ScopeTreeMiner(ScopeTreeMiner):
    def get_derivation_tree(self):
        tree = (START_SYMBOL, [(self.my_input, [], 0)], 0)
        for var, value in self.my_assignments:
            self.log(0, "%s=%s" % (var, repr(value)))
            self.apply_new_definition(tree, var, value)
        return tree

```

#### 示例1：恢复URL解析树

我们验证我们的URL解析树恢复是否仍能按预期进行。

```py
url_dts = []
for inputstr in URLS_X:
    clear_cache()
    with Tracer(inputstr, files=['urllib/parse.py']) as tracer:
        urlparse(tracer.my_input)
    sm = ScopeTracker(tracer.my_input, tracer.trace)
    for k, v in sm.my_assignments.defined_vars(formatted=False):
        print(k, '=', repr(v))
    dt = ScopeTreeMiner(
        tracer.my_input,
        sm.my_assignments.defined_vars(
            formatted=False))
    display_tree(dt.tree, graph_attr=lr_graph)
    url_dts.append(dt)

```

```py
('urlparse', 1, 'url', 361) = ('http://user:pass@www.google.com:80/?q=path#ref', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 412) = ('http', ('urlsplit', 3))
('urlsplit', 3, 'url', 413) = ('//user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//user:pass@www.google.com:80/?q=path#ref', ('_splitnetloc', 5))
('urlsplit', 3, 'url', 415) = ('/?q=path#ref', ('urlsplit', 3))
('urlsplit', 3, 'netloc', 415) = ('user:pass@www.google.com:80', ('urlsplit', 3))
('urlsplit', 3, 'url', 420) = ('/?q=path', ('urlsplit', 3))
('urlsplit', 3, 'fragment', 420) = ('ref', ('urlsplit', 3))
('urlsplit', 3, 'query', 422) = ('q=path', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('http', ('urlparse', 1))
('urlparse', 1, 'fragment', 369) = ('ref', ('urlparse', 1))
('urlparse', 1, 'query', 369) = ('q=path', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('user:pass@www.google.com:80', ('urlparse', 1))
('urlparse', 1, 'url', 361) = ('https://www.cispa.saarland:80/', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('https://www.cispa.saarland:80/', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('https://www.cispa.saarland:80/', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('https://www.cispa.saarland:80/', ('urlsplit', 3))
('urlsplit', 3, 'rest', 432) = ('//www.cispa.saarland:80/', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 435) = ('https', ('urlsplit', 3))
('urlsplit', 3, 'url', 435) = ('//www.cispa.saarland:80/', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//www.cispa.saarland:80/', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//www.cispa.saarland:80/', ('_splitnetloc', 7))
('urlsplit', 3, 'netloc', 438) = ('www.cispa.saarland:80', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('https', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('www.cispa.saarland:80', ('urlparse', 1))
('urlparse', 1, 'url', 361) = ('http://www.fuzzingbook.org/#News', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('http://www.fuzzingbook.org/#News', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://www.fuzzingbook.org/#News', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('http://www.fuzzingbook.org/#News', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 412) = ('http', ('urlsplit', 3))
('urlsplit', 3, 'url', 413) = ('//www.fuzzingbook.org/#News', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//www.fuzzingbook.org/#News', ('urlsplit', 3))
('_splitnetloc', 5, 'url', 386) = ('//www.fuzzingbook.org/#News', ('_splitnetloc', 5))
('urlsplit', 3, 'url', 415) = ('/#News', ('urlsplit', 3))
('urlsplit', 3, 'netloc', 415) = ('www.fuzzingbook.org', ('urlsplit', 3))
('urlsplit', 3, 'fragment', 420) = ('News', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('http', ('urlparse', 1))
('urlparse', 1, 'fragment', 369) = ('News', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('www.fuzzingbook.org', ('urlparse', 1))
('urlparse', 1, 'url', 361) = ('ftp://freebsd.org/releases/5.8', ('<start>', 0))
('urlparse', 1, 'url', 361) = ('ftp://freebsd.org/releases/5.8', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('ftp://freebsd.org/releases/5.8', ('urlparse', 1))
('urlsplit', 3, 'url', 394) = ('ftp://freebsd.org/releases/5.8', ('urlsplit', 3))
('urlsplit', 3, 'rest', 432) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))
('urlsplit', 3, 'scheme', 435) = ('ftp', ('urlsplit', 3))
('urlsplit', 3, 'url', 435) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))
('_splitnetloc', 7, 'url', 386) = ('//freebsd.org/releases/5.8', ('_splitnetloc', 7))
('urlsplit', 3, 'url', 438) = ('/releases/5.8', ('urlsplit', 3))
('urlsplit', 3, 'netloc', 438) = ('freebsd.org', ('urlsplit', 3))
('urlparse', 1, 'scheme', 369) = ('ftp', ('urlparse', 1))
('urlparse', 1, 'url', 369) = ('/releases/5.8', ('urlparse', 1))
('urlparse', 1, 'netloc', 369) = ('freebsd.org', ('urlparse', 1))

```

#### 示例2：恢复库存分析树

接下来，我们来看从`process_inventory()`恢复上次失败的解析树。

```py
with Tracer(INVENTORY) as tracer:
    process_inventory(tracer.my_input)

sm = ScopeTracker(tracer.my_input, tracer.trace)
for k, v in sm.my_assignments.defined_vars():
    print(k, '=', repr(v))
inventory_dt = ScopeTreeMiner(
    tracer.my_input,
    sm.my_assignments.defined_vars(
        formatted=False))
display_tree(inventory_dt.tree, graph_attr=lr_graph)

```

```py
process_inventory[1]:inventory@1 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('<start>', 0))
process_inventory[1]:inventory@1 = ('1997,van,Ford,E350\n2000,car,Mercury,Cougar\n1999,car,Chevy,Venture', ('process_inventory', 1))
process_inventory[1]:vehicle@3 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1 = ('1997,van,Ford,E350', ('process_inventory', 1))
process_vehicle[2]:vehicle@1 = ('1997,van,Ford,E350', ('process_vehicle', 2))
process_vehicle[2]:model@2 = ('E350', ('process_vehicle', 2))
process_vehicle[2]:company@2 = ('Ford', ('process_vehicle', 2))
process_vehicle[2]:kind@2 = ('van', ('process_vehicle', 2))
process_vehicle[2]:year@2 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1 = ('E350', ('process_vehicle', 2))
process_van[3]:company@1 = ('Ford', ('process_vehicle', 2))
process_van[3]:year@1 = ('1997', ('process_vehicle', 2))
process_van[3]:model@1 = ('E350', ('process_van', 3))
process_van[3]:company@1 = ('Ford', ('process_van', 3))
process_van[3]:year@1 = ('1997', ('process_van', 3))
process_inventory[1]:vehicle@3 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))
process_vehicle[4]:vehicle@1 = ('2000,car,Mercury,Cougar', ('process_vehicle', 4))
process_vehicle[4]:model@2 = ('Cougar', ('process_vehicle', 4))
process_vehicle[4]:company@2 = ('Mercury', ('process_vehicle', 4))
process_vehicle[4]:kind@2 = ('car', ('process_vehicle', 4))
process_vehicle[4]:year@2 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1 = ('Cougar', ('process_vehicle', 4))
process_car[5]:company@1 = ('Mercury', ('process_vehicle', 4))
process_car[5]:year@1 = ('2000', ('process_vehicle', 4))
process_car[5]:model@1 = ('Cougar', ('process_car', 5))
process_car[5]:company@1 = ('Mercury', ('process_car', 5))
process_car[5]:year@1 = ('2000', ('process_car', 5))
process_inventory[1]:vehicle@3 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1 = ('1999,car,Chevy,Venture', ('process_inventory', 1))
process_vehicle[6]:vehicle@1 = ('1999,car,Chevy,Venture', ('process_vehicle', 6))
process_vehicle[6]:model@2 = ('Venture', ('process_vehicle', 6))
process_vehicle[6]:company@2 = ('Chevy', ('process_vehicle', 6))
process_vehicle[6]:kind@2 = ('car', ('process_vehicle', 6))
process_vehicle[6]:year@2 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1 = ('Venture', ('process_vehicle', 6))
process_car[7]:company@1 = ('Chevy', ('process_vehicle', 6))
process_car[7]:year@1 = ('1999', ('process_vehicle', 6))
process_car[7]:model@1 = ('Venture', ('process_car', 7))
process_car[7]:company@1 = ('Chevy', ('process_car', 7))
process_car[7]:year@1 = ('1999', ('process_car', 7))

```

<svg height="683pt" viewBox="0.00 0.00 1839.00 683.00" width="1839pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g class="graph" id="graph0" transform="scale(1 1) rotate(0) translate(4 679)"><title>%3</title> <g class="node" id="node1"><title>0</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="20" y="-333.8"><start></text></g> <g class="node" id="node2"><title>1</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="174" y="-333.8"><process_inventory@1:inventory></text></g> <g class="edge" id="edge1"><title>0->1</title></g> <g class="node" id="node3"><title>2</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="406" y="-333.8"><process_inventory@1:inventory></text></g> <g class="edge" id="edge2"><title>1->2</title></g> <g class="node" id="node4"><title>3</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="631.5" y="-458.8"><process_inventory@3:vehicle></text></g> <g class="edge" id="edge3"><title>2->3</title></g> <g class="node" id="node24"><title>23</title></g> <g class="edge" id="edge23"><title>2->23</title></g> <g class="node" id="node25"><title>24</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="631.5" y="-333.8"><process_inventory@3:vehicle></text></g> <g class="edge" id="edge24"><title>2->24</title></g> <g class="node" id="node45"><title>44</title></g> <g class="edge" id="edge44"><title>2->44</title></g> <g class="node" id="node46"><title>45</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="631.5" y="-245.8"><process_inventory@3:vehicle></text></g> <g class="edge" id="edge45"><title>2->45</title></g> <g class="node" id="node5"><title>4</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="844" y="-494.8"><process_vehicle@1:vehicle></text></g> <g class="edge" id="edge4"><title>3->4</title></g> <g class="node" id="node6"><title>5</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1050" y="-547.8"><process_vehicle@1:vehicle></text></g> <g class="edge" id="edge5"><title>4->5</title></g> <g class="node" id="node7"><title>6</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-663.8"><process_vehicle@2:year></text></g> <g class="edge" id="edge6"><title>5->6</title></g> <g class="node" id="node11"><title>10</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-630.8">,</text></g> <g class="edge" id="edge10"><title>5->10</title></g> <g class="node" id="node12"><title>11</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-597.8"><process_vehicle@2:kind></text></g> <g class="edge" id="edge11"><title>5->11</title></g> <g class="node" id="node14"><title>13</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-564.8">,</text></g> <g class="edge" id="edge13"><title>5->13</title></g> <g class="node" id="node15"><title>14</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-531.8"><process_vehicle@2:company></text></g> <g class="edge" id="edge14"><title>5->14</title></g> <g class="node" id="node19"><title>18</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-498.8">,</text></g> <g class="edge" id="edge18"><title>5->18</title></g> <g class="node" id="node20"><title>19</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-465.8"><process_vehicle@2:model></text></g> <g class="edge" id="edge19"><title>5->19</title></g> <g class="node" id="node8"><title>7</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-663.8"><process_van@1:year></text></g> <g class="edge" id="edge7"><title>6->7</title></g> <g class="node" id="node9"><title>8</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-663.8"><process_van@1:year></text></g> <g class="edge" id="edge8"><title>7->8</title></g> <g class="node" id="node10"><title>9</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-663.8">1997</text></g> <g class="edge" id="edge9"><title>8->9</title></g> <g class="node" id="node13"><title>12</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-597.8">van</text></g> <g class="edge" id="edge12"><title>11->12</title></g> <g class="node" id="node16"><title>15</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-531.8"><process_van@1:company></text></g> <g class="edge" id="edge15"><title>14->15</title></g> <g class="node" id="node17"><title>16</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-531.8"><process_van@1:company></text></g> <g class="edge" id="edge16"><title>15->16</title></g> <g class="node" id="node18"><title>17</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-531.8">Ford</text></g> <g class="edge" id="edge17"><title>16->17</title></g> <g class="node" id="node21"><title>20</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-465.8"><process_van@1:model></text></g> <g class="edge" id="edge20"><title>19->20</title></g> <g class="node" id="node22"><title>21</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-465.8"><process_van@1:model></text></g> <g class="edge" id="edge21"><title>20->21</title></g> <g class="node" id="node23"><title>22</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-465.8">E350</text></g> <g class="edge" id="edge22"><title>21->22</title></g> <g class="node" id="node26"><title>25</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="844" y="-333.8"><process_vehicle@1:vehicle></text></g> <g class="edge" id="edge25"><title>24->25</title></g> <g class="node" id="node27"><title>26</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1050" y="-333.8"><process_vehicle@1:vehicle></text></g> <g class="edge" id="edge26"><title>25->26</title></g> <g class="node" id="node28"><title>27</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-432.8"><process_vehicle@2:year></text></g> <g class="edge" id="edge27"><title>26->27</title></g> <g class="node" id="node32"><title>31</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-399.8">,</text></g> <g class="edge" id="edge31"><title>26->31</title></g> <g class="node" id="node33"><title>32</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-366.8"><process_vehicle@2:kind></text></g> <g class="edge" id="edge32"><title>26->32</title></g> <g class="node" id="node35"><title>34</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-333.8">,</text></g> <g class="edge" id="edge34"><title>26->34</title></g> <g class="node" id="node36"><title>35</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-300.8"><process_vehicle@2:company></text></g> <g class="edge" id="edge35"><title>26->35</title></g> <g class="node" id="node40"><title>39</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-267.8">,</text></g> <g class="edge" id="edge39"><title>26->39</title></g> <g class="node" id="node41"><title>40</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-234.8"><process_vehicle@2:model></text></g> <g class="edge" id="edge40"><title>26->40</title></g> <g class="node" id="node29"><title>28</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-432.8"><process_car@1:year></text></g> <g class="edge" id="edge28"><title>27->28</title></g> <g class="node" id="node30"><title>29</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-432.8"><process_car@1:year></text></g> <g class="edge" id="edge29"><title>28->29</title></g> <g class="node" id="node31"><title>30</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-432.8">2000</text></g> <g class="edge" id="edge30"><title>29->30</title></g> <g class="node" id="node34"><title>33</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-366.8">car</text></g> <g class="edge" id="edge33"><title>32->33</title></g> <g class="node" id="node37"><title>36</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-300.8"><process_car@1:company></text></g> <g class="edge" id="edge36"><title>35->36</title></g> <g class="node" id="node38"><title>37</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-300.8"><process_car@1:company></text></g> <g class="edge" id="edge37"><title>36->37</title></g> <g class="node" id="node39"><title>38</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-300.8">Mercury</text></g> <g class="edge" id="edge38"><title>37->38</title></g> <g class="node" id="node42"><title>41</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-234.8"><process_car@1:model></text></g> <g class="edge" id="edge41"><title>40->41</title></g> <g class="node" id="node43"><title>42</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-234.8"><process_car@1:model></text></g> <g class="edge" id="edge42"><title>41->42</title></g> <g class="node" id="node44"><title>43</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-234.8">Cougar</text></g> <g class="edge" id="edge43"><title>42->43</title></g> <g class="node" id="node47"><title>46</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="844" y="-172.8"><process_vehicle@1:vehicle></text></g> <g class="edge" id="edge46"><title>45->46</title></g> <g class="node" id="node48"><title>47</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1050" y="-119.8"><process_vehicle@1:vehicle></text></g> <g class="edge" id="edge47"><title>46->47</title></g> <g class="node" id="node49"><title>48</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-201.8"><process_vehicle@2:year></text></g> <g class="edge" id="edge48"><title>47->48</title></g> <g class="node" id="node53"><title>52</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-168.8">,</text></g> <g class="edge" id="edge52"><title>47->52</title></g> <g class="node" id="node54"><title>53</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-135.8"><process_vehicle@2:kind></text></g> <g class="edge" id="edge53"><title>47->53</title></g> <g class="node" id="node56"><title>55</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-102.8">,</text></g> <g class="edge" id="edge55"><title>47->55</title></g> <g class="node" id="node57"><title>56</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-69.8"><process_vehicle@2:company></text></g> <g class="edge" id="edge56"><title>47->56</title></g> <g class="node" id="node61"><title>60</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-36.8">,</text></g> <g class="edge" id="edge60"><title>47->60</title></g> <g class="node" id="node62"><title>61</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1261.5" y="-3.8"><process_vehicle@2:model></text></g> <g class="edge" id="edge61"><title>47->61</title></g> <g class="node" id="node50"><title>49</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-201.8"><process_car@1:year></text></g> <g class="edge" id="edge49"><title>48->49</title></g> <g class="node" id="node51"><title>50</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-201.8"><process_car@1:year></text></g> <g class="edge" id="edge50"><title>49->50</title></g> <g class="node" id="node52"><title>51</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-201.8">1999</text></g> <g class="edge" id="edge51"><title>50->51</title></g> <g class="node" id="node55"><title>54</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-135.8">car</text></g> <g class="edge" id="edge54"><title>53->54</title></g> <g class="node" id="node58"><title>57</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-69.8"><process_car@1:company></text></g> <g class="edge" id="edge57"><title>56->57</title></g> <g class="node" id="node59"><title>58</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-69.8"><process_car@1:company></text></g> <g class="edge" id="edge58"><title>57->58</title></g> <g class="node" id="node60"><title>59</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-69.8">Chevy</text></g> <g class="edge" id="edge59"><title>58->59</title></g> <g class="node" id="node63"><title>62</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1468.5" y="-3.8"><process_car@1:model></text></g> <g class="edge" id="edge62"><title>61->62</title></g> <g class="node" id="node64"><title>63</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1665.5" y="-3.8"><process_car@1:model></text></g> <g class="edge" id="edge63"><title>62->63</title></g> <g class="node" id="node65"><title>64</title> <text fill="#000000" font-family="Times,serif" font-size="14.00" text-anchor="middle" x="1806.5" y="-3.8">Venture</text></g> <g class="edge" id="edge64"><title>63->64</title></g></g></svg>

恢复的解析树似乎是合理的。

从示例（2）中可能会注意到的一件事是`vehicle[2:1]`，`vehicle[4:1]`和`vehicle[6:1]`这三个子树非常相似。 接下来，我们将研究如何利用它直接生成语法。

### 语法挖掘

现在将`tree_to_grammar()`重新定义如下，以解决节点中的额外作用域。

```py
class ScopedGrammarMiner(GrammarMiner):
    def tree_to_grammar(self, tree):
        key, children, scope = tree
        one_alt = [ckey for ckey, gchildren, cscope in children if ckey != key]
        hsh = {key: [one_alt] if one_alt else []}
        for child in children:
            (ckey, _gc, _cscope) = child
            if not is_nonterminal(ckey):
                continue
            chsh = self.tree_to_grammar(child)
            for k in chsh:
                if k not in hsh:
                    hsh[k] = chsh[k]
                else:
                    hsh[k].extend(chsh[k])
        return hsh

```

语法为规范形式，需要按摩才能显示。 首先，为清单恢复语法。

```py
si = ScopedGrammarMiner()
si.add_tree(inventory_dt)
syntax_diagram(readable(si.grammar))

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 386.5 62" width="386.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="193.25" y="35">process_inventory@1:inventory</text></g></g></g></g></svg>

```py
process_inventory@1:inventory

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 1005.5 62" width="1005.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="184.75" y="35">process_inventory@3:vehicle</text></g><g class="non-terminal"><text x="502.75" y="35">process_inventory@3:vehicle</text></g><g class="non-terminal"><text x="820.75" y="35">process_inventory@3:vehicle</text></g></g></g></g></svg>

```py
process_inventory@3:vehicle

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 352.5 62" width="352.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="176.25" y="35">process_vehicle@1:vehicle</text></g></g></g></g></svg>

```py
process_vehicle@1:vehicle

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 1187.5 62" width="1187.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="163.5" y="35">process_vehicle@2:year</text></g> <g class="terminal"><text x="301.25" y="35">,</text></g> <g class="non-terminal"><text x="439.0" y="35">process_vehicle@2:kind</text></g> <g class="terminal"><text x="576.75" y="35">,</text></g> <g class="non-terminal"><text x="727.25" y="35">process_vehicle@2:company</text></g> <g class="terminal"><text x="877.75" y="35">,</text></g> <g class="non-terminal"><text x="1019.75" y="35">process_vehicle@2:model</text></g></g></g></g></svg>

```py
process_vehicle@2:year

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 293.0 92" width="293.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="146.5" y="35">process_car@1:year</text></g></g> <g><g class="non-terminal"><text x="146.5" y="65">process_van@1:year</text></g></g></g></g></svg>

```py
process_van@1:year

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">1997</text></g></g></g></g></svg>

```py
process_vehicle@2:kind

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 165.5 92" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="82.75" y="35">van</text></g></g> <g><g class="terminal"><text x="82.75" y="65">car</text></g></g></g></g></svg>

```py
process_vehicle@2:company

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 318.5 92" width="318.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="159.25" y="35">process_van@1:company</text></g></g> <g><g class="non-terminal"><text x="159.25" y="65">process_car@1:company</text></g></g></g></g></svg>

```py
process_van@1:company

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">Ford</text></g></g></g></g></svg>

```py
process_vehicle@2:model

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 301.5 92" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">process_van@1:model</text></g></g> <g><g class="non-terminal"><text x="150.75" y="65">process_car@1:model</text></g></g></g></g></svg>

```py
process_van@1:model

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">E350</text></g></g></g></g></svg>

```py
process_car@1:year

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">2000</text></g></g> <g><g class="terminal"><text x="87.0" y="65">1999</text></g></g></g></g></svg>

```py
process_car@1:company

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 199.5 92" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Chevy</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Mercury</text></g></g></g></g></svg>

```py
process_car@1:model

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 199.5 92" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Cougar</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Venture</text></g></g></g></g></svg>

URL的恢复语法。

```py
su = ScopedGrammarMiner()
for url_dt in url_dts:
    su.add_tree(url_dt)
syntax_diagram(readable(su.grammar))

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 276.0 62" width="276.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="138.0" y="35">urlparse@361:url</text></g></g></g></g></svg>

```py
urlparse@361:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 276.0 62" width="276.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="138.0" y="35">urlsplit@394:url</text></g></g></g></g></svg>

```py
urlsplit@394:url

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 534.5 92" width="534.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">urlsplit@435:scheme</text></g> <g class="terminal"><text x="275.75" y="35">:</text></g> <g class="non-terminal"><text x="392.25" y="35">urlsplit@432:rest</text></g></g> <g><g class="non-terminal"><text x="155.0" y="65">urlsplit@412:scheme</text></g> <g class="terminal"><text x="280.0" y="65">:</text></g> <g class="non-terminal"><text x="392.25" y="65">urlsplit@413:url</text></g></g></g></g></svg>

```py
urlsplit@412:scheme

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 301.5 62" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">urlparse@369:scheme</text></g></g></g></g></svg>

```py
urlparse@369:scheme

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 182.5 122" width="182.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="91.25" y="35">ftp</text></g></g> <g><g class="terminal"><text x="91.25" y="65">https</text></g></g> <g><g class="terminal"><text x="91.25" y="95">http</text></g></g></g></g></svg>

```py
urlsplit@413:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 310.0 62" width="310.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="155.0" y="35">_splitnetloc@386:url</text></g></g></g></g></svg>

```py
_splitnetloc@386:url

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 534.5 122" width="534.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="142.25" y="35">//</text></g> <g class="non-terminal"><text x="271.5" y="35">urlsplit@438:netloc</text></g> <g class="terminal"><text x="396.5" y="35">/</text></g></g> <g><g class="terminal"><text x="78.5" y="65">//</text></g> <g class="non-terminal"><text x="207.75" y="65">urlsplit@438:netloc</text></g> <g class="non-terminal"><text x="396.5" y="65">urlsplit@438:url</text></g></g> <g><g class="terminal"><text x="78.5" y="95">//</text></g> <g class="non-terminal"><text x="207.75" y="95">urlsplit@415:netloc</text></g> <g class="non-terminal"><text x="396.5" y="95">urlsplit@415:url</text></g></g></g></g></svg>

```py
urlsplit@415:netloc

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 301.5 62" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">urlparse@369:netloc</text></g></g></g></g></svg>

```py
urlparse@369:netloc

```

 <svg class="railroad-diagram" height="152" viewBox="0 0 369.5 152" width="369.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="184.75" y="65">www.cispa.saarland:80</text></g></g> <g><g class="terminal"><text x="184.75" y="35">freebsd.org</text></g></g> <g><g class="terminal"><text x="184.75" y="95">user:pass@www.google.com:80</text></g></g> <g><g class="terminal"><text x="184.75" y="125">www.fuzzingbook.org</text></g></g></g></g></svg>

```py
urlsplit@415:url

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 543.0 92" width="543.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="162.25" y="35">/#</text></g> <g class="non-terminal"><text x="300.0" y="35">urlsplit@420:fragment</text></g></g> <g><g class="non-terminal"><text x="138.0" y="65">urlsplit@420:url</text></g> <g class="terminal"><text x="250.25" y="65">#</text></g> <g class="non-terminal"><text x="383.75" y="65">urlsplit@420:fragment</text></g></g></g></g></svg>

```py
urlsplit@420:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 350.0 62" width="350.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">/?</text></g> <g class="non-terminal"><text x="203.5" y="35">urlsplit@422:query</text></g></g></g></g></svg>

```py
urlsplit@422:query

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 293.0 62" width="293.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="146.5" y="35">urlparse@369:query</text></g></g></g></g></svg>

```py
urlparse@369:query

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 191.0 62" width="191.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="95.5" y="35">q=path</text></g></g></g></g></svg>

```py
urlsplit@420:fragment

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 318.5 62" width="318.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="159.25" y="35">urlparse@369:fragment</text></g></g></g></g></svg>

```py
urlparse@369:fragment

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">News</text></g></g> <g><g class="terminal"><text x="87.0" y="65">ref</text></g></g></g></g></svg>

```py
urlsplit@435:scheme

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 301.5 62" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">urlparse@369:scheme</text></g></g></g></g></svg>

```py
urlsplit@432:rest

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 276.0 62" width="276.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="138.0" y="35">urlsplit@435:url</text></g></g></g></g></svg>

```py
urlsplit@435:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 310.0 62" width="310.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="155.0" y="35">_splitnetloc@386:url</text></g></g></g></g></svg>

```py
urlsplit@438:netloc

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 301.5 62" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">urlparse@369:netloc</text></g></g></g></g></svg>

```py
urlsplit@438:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 276.0 62" width="276.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="138.0" y="35">urlparse@369:url</text></g></g></g></g></svg>

```py
urlparse@369:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 250.5 62" width="250.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="125.25" y="35">/releases/5.8</text></g></g></g></g></svg>

可能会注意到，语法不是完全人类可读的，具有许多单个标记定义。

因此，难题的最后一部分是清理方法`clean_grammar()`，它可以清理此类定义。 想法是寻找单个令牌定义，以便一个密钥由另一个密钥（单个替代，单个令牌，非终结符）准确地定义。

```py
class ScopedGrammarMiner(ScopedGrammarMiner):
    def get_replacements(self, grammar):
        replacements = {}
        for k in grammar:
            if k == START_SYMBOL:
                continue
            alts = grammar[k]
            if len(set([str(i) for i in alts])) != 1:
                continue
            rule = alts[0]
            if len(rule) != 1:
                continue
            tok = rule[0]
            if not is_nonterminal(tok):
                continue
            replacements[k] = tok
        return replacements

```

一旦有了这样的列表，就用我们先前找到的令牌迭代地替换原始密钥。 重复进行，直到没有剩余。

```py
class ScopedGrammarMiner(ScopedGrammarMiner):
    def clean_grammar(self):
        replacements = self.get_replacements(self.grammar)

        while True:
            changed = set()
            for k in self.grammar:
                if k in replacements:
                    continue
                new_alts = []
                for alt in self.grammar[k]:
                    new_alt = []
                    for t in alt:
                        if t in replacements:
                            new_alt.append(replacements[t])
                            changed.add(t)
                        else:
                            new_alt.append(t)
                    new_alts.append(new_alt)
                self.grammar[k] = new_alts
            if not changed:
                break
            for k in changed:
                self.grammar.pop(k, None)
        return readable(self.grammar)

```

`clean_grammar()`的用法如下：

```py
si = ScopedGrammarMiner()
si.add_tree(inventory_dt)
syntax_diagram(readable(si.clean_grammar()))

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 386.5 62" width="386.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="193.25" y="35">process_inventory@1:inventory</text></g></g></g></g></svg>

```py
process_inventory@1:inventory

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 954.5 62" width="954.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="176.25" y="35">process_vehicle@1:vehicle</text></g><g class="non-terminal"><text x="477.25" y="35">process_vehicle@1:vehicle</text></g><g class="non-terminal"><text x="778.25" y="35">process_vehicle@1:vehicle</text></g></g></g></g></svg>

```py
process_vehicle@1:vehicle

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 1187.5 62" width="1187.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="163.5" y="35">process_vehicle@2:year</text></g> <g class="terminal"><text x="301.25" y="35">,</text></g> <g class="non-terminal"><text x="439.0" y="35">process_vehicle@2:kind</text></g> <g class="terminal"><text x="576.75" y="35">,</text></g> <g class="non-terminal"><text x="727.25" y="35">process_vehicle@2:company</text></g> <g class="terminal"><text x="877.75" y="35">,</text></g> <g class="non-terminal"><text x="1019.75" y="35">process_vehicle@2:model</text></g></g></g></g></svg>

```py
process_vehicle@2:year

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 293.0 92" width="293.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="146.5" y="35">process_car@1:year</text></g></g> <g><g class="non-terminal"><text x="146.5" y="65">process_van@1:year</text></g></g></g></g></svg>

```py
process_van@1:year

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">1997</text></g></g></g></g></svg>

```py
process_vehicle@2:kind

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 165.5 92" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="82.75" y="35">van</text></g></g> <g><g class="terminal"><text x="82.75" y="65">car</text></g></g></g></g></svg>

```py
process_vehicle@2:company

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 318.5 92" width="318.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="159.25" y="35">process_van@1:company</text></g></g> <g><g class="non-terminal"><text x="159.25" y="65">process_car@1:company</text></g></g></g></g></svg>

```py
process_van@1:company

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">Ford</text></g></g></g></g></svg>

```py
process_vehicle@2:model

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 301.5 92" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">process_van@1:model</text></g></g> <g><g class="non-terminal"><text x="150.75" y="65">process_car@1:model</text></g></g></g></g></svg>

```py
process_van@1:model

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">E350</text></g></g></g></g></svg>

```py
process_car@1:year

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">2000</text></g></g> <g><g class="terminal"><text x="87.0" y="65">1999</text></g></g></g></g></svg>

```py
process_car@1:company

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 199.5 92" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Chevy</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Mercury</text></g></g></g></g></svg>

```py
process_car@1:model

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 199.5 92" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Cougar</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Venture</text></g></g></g></g></svg>

我们更新了`update_grammar()`以使用正确的跟踪器和矿机。

```py
class ScopedGrammarMiner(ScopedGrammarMiner):
    def update_grammar(self, inputstr, trace):
        at = self.create_tracker(inputstr, trace)
        dt = self.create_tree_miner(
            inputstr, at.my_assignments.defined_vars(
                formatted=False))
        self.add_tree(dt)
        return self.grammar

    def create_tracker(self, *args):
        return ScopeTracker(*args)

    def create_tree_miner(self, *args):
        return ScopeTreeMiner(*args)

```

`recover_grammar()`使用正确的矿工，并返回已清除的语法。

```py
def recover_grammar(fn, inputs, **kwargs):
    miner = ScopedGrammarMiner()
    for inputstr in inputs:
        with Tracer(inputstr, **kwargs) as tracer:
            fn(tracer.my_input)
        miner.update_grammar(tracer.my_input, tracer.trace)
    return readable(miner.clean_grammar())

```

```py
url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])

```

```py
syntax_diagram(url_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 276.0 62" width="276.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="138.0" y="35">urlsplit@394:url</text></g></g></g></g></svg>

```py
urlsplit@394:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 560.0 62" width="560.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">urlparse@369:scheme</text></g> <g class="terminal"><text x="275.75" y="35">:</text></g> <g class="non-terminal"><text x="405.0" y="35">_splitnetloc@386:url</text></g></g></g></g></svg>

```py
urlparse@369:scheme

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 182.5 122" width="182.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="91.25" y="35">ftp</text></g></g> <g><g class="terminal"><text x="91.25" y="65">https</text></g></g> <g><g class="terminal"><text x="91.25" y="95">http</text></g></g></g></g></svg>

```py
_splitnetloc@386:url

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 534.5 122" width="534.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">//</text></g> <g class="non-terminal"><text x="207.75" y="35">urlparse@369:netloc</text></g> <g class="non-terminal"><text x="396.5" y="35">urlsplit@415:url</text></g></g> <g><g class="terminal"><text x="78.5" y="65">//</text></g> <g class="non-terminal"><text x="207.75" y="65">urlparse@369:netloc</text></g> <g class="non-terminal"><text x="396.5" y="65">urlparse@369:url</text></g></g> <g><g class="terminal"><text x="142.25" y="95">//</text></g> <g class="non-terminal"><text x="271.5" y="95">urlparse@369:netloc</text></g> <g class="terminal"><text x="396.5" y="95">/</text></g></g></g></g></svg>

```py
urlparse@369:netloc

```

 <svg class="railroad-diagram" height="152" viewBox="0 0 369.5 152" width="369.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="184.75" y="65">www.cispa.saarland:80</text></g></g> <g><g class="terminal"><text x="184.75" y="35">freebsd.org</text></g></g> <g><g class="terminal"><text x="184.75" y="95">user:pass@www.google.com:80</text></g></g> <g><g class="terminal"><text x="184.75" y="125">www.fuzzingbook.org</text></g></g></g></g></svg>

```py
urlsplit@415:url

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 543.0 92" width="543.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="162.25" y="35">/#</text></g> <g class="non-terminal"><text x="300.0" y="35">urlparse@369:fragment</text></g></g> <g><g class="non-terminal"><text x="138.0" y="65">urlsplit@420:url</text></g> <g class="terminal"><text x="250.25" y="65">#</text></g> <g class="non-terminal"><text x="383.75" y="65">urlparse@369:fragment</text></g></g></g></g></svg>

```py
urlsplit@420:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 350.0 62" width="350.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="78.5" y="35">/?</text></g> <g class="non-terminal"><text x="203.5" y="35">urlparse@369:query</text></g></g></g></g></svg>

```py
urlparse@369:query

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 191.0 62" width="191.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="95.5" y="35">q=path</text></g></g></g></g></svg>

```py
urlparse@369:fragment

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">News</text></g></g> <g><g class="terminal"><text x="87.0" y="65">ref</text></g></g></g></g></svg>

```py
urlparse@369:url

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 250.5 62" width="250.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="125.25" y="35">/releases/5.8</text></g></g></g></g></svg>

```py
f = GrammarFuzzer(url_grammar)
for _ in range(10):
    print(f.fuzz())

```

```py
ftp://user:pass@www.google.com:80/?q=path#ref
http://freebsd.org/
ftp://freebsd.org/#News
http://www.cispa.saarland:80/?q=path#ref
http://user:pass@www.google.com:80/#ref
https://freebsd.org/
ftp://freebsd.org/#ref
ftp://freebsd.org/
ftp://freebsd.org/releases/5.8
ftp://user:pass@www.google.com:80/#News

```

```py
inventory_grammar = recover_grammar(process_inventory, [INVENTORY])

```

```py
syntax_diagram(inventory_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 386.5 62" width="386.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="193.25" y="35">process_inventory@1:inventory</text></g></g></g></g></svg>

```py
process_inventory@1:inventory

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 954.5 62" width="954.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="176.25" y="35">process_vehicle@1:vehicle</text></g><g class="non-terminal"><text x="477.25" y="35">process_vehicle@1:vehicle</text></g><g class="non-terminal"><text x="778.25" y="35">process_vehicle@1:vehicle</text></g></g></g></g></svg>

```py
process_vehicle@1:vehicle

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 1187.5 62" width="1187.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="163.5" y="35">process_vehicle@2:year</text></g> <g class="terminal"><text x="301.25" y="35">,</text></g> <g class="non-terminal"><text x="439.0" y="35">process_vehicle@2:kind</text></g> <g class="terminal"><text x="576.75" y="35">,</text></g> <g class="non-terminal"><text x="727.25" y="35">process_vehicle@2:company</text></g> <g class="terminal"><text x="877.75" y="35">,</text></g> <g class="non-terminal"><text x="1019.75" y="35">process_vehicle@2:model</text></g></g></g></g></svg>

```py
process_vehicle@2:year

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 293.0 92" width="293.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="146.5" y="35">process_car@1:year</text></g></g> <g><g class="non-terminal"><text x="146.5" y="65">process_van@1:year</text></g></g></g></g></svg>

```py
process_van@1:year

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">1997</text></g></g></g></g></svg>

```py
process_vehicle@2:kind

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 165.5 92" width="165.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="82.75" y="35">van</text></g></g> <g><g class="terminal"><text x="82.75" y="65">car</text></g></g></g></g></svg>

```py
process_vehicle@2:company

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 318.5 92" width="318.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="159.25" y="35">process_van@1:company</text></g></g> <g><g class="non-terminal"><text x="159.25" y="65">process_car@1:company</text></g></g></g></g></svg>

```py
process_van@1:company

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">Ford</text></g></g></g></g></svg>

```py
process_vehicle@2:model

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 301.5 92" width="301.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="150.75" y="35">process_van@1:model</text></g></g> <g><g class="non-terminal"><text x="150.75" y="65">process_car@1:model</text></g></g></g></g></svg>

```py
process_van@1:model

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 174.0 62" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">E350</text></g></g></g></g></svg>

```py
process_car@1:year

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 174.0 92" width="174.0" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="87.0" y="35">2000</text></g></g> <g><g class="terminal"><text x="87.0" y="65">1999</text></g></g></g></g></svg>

```py
process_car@1:company

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 199.5 92" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Chevy</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Mercury</text></g></g></g></g></svg>

```py
process_car@1:model

```

 <svg class="railroad-diagram" height="92" viewBox="0 0 199.5 92" width="199.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="99.75" y="35">Cougar</text></g></g> <g><g class="terminal"><text x="99.75" y="65">Venture</text></g></g></g></g></svg>

```py
f = GrammarFuzzer(inventory_grammar)
for _ in range(10):
    print(f.fuzz())

```

```py
1997,car,Chevy,E350
1997,van,Ford,Cougar
2000,car,Chevy,Venture
1999,car,Ford,E350
1997,car,Mercury,E350
2000,van,Mercury,E350
1997,van,Mercury,Venture
2000,van,Mercury,E350
1997,van,Ford,E350
1997,car,Mercury,Venture
1997,car,Mercury,E350
1997,car,Mercury,E350
1997,van,Chevy,E350
1997,van,Mercury,E350
1997,van,Chevy,E350
2000,car,Ford,E350
2000,van,Chevy,E350
1997,van,Mercury,E350
1999,car,Chevy,E350
2000,van,Mercury,E350
2000,van,Ford,E350
2000,car,Mercury,E350
1999,car,Ford,E350
2000,van,Chevy,Cougar
1997,car,Ford,E350
1997,car,Mercury,E350
1997,car,Chevy,Venture
1997,car,Ford,Venture
1997,car,Mercury,E350
1997,van,Mercury,E350

```

我们看到跟踪范围如何帮助我们提取更精确的语法。

请注意，我们使用*字符串*包含测试作为确定特定字符串片段是否来自原始输入字符串的方法。 与动态污点相比，这似乎容易出错，但我们注意到，许多跟踪工具（例如`dtrace()`和`ptrace()`）允许人们直接在不同平台上执行二进制文件来获取我们想要的信息。 但是，获取动态污点的方法几乎总是涉及在使用二进制文件之前对其进行检测。 因此，这种包含字符串的方法可以比动态污染方法更普遍地应用。 此外，动态污点经常由于隐式传输而丢失，或者在 *Python* 和 *C* 代码之间的边界处丢失。 字符串包含没有这样的问题。 因此，与依靠动态污染相比，我们的方法通常可以获得更好的结果。

## 经验教训

*   给定程序的一组示例输入，如果程序依赖于手写解析器，我们可以通过在执行过程中检查变量值来学习输入语法。
*   简单的字符串包含检查足以从实际程序中获取合理准确的语法。
*   生成的语法可以直接用于模糊测试，并且可以对您拥有的所有样本产生乘数效应。

## 后续步骤

*   了解如何使用[信息流](InformationFlow.html)进一步改善将输入映射到状态。

## 背景

从一组*样本*中恢复语言（即，不考虑可能处理它们的程序）是一个经过充分研究的主题。 Higuera [De la Higuera *等人*，2010。]的优秀参考资料涵盖了所有经典方法。 Clark [Clark *等人*，2013。]描述了黑盒语法挖掘的最新技术。

从*程序*中学习输入语言（有或没有样本），尽管可能会引起混淆，但仍是一个新兴话题。 该领域的开拓性工作是由Lin等人完成的。 [ [Lin *等人*，2008。](https://doi.org/10.1145/1453101.1453114)]发明了一种从上向下和下向上的解析器检索解析树的方法。 本章中描述的方法直接基于Hoschele等人的AUTOGRAM工作。 [[Höschele*等人*，2017年。](https://doi.org/10.1109/ICSE-C.2017.14)]。

## 练习

### 练习1：展平复杂对象

我们的语法挖掘者仅检查字符串片段。 但是，程序可能经常传递包含输入片段的容器或自定义对象。 例如，考虑对清单处理器进行合理的修改，其中我们使用自定义对象`Vehicle`携带片段。

```py
class Vehicle:
    def __init__(self, vehicle):
        year, kind, company, model, *_ = vehicle.split(',')
        self.year, self.kind, self.company, self.model = year, kind, company, model

```

```py
def process_inventory(inventory):
    res = []
    for vehicle in inventory.split('\n'):
        ret = process_vehicle(vehicle)
        res.extend(ret)
    return '\n'.join(res)

```

```py
def process_vehicle(vehicle):
    v = Vehicle(vehicle)
    if v.kind == 'van':
        return process_van(v)

    elif v.kind == 'car':
        return process_car(v)

    else:
        raise Exception('Invalid entry')

```

```py
def process_van(vehicle):
    res = [
        "We have a %s  %s van from %s vintage." % (vehicle.company,
                                                  vehicle.model, vehicle.year)
    ]
    iyear = int(vehicle.year)
    if iyear > 2010:
        res.append("It is a recent model!")
    else:
        res.append("It is an old but reliable model!")
    return res

```

```py
def process_car(vehicle):
    res = [
        "We have a %s  %s car from %s vintage." % (vehicle.company,
                                                  vehicle.model, vehicle.year)
    ]
    iyear = int(vehicle.year)
    if iyear > 2016:
        res.append("It is a recent model!")
    else:
        res.append("It is an old but reliable model!")
    return res

```

我们像以前一样恢复语法。

```py
vehicle_grammar = recover_grammar(
    process_inventory,
    [INVENTORY],
    methods=INVENTORY_METHODS)

```

缺少新的车辆语法，尤其是在面包车和轿车的不同型号和公司方面。

```py
syntax_diagram(vehicle_grammar)

```

```py
start

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 386.5 62" width="386.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="193.25" y="35">process_inventory@1:inventory</text></g></g></g></g></svg>

```py
process_inventory@1:inventory

```

 <svg class="railroad-diagram" height="62" viewBox="0 0 954.5 62" width="954.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="non-terminal"><text x="176.25" y="35">process_vehicle@1:vehicle</text></g><g class="non-terminal"><text x="477.25" y="35">process_vehicle@1:vehicle</text></g><g class="non-terminal"><text x="778.25" y="35">process_vehicle@1:vehicle</text></g></g></g></g></svg>

```py
process_vehicle@1:vehicle

```

 <svg class="railroad-diagram" height="122" viewBox="0 0 335.5 122" width="335.5" xmlns="http://www.w3.org/2000/svg"><g transform="translate(.5 .5)"><g><g><g class="terminal"><text x="167.75" y="35">2000,car,Mercury,Cougar</text></g></g> <g><g class="terminal"><text x="167.75" y="65">1999,car,Chevy,Venture</text></g></g> <g><g class="terminal"><text x="167.75" y="95">1997,van,Ford,E350</text></g></g></g></g></svg>

问题是，我们正在专门寻找在跟踪过程中包含输入字符串片段的字符串对象。 您是否可以修改我们的语法挖掘器以正确解决复杂对象？

[Use the notebook](https://mybinder.org/v2/gh/uds-se/fuzzingbook/master?filepath=docs/notebooks/GrammarMiner.ipynb#Exercises) to work on the exercises and see solutions.

### 练习2：整合InformationFlow 中的污点

我们一直在使用*包含字符串*来检查特定片段是否来自输入字符串。 这是不令人满意的，因为它要求我们在跟踪的字符串的大小上做出折衷，这仅限于大于`FRAGMENT_LEN`的字符串。 此外，单个方法可能会处理片段重复的字符串，但该字符串是不同标记的一部分。 例如，CSV文件中的嵌入式逗号会导致我们的解析器失败。 避免这种情况的一种方法是依靠*动态污点*并检查污点包含而不是字符串包含。

关于[信息流](InformationFlow.html)的章节详细介绍了如何合并动态污点。 您能否根据范围更新我们的语法挖掘器，以改用*动态污点*？

[Use the notebook](https://mybinder.org/v2/gh/uds-se/fuzzingbook/master?filepath=docs/notebooks/GrammarMiner.ipynb#Exercises) to work on the exercises and see solutions.