不应过度追求coverage

这是我在我想这样去逼近白盒测试的一个回帖。

gigix 写道

tianxinet 写道

code review 是最好的白盒测试方式之一。

视图覆盖测试所有代码是不值得滴。
http://www.infoq.com/news/dangers-code-coverage

不错。也看了John Casey的原文。这应该是John Casey重构“Maven's assembly plugin”后写的一篇blog。

刚好，结合John Casey的原文谈下自己的一些看法：

引用

when you're seeking confidence through testing, perhaps the worst thing you can do is to look at a test coverage report.

开篇，John Casey说明不要把注意力集中到coverage report，他在文中逐步给出了理由。
我完全同意，code coverage有必要做，但到底多高的覆盖率是好，100％，90％？一个应用中的很多代码可能根本没必要去coverage，只对关键代码编写详细的测试代码就够了。至于什么是关键代码，可能是最重要的功能，或逻辑最复杂的部分...，一个成熟的团队完全有能力根据具体情况进行判断。

引用

Recently, I spent quite a bit of time refactoring Maven's assembly plugin. I'm sorry to say that I was fooled when I started by the siren song of coverage reporting,.........the plugin had a suite of tests that were so rigidly constructed that even fixing a simple bug would result in hours of debugging the test cases to find out what assumption had been violated.

当开始重构“Maven's assembly plugin”时，John Casey被coverage report愚弄了。修正一个简单bug就要花数小时的时间去检查原来的test case。
我不能怀疑“Maven's assembly plugin”老版的开发者水平很糟糕，只能怀疑用于单元测试的test case写的太复杂了（或许就是试图追求过高的覆盖率？），而且已经本末倒置了，到底是要测试function code还是test code？

引用

I planned to move the existing tests to an undisclosed location for safe keeping , then construct a completely new suite of tests to focus on small, specific units of the new plugin's implementation..........As I went, I planned to use the coverage report  to mark my progress, and let me know how much I had left to do.

John Casey抛弃了原来的测试代码，重新写了针对少量的、特定单元的测试代码。不过John Casey还是要做coverage report。
我绝对赞同这种做法，只对必要的unit编写详细的测试代码。而相对coverage我认为code review的效果更好，一个团队一起对代码进行code review，这样白盒测试的效果要比依赖coverage report好。这也使团队保持沟通（而且是在开发过程中就代码质量持续沟通），对代码质量有更一致的认知。考虑到有些开发是单独完成的，并没有一个团队，所以只做coverage report也是可行的。John Casey也没有说明他的coverage是什么级别的，line coverage？method coverage？class coverage？我认为配合code review，method coverage对多数情况已经足够（不排除有特殊情况）。

......but by this time I was badly shaken. I had the coverage numbers, but I was starting to see that they weren't telling me the whole story.
......I stopped looking at the report, ......to capture real-world use cases, and test whether they were supported properly.

John Casey开始怀疑coverage report并最终停止关注它，转而去捕获real-world的use case。
过度依赖单元测试是不好的。我之所以这样说，背景是现在单元测试已经受到了足够的重视，恐怕没有哪个开发者或团队会忽视单元测试（其实我还是不太确定单元测试是否已经受到普遍重视）。比如试图对一个应用的所有代码做line coverage就是”过度测试“（好像没这个术语，表达一下意思吧）。

引用

......In the end, the combination of these two approaches resulted in an easy-to-maintain, comprehensive testing strategy.
......integration testing allowed me to capture error reports directly from users, and put them to the test. This provides a quick way to verify a bug - even at a high level - and gives that all-important direction to the debugging effort.

John Casey最终选择coverage、集成测试平衡运用。
code-world 的 test case（白盒测试代码）；
real-world 的 use case（黑盒测试用例）；
这两者确实要平衡运用，引用一个对John Casey的回复：" Coverage tests are great at telling you when you don't have enough tests.  They're terrible at telling you that you have enough, that they're good enough, etc."

引用

......Bumping the progress bar of your coverage report up to account for a line of code which has been touched by one test is a fatally flawed concept, and it can lead to misperceptions on the parts of managers and developers alike. And, since there is no hard-and-fast rule about how many test passes it will take to test any given line of code adequately, the concept of a coverage report is itself fatally flawed.

这是作者的一个总结，让coverage report统计一个测试是否到达一行代码，是一个有致命缺陷的观点。
没什么说的，我完全赞成。覆盖测试每一行代码真的没多大意义。对有些不必要的代码进行coverage，最大的好处或许只是心理安慰。但没有来自real-world的反馈，这样的心理安慰或信心其实很脆弱。

大家有兴趣可以看看：Testing: Coverage Reports Considered Dangerous
关于这篇文章的讨论：Opinion: Code Coverage Stats Misleading