ZTC/PTC are integration tests with a naming problem ;)
Posted bylimiat
2008-05-29 01:11 AM
In my opinion, PloneTestCase/ZopeTestCase is more a victim of the circumstances than anything else — they really aren't unit tests, they are integration tests. Which (at least in the case of Plone) is just as important.
That of course doesn't mean that we shouldn't have real, lightweight unit tests too, but that's exactly what you're pointing out. Of course, maybe we should call them by their correct name, but that's ultimately something others have to decide on. I don't actually know if they should all be under the same testrunner or not. :)
the problem is...
Posted bychrismat
2008-05-29 01:28 AM
... I don't think most people "get the joke". They are writing things that *are* unit (and not integration) tests using the ZTC/PTC framework. A test that tests, say, a simple accessor method takes 500ms because they've inherited from PloneTestCase ("everybody does it, I must need it!"). It's good that there are tests, but to assume that all users who run the tests should wait five minutes for each test run to finish when the tests really should take about 30 seconds is silly. There are definitely things that need integration tests. Testing adding an action to the actions tool's ("testAddAction" at https://svn.plone.org/svn/plone/Products.CMFPlone/trunk/Products/CMFPlone/tests/testActionsTool.py, chosen randomly) is not one of these, but we still pay that 500ms each time it runs. Multiply that by forty bazillion (estimated)...
argh...
Posted bychrismat
2008-05-29 01:58 AM
Even worse, that particular testAddAction test isn't even testing *any implementation defined by the tool*. It should be moved into the CMF tests, or maybe just deleted. God knows what its author was trying to cover. He *might* be trying to test listActions (which the tool does define), but if so he's doing a really shitty job because he doesn't actually test any of its conditionals.
So... are you going to delete it? No. Because you don't know why it's there, and neither do I, and it's easier to not think about it and just leave it there. Are you going to add tests that actually test the functionality *of the implementation*? No. Because "it's tested, it has tests!". But a lot of these tests are actually complete decoys. And on and on it goes, running useless tests that just spin the CPU for no apparent purpose, neglecting to revisit old tests and rewrite them in a way that *actually* tests the stuff that is written in the implementation... but, sure, features? Did you say features? We can add more features! So much technical debt...
That of course doesn't mean that we shouldn't have real, lightweight unit tests too, but that's exactly what you're pointing out. Of course, maybe we should call them by their correct name, but that's ultimately something others have to decide on. I don't actually know if they should all be under the same testrunner or not. :)
So... are you going to delete it? No. Because you don't know why it's there, and neither do I, and it's easier to not think about it and just leave it there. Are you going to add tests that actually test the functionality *of the implementation*? No. Because "it's tested, it has tests!". But a lot of these tests are actually complete decoys. And on and on it goes, running useless tests that just spin the CPU for no apparent purpose, neglecting to revisit old tests and rewrite them in a way that *actually* tests the stuff that is written in the implementation... but, sure, features? Did you say features? We can add more features! So much technical debt...
Replies to this comment