t+1 - Latest Comments in I need to write faster tests

Re: I need to write faster tests

Matt Wilson — Fri, 18 Jul 2008 20:27:28 -0000

The other thing that unit tests won't catch is when one unit alters its interface but the second unit doesn't find out about it.

Incidentally, I don't agree with the idea that interfaces shouldn't change. If I come up with a better way of doing something, I'm going to do it.

Re: I need to write faster tests

Matt Wilson — Fri, 18 Jul 2008 20:25:22 -0000

Yeah, I think maybe there's nothing wrong with these narrative-based tests, but I would like to separate them from my other split-second tests.

Re: I need to write faster tests

Aaron Oliver — Fri, 18 Jul 2008 15:51:55 -0000

I like to think of tests in terms of cost/benefit.

Longer (integration) tests cost more because they eat time, but in theory have greater benefit, because they reflect actual use more closely. ( http://googletesting.blogsp....

Teeny unit tests are cheap, but don't verify as much as the longer integration tests.

So running cheap tests all the time is ok, just like paying for electricity in your house all the time is ok. It doesn't cost much.

Expensive tests can run less often but give better payoff, just like only buying a car every couple of years.

Re: I need to write faster tests

Michael Foord — Fri, 18 Jul 2008 15:11:21 -0000

+1 to separating unit and integration tests.

Integration tests *should* use a real database (etc) - but will take long er and longer as your tests cover more things (although there are obviously techniques you can use to minimize this).

Unit tests should mock out every they can in terms of contact with the 'outside world' - as a result they can run much faster.

You can run your functional tests in a loop on your integration machine (using a fresh checkout each time and emailing you failures - use something like CC.NET or just knock up an integration script tied to a post commit hook yourself).

You can run your unit tests whenever you make a change (preferably before to see the test fail and then afterwards to see it pass of course).

Re: I need to write faster tests

Christian Wyglendowski — Fri, 18 Jul 2008 15:06:16 -0000

In the main application that I work on, we have both unit (py.test) and functional (Selenium) tests.

The unit tests are quite fast and do all of the low-level library testing and stuff. I run them often.

The Selenium tests take longer, but they really work the system over.

I think that is something that you can't avoid, really. If you want to test your application like users are going to use it, it is bound to be slower, even with a computer at the wheel.

Re: I need to write faster tests

Matt Wilson — Fri, 18 Jul 2008 14:26:13 -0000

Hi Kumar, thanks for the comment! I saw your fixture project earlier today on planetpython. Right now, my setup code doesn't go through the app to setup. It makes a database connection directly and monkeys around.

All my tests use the same setup and teardown, so each test starts at the same state. That still might not let me run tests in parallel, however, since they're all working on the same user objects. If I could run all these in parallel, that would likely cut time way down. Maybe I could change my setup code so that each test gets objects created with names just for them.

I'll be the first one to admit that I have a lot more to learn in terms of testing. Right now, I feel a lot of frustration waiting for these things to finish execution.

I'll check out fixture over the weekend. Thanks for the tips.

Re: I need to write faster tests

Matt Wilson — Fri, 18 Jul 2008 14:25:47 -0000

...

Re: I need to write faster tests

Kumar McM illan — Fri, 18 Jul 2008 14:12:42 -0000

I've found that two things you mentioned here tend to be problematic for functional testing.

First is using an actual interface to setup the data you need to test with. Although this hides the implementation of how your data gets stored, which is nice, it is often slower than necessary. You just need data in the db, right? I actually created a module called fixture to load up just the data for such a test: http://farmdev.com/projects... :)

Second, it's not a good idea to insert data in one test that a later test depends on. This is because your tests are no longer modular so you couldn't, for example, re-run only one test that is failing or use a grid solution (like http://selenium-grid.openqa... or like http://code.google.com/p/py... -- running tests in parallel with nose).