The COSC doesn't rubber-stamp all movements, even though something like 95% of them pass. Since the companies pay a hefty price for the testing (I think it is the equivalent of about $200-$250 per movement), they will have passed very stringent quality control by the manufacturer first--the manufacturer is only going to send movements that they are sure will pass, since they still need to pay for the ones that fail. There is an early grace period, so if a movement has problems and it is obvious after the first couple of days that it will not pass, it can be withdrawn and only a partial fee must be paid. Watches that fail after the grace period pay the entire fee. The most common reason for failure is that the movement stopped during testing. I would imagine that many problems are the result of all the shipping and handling that these movements would get.
COSC testing doesn't really mean anything significant. It is only an independent test with a certificate awarded for passing. A watch from a decent manufacturer might perform as well as, or better than, than the COSC standards, whether or not is has a certificate. JLC's Master series watches undergo much more stringent tests than the COSC administers, and are tested in their actual cases.
Movements that go to the COSC are mounted in temporary cases, with hands and dials, for testing purposes. After they come back from testing, they are put into their real cases. Watches with no second hand are sent for testing with a temporary second hand fitted.
There is a picture in the Chronometer Watches and Their Testing book which shows chronometer movements under test being wound with a small motor. All watches under test get wound by their crowns, even automatics. The winding device is smallish and looks like a dentist's drill or a Dremmel motor tool. The business end is a cup that fits over the crown, like a socket wrench.
The movements cases are threaded one-after-the-other on long straps, about 10 to a strap. There are about 10 of these straps mounted in a rack or frame, so that if you look at the frame from the front, you see 100 watch faces equally spaced in rows and columns.
During testing, the entire rack, with cased movements attached, is placed in the various positions, and moved into the hot and cold testing areas. (Very old chronomter certificates refer to the "oven" and the "refridgerator.") Instead of trying to read the time on each face when the daily test results are recorded, in the past they took a photograph of each rack, including a reference timepiece in the photo. This would record the position of each second hand in the rack, at the same instant, and would make it possible to read the indicated time on each face to a fraction of a second, as well as documenting the correct time on the reference timepiece. The photographic negative would be read under magnification, and the test sheet for each movement would be filled out.
Today, a laser scans across the rack and reads the position of each second hand, and the test results are stored as a computer file.
Automatic watches are wound once a day, by their crowns. They are happier in real life, where they are constantly wound during the day by their rotors, and their mainspring tension stays fairly constant. Because of this, they tend to perform slightly less well in testing than they do in real life--they run a little slow. The opposite is true for handwinds, which do slightly better in testing than they do in real life. Some companies regulate their autos slightly fast before they go to the COSC, to compensate for the once-a-day winding, then regulate them slower again when they come back.
Rather than absolute accuracy, which can be regulated easily if the watch runs slightly fast or slow, the most significant aim of COSC testing is to test for consistancy in different positions and temperatures. This is primarily a test of the balance and hairspring. I had a conversation about this with Walt O. once, and he said that the balance and hairspring are part of a module which is not taken apart even when the watch is serviced, so a watch that performs well in testing should do so as long as this is not tampered with. In the Chronometer book, the author reports on a large number of vintage chronometers, mostly from the 1950s and 60s, which he has repaired and serviced and then subjected to simulations of their original COSC tests. Most of them could still pass, once they were returned to good running condition.
One thing that the book impressed upon me is the extreme difficulty of making a mechanical watch perform well in all positions and at different temperatures. The COSC standards are compromised from an ideal, so that the standards fall within ranges that can be met realistically in mass production. The standards have been influenced by the companies, and were loosened slightly during the early years of the COSC, because they were too difficult for the movements of the time, and too many watches failed.
There were once "passed" and "passed with distinction" ratings, but today movements either pass or fail. The standards have gotten tighter over the years.
In some instances, the methods of calculating and recording a watch's performance, rather than the actual test itself, can be questioned. For instance, if the watch runs -5 seconds in one position, and +8 in another, this falls within the test range because the deviation is measured from 0. However, the range of variation is 13 seconds. The deviations caused by temperature change are listed not by total variation, but by variation per degree of temperature change. A .5 second change per degree looks pretty good, but if you multiply it by 20 degrees, 10 seconds doesn't look so good. The results are not fudged, but the way they are reported makes most watches sound better than they are.
Just to repeat, the book explains how difficult it is to build a mechanical watch, in large quantities, which will perform consistantly under changes of position and temperature. Seeing the issue from both sides, I personally don't have any problem with the test standards. However, I also don't think the certificate is as significant as some people do.
You can be smart and well-educated, and yet not have a college degree, but some people are more likely to trust what you say if you have diplomas and certificates on your wall. If your watch has a COSC certificate, it means that it was able to pass a bunch of tests under artificial conditions. Someone with a diploma was also able to pass a bunch of tests under artificial conditions. In both cases, your real-world results may differ.