There are many websites on the Internet that test lenses and publish test results in the form of MTF graphs, describe how strong aberrations are and how effective autofocus is.
Some of these websites disclose the methodology by which they conduct tests. However, even if this methodology looks reasonable enough, is it possible to obtain reliable results with its help? Let's try to figure it out, but first we will study the methodology of some well-known lens testers.
Charts of different sizes are used for each specific lens. RAW images are developed using Photoshop ACR version 3.2 or better (default settings, no automatic correction, contrast 0). The images are evaluated using the Imatest program.
Source: https://opticallimits.com/Reviews/lens-test-faq
The lens is mounted on a tripod, a cable or a remote control is used to release the shutter, and the mirror is locked up if the lens is tested on a SLR camera.
Charts of four sizes are used for testing resolution. Typically, during testing, three of the four charts are selected, measurements are taken at different focusing distances, and then the results are averaged. The wider the angle of view of the lens, the larger the charts are used.
In the case of testing an autofocus lens, back- and front-focus check is carried out first, and then, if necessary, autofocus microadjustents are performed. Even so, half of the test shots at each aperture are taken in manual focus mode, which eliminates the influence of possible autofocus errors on the test results.
The resulting images are saved in RAW and then developed by dcraw with output in TIFF or PPM format. The analysis of these images with obtaining the values of the MTF50 function is carried out using the Imatest program (however, it would be more correct to use the MTF30 to estimate the resolution, and the MTF80 to estimate the contrast).
The MTF50 values obtained are not the result of the lens performance per se, but always the result of the camera-lens pairing performance. In other words, the measured resolution depends on the sensor of the camera used.
Only one copy of the lens is tested. As an advantage of this approach, it is pointed out that it allows to get more relevant results (since the lens is actually chosen at random) and publish the results on the website faster.
Source:
As you can see, not all websites prefer to disclose their lens testing methodologies in detail. For example, opticallimits.com did not indicate whether they are testing one or more copies of the lens. However, if they do test multiple copies, they would definitely indicate this, as it makes the test results more reliable.
Each copy of a particular lens is different from others. It can be sharper or, conversely, less sharp at the same focal lengths and apertures. The lens you buy from the store will definitely be different from the lens tested on the popular website. For this reason, it still seems advisable to test several copies and then average the results. Of course, this is a laborious, lengthy and, which is also important, financially costly process, but it will provide truly reliable results. Otherwise, it may turn out that the tester got a bad copy of the lens, and the results of his tests may not reflect the actual optical performance of this lens model.
None of the testers indicated specific sizes of charts they have. And as you will be able to see, this silence is not without reason.
As you may know, almost all lenses (both modern and those produced in the 20th century) are optimized to provide the best image quality when focusing at infinity. At this focusing distance, the lens provides maximum resolution across the frame, and aberrations are corrected in the best way.
Of course, there are a number of exceptions to this rule:
However, in general, again, most lenses are still optimized by manufacturers for shooting at infinity, so resolution and correction of aberrations may be worse over the rest of the focusing distance range.
In order to test wide-angle lenses at infinity, you need an oversized chart due to the large field of view provided by this class of lenses. Do not forget that the entire image on the chart must be completely inscribed in the frame: the edges and corners of the chart must be at the edges and in the corners of the frame, respectively. In other words, the test image should not contain an extra image that is not related to the chart (for example, the image of the wall on which it hangs). Conversely, the chart image must not go beyond the frame, otherwise the test results will be unreliable. So, most likely the above testers use charts of the largest available size (187.96x111.76cm in the case of the Imatest SFRplus) to test wide-angle lenses, and the focusing distance is only a few meters. But how often do you shoot with wide-angle lenses at such distances? In the end, the specifics of shooting with lenses of this class is that the subject (landscape, interior, building) is quite large (much larger than the largest Imatest chart), so to capture it in full, you need to shoot at as great distance as possible. In that case, how useful to you are the test results obtained when shooting a relatively small chart at a focusing distance of several meters?
For example, when testing the Sigma AF 12-24mm f/4.5-5.6 EX DG HSM wide-angle zoom lens, opticallimits.com were forced to carry out it on an APS-C camera, and as a result the field of view formed by the lens was 1.5 times less than if the lens would be used on a full frame camera. To measure the distortion, a chart with a size of only 120x80cm was used. The focusing distance, at which the chart was shot, was not indicated. Let's try to calculate it.
It is known that the Sigma AF 12-24mm f/4.5-5.6 EX DG HSM lens has magnification ratio of 1:7.1 at focal length of 24mm and a closest focusing distance of 0.28m. This means that at this combination of focal length and focusing distance, an object with a size of 25.56x17.04cm will completely fill the 3.6x2.4cm frame with its image. In order for image of a 120x80cm chart to completely fill the frame, it is necessary to increase the focusing distance by 4.7 times (120/25.56 or 80/17.04). Therefore, if the shooting was carried out on a full-frame camera, the focusing distance would be 1.316m (0.28*4.7). However, since testing was carried out on an APS-C camera with a crop factor of 1.53, as a result of which the field of view of the lens became more narrow, the focusing distance must be increased by 1.53 times, and it will be 2.01m. Of course, this is an approximate calculation, since the magnification ratio may decrease non-linearly with increasing focusing distance, but you can still get a rough idea of the focusing distance used.
Accordingly, the distortion value obtained by opticallimits.com is relevant only for this focusing distance (rarely used with this lens in practice).
You may ask: perhaps test results are the most useful for macro lenses, because lenses of this class are optimized to provide the best image quality for close-ups, so shooting charts at short focusing distances is quite justified in this case? However, the problem is that for the focusing distances at which macro lenses are used in practice, even the smallest charts are too large! In order for the chart image to completely fill the frame, when testing macro lenses, the focusing distance must be increased, which, as in the case of wide-angle lenses, leads to useless results. For example, if a full-frame macro lens is capable of shooting at its closest focusing distance with a magnification ratio of 1:1, this means that a 3.6x2.4cm object at the closest focusing distance will completely fill the frame. However, the minimum size of the Imatest (SFRplus) chart is 38.1x21.59cm (working area is 35.56x19.76cm), which is almost 10 times larger and requires a significant increase in the focusing distance, at which macro photography is usually no longer carried out due to insufficient magnification ratio (as you may know, it decreases with increasing focusing distance).
All of the above is true for both resolution measurements and lens distortion measurements, since they both depend on the focusing distance.
We will deliberately not draw any conclusions at the end of this article. We believe that based on the information above, you will make them yourself. We just consider it necessary to draw your attention to the following.
If you buy a car after having carried out a test drive, buy or rent a house or apartment after having inspected them, why should the approach be different in the case of camera lenses? Always choose lenses personally, especially if they are expensive models. Visit a store and test a few copies at the focal lengths and apertures at which you usually shoot or plan to shoot with this lens. During testing, enable those lens functions that you prefer to use during normal shooting (image stabilizer, focus range limiter, autofocus, etc.), and disable those you do not intend to use.
If you nevertheless decide to ignore our advice and do not mind to purchase lenses online, then at least find a store with a favorable return policy.
And the last thing. It doesn't matter if other people don't like a certain lens, if it suits your tasks. Conversely, if you think that a lens is not worth its money, then you can safely ignore other people's opinions that this is one of the best lenses that money can buy.
Copyright © 2012-2024 Evgenii Artemov. All rights reserved. Translation and/or reproduction of website materials in any form, including the Internet, is prohibited without the express written permission of the website owner.