Quality Report Help

 
Video: Watch the instructional video related to Quality Assessmenthere.
 
Important: 
  • For a description about how to analyze the Quality Report: 202557339.
  • For a detailed description about any parameter described in the Quality Report: 202558679
  • Example of a Quality Report available at the following link: Quality Report.

 

 

Processing Failed in Step 1 

If processing failed during step 1, the Failed Processing Report displays the following information:

  • Error: Description of the error that made the processing fail.
  • Substep: The substep of Initial Processing at which the processing failed.
  • Cause: Description of the possible causes of the failure.
  • Solution:Description of the possible solutions with a link to step by step instructions.

 

Quality Check

Images    

Keypoints Image Scale > 1/4: More than 10'000 keypoints have been extracted per image.


Keypoints Image Scale ≤ 1/4: More than 1'000 keypoints have been extracted per image.

Images have enough visual content to be processed.

Keypoints Image Scale > 1/4: Between 500 and 10'000 keypoints have been extracted per image.

Keypoints Image Scale ≤ 1/4: Between 200 and 1'000 keypoints have been extracted per image.

Not much visual content could be extracted from the images. This may lead to a low number of matches in the images (for more information: Matching Quality Check) and incomplete reconstruction or low quality results. This may occur due to several factors:

  • Image content: Large uniforms areas such as deserts, snow, fog, etc.
    What to do: In such cases, a high overlap is required. Flying at a different altitude may also have a positive influence on the visual content of the images.
  • Image quality: Images are over/under exposed, blurry, or noisy.
    What to do: Camera parameters need to be adjusted (shutter speed, exposure time). For more information about the camera settings: 202557419.
  • Image size: The likelihood of extracting many features increases with the image size.
    What to do: Images smaller than one megapixel have very few features and require a large amount of overlap (>80%). Doubling the image size used to extract the features could also help: 202557759.

Keypoints Image Scale > 1/4: Less than 500 keypoints have been extracted per image.

Keypoints Image Scale ≤ 1/4: Less than 200 keypoints have been extracted per image.

What to do: Same as above, increasing the overlap (> 90%). 
Dataset    

More than 95% of enabled images are calibrated in one block.

All or almost all images have been calibrated in a single block. 

Between 60% and 95% of enabled images are calibrated

or

more than 95% of enabled images are calibrated in multiple blocks.

Many images have not been calibrated (A) or multiple blocks have been generated (B).

A) Uncalibrated images are not used for processing. This may occur due to several factors:

  • Dataset with low overlap or images not taken in a systematic way. Overlap can be assessed in figure 4 and figure 5 of the Quality Report.
    What to do: Increase the overlap. For more information about the flight plan: 202557459
  • Repetitive or complex dataset (trees, forest, fields).
    What to do: The overlap might need to be increased (>80%). Flying at a higher altitude often reduces visual complexity and improves the results, especially in forest and dense vegetation environments. For more information about the flight plan: 202557459
  • Dataset made from multiple flights with images not similar enough (different time of capture, moving objects, different temperature, different lens).
    What to do: Process each flight individually and combine the projects together in a second step. For more information about how to merge projects: 202558529.
  • Dataset containing multiple images shot from the same position, or images taken during take-off or landing phase.
    What to do: These images should be manually removed.
  • Image quality not sufficient: Camera parameters need to be adjusted (shutter speed, exposure time). For more information about the camera settings: 202557419.

 

B) Multiple blocks: A block is a set of images that were calibrated together. Multiple blocks indicate that there were not enough matches between the different blocks to provide a global optimization (see the graph 2D Keypoint Matches). The different blocks might not be perfectly georeferenced with respect to each other..
What to do:

  • Enabling the option Rematch may improve the connection between the blocks. For more information about the Rematch option: 205327965.
  • Adding Manual Tie Points between the blocks and reoptimizing can connect the blocks together. For more information: 202560349.
  • Capturing new images with more overlap may be required. 

Less than 60% of enabled images are calibrated.

What to do: Same as above.
Such a low score may also indicate a severe problem in:

  • The type of terrain: Water surface, oceans, mirrors and glass surfaces, moving lava, and moving landscapes do not contain the needed visual content for processing. To obtain results, these terrains need to be combined with areas that are easy to reconstruct. Flying at a higher altitude is recommended to map areas close to water: 202557459.
  • Image acquisition process: Wrong image geolocation, inappropriate flight plan, insufficient overlap, corrupted images, etc.
  • Project setup: Wrong coordinate system definition, wrong images, etc.
Camera Optimization     

Perspective lens: The percentage of difference between initial and optimized focal length is less than 5%.

Fisheye lens: The percentage of difference between initial and optimized affine transformation parameters C and F is less than 5%.

The focal length/affine transformation parameters are a property of the camera's sensor and optics. It varies with temperature, shocks, altitude, and time. The calibration process starts from an initial camera model and optimizes the parameters. It is normal that the focal length/affine transformation parameters are slightly different for each project. An initial camera model should be within 5% of the optimized value to ensure a fast and robust optimization.

Perspective lens: The percentage of difference between initial and optimized focal length is between 5% and 20%.

Fisheye lens: The percentage of difference between initial and optimized affine transformation parameters C and F is between 5% and 20%.

What to do:

  • If the dataset completeness is above 50%, the initial parameters need to be optimized again, using the optimized parameters as initial parameters in order to improve the quality. For step by step instructions: 202560059.
  • If the completeness is low, this may indicate either a problem in the project (overlap too low, image quality too low, wrong image geolocation) or that the initial camera model was not entered properly. In this case, check that the camera parameters are good.

Perspective lens: The percentage of difference between initial and optimized focal length is more than 20%.

Fisheye lens: The percentage of difference between initial and optimized affine transformation parameters C and F is more than 20%.

Failed Processing Report: always displayed as the information is not available.t

What to do: same as above.
Matching     

Keypoints Image Scale > 1/4: More than 1'000 matches have been computed per calibrated image.

Keypoints Image Scale ≤ 1/4: More than 100 matches have been computed per calibrated image.

This indicates that the results are likely to be of high quality in the calibrated areas. Figure 5 of the Quality Report is useful to assess the strength and quality of matches.

Keypoints Image Scale > 1/4: Between 100 and 1'000 matches have been computed per calibrated image.

Keypoints Image Scale ≤ 1/4: Between 50 and 100 matches have been computed per calibrated image.

Low number of matches is in the calibrated images may indicates that the results are not very reliable: achanges in the initial camera model parameters or in the set of images may lead to improvements in the results. Figure 5 of the Quality Report shows the areas with very weak matches. A low number of matches is very often related to low overlap between the images.


What to do: See the Dataset Quality Check section to improve the results. There might be needed to restart the calibration a few times with different settings (camera model, Manual Tie Points) to get more matches. To avoid this situation, it is recommended to acquire images with more systematic overlap. For more information about the flight plan: 202557459

Keypoints Image Scale > 1/4: Less than 100 matches have been computed per calibrated image.

Keypoints Image Scale ≤ 1/4: Less than 50 matches have been computed per calibrated image.

Failed Processing Report: always displayed as the information is not available.

What to do: Same as above. The minimum number of matches to calibrate an image is 25.
Georeferencing    

GCPs are used and the GCP error is less than the average GSD.

For optimal results, GCPs should be well distributed over the dataset area. Optimal accuracy is usually obtained with 5 - 10 GCPs. For more information about GCPs: 202557489.

GCPs are used and the GCP
error is less than 2 times the average GSD.

or

no GCPs are used

Failed Processing Report: always displayed whether GCPs are used or not.

GCPs are used
GCPs might not have been marked very precisely. Verify the GCPs marks and if needed, add more marks in more images. If possible, select images with a big base (distance) as it helps to compute the GCPs 3D position more precisely.

No GCPs are used

There are two cases where No GCPs are displayed:

  • No GCPs were entered. This means that the project is georeferenced using the position of the computed image positions. GPS devices used to geolocate the original images may suffer from a global shift, leading to a global shift in the project of several meters.
  • The GCPs were discarded by the software due to errors with the GCPs (e.g. wrong GCP coordinate system, wrong GCP coordinates, GCPs not marked correctly on the images). For more information: Ground Control Points.

GCPs are used and the GCP error is more than 2 times the average GSD.

A GCP error superior to 2 times the Ground Sampling Distance may indicate a severe issue with the dataset or more likely an error when marking or specifying the GCPs. 

 

Preview

The displayed images are the low resolution preview of the Orthomosaic and the DSM before step 2. They allow a visual inspection of the quality of the initial calibration. If the orthomosaic is skewed, there might be an error with the project orientation and GCPs may be required. If the DSM contains large seams or artifacts, there might be due to multiple blocks in the reconstruction. If there are holes in the Orthomosaic and DSM, check the Quality Checks section and the 2D Keypoint Matches graph.

 

Initial Image Positions

This graph is useful to review the images geolocation. If this graph does not correspond to the flight plan, there might be problems with the matching, and the orientation, scale and/or the geolocation of the results. Check that the images' coordinate system and the images coordinates are correct.

 

Computed Image/GCPs/Manual Tie Points Positions

This graph shows the difference between the initial and computed image positions,  the difference between the initial and computed GCPs/Check Points positions (if any), the MTPs positions (if any) and the uncertainty ellipses of the absolute camera positions. 

Images: There might be a small offset between the initial and computed image positions because of image geolocation synchronization issues or GPS noise. If the offset is very high for many images, it may affect the quality of the reconstruction and may indicate severe issues with the image geolocation (missing images, wrong coordinate system, and/or coordinate inversions). 

A bended/curved shape in the side and front view may indicate a problem in the camera parameters optimization. Ensure that the correct camera model is used. If the camera parameters are wrong, correct them and reprocess. If they are correct, the camera calibration can be improved by:

  • Increasing overlap/image quality.
  • Removing ambiguous images (shot from same position, take-off or landing, too much angle, image quality too low).
  • Introducing Ground Control Points.

GCPs/Check Points: An offset between initial and computed position may indicate severe issues with the geolocation due to wrong GCP/Check Point initial positions, wrong coordinate system, and/or coordinate inversions, wrong marks on the images, wrong point accuracy.

Uncertainty Ellipses: The absolute size of the uncertainty ellipses does not indicate their absolute value because they have been magnified by a constant factor noted in the figure caption. In projects with GCPs, the uncertainty ellipses close to the GCPs should be very small and increase for images further away. This can be improved by distributing the GCPs homogeneously in the project. 

In projects only with image geolocation, all ellipses should be similar in size. Exceptionally large ellipses may indicate calibration problems of a single image or all images in an area of the project. This can be improved by:

  • Adding Manual Tie Points in the area.
  • Rematching and optimizing the project.
  • Removing images of low quality.

 

Absolute Camera Position and Orientation Uncertainties

In projects only with image geolocation, the absolute camera position uncertainty should be similar to the expected GPS accuracy. As all images are positioned with similar accuracy, the sigma reported in the table should be small compared to the mean. In such projects, the absolute camera position uncertainties may be bigger than the relative ones in the table “Relative position and orientation uncertainties”.

In projects with GCPs, a large sigma can signify that some areas of the project (typically those far away from any GCPs) are less accurately reconstructed and may benefit from additional GCPs.

 

Overlap

This graph shows the number of overlapping images for each pixel of the orthomosaic. It only takes into account the calibrated images. Red areas indicate too low overlap. This may lead to low quality 3D reconstruction in these areas. The overlap is an important parameter for the overall quality. For precise 3D modeling and mapping applications, the overlap should be in green, i.e. each pixel should be visible in more than 5 images.

 

Internal Camera Parameters

Perspective lens: The principal point should be around half the resolution of the camera, and that the radial distortion values R1, R2, R3 should each be smaller than 1. The uncertainties on the focal length and the principal point should be only a few pixel. The uncertainties on the distortion parameters should be close to zero. For more information about how to edit the initial camera parameters: 202560169.

Fisheye lens: The principal point should be around half the resolution of the camera. The optimized values for the affine transformation parameters C and F should be close to each other. The optimized values for the affine transformation parameters D and E should be close to 0. The uncertainties on the affine transformation parameters C, D, E and F should be only a few pixel. The uncertainties on the polynomial parameters should be close to zero. For more information about how to edit the initial camera parameter: 202560169.

For more information: Camera Optimization section in the Quality Check table.

Parameters Correlation: The highest quality calibration is characterized by decorrelated parameters. However, having some type of correlation between parameters is sometimes expected: 

  • In projects with nadir images, a correlation between the radial distortion parameters and between the coordinates of the principal point is expected. 
  • In close-range projects with oblique images, a correlation between the focal length and the coordinates of the principal point is expected as well as between the coordinates of the principal point and the tangential distortions.

What helps with decorrelation:

  • Accurate camera positions (e.g. RTK, or at least GPS), combined with GCPs.
  • MTPs at multiple depths and near image edges for oblique projects.
  • Different camera orientations (e.g. rotate at ends of grid rows, so images will be at 180°), especially when the coordinates of the principal point are correlated.
  • Using All Prior for the internal parameters: 205327965

  

2D Keypoints Table - 2D Keypoint Matches graph

See the Matching section in the Quality Check.

The 2D Keypoints Table displays some statistics of the keypoints and the matches of the project. Keypoints are points of interest (high contrast, interesting texture) on the images that can be easily recognized. The number of keypoints depends on:

  • The size of the images.
  • The visual content.

A 14MP image will generate between 5'000 and 50'000 keypoints per image. If the number of keypoints is less than 1'000, the image may not have enough content to be calibrated (Images section in the Quality Check). The number of matches will be very low if:

  • Number of keypoints is low.
  • Visual content is too repetitive.
  • Overlap between the images is too low.
  • There are too many changes in the scene during the image acquisition (moving shadows, cars, etc).

The minimum number of matches to calibrate an image is 25. The recommended number of matches is at least 1'000 per image. In the 2D Keypoint Matches graph, it is possible to visualize the areas with weak matches. It might be necessary to acquire images again in these areas to increase the image overlap.

The 2D Keypoint Matches graph displays the matches between the images and the uncertainties of the relative camera positions. Each block is displayed with a different color. For more information about the blocks: Dataset

The Uncertainty Ellipses describe how precisely each image is located with respect to the other images by means of the Manual and Automatic Tie Points. Usually the ellipses in the center of the project are smaller than at the outside, as these images have more matches that bind them to the surrounding images. Large ellipses in parts of the project may indicate problems calibrating these parts of the project and typically correspond to areas with few matches.

 

Relative Camera Position and Orientation Uncertainties

The mean relative camera position uncertainty should be within a few multiples of the GSD, the mean orientation uncertainty should be less than 0.1 degree. A large sigma may indicate that some parts of the project are not well calibrated.

In projects with RTK-GPS or with many GCPs the relative camera position and orientation uncertainties reported in this table may be worse than the ones reported in the table “Absolute Camera Position and Orientation Uncertainties”. This is expected as this table provides information on how well the Tie Points constrain the images.

 

2D Keypoint Table for Camera

This table is displayed if more than one camera model is used. It displays some statistics of the keypoints and the matches for each camera model. The same analysis as above applies.

 

Median / 75% / Maximal Number of Matches Between Camera Models

This table is displayed if more than one camera model is used. It shows the median, 75% quartile, maximum number of matches between two different cameras models and between the images of one camera model. The same analysis as above applies. 

 

3D Points from 2D Keypoint Matches

Multiple 2D matching keypoints are triangulated together using the camera parameters to generate a 3D point. 3D points generated from 2-3 images are less precise than 3D points generated from a higher number of images. 

 

Manual Tie Points

This section is displayed if MTPs have been used. For a good calibration Manual Tie Points should have an error around 1 pixel. The Manual Tie Points image marks should also be Verified. High Projection Error and/or many not Verified marks may indicate an issue with the marking or the calibration.

 

Ground Control Points

This section is displayed if GCPs have been used. GCPs are used to assess and correct the georeference of a project. For more information: Georeferencing. 3 GCPs is the minimum to geolocate (scale, orient, position) a project. Optimal accuracy is usually obtained with 5 - 10 GCPs. For more information about GCPs: 202557489.

For more information about how to include GCPs in the project: 202560239.

 

Scale Constraints

This table is displayed if a the project has Scale Constraints. It displays the Computed Length Error for the scale constraint. Verify that the mean error is close to 0 and the sigma error is close to 1. If not, verify that:

  • The Initial Length accuracy for the scale constraint is correct.
  • The image geolocation accuracy is correct if the images are geolocated.
  • The GCPs accuracy is correct if the project has GCPs.

 

Orientation Constraints

This table is displayed if a the project has Orientation Constraints. It displays the Computed Angular Error in degrees for the orientation constraint. Verify that the mean error is close to 0 and the sigma error is close to 1. If not, verify that:

  • The Angular Accuracy for the orientation constraint is correct.
  • The image geolocation accuracy is correct if the images are geolocated.
  • The GCPs accuracy is correct if the project has GCPs.

 

Absolute Geolocation Variance

This table displays the percentage of geolocated and calibrated images with a geolocation error in X,Y,Z within a predefined error interval. There are ten predefined intervals between -1.5 and 1.5 times the maximum accuracy Amax of all images. If the percentage of images with an error lower than -1.5 × Amax or higher than 1.5 × Amax is big, the Accuracy values might not have been set correctly. Verify if the Accuracy of the image geolocation and the GCPs needs to be adapted.

This table also assess the quality of the image geolocation. High percentage of images with high error may indicate:

  • Noise in the GPS device.
  • Poor synchronization between the GPS device and the camera.
  • Errors in the geotagging process.

 

Geolocation Coordinate System Transformation

This table is displayed if a Site Calibration transformation is defined and enabled and if the output coordinate system is an arbitrary coordinate system. It defines the transformation from the input coordinate system to the output arbitrary coordinate system.

It can be used in projects where the images are in a known coordinate system and no GCPs are used in order to define the transformation to an arbitrary output coordinate system. For more information: 204515169.

 

Relative Geolocation Variance

This table displays the percentage of geolocated and calibrated images with a Relative Geolocation Error between -1 and 1, -2 and 2 and -3 and 3. High percentage of images with a Relative Geolocation Error lower than -3 or higher than 3 may indicate an incorrect value for the Accuracy of the image geolocation (given by the user). Verify if the Accuracy of the image geolocation and the GCPs needs to be adapted.

In projects without GCPs:

  • If less than 99.6% of the geolocated and calibrated images have an error between -3 and 3, then the geolocation Accuracy might be overestimated. Try to increase the value for the geolocation Accuracy.

 

Rolling Shutter Statistics 

This section is displayed if the Linear Rolling Shutter model has been selected. The graph demonstrates the vector between the initial camera position and the final camera position during the readout time (time needed to take the image). The vector should have the same direction as the flight direction. If this is not the case, there might be a problem with the calibration. Check the number and quality of matches, camera parameters, overlap of the project.

The table displays the

  • Median camera speed
  • Median rolling shutter displacement(during sensor readout)
  • Median rolling shutter time

Make sure that the Median camera speed corresponds to the drone speed during flight. If this is not the case, there might be a problem with the calibration. Check the number and quality of matches, camera parameters, overlap of the project.

 

 

Was this article helpful?
2 out of 2 found this helpful


Powered by Zendesk