Processing Large Datasets

Pix4Dmapper is able to simultaneously process an unlimited number of images. However, depending on the dataset and the processing resources, there might be some issues with datasets larger than 2000 images. This is an indicative value, as processing depends on the image resolution, image content, overlap between images, chosen output resolution and computer used. 

Large datasets can be taken with:

  • One flight for aerial projects.
  • Multiple flights for aerial projects.
  • A car for terrestrial projects (mobile mapping).
  • A tripod or by hand for terrestrial projects.

If the dataset consists of more than 2'000 images, it is recommended to divide the dataset into multiple datasets, process them separately and then merge them.
Two cases can be distinguished:

Having image geolocation 
Not having image geolocation

Having image geolocation

When having image geolocation, the Split into Subprojects option can be used to create the multiple subprojects.

1. Create a new project that consists of all the images of the large dataset. For more information: 202557309.
2. Split the project into subprojects. For more information: 204217299.
3. Run step 1. Initial Processing for each sub project.
4. Check the Quality Report for each subproject. For more information about how to analyze the Quality Report202558679:

    • If the quality of the reconstruction of all subprojects is good and consists of one single block, go to step 5.
    • If  the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: 202558689.
 
Tip: Add GCPs (202560239) and/or Manual Tie Points (360000395103) on each subproject. The GCPs and/or Manual Tie Points should be homogeneously distributed in the common area of the subprojects. There should be at least three common GCPs and/or common MTPs marked on each subproject. Then, Reoptimize (202557809) and check the new Quality Report (202558319) for each subproject.

5. Merge the subprojects. For more information: 202558529.
6. (optional) Click Process > Reoptimize.

 
Important: The different subprojects created with the Split into Subprojects option contain common images in order to ensure overlap.
 
Note: If there are GCPs in the project, it is possible to include them in the subprojects or in the merged project. For more information about how to include the GCPs in a project: 202560239.

Not having image geolocation

When there is no image geolocation, the dataset should be divided into subprojects manually. For more information: 204217299.

Having more than 3 GCPs per Subproject

1. Create a new project for each subproject. For more information: 115002472026.
2. Run step 1. Initial Processing for each subproject with the same Keypoints Image Scale for all subprojects. For more information: 202557759.
3. Add at least three GCPs on each subproject. The GCPs should be homogeneously distributed on each sub project. For more information: 202560239.

 
Warning: Only GCPs that are common in the different subprojects should have the same label. GCPs that do not represent the same point in the different subprojects should have different labels.


4. Click Process > Reoptimize for each subproject. 
5. Verify the Quality Report for each subproject. For more information about how to analyze the Quality Report: 202558679:

    • If the quality of the reconstruction is good for each subproject and consists of one single block, go to step 6.
    • If  the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: 202558689.

6. Merge the sub projects. Fore more information: 202558529
7. (optional) Click Process > Rematch and Optimize.

 
Important: Projects can be merged only if the GCPs defined in the different subprojects are in the same coordinate system.

Having less than 3 GCPs per Subproject

1. Create a new project for each subproject. For more information: 115002472026.
2. Run step 1. Initial Processing for each subproject with the same Keypoints Image Scale for all subprojects. For more information: 202557759
3. Add GCPs (202560239) and/or Manual Tie Points (360000395103) on each subproject. The GCPs and/or Manual Tie Points should be homogeneously distributed in the common area of the subprojects. There should be at least three common GCPs and/or common MTPs marked on each subproject.
4. Click Process > Reoptimize for each subproject. 

 
Example: For an aerial project:

 
Important:
  • Use the same label for the common GCPs/Manual Tie Points for all subprojects. For more information about how Pix4Dmapper treats the MTPs during merging: 202559439.
  • If less than three GCPs are used for the merged project, then they are not taken into account as GCPs. 

5. Verify the Quality Report for each subproject. For more information about how to analyze the Quality Report202558679:

    • If the quality of the reconstruction is good for each subproject and consists of one single block, go to step 5.
    • If  the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: 202558689.

6. Merge the subprojects. Fore more information: 202558529.
7. (optional) Click Process > Reoptimize.

Was this article helpful?
4 out of 5 found this helpful

Article feedback (for troubleshooting, post here)

2 comments

  • Achyut Rao

    Hello, We are attempting to process very large UAV dataset (~ 400 sqkm) captured using Sony A7R2 (42 MP). This project included multiple phase areas with multiple flights - individual phase areas were successfully processed using Pix4D Mapper Pro and now we attempting to process one single monolithic orthomosaic, DSM, DTM & contours.

    Since we have processed individual phases; we are attempting to merge the Initial results and process the monolithic datasets.

    Fortunately we have a very high end hardware:

    - Dual Xeon Gold 6136 CPU @ 3.00 GHz (2 x 24 cores = 48 Cores)

    - 1024 GB (1 TB) DDR-4 RAM

    - 8 TB SATA SSD (RAID 5 configured - 6 TB available)

    - NVIDIA P5000 GPU (configured as 3D App - Game Development - as recommended by Pix4D)

     

    Undesirably, Pix4D mapper is unable to merge these datasets and bails out while trying to merge the projects; i.e., while merging the projects, it gets stuck at Generating the Quality Report (displays 0%) and after 6-7 hours simply shuts down Pix4D. The log file too does not register any error.

    We have done this twice now & are unable to figure out the reasons. The hardware usage maximum is 199 GB RAM and max 20% CPU usage with hardly any GPU usage.

    We have complied all the initial reports of individual phases and flights along with the P4D file as well as the log file for your analysis.

    Since the size is about 50 MB - the Google Drive link is:

    https://drive.google.com/file/d/1ZEHxfP622v8Rz5nnV7Q_JIE2-gjhJo3j/view?usp=sharing

    I hope you will be able to download these files and critically analyse them.

    We look forward to your valuable support and insights, for us to produce such a large UAV datasets for the first time in India.

    Thanks a ton in advance!

    Best regards,

     

    Achyut Rao

    Mobile: +91-9810473821

  • Blaž (Pix4D)

    Hi Achyut,

    In your case, if you want to increase the processing speed of a project, you could also try changing the processing options to reduce processing time: https://support.pix4d.com/hc/en-us/articles/202560419. In the case where the dataset is very good, reducing the processing options will not have an impact or very little on the results.

    Also, we would recommend checking the suggestions in the How to change the Processing Options to generate fewer 3D Points article and process the project again.

    Best,

Please sign in to leave a comment.