Processing Large Datasets

Pix4Dmapper is able to simultaneously process an unlimited number of images. However, depending on the dataset and the processing resources, there might be some issues with datasets larger than 2000 images. This is an indicative value, as processing depends on the image resolution, image content, overlap between images, chosen output resolution and computer used. 

Large datasets can be taken with:

  • One flight for aerial projects.
  • Multiple flights for aerial projects.
  • A car for terrestrial projects (mobile mapping).
  • A tripod or by hand for terrestrial projects.

If the dataset consists of more than 2'000 images, it is recommended to divide the dataset into multiple datasets, process them separately and then merge them.
Two cases can be distinguished:

Having image geolocation 
Not having image geolocation

Having image geolocation

When having image geolocation, the Split into Subprojects option can be used to create the multiple subprojects.

1. Create a new project that consists of all the images of the large dataset. For more information: Step 2. Creating a Project.
2. Split the project into subprojects. For more information: How to split a project into subprojects.
3. Run step 1. Initial Processing for each sub project.
4. Check the Quality Report for each subproject. For more information about how to analyze the Quality ReportQuality report specifications:

    • If the quality of the reconstruction of all subprojects is good and consists of one single block, go to step 5.
    • If  the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: Quality Report Help.
 
Tip: Add GCPs (How to include GCPs in the project) and/or Manual Tie Points (How to import and mark Manual Tie Points (MTPs)) on each subproject. The GCPs and/or Manual Tie Points should be homogeneously distributed in the common area of the subprojects. There should be at least three common GCPs and/or common MTPs marked on each subproject. Then, Reoptimize (Menu Process > Reoptimize) and check the new Quality Report (Menu Process > Generate Quality Report) for each subproject.

5. Merge the subprojects. For more information: Merging projects.
6. (optional) Click Process > Reoptimize.

 
Important: The different subprojects created with the Split into Subprojects option contain common images in order to ensure overlap.
 
Note: If there are GCPs in the project, it is possible to include them in the subprojects or in the merged project. For more information about how to include the GCPs in a project: How to include GCPs in the project.

Not having image geolocation

When there is no image geolocation, the dataset should be divided into subprojects manually. For more information: How to split a project into subprojects.

Having more than 3 GCPs per Subproject

1. Create a new project for each subproject. For more information: New project in Pix4Dmapper.
2. Run step 1. Initial Processing for each subproject with the same Keypoints Image Scale for all subprojects. For more information: Menu Process > Processing Options... > 1. Initial Processing > General.
3. Add at least three GCPs on each subproject. The GCPs should be homogeneously distributed on each sub project. For more information: How to include GCPs in the project.

 
Warning: Only GCPs that are common in the different subprojects should have the same label. GCPs that do not represent the same point in the different subprojects should have different labels.


4. Click Process > Reoptimize for each subproject. 
5. Verify the Quality Report for each subproject. For more information about how to analyze the Quality Report: Quality report specifications:

    • If the quality of the reconstruction is good for each subproject and consists of one single block, go to step 6.
    • If  the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: Quality Report Help.

6. Merge the sub projects. Fore more information: Merging projects
7. (optional) Click Process > Rematch and Optimize.

 
Important: Projects can be merged only if the GCPs defined in the different subprojects are in the same coordinate system.

Having less than 3 GCPs per Subproject

1. Create a new project for each subproject. For more information: New project in Pix4Dmapper.
2. Run step 1. Initial Processing for each subproject with the same Keypoints Image Scale for all subprojects. For more information: Menu Process > Processing Options... > 1. Initial Processing > General
3. Add GCPs (How to include GCPs in the project) and/or Manual Tie Points (How to import and mark Manual Tie Points (MTPs)) on each subproject. The GCPs and/or Manual Tie Points should be homogeneously distributed in the common area of the subprojects. There should be at least three common GCPs and/or common MTPs marked on each subproject.
4. Click Process > Reoptimize for each subproject. 

 
Example: For an aerial project:

 
Important:
  • Use the same label for the common GCPs/Manual Tie Points for all subprojects. For more information about how Pix4Dmapper treats the MTPs during merging: Manual Tie Points and GCPs in merged project.
  • If less than three GCPs are used for the merged project, then they are not taken into account as GCPs. 

5. Verify the Quality Report for each subproject. For more information about how to analyze the Quality ReportQuality report specifications:

    • If the quality of the reconstruction is good for each subproject and consists of one single block, go to step 5.
    • If  the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: Quality Report Help.

6. Merge the subprojects. Fore more information: Merging projects.
7. (optional) Click Process > Reoptimize.

Was this article helpful?
4 out of 5 found this helpful

Article feedback (for troubleshooting, post here)

12 comments

  • Achyut Rao

    Hello, We are attempting to process very large UAV dataset (~ 400 sqkm) captured using Sony A7R2 (42 MP). This project included multiple phase areas with multiple flights - individual phase areas were successfully processed using Pix4D Mapper Pro and now we attempting to process one single monolithic orthomosaic, DSM, DTM & contours.

    Since we have processed individual phases; we are attempting to merge the Initial results and process the monolithic datasets.

    Fortunately we have a very high end hardware:

    - Dual Xeon Gold 6136 CPU @ 3.00 GHz (2 x 24 cores = 48 Cores)

    - 1024 GB (1 TB) DDR-4 RAM

    - 8 TB SATA SSD (RAID 5 configured - 6 TB available)

    - NVIDIA P5000 GPU (configured as 3D App - Game Development - as recommended by Pix4D)

     

    Undesirably, Pix4D mapper is unable to merge these datasets and bails out while trying to merge the projects; i.e., while merging the projects, it gets stuck at Generating the Quality Report (displays 0%) and after 6-7 hours simply shuts down Pix4D. The log file too does not register any error.

    We have done this twice now & are unable to figure out the reasons. The hardware usage maximum is 199 GB RAM and max 20% CPU usage with hardly any GPU usage.

    We have complied all the initial reports of individual phases and flights along with the P4D file as well as the log file for your analysis.

    Since the size is about 50 MB - the Google Drive link is:

    https://drive.google.com/file/d/1ZEHxfP622v8Rz5nnV7Q_JIE2-gjhJo3j/view?usp=sharing

    I hope you will be able to download these files and critically analyse them.

    We look forward to your valuable support and insights, for us to produce such a large UAV datasets for the first time in India.

    Thanks a ton in advance!

    Best regards,

     

    Achyut Rao

    Mobile: +91-9810473821

  • Blaž (Pix4D)

    Hi Achyut,

    In your case, if you want to increase the processing speed of a project, you could also try changing the processing options to reduce processing time: https://support.pix4d.com/hc/en-us/articles/202560419. In the case where the dataset is very good, reducing the processing options will not have an impact or very little on the results.

    Also, we would recommend checking the suggestions in the How to change the Processing Options to generate fewer 3D Points article and process the project again.

    Best,

  • Kevin Robar

    We are having the same problem trying to process several large datasets (though not a large as yours) where it crashes out while generating the quality report at the end on step 1, but there are no warnings or indications of a problem in the log file. Did you ever find a solution to the problem?

    Thanks

  • Beata (Pix4D)

    Hi Kevin,

    Does the software crash, reboot itself, or stop at a particular step? Which version of Pix4Dmapper are you using? In both cases; yours and Achyut I would recommend unselecting Generate Orthomosaic Preview in Quality Report and reprocessing the project again.

    Let me know how it goes. 

    Best

  • Kevin Robar

    We are using version 4.4.12. We have tried turning off the ortho preview option, turning Nvidia Cuda cores on and off, and run several different data sets on two different machines but it still crashes out in the quality report generation phase of step one. We have run several large projects in the past, on versions prior to 4.3.33, with no problems so I suspect there's something going on in one of the updates. I already have a support ticket opened (#72690) for the issue but found Achyut's post which sounded like they had the same problem and just wondered if they had already found a resolution.

  • Beata (Pix4D)

    Hi Kevin,

    I see in the ticket #72690 that you are in contact with Holden and now you're reprocessing the project with a new processing setup. Therefore we're waiting for the updates. If the issue persists, we'll help troubleshoot further.

    Best

  • Garuda Robotics

    Same issue here (#72487).

    I am processing 8,614 (20MP) photos. Keep failing at Step 3. 

    The error is: 
    [Error]: Load point cloud: the mean resolution was not set or invalid! 
    [Error]: Point cloud loader: Initialisation failed with error: [Load point cloud: the mean resolution was not set or invalid!] 
    [Error]: Load point cloud failed!

  • Achyut Rao

    Hello Friends, 

    Apologies for delayed response.

    Based on our various interaction with Pix4D technical team and the suggestions offered, we did not process the large dataset using Pix4D. I am not too sure if it would be appropriate to name another competing software on this forum - folks can PM me on arao@geoepc.com to know more on this issue. We successfully created a 400 sqkm monolithic orthomosaic using another software.

    Processing large to very large datasets using Pix4D is challenging - the resources required are far too large, with a workstation as powerful as:

    Dual Xeon Gold 6136 CPU @ 3.00 GHz (2 x 24 cores = 48 Cores)

    - 1024 GB (1 TB) DDR-4 RAM

    - 8 TB SATA SSD (RAID 5 configured - 6 TB available)

    - NVIDIA P5000 GPU (configured as 3D App - Game Development - as recommended by Pix4D).... too failed.

    Small to medium size projects with good configuration workstation works fine with Pix4D - however I am yet to find a solution to the watermarks surrounding the buildings in orthomosaic produced by Pix4D - the mosaic editor with Planar defeats the very purpose of creating orthomosaic with GCPs etc.

    Thanks & regards,

     

    Achyut Rao

  • Beata (Pix4D)

    Hi Achyut!

    Thank you for not forgetting about us and giving the update.

    At the same time, I'm sorry to hear that Pix4D didn't work with your project as expected. However, I would like to ask if you managed to process that project with our Enterprise Solution? The last thing that it's registered in the conversation between you and our Sales Team is that they advise you to process your large dataset with that product. 

    Thanks in advance,

    Regards

     

  • Roland Cory-Wright

    Hi Guys,

    I'm having the same problem when merging a large project. The Merge gets stuck at "Generating Quality Report (orthomosaic) [Group1](0%)"

    I'm merging 3 sub projects of about 3000 photos back into a single project of about 9000 20Mb Photos.

    I have also disabled the "Generate Orthomosaic Preview in Quality Report" option and only "1. Initial Processing" box is checked. 

    I'm on V 4.4.12.

    Just to be clear on my workflowup to this point, I :

    1. started with single large project
    2. split it into 10 sub projects
    3. ran step 1, marked the GCPs, and re-optimised.
    4. Merged these into 3 larger projects, and re-optimised. Checked quality reports etc..
    5. Now i'm merging these 3 large sub projects into a single project.

    Any clues?

    Thanks

    Edited by Roland Cory-Wright
  • Achyut Rao

    Hi Roland,

    Can you inbox me on arao@geoepc.com... we have started to provide data processing services and we maybe able to help you.

    Regards,

    Achyut Rao

     

     

     

  • Roland Cory-Wright

    Found it.

    --------

    Version 4.5

    Bug Fix

    • Fixed Quality Report generation option in Initial processing.

    ----------

    It was a bug. Back to work....

Please sign in to leave a comment.