This article relates to processing large datasets in PIX4Dmapper specifically. The number of images PIX4Dmapper is capable of processing in a single project depends on the hardware resources available to the software. See our "Hardware and Pix4D" article for a detailed discussion about the role of hardware and processing.
Projects larger than 2000 images will generally require a premium workstation-class computer that has high-quality GPU and CPU along with 128GB of RAM. Projects this large can fail due to insufficient storage or RAM so it is important to ensure best practices are followed in order to avoid loss of work.
Large datasets can be taken with:
- One flight for aerial projects.
- Multiple flights for aerial projects.
- A car for terrestrial projects (mobile mapping).
- A tripod or by hand for terrestrial projects.
If the dataset consists of more than 2'000 images, it is recommended to divide the dataset into multiple projects and process them separately and then merge them.
For more information: Hardware components usage when processing with Pix4D
Two cases can be distinguished:
Having image geolocation
Not having image geolocation
When having image geolocation, the Split into Subprojects option can be used to create the multiple subprojects.
1. Create a new project that consists of all the images of the large dataset. For more information: Step 2. Creating a Project.
2. Split the project into subprojects. For more information: How to split a project into subprojects.
3. Run step 1. Initial Processing for each sub project.
4. Check the Quality Report for each subproject. For more information about how to analyze the Quality Report: Quality report specifications:
- If the quality of the reconstruction of all subprojects is good and consists of one single block, go to step 5.
- If the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: Quality Report Help.
5. Merge the subprojects. For more information: Merging projects.
6. (optional) Click Process > Reoptimize.
When there is no image geolocation, the dataset should be divided into subprojects manually. For more information: How to split a project into subprojects.
Having more than 3 GCPs per Subproject
1. Create a new project for each subproject. For more information: New project in PIX4Dmapper.
2. Run step 1. Initial Processing for each subproject with the same Keypoints Image Scale for all subprojects. For more information: Menu Process > Processing Options... > 1. Initial Processing > General.
3. Add at least three GCPs on each subproject. The GCPs should be homogeneously distributed on each sub project. For more information: How to include GCPs in the project.
4. Click Process > Reoptimize for each subproject.
5. Verify the Quality Report for each subproject. For more information about how to analyze the Quality Report: Quality report specifications:
- If the quality of the reconstruction is good for each subproject and consists of one single block, go to step 6.
- If the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: Quality Report Help.
6. Merge the sub projects. Fore more information: Merging projects.
7. (optional) Click Process > Rematch and Optimize.
Having less than 3 GCPs per Subproject
1. Create a new project for each subproject. For more information: New project in PIX4Dmapper.
2. Run step 1. Initial Processing for each subproject with the same Keypoints Image Scale for all subprojects. For more information: Menu Process > Processing Options... > 1. Initial Processing > General.
3. Add GCPs (How to include GCPs in the project) and/or Manual Tie Points (How to import and mark Manual Tie Points (MTPs)) on each subproject. The GCPs and/or Manual Tie Points should be homogeneously distributed in the common area of the subprojects. There should be at least three common GCPs and/or common MTPs marked on each subproject.
4. Click Process > Reoptimize for each subproject.
- Use the same label for the common GCPs/Manual Tie Points for all subprojects. For more information about how PIX4Dmapper treats the MTPs during merging: Manual Tie Points and GCPs in merged project.
- If less than three GCPs are used for the merged project, then they are not taken into account as GCPs.
5. Verify the Quality Report for each subproject. For more information about how to analyze the Quality Report: Quality report specifications:
- If the quality of the reconstruction is good for each subproject and consists of one single block, go to step 5.
- If the Quality Report indicates that there are problems with the reconstruction of one subproject, check the Quality Report Help: Quality Report Help.
6. Merge the subprojects. Fore more information: Merging projects.
7. (optional) Click Process > Reoptimize.
Hello, We are attempting to process very large UAV dataset (~ 400 sqkm) captured using Sony A7R2 (42 MP). This project included multiple phase areas with multiple flights - individual phase areas were successfully processed using Pix4D Mapper Pro and now we attempting to process one single monolithic orthomosaic, DSM, DTM & contours.
Since we have processed individual phases; we are attempting to merge the Initial results and process the monolithic datasets.
Fortunately we have a very high end hardware:
- Dual Xeon Gold 6136 CPU @ 3.00 GHz (2 x 24 cores = 48 Cores)
- 1024 GB (1 TB) DDR-4 RAM
- 8 TB SATA SSD (RAID 5 configured - 6 TB available)
- NVIDIA P5000 GPU (configured as 3D App - Game Development - as recommended by Pix4D)
Undesirably, Pix4D mapper is unable to merge these datasets and bails out while trying to merge the projects; i.e., while merging the projects, it gets stuck at Generating the Quality Report (displays 0%) and after 6-7 hours simply shuts down Pix4D. The log file too does not register any error.
We have done this twice now & are unable to figure out the reasons. The hardware usage maximum is 199 GB RAM and max 20% CPU usage with hardly any GPU usage.
We have complied all the initial reports of individual phases and flights along with the P4D file as well as the log file for your analysis.
Since the size is about 50 MB - the Google Drive link is:
[link removed]
I hope you will be able to download these files and critically analyse them.
We look forward to your valuable support and insights, for us to produce such a large UAV datasets for the first time in India.
Thanks a ton in advance!
Best regards,
Achyut Rao
Mobile: +91-9810473821
Hi Achyut,
In your case, if you want to increase the processing speed of a project, you could also try changing the processing options to reduce processing time: https://support.pix4d.com/hc/en-us/articles/202560419. In the case where the dataset is very good, reducing the processing options will not have an impact or very little on the results.
Also, we would recommend checking the suggestions in the How to change the Processing Options to generate fewer 3D Points article and process the project again.
Best,
We are having the same problem trying to process several large datasets (though not a large as yours) where it crashes out while generating the quality report at the end on step 1, but there are no warnings or indications of a problem in the log file. Did you ever find a solution to the problem?
Thanks
Hi Kevin,
Does the software crash, reboot itself, or stop at a particular step? Which version of Pix4Dmapper are you using? In both cases; yours and Achyut I would recommend unselecting Generate Orthomosaic Preview in Quality Report and reprocessing the project again.
Let me know how it goes.
Best
We are using version 4.4.12. We have tried turning off the ortho preview option, turning Nvidia Cuda cores on and off, and run several different data sets on two different machines but it still crashes out in the quality report generation phase of step one. We have run several large projects in the past, on versions prior to 4.3.33, with no problems so I suspect there's something going on in one of the updates. I already have a support ticket opened (#72690) for the issue but found Achyut's post which sounded like they had the same problem and just wondered if they had already found a resolution.
Hi Kevin,
I see in the ticket #72690 that you are in contact with Holden and now you're reprocessing the project with a new processing setup. Therefore we're waiting for the updates. If the issue persists, we'll help troubleshoot further.
Best
Same issue here (#72487).
I am processing 8,614 (20MP) photos. Keep failing at Step 3.
The error is:
[Error]: Load point cloud: the mean resolution was not set or invalid!
[Error]: Point cloud loader: Initialisation failed with error: [Load point cloud: the mean resolution was not set or invalid!]
[Error]: Load point cloud failed!
Hello Friends,
Apologies for delayed response.
Based on our various interaction with Pix4D technical team and the suggestions offered, we did not process the large dataset using Pix4D. I am not too sure if it would be appropriate to name another competing software on this forum - folks can PM me on arao@geoepc.com to know more on this issue. We successfully created a 400 sqkm monolithic orthomosaic using another software.
Processing large to very large datasets using Pix4D is challenging - the resources required are far too large, with a workstation as powerful as:
Dual Xeon Gold 6136 CPU @ 3.00 GHz (2 x 24 cores = 48 Cores)
- 1024 GB (1 TB) DDR-4 RAM
- 8 TB SATA SSD (RAID 5 configured - 6 TB available)
- NVIDIA P5000 GPU (configured as 3D App - Game Development - as recommended by Pix4D).... too failed.
Small to medium size projects with good configuration workstation works fine with Pix4D - however I am yet to find a solution to the watermarks surrounding the buildings in orthomosaic produced by Pix4D - the mosaic editor with Planar defeats the very purpose of creating orthomosaic with GCPs etc.
Thanks & regards,
Achyut Rao
Hi Achyut!
Thank you for not forgetting about us and giving the update.
At the same time, I'm sorry to hear that Pix4D didn't work with your project as expected. However, I would like to ask if you managed to process that project with our Enterprise Solution? The last thing that it's registered in the conversation between you and our Sales Team is that they advise you to process your large dataset with that product.
Thanks in advance,
Regards
Hi Guys,
I'm having the same problem when merging a large project. The Merge gets stuck at "Generating Quality Report (orthomosaic) [Group1](0%)"
I'm merging 3 sub projects of about 3000 photos back into a single project of about 9000 20Mb Photos.
I have also disabled the "Generate Orthomosaic Preview in Quality Report" option and only "1. Initial Processing" box is checked.
I'm on V 4.4.12.
Just to be clear on my workflowup to this point, I :
Any clues?
Thanks
Hi Roland,
Can you inbox me on arao@geoepc.com... we have started to provide data processing services and we maybe able to help you.
Regards,
Achyut Rao
Found it.
--------
Version 4.5
Bug Fix
----------
It was a bug. Back to work....
HI RONALD
I have installed latest version of Pix4dMapper (4.5.2) yesterday and merged 3 sub projects that contains of about 4000 images(20 MP) per each one (totally 11000 images)
But this problem exists yet and system crashes when generating quality report.....
System sps:
windows 8.1 64-bit
64 GB of RAM
GTX1050 TI
INTEL I7-6950X
Roland,
I'm curious if the 4.5.2 update actually solved the problem for you? We did a fresh install of 4.5.2 and re-ran our data set, attempting to merge 4 sub-projects into a single project. It crashed out for us the same as it did before. We have tried to run multiple data sets of varying size through versions 4.3.33, 4.4.12, and now 4.5.2, on two very different systems, and the largest data set we have been able to successfully process thru step 1 is about 7000 50mp images. We were only able to accomplish that on one of our two system (the older and slower of the two) and only with version 4.3.33 with Nvidia Cuda acceleration turned off. The largest data set we were able to process with 4.4.12 has been 381 20mp images. Both systems are running windows 10. The windows application error reports for each crash on both systems point to a library file, libtcmalloc_minimal.dll as the common cause of failure. I would be interested to know if you are still experiencing the crash and if your errors relate to the same library file.
Thanks
Dear Kevin
I had the similar problem and found that the source of error is libtcmalloc_minimal.dll.
I sent all of information about this error to Pix4d support team yesterday and this is their answer:
. There is some issue with quality report generation for big projects when orthomosiac preview is selected. We would recommend you the following:
- Open the Processing Options and uncheck the box Generate Orthomosaic Preview in Quality Report
- Open the menu Process and click Generate Quality Report.
-Start processing the other steps.
Then I tested this instruction and my problem has been solved...
At this time I have a project that has been merged from 3 sub projetcs and it has about 10400 images(20mp).Quality Report has been generated successfully after merging procedure...
Emamzade,
Thank you for your response! We have tried every configuration of the Orthomosaic Preview Option and still have not been able to get thru step 1. Our project has been broken down in to 4 sub-projects each around 7000 50mp images and will only process on one of our systems, using version 4.3.33.
I have a project with over geolocated 3200 images. Following Pix4D recommendations, I split the images into 4 sub-projects. My question is about merging the processed subs. It is unclear from the documentation if it is necessary or not to use mtps (or gcps) to merge the subs back together. I realize you always can, and that might even help, but since came from the same origin, is it a requirement?
Due to the nature of the project, I had only very few GCP's so would have to pull mtp's in the overlap areas. Just not sure if I need to or not.
Hi Austen. This needs to be evaluated on a case by case basis. If you have 32GB of RAM or more you may be able to get away processing as a single project depending on your processing options. GCPs are the best way to ensure that a project lines up when processed in sub-blocks. It is a good bet to place 2-3 MTPs in the areas where they overlap but it is not always possible to know if they are needed.
Same issues with a 20,000 picture project. Crashed Pix4D twice. I divided into subprojects. Each had good results. When I merged them using common GCPs and creating MTPs the merged results were 157 feet off.
Hi Nathan. If you are planning on processing up to step 3 you will still likely experience stability issues. How much RAM do you have? You may want to consider merging the datasets outside of Pix4D. FOr example, merging the orthomosiacs in GIS.
Hi Holden,
I have another project (5000 images more manageable) that is experiencing the same issues. 90% of GCPs come in great. The other 10% very very bad - 2-3 feet. I added MTPs and now those same GCPs are 10-15 feet. I added more MTPs and re ran Step 1. Same GCPs are now 3-5 feet. It seems as though anything over 5000 images is producing poor results.
Hi nathan. Can you confirm you are following the workflow descrbed below?
Can you send a screenshot of what the project looks like after this? I would suggest creating a support request so that we can take a look at things like the quality reports etc.
Holden --- yes I did all of those things (merged many projects and have used Pix4D for years).
I was so upset with the results I put them in the trashcan with a firm press of delete.
Hi,
The preview of the orthomosaic in the initial processing report is a good thing, as it gives you a visualization of the output. I presume Pix4D needs to refine its software code to be more efficient. Usage of 100% CPU will degrade the workstation performance on the long run.
Hi Nathan. Sorry to hear the project was giving you trouble. Large projects can be complex and in some cases frustrating for sure. If you process again and follow the workflow i described above, consider opening a support request and providing a quality report for us to review.
Hello Friends,
We have been using Pix4D since early 2015 and it has improved substantially on various fronts including the capability to process large project sizes. I agree that it can be improved even better.
However, based on our experiences, we find that the image information, its quality, overlaps, etc., do play a major role in performance. Also, the Area of Interest (AOI) also does play an important role e.g., a rural area result /time would be very different from dense urban areas and areas with high rise buildings will give different results/timelines.
Therefore, handling multiple GBs of raw data for a particular type of AOI requires multiple strategies at the flight planning level, proper camera settings (aperture, ISO, high-speed SD card and ability of the camera to properly click with given overlaps at a particular flying altitude). Also, hardware does play an important role as Pix4D is resource hungry. Especially Step # 2 and Step # 3 requires substantial RAM and fast disk(read SSD). Appropriate GPU with updated drivers and updated Windows 10 is important too.
The biggest strength of Pix4D is to generate accurate EO files during Step # 1 - please have a look at the calibrated Kappa values (means proper orientation of images). This substantially improves the time and quality of the final results.
Ultra-high resolution images are heavy in size (we talk GBs and TBs here); normal desktop with standard GPUs will take a very long time to process and may result in multiple crashes leading to frustration. The beauty of drone mapping technologies is to create precision orthomosaic - this does require efforts and investments!
Happy flying!
Achyut Rao
+91-9810473821
arao@geoepc.com