The study site was located in the northwest corner of Webster County, Iowa, USA (Fig. 1). Farming is the dominant enterprise in Webster County. The crop rotation was a corn–soybean rotation. Total annual precipitation is about 873 mm, 70 % of which usually falls between April and September. The area of interest (AOI) within the field survey was a small reach (1.9 × 1.3 m; 2.47 m2) of a 150 m long ephemeral gully oriented south to northwest with eroding Clarion loam (fine-loamy, mixed, superactive, mesic Typic Hapludolls) at the upper (south) extent, Terril loam (fine-loamy, mixed, superactive, mesic Cumulic Hapludolls) on the intermediate slopes, and Webster clay loam (fine-loamy, mixed, superactive, mesic Typic Endoquolls) within the lower relief section of the field (Fig. 2). Within the AOI, the soil was Clarion loam (Fig. 2b).

Field surveys were conducted using three independent modes: ground-based and terrestrial lidar, ground-based and terrestrial photogrammetry, and airborne photogrammetry. The surveys yielded 12 datasets (Table 1; dashed lines represent the delineation between survey modes). The terrestrial lidar was considered the reference dataset due to perceived superior accuracy. All surveys were run independently of each other and completed on the same day. Each dataset was represented using the NAD83 UTM 15N coordinate system.

In this study, the terrestrial lidar point cloud was generated using Topcon ScanMaster software (https://www.topconpositioning.com/software/mass-data-collection/scanmaster), all terrestrial photogrammetric point clouds were generated using PhotoModeler Scanner software (www.photomodeler.com/products/scanner/default.html), and all airborne photogrammetric point clouds were generated using Pix4Dmapper Pro software (https://pix4d.com/pix4dmapper-pro/). It is acknowledged that the selection of input parameters influences the sampling intensity and local elevation variance; however, the quantification of the influence of input parameters on the output is beyond the scope of this study. Here, similar survey methods used the same input parameters to generate point clouds.

To ensure the highest three-dimensional alignment among all point clouds and consistent spatial coverage by all methods, a three-step preprocessing approach was developed. First, the set of four channel GCPs (white square targets in Fig. 2b) were used to generate a rectangular polygon to subset the point clouds in all surveys and ensure that all surveys cover exactly the same ground position. Second, a smaller polygon subset was created by generating a polygon with a 50 mm reduction on all sides. This was performed to exclude areas close to the channel GCPs and ensure the elimination of shadowing and occlusion created by the slightly elevated channel GCPs. Third, the sampling void within the lidar dataset, created by the presence of a thin film of water within the channel (Fig. 3a), was manually digitized into another polygon and used to remove points from all photogrammetrically generated datasets to ensure uniformity among all datasets. Essentially, instead of using interpolated data within this void in the reference dataset, we simply placed a void in all datasets; therefore, we do not introduce bias into the calculations with regard to the water film void within the lidar data (e.g., Gómez-Gutiérrez et al., 2014).

Subsequently, since each surveying method was performed using the same set of field and channel GCPs, a manual inspection of measured points located coincident with channel GCPs (white square targets in Fig. 2b) was used to generate planes (10 total), one for each dataset with the exception of the Fixed_61m and Fixed_122m datasets, in which no points were located on top of the channel GCPs. The four GPS-surveyed coordinates of the center location of the channel GCPs were used to fit a reference plane to be matched by all surveys (black squares in Fig. 4). Three-dimensional locational differences between the reference plane generated using the GPS survey (black squares in Fig. 4) and the planes of each surveyed dataset (lidar and photogrammetry) were calculated using the iterative closest point (ICP) algorithm (Besl and Mckey, 1994; James and Robson, 2012; Micheletti et al., 2015) implemented in Matlab (MathWorks Inc., Natick, Massachusetts). Since no scale issues were observed, no scaling factor was implemented in the ICP. The ICP algorithm minimizes the locational differences between two sets of three-dimensional point clouds and outputs a 3 × 3 rotation angle matrix, R, and a 1 × 3 translation vector, T (Eq. 1). These matrices were used to three-dimensionally transform, through rotation and translation, the measured point clouds to best match the reference plane: XYZref=T+Rωφκ⋅XYZmeas.

One of the problems in looking at the data in the original point cloud format was the large difference between the total number of points within datasets (i.e., hundreds to millions), which tends to bias the results; therefore, in the sections that follow, the investigation of point cloud data is complemented by an analysis of gridded data. In the point cloud analysis, each point within the photogrammetry surveys was compared to that within the lidar survey. The analysis is carried out in two ways: point normal to the plane (each photogrammetry point was projected normal to a fitted plane of lidar points at the nearby position) and spot elevation to triangular irregular network (TIN; each photogrammetry point was projected up or down to intersect the TIN surface of the lidar points). In the gridded data analysis, a volume difference and cross-sectional assessment are performed. This type of data structure (i.e., raster grid) is a common format used to estimate soil loss volumes and generate cross sections for modeling exercises (Dabney et al., 2014). The gridded data introduce a common means of discussing differences between the survey methods.

Since there is no true standard of judging the performance of the measurements provided herein, a system of scoring was developed to grade the photogrammetry data with regard to the lidar data (Table 2). Scores between 1 and 11 were assigned to each evaluation category using both point cloud and gridded data. For example, the difference in absolute volume was assigned decreasing scores (1 → 11) for increasing volume difference, and correlation coefficients were assigned decreasing scores (1 → 11) for decreasing correlation. Put simply, if a variable had a positive impact, it received a higher score and all variables were equally weighted. Each score is defined in Table 2.

The point clouds evaluated here had a large variability in sampling intensity (from 1 to > 250 points cm-2; Fig. 6). The difference in point sampling influences micro-topography and apparent roughness and may lead to bias in volume estimation. If the surface is rough, the effect will be greater (Fig. 7). Point counts are very low for the fixed-wing flights in comparison to the other methods, and the sparse point count leads to interpolation (filling) during raster gridding (Fig. 6), while the elevation range is very similar for all methods with the exception of the fixed-wing datasets (Fig. 7).

Point pattern analysis was examined using the G and F functions (Fig. 8). Each analysis tests through an assumption of complete spatial randomness (homogeneous Poisson process), although interpretations for clustering and regularity are in opposition for each test (i.e., the regular point spacing outcome for the G function is below the gray confidence bounds, and for the F function it is above them). At first, the confidence bounds (gray envelope bounding the theoretical values; red dashed line) show that the Fixed_122 has a sparse point count. Looking at the G function results, the data are clustered at distances of 0.02 m (Fixed_122), 0.002 m (Quad_20), and 0.001 m (Ground_8A) and then regularly distributed. This indicates that small distances occur less often than expected under the assumption of spatial randomness. The F function results indicate that the data are randomly distributed to 0.06 m (Fixed_122) and 0.025 m (Quad_20), and clustered for Ground_8A (i.e., at short distances, fewer points are encountered than for a random pattern); however, the scale of r should be acknowledged (< 3 mm). All 12 datasets yielded observed G function values below the theoretical values, indicating a regular sampling pattern. The terrestrial photogrammetric surveys showed slight clustering at small distances (< 3 mm). Therefore, based on these metrics, a regular sampling pattern was observed, indicating that all locations within the study area were sampled with a similar sampling pattern (no areas were oversampled or undersampled) for all 12 datasets in this study.

Graphical representations of both spot elevation to TIN and normal to fitted plane for the fixed-wing flight at 122 m of altitude (Fixed_122; Fig. 9) and the four-photo pair (Ground_8A; Fig. 10) are provided for comparative purposes. In the Fixed_122 spot and normal analysis (Fig. 9), the range was larger for the spot (±0.06 m) than for the normal (±0.02 m); the residuals suggest a nonlinear response, potentially attributed to over-smoothing of the surface and lack of preprocessing (slope of the blue line). Residuals for the Ground_8A (Fig. 10) have a constant variance and do not show an x or y axis bias. Clearly, outliers can be identified (Fig. 10; ∼ 0.01 m) and it seems as though the spot analysis provides a larger variance (i.e., amplified residual signature) than the normal analysis, which may be attributed to the 2.5 cm sphere used to define the plane in the normal analysis (i.e., smoothing).

Very similar results were obtained with these two methods. All datasets had negligible mean displacement in the x and y directions (Table 3). The standard error of Z in the normal to plane analysis (Table 3) ranged from 0.3 mm (Ground_8A, Ground_6A, Ground_4B, Ground_4C, Ground_2B) to 2.9 mm (Fixed_122). The mean displacement for the normal to plane analysis in the z direction (Table 3) ranged from 0.2 mm (Quad_35) to 7.4 mm (Fixed_61). For the vertical spot to TIN analysis (Table 4), the standard error ranged from 0.4 mm (Ground_8A, Ground_6A, Ground_4B, Ground_2B) to 2.1 mm (Fixed_122), and the mean displacement ranged from 0.1 mm (Ground_8A) to 35 mm (Fixed_122). With the spot to TIN analysis, the lidar point cloud was evaluated, resulting in a standard error of 0.3 mm and a mean displacement of 0.1 mm; both results are similar to Ground_6A and Ground_8A for standard error and Quad_35 and Ground_4A for mean displacement. The mean elevation difference between photogrammetry and lidar was approximately 5.3 mm; between the quadrotor and lidar it was 3 mm, and between the fixed wing and lidar it was 25 mm. Perhaps the 10-fold increase derived from the fixed-wing flights was simply because they were not preprocessed (affine transformation was not applied); however, this result shows the importance of common reference points between surveys.

Most of what is reported here is due to point cloud alignment during the preprocessing step discussed earlier (i.e., GCP alignment for each, except fixed-wing flights). The mean elevation difference was approximately 5 mm for all datasets (Tables 3 and 4). Positive values indicate that the lidar data was, on average, higher than the photogrammetry data and negative values indicate that the lidar data was, on average, lower than the photogrammetry data. Similarly, when contrasting the photogrammetry elevation with the elevation of the normal point through linear regression (Table 3), the slope of the fitted line is very close to unity for all methods except Fixed_122, indicating spatially variable discrepancies (higher elevation differences at one region than the rest of the study site).

The conversion of point clouds with irregularly spaced points and spatially varying sampling intensity point clouds into regular raster grids affected each dataset differently (Fig. 11). For example, the lidar dataset contained a high sampling intensity (>100 points cm-2) with relatively large elevation variability in the points within a raster grid cell. Therefore, the interpolation procedure generated a significantly smoothed surface (e.g., Eitel et al., 2011). Conversely, the fixed-wing surveys had a low sampling intensity and the interpolation procedure linearly filled the gaps, potentially generating a significantly smoothed surface that differs from the “natural” surface. The variance shows that roughness is similar for most of the surveys, with the exception of Fixed_122 for which the variance is extremely low (0.0005; Table 5). This result merely points out that the Fixed_122 is extremely smooth in comparison to the other surveys due to the low sampling density (Fig. 6) and enhanced interpolation between points for the high spatial resolution raster grid (0.005 m cell size).

One of the most important measurements for gully monitoring is the volume difference between surfaces (Table 6). Given the small scale of this type of erosional feature (on the order of a few centimeters), it is vital to have a good understanding of the expected error for each method. Among similar collection methods (terrestrial photogrammetry), the absolute volume difference (Table 6) ranged from 1 to 52 % in comparison to Ground_8A, although these differences were extremely small in reality (i.e., a range of 0.0062 to 0.0105 m3). The performance ranking, in terms of absolute volume difference, was Quad_35, Ground_8A, Ground_4A, Ground_6A, Quad_20, Ground_4C, Ground_4B, Ground_2B, Ground_2A, Fixed_61, and Fixed_122 (10 to 164 % absolute volume difference for Ground_8A and Fixed_122, respectively, in comparison to Quad_35; Table 6). The variances in elevation difference between the lidar and photogrammetry data were all quite similar (Table 5), with the exception of the fixed-winged flights (effect of interpolation). In terms of the elevation range of the data (Table 5), the terrestrial photogrammetry and quadrotor flights were within 1.15 % of the lidar range and appeared to be very similar (Fig. 6), with the exceptions of Ground_2A (too rough; 8.2 % roughness increase) and Fixed_122 (too smooth; 84 % roughness decrease).

In the gridded elevation evaluations (min, max, mean; Table 5; Fig. 12), the absolute difference from lidar was less than 0.02 %, and these differences were only seen in the fixed-wing flights. The variance (i.e., roughness), however, shows that the absolute differences from lidar were 131 % (Fixed_122), 23 % (Fixed_61), and 9 % (Grround_4A). A comparison of the elevation information (Table 7; Fig. 12) between photogrammetric cross sections and lidar cross sections through linear regression indicates a coefficient of determination larger than 0.98 for all datasets, excluding the two fixed-wing flights. The standard error for this regression was less than 10 mm for Quad_20, Quad_35, Ground_2B, Ground_4B, Ground_4C, Ground_6A, and Ground_8A. Following that, Ground_2A and Ground_4A had a standard error of approximately 17 mm and the two fixed-wing flights had a standard error of 25 mm. The average area percent differences for all cross sections were within 1.5 %, while the two fixed-wing flights had 3 % (Fixed_61) and 8 % (Fixed_122). It is important to mention that the range of area percent difference is within ±2 %, while the fixed-wing systems had up to 15 % difference. The error is huge, for instance, if this dataset was intended to be used for the development, calibration, and validation of a soil erosion model.

The gridded data performance was led by Ground_8A and Quad_35. Combined category score points ranged from 49 (Ground_8A) to 6 (Fixed_122), terrestrial photogrammetry ranged from 49 to 20, quadrotor ranged from 42 to 35, and the fixed wing ranged from 9 to 6 (Table 8). For the point cloud analysis, the scoring results were, for the most part, very similar. The terrestrial photogrammetry surveys all score very high, with the quadrotor falling in the middle and the fixed wing at the bottom. Overall, scoring ranged from 140 to 24 with the terrestrial photogrammetry leading the group. As the number of photos increased, so did the sample density; however, the four-photo pair (Ground_8A) was less dense than the six-photo pair (Ground_6A) or the two-photo pair (Ground_4B, Ground_4C), which may be associated with higher accuracy in pixel matching or the addition of inferior images to the project. However, it is noteworthy to add that sampling intensity increases as the UAV altitude decreased, although the Quad_35 outperformed the Quad_20 in a number of categories.

Two photogrammetric software packages (Pix4DMapper Pro and PhotoModeler Scanner) were used to generate solutions for the UAV platform and terrestrial photogrammetry surveys. Pix4DMapper Pro uses a larger number (≥ 3) of overlapping photos, while PhotoModeler Scanner can offer solutions with only two overlapping photos. These software packages differ in the level of user control options for processing and point cloud generation. Point clouds processed by different software packages and/or users could yield very different solutions; however, this aspect was not investigated here.

An alarming concern in this analysis was the realization that rotation and translation (Fig. 4) were required to ensure that all data were properly aligned. The lidar global coordinates were the same as those used for the fixed-wing and quadrotor flights (i.e., field GCPs). The channel GCPs were also utilized to optimize the lidar point cloud solution. In all terrestrial photogrammetry point cloud solutions, the same set of global coordinates (channel GCPs) were used. One might expect the solutions to converge without the need to manipulate the point clouds in postprocessing; however, not one of the solutions contained the exact positions of the channel GCPs, including the solutions generated using the same platform but with varying processing parameters. For example, three-dimensional registration discrepancies were detected between lidar solutions and solutions from the quadrotor platform at 20 and 35 m, the fixed-wing platform at 61 and 122 m, and the terrestrial photogrammetry surveys. This realization presents extreme difficulty for temporal studies of ephemeral erosion processes, no matter the choice of resolution, platform, or processing parameters.

Initially, an attempt was made to analyze all datasets in their original form; however, two limitations to the approach were noted: the lack of three-dimensional registration between the datasets skewed efforts to quantify individual point accuracy and, more importantly, reduced confidence in the geomorphology information generated. The difference in point sampling density, ranging from hundreds (fixed-wing platform) to millions (lidar), biased the results. Therefore, the discussion presented herein relates to solutions that have been altered from the original solutions produced by the respective software packages. A comparison of the measured F and G functions with the estimated theoretical spatial distributions under the complete spatial randomness assumption suggested that all datasets did not present any spatial clustering, therefore indicating that the study site was sampled uniformly (regular spatially distributed data throughout the study site; Fig. 8). The main difference between datasets was the scale, at which terrestrial photogrammetry and quadrotor airborne photogrammetry yielded sub-centimeter distances as a result of the large number of samples when compared to the fixed-wing airborne photogrammetry. The results from the point evaluations suggest that the Fixed_122 data are clustered below r=20 mm; however, the very same point cloud was interpreted to a 5 mm raster, so part or all of the metrics associated with these flights may be biased.

The normal projection and vertical spot analysis place the mean elevation for quadrotor flights at 2.9 mm below the lidar, for terrestrial photogrammetry at 5.0 mm below the lidar, and for the fixed-wing flights at 16 mm above the lidar. The range of cross-sectional elevation from all terrestrial photogrammetry was within 14 % of the lidar and, if we drop the rough sample (Ground_2A), the difference falls below 0.24 % (e.g., Gómez-Gutiérrez et al., 2014; Di Stefano et al., 2016). This is all seemingly acceptable for terrain mapping and perhaps even process development; however, if it is assumed that the bulk density of the soil is 1500 kg m-3, then the soil mass difference for the quadrotor flights is 5.6 kg (erosion and elevation depletion mass). For terrestrial photogrammetry it is 9.8 kg (erosion and elevation depletion mass), and for the fixed-wing flights it is 44.8 kg (deposition and elevation enhancement mass). Furthermore, if the size of the study area (2.47 m2) is projected onto a 1 ha field, the elevation distortion is anywhere from 23 000 to 181 000 kg of material, which is substantial. Another way to visualize these data would be to look at the area calculations, from which there is a 1.064 % decrease in cross-sectional area (Fig. 10) for Ground_8A over that of the lidar. One percent is a very low difference and amounts to an impacted area of 0.03 m2. Again, when considering the site projected onto the 1 ha field, the impacted area is on the order of 50 times the original measurement area (121.5 m2). These findings are reflective of the decision to keep all collected data and further promote the importance of data uncertainty analysis (Wheaton et al., 2010).

Another interesting finding was the difference between solutions from terrestrial photogrammetry (varying number and/or orientation of photo pairs). The solutions from the Ground_2A and Ground_2B datasets both used only one photo pair; however, the results from the analysis indicate a superior solution generated from the Ground_2B pairing (i.e., upstream or downstream orientation; Table 1). This could be potentially attributed to the orientation of the images in relation to the channel, in which differences in illumination could hamper the photogrammetric process of automated pixel matching between each photo pair (Marzolff and Poesen, 2009). Additionally, increasing the number of photo pairs used in the solution seems to yield improved solutions. Results from the volumetric analysis show that the Quad_35 was a very close approximation (0.0002 m3) to the lidar, and Ground_4A (i.e channel left and right with corner left and right photo pair; Table 1) was within 0.0001 m3 of the Quad_35. Solutions obtained with the “corner left and right photo pair” tended to improve the estimates. However, within the overall assessment of data performance, the Ground_4A data finished eighth, and small differences between Ground_6A and Ground_8A suggest a potential threshold in the number of photo pairs to which including additional photo pairs adds marginally to the final quality of the solution. Whether or not the datasets were adjusted spatially in accordance with the channel GCP positions, the absolute volume differences were similarly ranked between photogrammetry datasets. Point clouds built from higher photo pairs and flights at lower altitudes produced better results when compared to terrestrial lidar.

The long-term photogrammetric monitoring of ephemeral gullies should be performed with systems and procedures that strongly consider the following.

Provide a minimum sampling density to capture the overall and local terrain characteristics based on the study objectives (i.e., a temporal headcut migration process understanding may require data with sub-centimeter resolution, while a temporal channel meander process understanding may only require decimeter resolution; James and Robson, 2012; Gómez-Gutiérrez et al., 2014). The planning phase of the project must consider the physical characteristics of the process to be investigated, the study site physical and environmental variables, and the available hardware and software.