I'd like thank the author for this contribution, which feels particularly relevant as machine learning approaches become increasingly common in geomorphology research. This manuscript explores the evaluation of the accuracy of binary classification of raster datasets, using mapping of bedrock tors as a case study. There is some discussion and presentation of the methods used in classifying real data, distinguishing between bedrock and soil within forest canopy gaps. A series of synthetic datasets are generated with known error properties and bedrock to soil ratios. These synthetic datasets are used to identify trends in two accuracy metrics, F1 Score and Matthews Correlation Coefficient, at a range of bedrock fractions when errors are random, systematic or both. A key result is the elegant demonstration of the unsuitability of using F1 Score as an accuracy metric in most settings, with the alternative Matthews Correlation Coefficient performing more robustly. There is then analysis of changes in bedrock tor shape and size with systematic error, demonstrating the sensitivity of these metrics to shape and size variation.

I believe that this Short Communication fits well into the Earth Surface Dynamics remit, and will be a valuable addition to the broader computational geomorphology literature. Overall, I am in favour of this manuscript being published, and look forward to seeing a final version in due course.

General comments

Overall, this is a well written and presented manuscript which is methodologically and theoretically sound. I do not have recommendations or requests for additional analysis, but have some observations on the presentation and structure of the manuscript that I hope the author will find useful.

Taken as a whole, for a short communication, the manuscript feels quite long and in places dense. One of the things that I struggled with when reading this manuscript, was whether it mattered that this was a manuscript about bedrock tors and the emergence of bedrock. Indeed, the paper's title does not mention bedrock at all. I wonder if it would streamline the paper to remove much of the discussion of bedrock tors, in favour of speaking more generically about binary classification of geomorphic data. I realise there is a fine line to tread here to keep the geomorphic relevance and novelty of the work, but I feel that there are a lot of tangential issues that could be raised around this paper regarding the correct mapping of tors, fuzzy boundaries, mixed pixels, etc, none of which are relevant to the evaluation of binary classifications. Particularly when approaching the impact that feature shape has on classification and error - this is really cool and could have broad implications for a range of applications but risks being bogged down in discussion of bedrock structure and emergence, rather than the more theoretical advancements being focused on.

A potentially less extreme restructure of the manuscript could be to present all of the synthetic landscapes first, without any reference to bedrock, simply setting the work up as an evaluation of binary classification metrics. Then the final section of the manuscript can bring in the bedrock mapping as a case study, thereby tying the synthetic data and theoretical work to a geomorphic context, without introducing as much ambiguity and complexity into the earlier sections of the manuscript.

When presenting the results of the comparison between MCC and F1 scores, in the case of asymmetric metrics (eg Table 2), more could be made of this result. This may be a well known issue in the data science literature, but I think it can be highlighted in a disciplinary context here, and the implications of (mis-)use of F1 scores explored in a geomorphic context.

The work on tor shape in Section 6.2 and Appendix B is very exciting. I would prefer to see it all within the main manuscript rather than split out into an Appendix. There are a lot of interesting factors at play here, where binary classifications have to contend with systematic errors, data resolution and feature size. As data resolution improves, features such as bedrock tors are represented by more pixels, creating the potential for more complex shapes with increased edge to area ratios, and therefore these results are critical in beginning our understanding of how to best classify and evaluate those classifications.

A final recommendation for this manuscript would be the addition of a clear section of recommendations for the types of evaluation method to use under different generalised circumstances. I think this would help with the impact of the manuscript, by giving readers clear direction which they can feed back into their own work.

Line by line comments

In addition to the comments above, I have some more general minor line by line comments:

Line 54 - Missing 'as' between 'long their'.

Line 108-110 - This sentence needs a citation.

Code

It is great to see the code associated with this manuscript available online, with the author detailing how they will archive it once the manuscript is accepted. I have gone through the code on github and it is well written and structured, and after some brief testing it appears to do what is described in the manuscript. It would be ideal if the repository had a licence file included, to ensure that people can use the code in the future. If you need help with software licences, you can look at https://choosealicense.com/ to guide you through the process. To aid reproducibility it would also be helpful to record the numpy, scipy, matplotlib and python versions you are using within the readme, in case future upgrades break things in your code.

-- Stuart Grieve

Review of the manuscript “Short Communication: Evaluating the accuracy of binary classifiers for geomorphic applications” from Rossi Submitted to Earth Surface Dynamics.

I would like to know the author for this very nice paper. This paper attempts to evaluate the accuracy of binary classifiers commonly used in raster datasets, in this case applied to the mapping of rock formations. In this study, the synthetic datasets (with known errors) were used to test two accuracy metrics F1 score and Matthews correlation coefficient on a range of bedrock fractions when errors are random, systematic and both, as well as variable shape and size, showing that Matthews correlation performs more robustly.

I think this short communication fits well into the Earth Surface Dynamics journal. However, I believe that this type of work would fit better in a broader computational journal (e.g. IEEE) as it makes a valuable contribution as machine learning approaches become increasingly common in geomorphological research.

General comments:

The paper is well written and presented. I don't have any further recommendations or additional analysis in the methodology and results section. However, I have very few concerns, mainly related to the way this paper is presented.

The title refers to a "short communication". However, the text is quite long. There are some places where it is not easy to read, mainly because the focus is on the binary classifiers (which could be very ambiguous) and very little on the geomophical implications (i.e. bedrock). I think this paper would be better suited as a research paper rather than a short communication. I would suggest reorganizing the different sections, showing the methodology first and then presenting the geomorphological applications as a study case.

I suggest changing the verb conjugation in some sentences, especially where it says 'I did' or 'I presented', to more neutral third person verb tenses (e.g. This study, on this work, etc).

Appendix B should appear in the discussion section. This is a valuable contribution to the newer remote sensing products (e.g. higher resolution) from which the patterns presented in Figure B1 originate. Finally, I suggest adding a recommendation section regarding the perspectives of this very nice work, especially regarding the limitations.