From visual diagnostics to deep learning: automatic mineral identification in polished section images

Introduction

Studying mineralogical composition of ores is a fundamental step in the exploration of new deposits, as it allows determining the forms in which useful components are found, the processes of ore formation, and the potential recoverability of valuable elements. The mineral associations, textures, and structures of ores not only provide key information about the geology of a deposit, but also determine the choice of beneficiation methods.

Despite the development of modern analytical tools and existing solutions for automatic mineral diagnosis, such as those based on the SEM-EDS method [1, 2], optical microscopy remains the most accessible means of quantitative mineralogical analysis. However, it remains labor-intensive and requires highly skilled specialists. In addition, its visual nature limits the accuracy and reproducibility of results, creating a need for more effective approaches.

A promising area is the automation of ore mineral identification based on photographs of polished sections. This approach not only reduces span time, but also minimizes subjective errors associated with visual diagnostics and enables the implementation of accurate statistical analysis methods. The aim of this work is to describe our experience in developing a segmentation model for automatic detection of minerals in photographs of polished sections and solving a number of related problems that arose during the research. The paper systematically outlines the main problems encountered by the authors and the solutions they propose.

Current state of the problem

The first attempts to create tools for the automatic diagnostics of ore minerals under a microscope were made in the second half of the 20th century [3, 4]. At that time, spectrophotometers were used to measure the color of minerals, in particular, a mineral type was interpreted based on the absorption spectra of light in the visible range. Due to its low accuracy, this method was not widely used. More advanced methods of automatic mineral identification were developed in the second half of the 1990s and were based on the analysis of photographs of polished sections under a microscope [5, 6].

Attempts were made to automatically analyze mineral associations using cluster analysis in order to find patterns between different objects in photographs [7]. Special mention should be made of the attempt by the authors [8] to compile a digital atlas of all minerals and to identify the minerals themselves using a dendrogram based on a digital questionnaire.

To date, existing classical solutions (without the use of deep learning) for automatic mineral identification can be divided into two main groups:

1. Based on reflected light intensity in conjunction with color characteristics expressed in the RGB or LAB color space [9];
2. Based on statistical principles of color palette separation to identify minerals in a specific sample [10–12].

Both approaches have significant limitations. Methods that use color and reflectivity are unable to distinguish between minerals with similar optical properties. Statistical methods, in turn, require recalibration for each new geological object, making their application "situational" and limited. This is well illustrated in [12], which shows the features of applying this principle to separate copper ore into three minerals and three lithological types at a specific deposit.

It is worth noting that there are also highly specialized solutions available in the form of extensions for popular image analysis software packages such as Fiji/ImageJ. For example, [13] describes a method for automatic determining  hematite grade, size, and intergrowth types in ore using this software. The problem with such solutions is that they solve a narrow, specific problem and lack the necessary level of versatility.

The most effective way to overcome the shortcomings of the classical methods and achieve fundamentally better results in the automatic analysis of such images is to use trainable deep models (e.g., convolutional neural networks) that are capable of extracting complex hierarchical features from images, taking into account not only local textures and shapes, but also global relationships between image fragments. Instead of manually selecting color and statistical characteristics, such models – whether traditional convolutional neural networks (CNN) [13–15], modern transformers with a self-attention mechanism [16], or hybrid architectures (e.g., Mamba [17]) – learn to identify the distinctive morphological and structural-textural features of each mineral.

For instance, convolutional neural networks were used to detect surface defects and examine polishing quality of metal products [18, 19] and to analyze carbon distribution in cast iron based on microphotographs of rough workpiece surfaces [20]. In [21], a method is presented for separating hematite and quartz in iron ore polished sections, with the determining their size classes to optimize the feed for a processing plant. It is also worth noting a number of works devoted to the assessment and classification of the dimensionality of individual mineral individuals [22, 23], as well as the analysis and typification of the morphology of intergrowths in a system with known mineral associations [24, 25]. The segmentation model proposed in these works achieved 98% accuracy in predicting iron ore quality and hematite recoverability, highlighting the potential of deep models in solving industrial problems.

In [14], the effectiveness of deep convolutional networks for three-dimensional mineral identification and free grain analysis was demonstrated, and in [26], the authors showed that combined analysis of optical micrographs using CNN improves the accuracy of mineral content estimation in charge. In [15], the authors improved the methods of feature downsampling, classifying rocks more accurately based on polished section images.

It is worth noting that applying modern deep learning approaches facilitates transition from image fragment classification [27] to full semantic segmentation, which allows for accurate pixel-level segmentation (decomposition) of images by mineral, see [17, 26] and [14, 28]. At the same time, works [14, 29] demonstrated the fundamental feasibility of creating high-quality ore mineral segmentation models with high identification accuracy (> 0.8 by the IoU metric).

The main advantage of using deep trainable neural networks when working with images of ore samples is their ability to take into account the context of an image and adapt to the variability of mineral associations. Most importantly, it allows reliable differentiation even between minerals with very similar characteristics (pyrite–marcasite, covellite–chalcocite, etc.) without the need for permanent recalibration of the algorithm for new samples, unlike other computer vision methods. However, there are still relatively few studies devoted directly to the diagnostics of mineral species using such approaches. Deep learning models can also be used in conjunction with domain adaptation methods, which allow the segmentation model to be retrained on "new" images – taken with different equipment or under different lighting conditions, and thus maintain high performance even with significant variations in input data. Extensive reviews on domain adaptation [14] and examples of successful application in semantic segmentation of geological and satellite images [30, 31] confirm that this approach provides versatility and stability in a wide variety of conditions. The fundamental feature of most deep learning methods is the need for complete image annotation for training. This is often a very labor-intensive process, but the use of specialized weak supervision methods, which appear to the user as annotation the image with of rough strokes (ScribbleSup [32], ScribbleSeg [33]) or clicks [34], allows in many cases to significantly speed up the collection and preparation of training data.

To build a reliable system based on deep learning, the following fundamental problems must be solved, which are discussed in detail in this paper:

1. Development of neural network methods for mineral segmentation.
2. Development of adaptive methods for image calibration and preprocessing.
3. Development of methods for joint processing of heterogeneous images.
4. Development of a method for creating panoramic images.
5. Development of auxiliary methods for processing and analyzing images of polished sections.

Research Materials and Techniques

This study used a collection of polished sections provided by the Department of Geology, Geochemistry, and Mineral Resources of the Faculty of Geology at Lomonosov Moscow State University. A Carl Zeiss AxioScope 40 polarizing microscope with a Canon PowerShot G10 camera was used to obtain images of the polished sections. All photos were taken with a magnification of ×50 and have a resolution of 3396×2547 pixels.

The main drawback of existing solutions that use deep neural network models in the considered problems of analyzing photos of polished sections [23, 35], according to the authors, are the proprietary image sets used and the proprietary code base, which makes it impossible to compare the methods being developed. Therefore, all annotated (indexed) image sets created as part of the work are presented as a single open dataset LumenStone¹, and the software implementation of all developed methods is published as a petroscope² library with open source code for the Python 3 programming language.

The LumenStone image dataset contains several subsets focused on solving various image analysis problems for polished sections. The main subsets are S1, S2, and S3, which are aimed at the problem of mineral segmentation (automatic identification) and are formed taking into account mineral associations and mineral properties:

• LumenStone S1 (84 images): complex ores (galena, sphalerite, chalcopyrite, bornite, fahlore);
• LumenStone S2 (39 images): sulfide copper-nickel ores (pyrrhotite, pentlandite, chalcopyrite);
• LumenStone S3 (35 images): minerals with strong anisotropic properties (arsenopyrite, covellite).

Pixel masks of the corresponding minerals were created for all images of the datasets using Supervisely and Adobe Photoshop software. The masks are necessary for training and testing deep learning models.

It should be noted that due to natural reasons (frequency of occurrence in nature), the collected set of images has a significant minerals imbalance (the percentage ratio is given in Table 1). This fact is an additional complication for the development of methods for automatic mineral segmentation and must be taken into account.

The authors also collected additional subsets of images necessary for solving related problems:

• LumenStone V1: a special dataset of images of the same 10 specimens (sections) with different shooting conditions, designed for developing and testing color adaptation methods. The images were obtained using the same equipment with blue and yellow light filters, as well as using a LOMO Microsystems PLM-215 microscope with a Canon EOS 40D camera.
• LumenStone P1: 875 images obtained for 35 polished sections. For each polished section, 25 photographs were taken with 20–30% overlap, intended for creating panoramic microscopic images.

To solve the problem of simultaneous analysis of anisotropic mineral photographs in PPL and XPL, "rotated" photographs of a single field of view were taken with the microscope stage rotation increment of 5 and 15° and additionally included in LumenStone S3.

Table 1

Distribution of minerals on labeled photographs of polished sections
in the LumenStone S1, S2, and S3 sets for solving segmentation problems
(the distribution when divided into training and test datasets is provided in square brackets)

¹ LumenStone Dataset. URL: https://imaging.cs.msu.ru/en/research/geology/lumenstone

² GitHub. URL: https://github.com/xubiker/petroscope

Problems and their solutions (discussion)

The descriptions of the problems in the field of image processing and analysis that are to be considered the complex problem of automatic mineral identification in microscopic images of polished sections and the approaches proposed by the authors to solve these problems are given below.

1. Neural network methods for mineral segmentation

In this work, we consider convolutional neural networks to solve segmentation problems. Transformer-based alternatives, although promising, remain excessively resource-intensive for standard laboratory conditions [36]. Despite the good generalization ability of convolutional neural networks, they are quite sensitive to class imbalance in training dataset [37, 38], which is characteristic of the collected data (Table 1). Furthermore, neural network methods cannot be directly applied to high-resolution images due to hardware limitations. To mitigate these shortcomings, we proposed a specialized method for sampling the training sample collection during the training process, which extracts small fragments from images (patches) and acts as a data balancer.

The objective of the developed sampling method is to equalize the distribution of mineral classes fed into a neural network during training. For each pair "training image – mineral type," a matrix containing the extracted area of a selected mineral at each point in the case of selecting a patch centered at that point is calculated. The resulting set of matrices is used as probability maps when selecting patches for training. For instance, at each sampling iteration, for the mineral that is currently the least represented, 1) an image from the training dataset is selected (proportional to the content of this mineral), 2) the center of the patch is selected in accordance with the previously calculated probability maps, 3) the patch is extracted, and 4) the information about the representation of minerals in the used data is updated. With moderate patch sizes (256–384 px), this method allows for a significant equalization of the distribution of minerals in the LumenStone S1, S2, and S3 sets that has a positive effect on the training speed of segmentation models and on the final segmentation quality metrics.

When developing neural network models for mineral segmentation, we reviewed and investigated a number of convolutional architectures, ranging from the traditional UNet [40] and its modification ResUNet [29] to more modern PSPNet [41] and UPerNet [42]. The advantage of the latter lies in the ability to analyze images at different scales, correctly identify both small and very large objects simultaneously, and take into account local and global context that significantly improves the quality of segmentation based on the available data.

To evaluate the quality of segmentation in this work, IoU (Intersection over Union) metric was used [43]. IoU is a key metric in object detection and segmentation that measures the overlap between predicted and ground truth boxes). This is one of the simplest and most common methods of geometric evaluation of segmentation when reference labeling is available. The metric takes values from the range [0, 1], where 1 corresponds to a complete match between the predicted and reference labelings (ideal case), and 0 corresponds to no intersections between the predicted and reference segmentation annotation. An IoU value greater than 0.7 is usually considered satisfactory, although this depends, of course, on the subject area.

In our case, training the PSPNet neural network with the ResNet18 encoder on the LumenStone S1 and S2 datasets, together with the class-balanced sampling method described above, allowed segmenting nine minerals and a generalized class of nonmetallic minerals with very high quality (the average IoU value on the test set was 0.88). The training uses a cross-entropy loss function, random augmentations (rotation, slight changes in scale, brightness, and color), an Adam optimizer with an initial learning rate of 0.001 and a decrease upon reaching a plateau. The training took approximately 3 hours using a Nvidia A6000 GPU. An example of applying the trained mineral segmentation model to an image from a test set is shown in Fig. 1.

Fig. 1. Example of a polished section image segmentation with the trained PSPNet model:
a – image; b – error map (correctly recognized areas are highlighted in green, segmentation errors are highlighted in red); c – mineral mask (expert annotation); d – model prediction

2. Adaptive methods for image calibration and preprocessing

One of the main problems encountered by the authors when working with primary data is high sensitivity of segmentation models to the color palette of images. Differences in color characteristics between training images and real images lead to a significant deterioration in the quality of mineral identification. The color and brightness characteristics of images are determined by many factors: microscope parameters, camera settings, lighting, etc.

A solution to this problem is to use automatic color correction based on the color difference between the received image and a known reference (e.g., [44]).

We proposed the method for correcting color distortions in [45]. The main idea is to construct a transition matrix (Color Correction Matrix, CCM) [46] between the color spaces of a distorted and reference images (images from the training set are taken as reference).

The process includes extracting the averaged colors of minerals and the background using partial labeling, linearizing colors through gamma correction (γ = 2.2), and calculating the affine transformation. The minimization problem is solved in LAB space, using the sum of the squares of color differences calculated using the CIEDE2000 formula [47] as a loss function. The work uses a 4x3 matrix with initial approximation initialization based on the "white balance" method [46]. The final step is to transform the distorted image through matrix multiplication by the previously calculated color correction matrix.

The proposed method allows preserving color differences that are critical for mineral identification (Fig. 2), while minimizing the influence of lighting changes and equipment settings. The algorithm supports two operating modes: an individual correction for each image and a "calibration" mode for a series of images, where the correction matrix is calculated once and applied to the entire group. The method does not require any prior training, and processing a single image takes less than 10 seconds on an Intel Xeon Gold 6226R CPU.

Fig. 2. An example of how the proposed color calibration method works:
a – original image taken with alternative equipment; b – reference image;
c – initial image after applying the method

3. Methods for joint processing of heterogeneous images

Many minerals are identified not only by their color and reflectivity, but also by the presence or absence of anisotropic properties. Anisotropy manifests itself in the ability of minerals to "fade" in doubly polarized light (crossed nicols) when the optical axes of a mineral coincide with the direction of the microscope polarizers. This property is a key to distinguishing minerals with similar reflectance and color parameters. For example, pyrite (isotropic) and marcasite (anisotropic) have similar optical characteristics but differ in the manifestation of anisotropy. Similarly, pyrite and arsenopyrite, although they have slightly different reflectivity and color, can also be reliably separated based on the manifestation of anisotropy by arsenopyrite.

We developed a neural network segmentation method that uses XPL and PPL images as additional input data for the segmentation neural network to improve the accuracy of mineral segmentation [48]. The key step in this method is to align images taken at different angles of rotation with the reference PPL image. For this purpose, SIFT algorithms [49] were used to detect stable key points in images, and RANSAC algorithm [50] was used to calculate the affine transformation between images based on the found matches. Thus, all images were referred to a single coordinate system (Fig. 3). Then XPL images referred to a single coordinate system were used as additional input channels for a neural network based on the architecture proposed earlier by the authors [29]. The used hyperparameters are described in [29], and the model training time is approximately 6 hours with the use of a NVidia A6000 GPU.

Fig. 3. Alignment of XPL images of arsenopyrite:
top row – images of arsenopyrite in different orientations; bottom row – images of arsenopyrite in different orientations after alignment. Four different orientations out of 24 are presented for each image with anisotropic minerals.

4. Methods for creating panoramic images

The average polished section area is several square centimeters, with typical ×50 magnification. Under such conditions, only a small part of a polished section, measuring a few square millimeters, is visible in each photograph. The use of photographs covering a large area of a sample would allow more accurate information to be obtained about the distribution of minerals in a sample and their relative positions, which would have a positive effect on the quality of the analysis.

Scanning electron microscopes (SEM) can be used to obtain large images in geology, but such equipment is very expensive, structural and textural features may be lost due to the nature of the method, and the identification of mineral phases requires additional effort. Therefore, like other researchers [51], we have opted for software stitching series of overlapping images into a single panorama.

Currently, there are many examples of software for automatic stitching disparate photos into a single panoramic image. These include Adobe Photoshop, Fiji/ImageJ, and many others. However, using third-party software has a number of disadvantages. Powerful tools such as Adobe Photoshop can overly transform a panorama (unnaturally change colors, remove important details, mistaking them for artifacts of stitching). Integrating a third-party implementation into your system is tricky, and it also makes it impossible to make the changes needed to the algorithm to fit the specifics of the problem you're trying to solve.

We developed our own algorithm for stitching photographs into a panoramic image of the surface of a polished section [52] (Fig. 4). The algorithm consists of two main stages: image alignment and further post-processing to improve visual perception. At the first stage, with the use of calibration images, geometric distortions of images are corrected using the Brown–Conrady model [53], and photometric distortions are corrected using flat field compensation [54]. Then, using the LoFTR neural network [55], common key points are found in images that have overlapping areas. These are used to calculate perspective transformations (homographies) for pairs of adjacent images using RANSAC [50], after which all images are transformed into the coordinates of a single image (reference image). Finally, global panorama optimization is performed to minimize the alignment errors. The result of this stage is a preliminary panorama, a collage. The second stage involves improving the initial panorama. The differences in exposure between images are compensated. The seams between images are masked by constructing the least noticeable seam using the graphcut method [56], taking into account the differences in color and gradients of neighboring pixels. The final step is to blend the images near the joints of the panorama tiles to remove any remaining stitching artifacts. The LumenStone P1 dataset, compiled for the panorama construction, was used to test the algorithm. The method does not require prior training, and the processing speed for a single panorama consisting of 25 images at an Intel Xeon Gold 6226R CPU is approximately 5 minutes.

Fig. 4. Illustration of the developed method for constructing panoramas: on the left, several images of the same polished section, taken with overlap; on the right, the constructed panorama

5. Additional methods for processing and analyzing images of polished sections

The application of deep learning methods for mineral segmentation in images requires accurate annotation of a large number of images, which is a labor-intensive process. To simplify the annotation process and create a segmentation model capable of recognizing the main ore minerals, the authors are developing a method of accelerated interactive annotation using superpixel clustering based on the SLIC [57] and Felzenswalb [58] methods. A geologist roughly annotates minerals with strokes, labeling entire areas of an image with the label of a particular mineral based on the scribble data and the superpixel map. The user adjusts the method's predictions until the final annotation is obtained. A distinctive feature of this approach is multi-scale clustering, which allows to quickly label both large homogeneous areas and small fragments, automatically breaking large clusters into smaller ones as needed.

One can also reduce the labor costs of data annotating by extending the training set with partially annotated data. The main idea behind this approach is to highlight areas of uncertainty (lack of confidence) in the trained segmentation model on images. The authors suggest highlighting areas of uncertainty in images [59] using a hyperbolic radius [60] that reduces the scope of annotation to 5–10% of the original image (Fig. 5).

Fig. 5. The result of uncertainty area assessment method:
a – original image; b – prediction of segmentation model uncertainty areas;
c – areas for manual labeling

The final stage after recognizing and segmenting all minerals in the images is the statistical analysis of an image. It is responsible for conducting quantitative analysis to assess the areal ratio of mineral phases and their particle size analysis with separating fractions by size class for each mineral. This stage is currently under development.

Findings

The result of the authors' research into the automatic analysis of microscopic images of geological polished sections to determine mineral composition was the creation of an open image dataset dataset called LumenStone and a number of algorithms and methods that solve the main problems encountered:

1. Neural network methods for mineral segmentation

A convolutional neural network model for mineral segmentation and a special method for sampling training data have been developed, allowing the existing class imbalance to be neutralized. The accuracy of mineral segmentation according to the IoU metric was as follows: non-metallic minerals – 0.912, bornite – 0.938, chalcopyrite – 0.899, galena – 0.905, magnetite – 0.650, pentlandite – 0.790, pyrrhotite – 0.928, pyrite – 0.964, sphalerite – 0.922, tennantite – 0.882. The overall pixel accuracy (PA) of the segmentation was 0.96. The differences in mineral identification results can be explained by the difference in the size of the training sets used for LumenStone S1 and LumenStone S2.

2. Adaptive methods for image calibration and preprocessing

An algorithm has been developed for adapting images of polished sections obtained under different shooting conditions using partial user annotation. Pixel segmentation accuracy for distorted images increased from 0.29 (before) to 0.87 after adaptation using annotation covering approximately 30–35% of the image.

3. Methods for joint processing of heterogeneous images

The developed algorithm for segmenting anisotropic minerals using additional rotated XPL images improved the quality of anisotropic mineral segmentation by 3–12%. It has been shown that the best results can be achieved by using 6 additional rotated images.

4. Methods for creating panoramic images

A method for constructing panoramic microscopic images of polished sections has been developed. The root mean square error of alignment of panorama tiles from 25 images was 0.5–0.6 px. The resulting panoramas have a resolution of 12000×8000 pixels and can be used for automatic mineral segmentation. The implemented method does not have the disadvantages of less specialized solutions such as Adobe Photoshop, Fiji, and Panorama Studio.

5. Additional methods for processing and analyzing images of polished sections

A prototype method for interactive annotation of polished section images has been developed, which significantly speeds up the process of preparing data for training segmentation models. A method for automatically searching for areas of uncertainty has also been developed, allowing image annotation to be prioritized and significantly reducing the scope of annotation required.

Conclusion

This paper presents the authors' experience in developing a set of methods for the automatic analysis of images of polished sections for the identification of ore minerals. The developed segmentation method based on a convolutional neural network is capable of identifying nine ore minerals (with correct distinguishing between ore minerals and non-metallic phases) with an IoU accuracy = 0.88 and PA = 0.96. The potential of using additional information from XPL images to increase the accuracy of anisotropic mineral identification was demonstrated.

The developed methods of interactive annotation and image adaptation significantly accelerate and improve the training and use of segmentation models on new data. It is worth noting the method developed by the authors for obtaining panoramic images of polished sections, which allows detailed images of the entire surface of polished sections to be obtained in high resolution without expensive equipment. Unlike existing software solutions, this method does not distort the final panorama that has a positive effect on the segmentation results. Working with large-format images opens up new possibilities for the automatic analysis of images of polished sections.

The results obtained justify the need for further development of the area under consideration and form the basis for the creation of an intelligent quantitative assessment system capable not only of identifying minerals and calculating their areal fractions and performing particle-size analysis by size class, but also determining the types of minerals intergrowths. The implementation of this methodology will open up new possibilities in digital petrography, enabling fast, economical, and reproducible mineralogical analysis on optical microscopes in reflected light. Ultimately, this will enable the formation of unified criteria for analyzing the structural and textural characteristics of mineral associations for genesis comparison of different deposits.

Currently, the authors are integrating most of the methods and algorithms described in this paper into their PathScribe software platform [61]. This platform is a cloud-based client-server solution with cross-platform clients for working with ultra-high-resolution images and is designed for universal use in both scientific and educational applications. The authors hope that the ability to work with panoramic images of polished sections using convenient tools for annotation and fully automatic analysis will be useful for geologists of various specializations.