A new internal cluster validity index for Fuzzy C-Means algorithm

Fuzzy c-means is a popular clustering algorithm that can partition a set of objects into groups so that objects within one group are similar and dissimilarto others. It is required to measure cluster quality formed by any clustering algorithm, either by external or internal validations. External validation is straightforward, requiring external information such as class labels and the number of classes. Thus, external cluster validations such as Rand, F-Measure, and Fowlkes-Mallows indices serveas the ground truth in measuring cluster quality. Most external validations are based on the bounded index model, indicatingtheir accuracy scores from a minimum of 0 to a maximum of 1. However, in the absence of external information, e.g., unknown class and number of classes, particularly in real-world data, internal validation is then required. Instead, some internal validations are based on the unbounded-index model, which relies on a single score, whether minimum or maximum scores andno boundary. For the bounded index models such as the Silhouette index, the Fuzzy Silhouette (FS) Index, and the Generalize In tra-In ter Silhouette (GIIS) Index, their accuracy scores range from -1 to +1, and they are different from the bounded-index model of external validationin which ranging from 0 to 1. Hence, it is difficultto benchmark any clustering algorithms or clustering results due to these constraints of the existing internal validations. In addition, no internal validation proposed thus far is associated with a model ranging from 0 to 1, like the external validation that serves as the clustering ground truth. Thus, to overcome these limitations, a new internal validation, namely, the fuzzy validity index (FVI), is proposed. The novel FVI was designed with a bounded-index model (0 to 1), ensuring interpretability and consistency with external validation indices. Beyond its boundedindex model, FVI introduces Adjusted Fuzzy Membership (AFM) and the Compactness-Separation Ratio (CSR), providinga more adaptive assessmentof FCM clustering quality. These components allow FVI to effectively balance membership uncertainty and cluster separability, making it particularly suitable for fuzzy clustering applications. Seven clustering properties such as "the number of clusters", "cluster overlap," "dimensions and cluster overlap", "cluster dimensions", "cluster structure", "unbalanced data" as well as real-world datasets (in a total of 3 5 datasets) were used to evaluate the performance of FVI thoroughly. Experimental results showed that the FVT is highly promising. Overall, the scores of the FVI were comparable to the scores obtained by the external validity index, i.e., F-measure. Statistically, the correlation coefficient between the FVI and F-measure was high (around 0.9 and above), indicating their similarity. Therefore, theFVI can be used as an internal validation index for FCM and could potentially serve as a ground truth for measuring cluster quality. In the future, with its strong performance and generalized formulation, FVI has the potential to be applied as an internal validation index for various fuzzy clustering techniques across different applications.