Feature Selection in Machine Learning

Introduction to Classification in Machine Learning

Classification is one of the essential tasks in machine learning whose purpose is to classify each instance in the dataset into different classes based on its features. It is often difficult to determine which features are useful without prior knowledge. As a result, a large number of features are usually introduced into the dataset that may be irrelevant or redundant. Feature selection is the process of selecting a small subset of relevant features from the original large set of features. This small subset of features may have less redundant or relevant features, making the machine learning process simpler with reduced learning process time and increased performance.

Benefits of Feature Selection

Other benefits of feature selection are improved prediction performance, scalability, understandability, and generalization capability of the classifier. It also reduces computational complexity and storage, provides a faster and more cost-effective model, and aids in knowledge discovery. Moreover, it offers new insights for determining the most relevant or informative features. The main challenge that occurs in feature selection is the large search space, where for n datasets, the solutions are 2^n. Feature selection consists of complex stages that are usually costly. Even the optimal model parameters of the full feature set might need to be redefined a few times to obtain the optimal model parameters for selected feature subsets.

Challenges in Feature Selection

Feature selection also involves two main objectives, which are to maximize the classification accuracy and minimize the number of features, both of which are conflicting objectives. Hence, feature selection is considered a multi-objective problem with some trade-off solutions that lie between these two objectives. Some examples of feature selection techniques are Information Gain, chi-square, lasso, and Fisher Score. Feature selection can be used to find key genes (i.e., biomarkers) from a large number of candidate genes in biological and biomedical problems, discover core indicators or features to describe the dynamic business environment, select key terms like words or phrases in text mining, and choose or construct important visual contents like pixel, color, texture, and shape in image analysis.

Comparison with Other Dimensionality Reduction Techniques

In comparison to other dimensionality reduction techniques, such as those based on projection, for example, principal component analysis (PCA) or compression, feature selection techniques do not modify the original representation of the variables but simply select a subset of them. Hence, they maintain the original semantics of the variables, offering interpretability (Guyon & Elisseeff, 2003; Saeys et al., 2007).

Feature Selection in Gene Expression Data

Feature selection used on gene expression data with a small sample size is called gene selection. Gene selection can be used to find key genes from biological and biochemical problems. This type of feature selection is important for disease detection and discovery, such as tumor detection and cancer discovery, which results in better diagnosis and treatment. Gene expression data can be expressed as fully labeled, unlabeled, or partially labeled. This leads to the development of supervised, unsupervised, and semi-supervised gene selection to discover biological patterns and classes.

Methods of Feature Selection

There are many feature selection methods, such as supervised, unsupervised, and semi-supervised feature selection. In supervised feature selection, it uses the labeled data for feature evaluation. But the data is large and continues to collect data at an increasing rate. Moreover, the labeled data is costly to obtain and may be undependable and mislabeled, which may cause overfitting in the learning process in supervised type feature selection by either removing relevant features or using irrelevant features. In the case of the supervised method, previous knowledge is taken into account.

Unsupervised and Semi-Supervised Feature Selection

Unsupervised feature selection is more difficult to work with than the other two approaches because it is unaided by labeled data. But advantages of this type of feature selection are unbiased and perform well with no previous knowledge. Unsupervised feature selection is useful in the discovery of diseases and the classification of disease types. The disadvantage of the unsupervised approach is that it ignores the connection between different features and depends on some mathematical principles with no guarantee that those principles are valid for all data. Semi-supervised feature selection is a combination of supervised and unsupervised feature selection. Semi-supervised feature selection is also being used for gene classification by jointly employing both labeled and unlabeled data (Tang et al., 2014).

Evaluation of Gene Expression Data

Gene expression data can be evaluated using microarray data methods, which are essential with different samples. These methods can be grouped into unsupervised, supervised, and semi-supervised methods. The microarray data has a large number of genes that are redundant. Thus, it needs to identify some important genes for a better understanding of the fundamental data, and also minimize the time taken for improved post-processing tasks such as classification, subset selection of genes (features), and so on.

Advantages of Feature Selection for Microarray Data

Using feature selection, a subset of relevant features can be selected from the original large set of features. For finding key genes from a large number of applicant genes in biological and biomedical problems using features like genes, biomarkers, and so on. A biomarker is a feature that gives an indication of a medical condition observed from the patient externally and this can be measured as well as reproducible, different from medical symptoms which show only the signs regarding disease or health that are understood only by the patients themselves.

Conclusion

Feature selection has several advantages for microarray data. First, dimension reduction to reduce the computational cost. Second, the reduction of noises to improve the classification accuracy. Finally, more interpretable features or characteristics that can be helpful to identify and monitor the target diseases. Biologically, only a few genetic alterations correspond to the malignant transformation of a cell. Determination of these regions from microarray data can allow for high-resolution global gene expression analysis of genes in these regions and better biological problem detection and classification for better diagnosis, prognosis, and correct treatment for corresponding biological problems (Golub et al., 1999; Tusher et al., 2001).