Download Dual Heuristic Feature Selection Based on Genetic Algorithm and Binary Particle Swarm Optimization and more Essays (high school) Art in PDF only on Docsity!
© Journal of University of Babylon for Pure and Applied Sciences (JUBES) by University of Babylon is licensed under a Creative
Commons Attribution 4. 0 International License
Dual Heuristic Feature Selection Based on Genetic
Algorithm and Binary Particle Swarm Optimization
Ali Hakem Jabor
University of Al-Qadisiyah
College of Computer Sciences and IT
[email protected]
Ali Hussein Ali
Al-Qadisiyah Education Directorate
[email protected]
Abstract
The features selection is one of the data mining tools that used to select the most important features of
a given dataset. It contributes to save time and memory during the handling a given dataset. According to
these principles, we have proposed features selection method based on mixing two metaheuristic
algorithms Binary Particle Swarm Optimization and Genetic Algorithm work individually. The K-Nearest
Neighbour (K-NN) is used as an objective function to evaluate the proposed features selection algorithm.
The Dual Heuristic Feature Selection based on Genetic Algorithm and Binary Particle Swarm Optimization
(DHFS) test, and compared with 26 well-known datasets of UCI machine learning. The numeric
experiments result imply that the DHFS better performance compared with full features and that selected
by the mentioned algorithms (Genetic Algorithm and Binary Particle Swarm Optimization).
Keyword: Data Mining, Features Selection, Genetic Algorithm, Binary Particle Swarm Optimization,
Metaheuristic Optimization.
I. Introduction
Over the last decades the devices, sensors, and users are on increasing, therefore,
the dimensions of the datasets are increasing. Logically, the size of data is directly
proportional to the execution time. As a result, reducing the dimensions of the data
becomes necessary to decrease execution time or processing. Using Data mining
technique in many fields such as Artificial intelligence [1], Databases [2], Image and
video processing [3], and others, make it in interesting topic for researchers. It is one of
the important techniques used for filtering data. The data mining searches of the data
(features or instances) that related to the objective of the dataset and removing garbage
data from the dataset [ 4 ]. The techniques that omitted unimportant features from the
dataset called features selection (FS). There are many algorithms used for FS such as
Filter [ 5 ], Wrapper [ 6 ], and Embedded [ 7 ] Methods. The metaheuristic algorithms use in
optimizing FS of verity styles. The advantages of the stochastic search are fast, flexible,
and succeed to solve many hard optimization problems, but disadvantages no grantee to
ARTICLE INFO
Submission date : 29/7/
Acceptance date : 4/10/
Publication date : 10/3/
find a global solution and may be suffering from stagnation at local optimum [8] [9]. The
Stagnation phenomena is a problem happens when the algorithm gave the same solution
(local optimum) during several search steps. To reduce stagnation need to decrease
convergence between candidate solutions. Therefore, went to increase the diversity of new
solutions. The increases randomness or number of mutation genes leads to make the
population more diversified.
[ 10 ] Have proposed a wrapper approach with Harmony Search algorithm (HS) that
adaptive for features selection. [ 11 ] The Binary Approach of Artificial Bee Colony
Optimization (BABC) for features selection. The binary vector generated by BABC
represents the FS, which were the features that the corresponding one have been selected.
[ 12 ] Modify Gravitation Search Algorithm (GSA) in order to find a subset of features that
maximize the Optimum-Path Forest ( OPF ) accuracy over a validation set.
The previous works [10] [11] [12] modify the stochastic search algorithms for FS
without trying to reduce the stagnation phenomena in these algorithms. The Binary
Particle Swarm Optimization (BPSO) may be suffering from stagnation problem [8] [9] in
some points of search (as in Evolutionary Algorithms EA). The hybrid algorithm is more
efficient than the algorithms it has built, because it combines the good features of them.
Therefore , in order to reduce stagnation in BPSO, we combine it with GA sequentially,
and uses both of them for features selection. The sequential of calling for BPSO and GA
individually makes the proposed algorithm more robustness with keeping on original
formatted of the mentioned algorithm by calling them sequentially. Increasing the number
of mutation genes and decreasing the crossover operations during search progress help the
proposed algorithm to be more robust to deal with stagnation problem. The wrapper
method uses a heuristic to rank the features [10] [11] [12], which are used for FS in the
proposed algorithm. The experiments have been performed in 26 well-known datasets of
UCI machine learning to compare the proposed algorithm (DHFS), full set classification,
and FS by GA and BPSO. The numeric experiments results imply that proposed DHFS is
better performance compared to mentioned algorithms.
The remainder of the paper is organized as follows: In Section II we explain the
Particle Swarm Optimization (PSO). Section III presents the Binary Particle Swarm
Optimization BPSO (BPSO), Genetic Algorithm (GA) is described in Section IV. Section
V illustrates the Features selection (FS). Dual Heuristic Feature Selection (DHFS) present
in Section VI. Section IX discusses Validate and test algorithm. Finally, the conclusion
and future works stated in Section X.
II. Particle Swarm Optimization (PSO)
The PSO is popular metaheuristic algorithm inspired by the behaviour of social
animals. The robustness, stability, and simplicity enough to be it quite use for enhancing
the different fields [ 13 ] such as Data mining [ 14 ] medical apply [ 15 ] image processing
[ 3 ] speech recognition [ 16 ]. There are many similar features between PSO and other
Evolutionary Algorithms (EA) [ 17 ]. All EA start with a random population and calculate
the fitness of each participant (candidate solution) to evaluate the performance of the
population (all candidate solution). It uses the random mathematical model to update the
IV. Genetic Algorithm (GA)
The genetic algorithm is a discreet population metaheuristic algorithm inspired by
the genetic behaviour of natural life according to Charles Darwin’s theory of natural
evolution [ 18 ]. It follows the rule of natural selection where the good individuals are
contributing produce offspring. The mechanism search of GA essentially based on three
genetic operations: a parent selection, crossover, and mutation. The parent selection is a
process of selecting two or more parents from the crossover pool to produce new
offspring (new candidate solution). The good parents have more chance to be selected for
reproduction according to Darwin’s theory [ 20 ]. There are other methods to select parents
of GA such as a roulette wheel, tournament [ 21 ]. The crossover is mix genes of parents
that elected for crossover operation. There are many crossover techniques such as one
point crossover, two-point crossover [ 9 ], Arethematic Crossover[ 22 ], heuristic crossover
[ 23 ]… etc. The mutation is tweak change in genes of offspring. The parent selection and
crossover are not enough to solve the stagnation in local optimum [ 3 ]. Therefore, the
mutation process in GA is important to make diversity and reduce stagnation effect on the
search processing [ 17 ].
V. Features selection (FS)
During the last decades, many datasets have huge information and high dimensional
with hundreds or ten thousand features. Some of these features may be not important to
the main object of the dataset [ 4 ]. The selection of important features that relate to a
dataset goal called features selection. It employs the specific technique to remove
garbage features from the dataset. The features selection technique is important tools to
save train time and enhance the accuracy ratio of machine learning algorithms [ 24 ]. As a
result makes the features selection a hot topic area for researchers. There are many
features of selection methods: Filter, Wrapper, and Embedded [ 25 ]. Filter Methods
depends on the relationship between the features and the target of the dataset to select the
importance of features [ 5 ]. Embedded method for feature selection, which achieves by
the insights using in some Machine Learning models such as LASSO Linear Regression
and Tree-based models [ 7 ]. Wrapper Methods generate models with a subset of feature
and gauge their model performances [ 6 ]. The stochastic search for important features can
be select subset features of the wrapper model [ 10 ]. The time complexity of running an
algorithm depends on data domination.
VI. Dual Heuristic Feature Selection (DHFS)
In this section, we focus on features selected based on a combination of two
population metaheuristic techniques: Genetic algorithm (GA) and Binary Particle Swarm
Optimization (BPSO). The hybrid algorithm has the advantages of multiple algorithms
when deploy to solve optimization problems [ 26 ]. The search process of GA depends on
three main operations: parent selection, crossover, and mutation. The parent selection and
crossover are not enough to solve stagnation in local optimum [ 3 ]. Therefore, the
mutation operation of GA tries to reduce stagnation at local optimum by increasing the
chance to product diversity solutions [ 17 ]. The BPSO also suffering from stagnation
problem [8] [9] [19]. The stagnation in BPSO happens when the local best solution P best
and global best solution G bset
has no change during several steps of the search process.
The proposed algorithm reduces the stagnation in BPSO by calling GA to decrease
convergence in newly candidate solutions (population). The proposed method (DHFS) as
shown in bellowing:
𝑠𝑜𝑙𝑢𝑎𝑡𝑖𝑜𝑛
1
2
1
𝑏𝑒𝑠𝑡
𝑏𝑒𝑠𝑡
𝑠𝑜𝑙𝑢𝑎𝑡𝑖𝑜𝑛
𝑏𝑒𝑠𝑡
𝑠𝑜𝑙𝑢𝑎𝑡𝑖𝑜𝑛
𝑏𝑒𝑠𝑡
1
2
1
2
1
2
1
𝑠𝑜𝑙𝑢𝑎𝑡𝑖𝑜𝑛
𝑠𝑜𝑙𝑢𝑎𝑡𝑖𝑜𝑛
1
2
1
2
1
2
𝑠𝑜𝑙𝑢𝑎𝑡𝑖𝑜𝑛
S
1
is a number of steps where no enhance in results when applied an algorithm
(BPSO or GA). It reset after calling another algorithm. S 2
is a number of calls an
algorithm (BPSO or GA) but without enhancing in results. The S 2
reset when any
algorithms (BPSO, BGA) find a new best solution. The (θ) represents the number of valid
steps without a change in results for both algorithms (BPSO or GA). The proposed
method consists of four parts: binary metaheuristics optimization algorithms (BPSO and
Where:𝑚
𝑚𝑖𝑛
is the minimum number of genes for mutation, 𝑚
𝑚𝑎𝑥
is the
maximum number of genes for mutation. The features that corresponding 1 in the vector
that been generating by BPSO or GA as shown in figure 2:
Figure 2 : Features selection strategies (10-dimensional problem. Here i
th
particle x i
= [1 0 111 00 1 0 0] indicate that the 1
st
, 3
rd
, 4
th
,
th
,
th
, 10
th
features are selected.)
The DHFS stops when satisfying one of the stop criteria: the algorithm reaches to
the optimal solution, No change in result during several iterations, or the algorithm gets
the maximum iterations. Each metaheuristic algorithm must have the cost function
(object function or benchmark function) to evaluate the performance of the search
processing on an algorithm. The cost function that uses with the proposed method is the
classification result of the dataset by K-NN. In addition, check the validated the proposed
method by CEC’15 benchmark functions.
VII. Algorithm Parameters
Table 1 illustrates the parameters of the algorithm
Table1: parameters of GA, BPSO, and DHFS
GA Crossover rate 0.8, α random [0 or 1], mutation rate 0.8, m_min=1, m_max =
60% of given vector.
BPSO
[29]
c1=c2=2 , maximum velocity =6 ,minimum velocity =0.4, inertia weight
𝑚𝑖𝑛
𝑚𝑎𝑥
DHFS θ = 20
VIII. Discusses Validate and test algorithm
A - Check Validate by CEC’ 15
After, combining two of the metaheuristic algorithms for features selection, we have to
test the proposed method whether successful or not. The CEC2015 is Congress
Evolutionary Computation function uses to test any given search algorithm [ 27 ] [ 28 ]. The
CEC2015 has fifteen functions divided on four-group unimodal, Simple multimodal,
hybrid multimodal, and composition multimodal. Four functions of CEC’15 use to check
the validate of the DHFS algorithm compared to mention algorithms. Figures 3 shows the
DHFS is best in Function3, Function11, and Function15. It overcomes on most stagnation
stages in search processing. The proposed algorithm failed to record better performance
in Function 7, mainwheel the BPSO get the best result.
Figure 3 : Compare the minimum value found by DHFS , BPSO , and GA over 4
functions of CEC2015, 500 iteration, 20 x100 population, and average 30 run times
Table2 illustrates the enhancing in the standard deviation (STD) of the BPSO when
combined with GA in the proposed method (DHFS). The same above function used in
comparative and same parameters of CEC’15 that set (500 iterations, 20 x100 population,
and average 30 runtimes.)
Table 2 : (STD) of BPSO comparing with DHFS
Function NO STD of DHFS STD of BPSO
Function 1 104094969.6 93759278.
Function 7 0.321599518 0.
Function 11 12.26498303 11.
Function 15 5.
B- Test by K-NN classification
The proposed method test over 26 datasets from UCI machine learning
[https://archive.ics.uci.edu]. The dataset that choice from UCI machine learning has
deferent domination (Features, sample, and classes). Table 3 shows the descriptions of
datasets that use in the comparative study.
Table 4: K-NN classification results of the dataset with full features and features selection
by BPSO, GA, and proposed DHFS (The K in K-NN is 5)
I.Conclusion and Future works
The stagnation phenomena increase as search progress due to new candidate solutions are convergence. The mutation operation in GA reduce these phenomena by produce diversity in the new population. The BPSO also suffering some time from stagnation at a local optimum. The Stagnation problem increasing as search progress, therefore, the metaheuristic search algorithms need to make diversity in the population for reducing the effects of stagnation on search processing. The DHSF save on the original format of both algorithms (GA and BPSO) by calling them sequentially and
- 1 HA 82. SQ Dataset Symbol full Set FS BPSO FS GA FS DHFS
- 2 OZD 97.23 97.23 97.63 100.
- 3 PA 91.38 92.10 89. - 93.
- 4 HE 77.78 86 .67 88.89 94.
- 5 SE 73.02 90.48 88. - 91.
- 6 SO 69.35 80.65 84.55 90.
- 7 SP 73.42 86.08 82. - 87.
- 8 WI 65.38 95.08 93.15 98.
- 9 WPBC 64.41 74.58 77. - 81.
- 10 AD 86.36 95.45 90.91 96.
- 11 BN 75.
- 12 BNT1 84.62 92.31 90.15 96.
- 13 BNT2 78.57 90.86 85. - 92.
- 14 BR2 64.55 78.18 81.82 84.
- 15 BR3 57.41 62.96 66. - 68.
- 16 CO 78.33 88.89 83.33 91.
- 17 DL 85.91 96.00 85. - 100.
- 18 LE 98.82 100.00 100.00 100.
- 19 LE1 87.50 90.00 87. - 95.
- 20 LE2 93.33 95.87 93.
- 21 LC 93.05 96.61 94. - 98.
- 22 LY 97.27 100.00 100.00 100.
- 23 NC 70.71 85.71 92. - 94.
- 24 PR 85.00 90.21 86.67 94.
- 25 PT 78.67 83.33 80. - 90.
- 26 SR 93.04 94.57 95.33 94.
share to find the optimum solution. The future work we suggest to use other
metaheuristic algorithms and adaptive it to work in the parallel model.
CONFLICT OF INTERESTS.
There are non-conflicts of interest.
II. Reference
[1] Mann, C.J.H. "Handbook of Approximation: Algorithms and
Metaheuristics" Kybernetes 37.2 (2008).
[2] Siarry, Patrick, and Zbigniew Michalewicz, eds. Advances in metaheuristics for hard
optimization. Springer Science & Business Media, 2007.
[3] Maitra, Madhubanti, and Amitava Chatterjee. "A hybrid cooperative–comprehensive
learning based PSO algorithm for image segmentation using multilevel thresholding."
Expert Systems with Applications 34.2 (2008): 1341-1350.
[4] Han, Jiawei, Jian Pei, and Micheline Kamber. Data mining: concepts and techniques.
Elsevier, 2011.
[5] Weston, Jason, et al. "Feature selection for SVMs." Advances in neural information
processing systems. 2010.
[6] Talavera Luis. "An evaluation of filter and wrapper methods for feature selection in
categorical clustering." International Symposium on Intelligent Data Analysis. Springer,
Berlin, Heidelberg, 2005.
[7] Chandrashekar, Girish, and Ferat Sahin. "A survey on feature selection
methods." Computers & Electrical Engineering 40.1 (2014): 16-28.
[8] Zubieta, Francisco Javier Orellana. Metaheuristics in requirements engineering: refining
the next release planning problem (meta-heurísticas en ingeniería de requisitos:
refinación del problema de planificación de la siguiente versión de software). Diss.
Universidad de Almería, 2015.
[9] Luke, Sean. Essentials of metaheuristics. Vol. 113. Raleigh: Lulu, 2009.
[10] C. Ramos, A. Souza, G. Chiachia, A. Falc˜ao, and J. Papa, “A novel algorithm for
feature selection using harmony search and its application for non-technical losses
detection,” Computers & Electrical Engineering, vol. 37, no. 6, pp. 886–894, 2011.
[11] Schiezaro, Mauricio, and Helio Pedrini. "Data feature selection based on Artificial Bee
Colony algorithm." EURASIP Journal on Image and Video Processing 2013.1(2013):
[12] E. Rashedi, H. Nezamabadi-pour, and S. Saryazdi, “GSA: A gravitational search
algorithm,” Information Sciences, vol. 179, no. 13, pp. 2232–2248, 2009.
[13] Chen, Wei, Mahdi Panahi, and Hamid Reza Pourghasemi. "Performance evaluation of
GIS-based new ensemble data mining techniques of adaptive neuro-fuzzy inference
system (ANFIS) with genetic algorithm (GA), differential evolution (DE), and particle
swarm optimization (PSO) for landslide spatial modelling." Catena 157 (2017): 310-324.
[14] Holden, Nicholas, and Alex A. Freitas. "A hybrid PSO/ACO algorithm for discovering
classification rules in data mining." Journal of Artificial Evolution and Applications
ان الفائدة من اختيار صفاتالمعطاة.ادوات تنقيب البيانات الذي يستخدم الختيار الصفات المهمة للبياناتأحد اختيار الصفات هو
الصفات على اساساختيارخوارزميةصممناحسب تلك المبادئالبيانات. ة المستخدمة في معالجةالبيانات هو توفير الوقت وتقليل الذاكر
أستخدممنفصل.والخوارزمية الجينية لتعمال معا ً بشكلاب الثنائيةاألسروارزميتين من خوارزميات البحث العشوائي هما خوارزمية دمج خ
فحصت وقورنت مع بيانات مصنفة بدون اختيار الصفات المهمةالمقترحة. ان كدالة لتقييم عمل الخوارزميةالتصنيف على اساس الجير
مجموعة 26 الثنائية والخوارزمية الجينية. استخدمت في عملية التصنيفاباألسرالصفات على اساس خوارزميةرباختيا وبيانات مصنفة
مقارنة مع البيانات بدون اختيار الصفات اوأفضلنتائج التجارب الرقمية بينت ان الخوارزمية المقترحة, UCI من البيانات التابعة للـ
الصفات للخوارزميات المشار اليها باختيار ً
.سابقا
.العشوائيطرق البحث–الثنائيةاباألسرخوارزمية–الخوارزمية الجينية–اختيار الصفات–تنقيب البيانات :الدالة الكلمات
