Internati
o
nal
Journal of Ele
c
trical
and Computer
Engineering
(IJE
CE)
V
o
l.
5, N
o
. 5
,
O
c
tob
e
r
201
5, p
p
. 1
164
~117
3
I
S
SN
: 208
8-8
7
0
8
1
164
Jo
urn
a
l
h
o
me
pa
ge
: h
ttp
://iaesjo
u
r
na
l.com/
o
n
lin
e/ind
e
x.ph
p
/
IJECE
Survey on Mutati
on-bas
ed
Test Data Gen
e
ration
Le Thi My H
a
nh,
Ngu
y
en T
h
an
h Binh,
K
huat Th
an
h T
ung
The Univ
ersity
o
f
Danang
, Univ
ersity
of Scien
c
e
and Technolo
g
y
, Vietn
a
m
Article Info
A
B
STRAC
T
Article histo
r
y:
Received Apr 27, 2015
Rev
i
sed
Jun
23,
201
5
Accepted
Jun 16, 2015
The c
r
iti
ca
l a
c
ti
vit
y
of t
e
sting
i
s
the s
y
st
em
ati
c
selec
tion of
suitabl
e t
e
st
cas
es
, whi
c
h is
a
b
le to
rev
eal
hig
h
l
y
the fau
lts.
Therefore, mutation cover
a
ge
is an eff
ect
ive
cr
iterion
for gen
e
r
a
ting
test d
a
ta
. S
i
nce
the
test d
a
ta
gener
a
tion
process is ver
y
labor intensiv
e,
ti
me-consuming and error-prone
when done
manually
,
the automation of
this proce
s
s
is
hig
h
l
y
as
pir
e
d.
The
res
ear
che
s
about automatic test data gener
a
tion c
ontr
i
buted a set of tools, approaches
,
development an
d
empirica
l res
u
lts. This p
a
peranaly
zes and
conducts a
comprehensive s
u
rvey
on g
e
ner
a
ting test
d
a
ta based on mutation. The pap
e
r
also an
al
yz
es th
e
trends
in
this fi
e
l
d.
Keyword:
C
onst
r
ai
nt
bas
e
d
t
e
st
i
n
g
Dynam
i
c sym
b
olic exec
ution
Mu
tatio
n
testing
Searc
h
-base
d
t
e
st data
gene
rat
i
o
n
Test
dat
a
ge
ner
a
t
i
o
n
Copyright ©
201
5 Institut
e
o
f
Ad
vanced
Engin
eer
ing and S
c
i
e
nce.
All rights re
se
rve
d
.
Co
rresp
ond
i
ng
Autho
r
:
Le Thi My
Ha
nh,
Inform
atio
n
Tech
no
log
y
Faculty,
Th
e
Un
iv
eristy
of Dan
a
ng
,
Un
iv
ersity o
f
Sci
e
n
ce an
d Techn
o
l
o
g
y
5
4
Ng
u
y
en
Luon
g Bang
, D
a
n
a
n
g
,
5
500
00
, V
i
et N
a
m
.
Em
a
il: l
t
m
h
a
n
h
@
du
t.udn
.v
n
1.
INTRODUCTION
So
ft
ware testing
is an
im
p
o
r
tan
t
m
ean
s u
s
ed
to
assu
re th
e qu
ality o
f
software.
Ho
wev
e
r,
testin
g
is
an
expe
nsi
v
e, t
e
di
ous
an
d t
i
m
e-
con
s
um
i
ng act
i
v
i
t
y
as i
t
can
consum
e
m
o
re than
50% of
th
e to
tal co
st
o
f
t
h
e
soft
ware
dev
e
l
opm
ent
[1]
.
A
u
t
o
m
a
t
i
c
t
e
st
dat
a
gene
rat
i
on
i
s
one
of t
h
e ap
pr
oac
h
es t
o
i
m
pr
o
v
e t
h
e acc
u
r
acy
as
well as to
red
u
ce th
e co
st o
f
testin
g
activ
ity. Th
e effectiv
eness o
f
test d
a
ta is
measu
r
ed
by th
e ab
ilit
y to
rev
e
al
the undetected faults i
n
s
o
ft
ware
. Howeve
r, c
o
m
p
lete
te
sting is i
n
feas
ible due to the vast i
n
put s
p
aces
in
vo
lv
ed
and
a lo
t
o
f
con
s
trai
n
t
s
o
n
th
e test
set. In
o
r
d
e
r to ov
erco
m
e
th
ese li
m
ita
tio
n
s
,
criteria are u
s
ed
t
o
provide a
requi
r
em
ent for test
data
ade
q
uacy, and so gi
ve a
measure for
imp
r
ov
ing
a test set. Th
e co
m
p
li
an
ce
with
a certain
stan
d
a
rd
will gen
e
rate ad
equ
a
te test sets and thu
s
it is ab
le
to
ex
po
se th
e
fau
lts ex
isting
i
n
t
h
e
pr
o
g
ram
und
er
t
e
st
(PUT
).
A
d
eq
uacy
cri
t
e
ri
a are u
s
ual
l
y
r
e
l
e
vant
t
o
t
e
st
cove
ra
ge. T
h
i
s
cove
ra
ge i
s
us
ed t
o
g
u
i
d
e
search
p
r
o
cess as
well as assess th
e q
u
a
lity o
f
t
h
e ob
tain
ed
test set. Co
v
e
rag
e
m
easu
r
es may b
e
cl
assi
fi
ed i
n
t
o
t
w
o
m
a
i
n
cat
egori
e
s:
st
r
u
ct
u
r
e
base
d a
n
d m
u
t
a
t
i
on-
base
d c
o
vera
ge c
r
i
t
e
ri
a.
The structure c
ove
ra
ge s
p
ecifies testing
requirem
e
nts
in
term
s o
f
th
e cov
e
rag
e
of a particu
l
ar set
of
ele
m
ents in the program
and include
s co
nt
r
o
l
-fl
ow a
n
d da
t
a
-fl
o
w
ba
sed
cri
t
e
ri
a. H
o
we
ver
,
t
h
ese c
r
i
t
e
ri
a
d
o
not
f
o
cus
o
n
t
h
e ca
use
of
p
r
og
ram
'
s fai
l
u
re
s, w
h
i
l
e
t
h
e mu
tatio
n
ad
equ
a
cy criterio
n
does. In
fact, m
u
tatio
n
cove
ra
ge
pr
o
v
i
d
es a
m
easure
o
f
t
e
st
dat
a
e
ffect
i
v
e
n
ess
by showi
n
g tha
t
the tests ca
n expose all
possible
si
m
p
le fau
lts o
f
a p
r
o
g
ram
.
De Millo
et al.,
[2
] p
r
opo
sed
m
u
ta
tio
n
testin
g
to
prov
id
e a
mean
s o
f
iterativ
ely
im
pro
v
i
n
g t
e
st
dat
a
ade
q
uacy
fo
r P
U
T.
Based
o
n
th
e m
u
ta
tio
n
ad
equ
acy
criteri
o
n
, fau
lt
indu
ced
variants of
the prog
ram
are e
x
e
c
uted
with a
test set to
fi
n
d
ou
t how m
a
n
y
v
a
rian
ts fail. Th
e m
o
re th
at
fail, the
great
er the
ade
q
ua
cy o
f
th
at test set. Th
e
t
e
st
er'
s
aim
i
s
to ge
ne
rat
e
ne
w t
e
st
dat
a
t
h
a
t
im
prove
s t
h
e ad
equ
acy of th
e ex
isting
tests. Mu
tatio
n
cov
e
rag
e
sub
s
um
es m
a
n
y
st
ruct
u
r
al
c
o
vera
ge c
r
i
t
e
ri
a.
It
m
a
y
det
ect
faul
t
s
t
h
at
st
r
u
ct
ural
t
e
st
i
n
g ca
n
not
di
sc
ove
r
[3
]
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 5
,
O
c
tob
e
r
20
15
:
116
4
–
11
73
1
165
There a
r
e som
e
excel
l
e
nt
sur
v
ey
s of t
e
st
dat
a
gene
rat
i
on t
e
chni
que
s, suc
h
as [4]
,
[5]
,
[
6
]
,
[7]
.
H
o
weve
r,
al
l
of
them used the
structure cove
rage to
ge
nerat
e
t
e
st
sui
t
e
s. Thi
s
w
o
r
k
co
nsi
d
ers t
h
e a
p
p
r
o
aches usi
ng m
u
t
a
t
i
on
an
alysis to
id
en
tify a set o
f
test d
a
ta th
at
max
i
mizes th
e n
u
m
b
e
r o
f
m
u
tan
t
s k
illed
.
In
oth
e
r wo
rd
s, th
e p
a
p
e
r
t
a
rget
s t
o
a
n
al
y
ze and
di
sc
u
ss an
o
v
er
vi
e
w
o
f
t
e
st
d
a
t
a
ge
nerat
i
o
n m
e
t
h
o
d
s
w
hi
c
h
h
a
ve
been
use
d
i
n
t
h
e
m
u
t
a
t
i
on t
e
st
i
n
g
dom
ai
n. O
u
r
researc
h
ai
m
s
t
o
a
n
sw
er t
w
o
f
o
l
l
o
wi
ng
q
u
est
i
ons:
+
Q1
:
What meth
od
sh
av
e b
e
en
u
s
ed
for test
d
a
ta g
e
n
e
ratio
n u
s
i
n
g m
u
tatio
n
an
alysis?
+
Q2
:
Ho
w can
th
ese test data g
e
n
e
ration
m
e
th
od
sb
e catego
r
ized
?
The o
r
ga
ni
zat
i
on
of t
h
e pa
pe
r i
s
as fol
l
o
ws.
Sect
i
on 2 i
n
t
r
od
uces t
h
e bac
k
g
r
ou
n
d
of m
u
t
a
t
i
on-
base
d
test data ge
ne
ration.
The
pa
rt
icular m
e
thods
will be
pres
ented
in section 3. In
sec
tion
4, som
e
challenges and
fu
t
u
re t
r
end
s
i
n
test
d
a
ta g
e
neratio
n will b
e
d
i
scu
sse
d. Sectio
n
5
is t
h
e
co
n
c
l
u
sion
and
g
e
n
e
ral assessm
en
t
abo
u
t
m
e
t
hods
.
2.
BACK
G
R
O
U
ND
OF
TEST
DAT
A GE
NE
RATI
O
N
BA
SED O
N
M
U
TATION
A
N
A
LYS
I
S
Mu
tatio
n
an
al
ysis su
p
ports so
ft
ware testin
g b
y
asse
ssin
g
th
e qu
ality o
f
t
e
st sets as wel
l
as creatin
g
th
e test d
a
ta t
h
at can
d
e
tect
fau
lts in
p
r
ogram
. In
th
is te
stin
g
pro
c
ess,
fau
lts are in
serted
in
to
PUT. Th
e
resu
lting
ch
ang
e
d v
e
rsion
s
of th
e test
p
r
og
ra
m
are called
m
u
tan
t
s.
Each
v
a
rian
t, o
r
m
u
tan
t
, d
i
ffers from
th
e
PUT
by
a sm
all
am
ount
, s
u
c
h
as a si
ngl
e
l
e
xe
m
e
, an
d i
s
ge
n
e
rat
e
d
by
a m
u
t
a
t
i
on
ope
rat
o
r.
M
u
t
a
t
i
on
ope
r
a
t
o
rs
can be see
n
as
rep
r
ese
n
t
i
ng c
o
m
m
on faul
t
s
usu
a
l
l
y
fou
n
d
i
n
so
ft
wa
re. T
h
us, m
u
t
a
t
i
on o
p
erat
ors a
r
e de
si
gne
d
by
basi
n
g
o
n
t
h
e m
o
st
co
m
m
on faul
t
s
com
m
i
t
t
e
d by
prog
ram
m
ers when usi
n
g a pr
o
g
ram
m
i
ng l
a
ngua
ge.
Fi
gu
re 1
prese
n
t
s
an exam
pl
e abo
u
t
t
h
e ori
g
i
n
al
pr
o
g
ram
and i
t
s
m
u
t
a
nt
pr
og
ram
when a
ppl
y
i
n
g
t
h
e rel
a
t
i
onal
ope
rat
o
r repl
ac
em
ent
ope
rat
o
r
.
M
u
t
a
t
i
on-
base
d t
e
st
dat
a
gen
e
rat
i
on
pr
ocee
ds by
i
n
ject
i
n
g
sy
nt
act
i
c
m
u
tat
i
ons i
n
t
o
a gi
ven
pr
og
ram
P
, gene
rat
i
n
g
f
r
om
P
a set
M
of m
u
tants. The goal of t
h
is a
c
tivity is to
find a
set of test-c
ases that
kill each
of
th
e m
u
tan
t
m
M
. This m
eans that the test-case
m
a
kes the m
u
ta
nt
m
pro
d
u
ce o
u
t
p
ut
s t
h
a
t
di
ffer
fr
om
t
hose o
f
t
h
e ori
g
i
n
al
pr
og
ram
P
. For the above e
x
ample, test data (
x
= 5,
y
= 5) is a
b
le to detect the
m
u
tant because the
o
u
t
p
u
t
of
th
e or
ig
in
al pr
ogr
am is
10
wh
ereas th
e o
u
t
pu
t of th
e
m
u
tan
t
is
0
. Ho
we
ver
,
test data (
x
= 5,
y
= 4)
can
no
t
d
e
tect th
is m
u
tan
t
du
e
to
th
e
sam
e
o
u
t
p
u
t
pr
odu
ced fo
r bo
th pr
ogr
am
s.
in
t fo
o (in
t
x
,
i
n
t y)
{
in
t z =
0
;
i
f
(
x
y)
z = x + y;
return z;
}
Orig
ina
l
Progra
m
in
t fo
o (in
t
x
,
i
n
t y)
{
in
t z =
0
;
i
f
(
x
>
y)
z = x + y;
return z;
}
Mu
ta
nt
Pr
ogr
am
Fi
gu
re
1.
A
m
u
t
a
nt
exam
pl
e
In the
case, there
is
no test data t
h
at are
able
t
o
det
e
c
t
m
u
t
a
nt
beca
use t
h
e m
u
t
a
n
t
pr
o
g
ram
i
s
fu
nct
i
o
nal
l
y
equi
val
e
nt
t
o
t
h
e
ori
g
i
n
al
o
n
e a
n
d cal
l
e
d e
qui
v
a
l
e
nt
m
u
t
a
nt
. A m
u
t
a
nt
i
s
sai
d
t
o
be eq
ui
va
l
e
nt
i
f
there is not such a test case able
t
o
di
f
f
ere
n
t
i
a
t
e
bet
w
een
t
h
e out
put
of
t
h
e
m
u
t
a
nt
and t
h
e o
u
t
p
ut
of t
h
e
ori
g
i
n
al
pr
og
ra
m
.
In
m
u
tatio
n
testin
g
,
each
test d
a
ta can
d
e
tect so
m
e
m
u
tan
t
s
in
PUT. It is u
n
lik
ely th
at on
e test will
k
ill all
m
u
tan
t
s, requ
iring
t
h
at, th
e
resu
lt o
f
test
d
a
ta
gen
e
ration
b
a
sed
o
n
m
u
tatio
n
m
u
st in
co
rporate a
su
fficien
t
nu
mb
er
o
f
tests. It is ex
p
ected
th
at th
is te
st set
ca
n
d
e
tect as
m
a
n
y
m
u
tan
t
s as p
o
s
sib
l
e. Th
e qu
ality
o
f
test sets is e
v
alu
a
ted
thro
ug
h
m
u
tatio
n
sco
r
e. It is th
e p
r
o
portio
n
of m
u
tan
t
s k
illed
o
u
t
o
f
all n
o
n
-equiv
a
len
t
m
u
t
a
nt
s t
h
at
are ge
nerat
e
d f
r
o
m
PUT by
ap
pl
y
i
ng m
u
t
a
t
i
on
ope
rat
o
rs.
I
f
t
h
i
s
pr
o
p
o
r
t
i
o
n i
s
eq
ual
t
o
1,
t
h
e
n
t
e
st
set is ad
eq
u
a
te
an
d cap
ab
le t
o
d
e
tect th
e
fau
l
t
s
in
PUT h
i
g
h
l
y.
It
i
s
st
at
ed t
h
at
m
u
t
a
t
i
on t
e
st
ing i
s
a
n
effect
i
v
e m
e
t
hod t
o
assess t
h
e q
u
al
i
t
y
of t
e
st
dat
a
. Ho
we
ver
,
m
o
st
of st
u
d
i
e
s
i
n
t
h
i
s
fi
el
d
f
o
cus
on
re
d
u
ci
n
g
c
o
m
put
at
i
o
n
a
l
expe
nse
.
T
h
ere
has
been
l
e
ss w
o
r
k
o
n
a
p
p
l
y
i
ng
m
u
ta
tio
n
cov
e
rag
e
i
n
th
e co
n
t
ex
t of test d
a
ta g
e
n
e
rati
o
n
. In
th
is
p
a
p
e
r, we
will an
alyze th
e test d
a
ta
g
e
n
e
ration
tech
n
i
q
u
e
s using m
u
tat
i
o
n
co
verag
e
t
o
assess
th
e
q
u
a
lity o
f
ob
tain
ed
test sets syste
m
atically.
These m
e
t
hod
s are
di
vi
de
d
i
n
t
o
s
o
m
e
grou
ps:
ra
n
dom
t
e
st
dat
e
ge
n
e
rat
i
o
n
,
co
nst
r
ai
nt
-base
d
t
e
st
dat
a
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Sur
vey on
Mut
a
tion-base
d
Te
st
Dat
a
Ge
ner
a
tion
(Le
Thi
My Hanh)
1
166
gene
rat
i
o
n, e
n
hance
d
c
o
nt
rol
fl
o
w
g
r
a
p
h
,
d
y
n
am
i
c
sym
bol
i
c
execut
i
o
n, s
earch
-ba
s
ed t
e
st
dat
a
ge
nerat
i
on
an
d
th
e h
y
b
r
i
d
techn
i
qu
es. Th
e d
e
t
a
ils o
f
t
h
ese m
e
th
od
s
will b
e
p
r
esen
ted in
n
e
x
t
section
.
3.
MUT
A
TION-BASED TEST DAT
A
GENERATION TECHNIQUE
S
Aft
e
r
c
o
l
l
ect
i
ng
t
h
e
t
e
st
dat
a
gene
rat
i
o
n
t
e
c
hni
que
s base
d on
m
u
t
a
t
i
on
t
e
st
i
ng, we cl
ass
i
fy
t
h
em
i
n
t
o
si
x cat
ego
r
i
e
s:
rand
om
t
e
st dat
a
generat
i
o
n
(G1
)
, c
ons
t
r
ai
nt
-
b
ased t
ech
n
i
ques (
G
2), e
n
hance
d
co
nt
r
o
l
fl
ow
gra
p
h (G
3)
, dy
nam
i
c sym
bol
ic execut
i
o
n (
G
4)
, searc
h
-
b
as
e
d
t
e
st
dat
a
gen
e
rat
i
on (
G
5)
, and
h
y
b
r
i
d
t
ech
n
i
ques
(G6). These c
a
tegories are deri
ved
based on the chara
c
teristics of each m
e
thod.
G1
gene
rates test data
random
l
y within the
input
space.
G2 co
m
p
rises the tec
hni
que
s
using c
onstraint
s
on test data t
o
kill mutants.
G3 c
o
nt
ai
ns
o
n
t
h
e st
udi
es
descri
bi
n
g
t
e
c
hni
que
s rel
y
i
n
g o
n
c
ode c
o
v
e
rage
, anal
y
s
i
s
of t
h
e dat
a
fl
ow a
n
d
co
n
t
ro
l flow to
g
e
n
e
rate test d
a
ta. G4
in
cl
u
d
e
s th
e
approache
s
that incorporat
e exec
ution
of the
program
un
de
r t
e
st
dat
a
wi
t
h
sel
ect
i
on
of pat
h
s f
r
om
it
s cont
r
o
l
fl
o
w
gra
p
h an
d sol
v
i
ng t
h
e co
nst
r
a
i
nt
s on i
n
put
d
a
t
a
t
o
k
ill
m
u
tan
t
. G5
con
s
ists
o
f
tech
n
i
q
u
e
s th
at ap
p
l
y th
e m
e
ta-h
euristic alg
o
ri
th
m
s
to
d
e
termin
e th
e
b
e
st so
l
u
tio
n
in a sea
r
c
h
s
p
ace of a
gi
ven problem
.
The
approac
h
es
i
n
G5 called the
searc
h
-b
ase
d
techniques
and the
p
r
o
cess
o
f
g
e
neratin
g
test d
a
t
a
k
illin
g
m
u
tan
t
s is gu
id
ed
by fitn
ess fu
n
c
ti
o
n
s
. G6
is th
e co
m
b
in
atio
n
o
f
G4
and
G5
becaus
e
it uses the
meta-he
u
ristic algorithm
s
and DSE (dynam
ic
sym
bolic
execution) techniques to
g
e
n
e
rate test sets.
3.
1. R
a
nd
om T
e
st
D
a
ta
Ge
nera
ti
on
R
a
nd
om
Test
ing i
s
o
n
e of t
h
e
m
o
st
fundam
e
nt
al
and p
o
p
u
l
ar
m
e
t
hods. It
i
s
sim
p
l
e
i
n
conce
p
t
,
easy
t
o
im
pl
em
ent
,
and ca
n be
u
s
ed o
n
i
t
s
o
w
n o
r
as a com
p
o
n
e
n
t
of m
a
ny
ot
he
r t
e
st
i
n
g m
e
t
hods. R
a
nd
o
m
approaches
ge
nerate test input vect
ors
wi
t
h
el
em
ent
s
rand
om
l
y
chosen
fr
o
m
ap
pr
op
r
i
ate do
m
a
in
s. In
put
v
ectors are
g
e
n
e
rated
un
til so
m
e
id
en
tified
criteria h
a
v
e
b
een
satisfied
su
ch
as th
e max
i
m
a
l n
u
m
b
e
r o
f
test
cases. R
a
nd
o
m
t
e
st
i
ng m
a
y be a
n
e
ffect
i
v
e m
eans o
f
gai
n
i
n
g a
n
a
d
equat
e
t
e
st
set
fo
r si
m
p
l
e
pr
og
ram
s
.
Ho
we
ver
,
it
m
a
y
sim
p
ly
fail to ge
ne
rate ap
p
r
o
p
riate d
a
ta in
any
reas
ona
bl
e tim
e
-fram
e
f
o
r c
o
m
p
lex so
f
t
ware
th
at h
a
s strict
requ
irem
en
ts o
f
the d
a
ta
d
o
main
sp
ecificatio
n
.
For th
e
v
a
st d
a
ta
d
o
m
ain
,
it is d
i
ffi
cu
lt to
rando
m
l
y g
e
n
e
rate test d
a
ta t
h
at can
d
e
tect h
a
rd
-t
o
-
k
ill m
u
tan
t
s. Fo
r exam
p
l
e in
Fig
u
re 1
,
th
e m
u
tan
t
is on
ly
k
illed
wh
en
x
is equ
a
l to
y
.
This is d
i
fficu
lt to b
e
ach
i
eved if
rando
m
test g
e
n
e
ration
i
n
a
vast d
a
ta
d
o
m
ai
n
.
Som
e
st
udi
es
have
bee
n
co
nd
uct
e
d
t
o
i
m
pr
o
v
e t
h
e
l
i
m
i
t
a
t
i
on o
f
ra
nd
om
appr
oac
h
e
s
. A
d
a
p
t
i
v
e
ran
d
o
m
t
e
sti
n
g
(AR
T
) was pr
op
ose
d
by
C
h
e
n
et al.
[8
] as an
en
h
a
n
cem
en
t to
p
u
r
e
r
a
ndom
testin
g
.
Ad
ap
tiv
e
random
testing seeks t
o
distribute test
cases m
o
re ev
en
ly
with
in
t
h
e inpu
t sp
ace.
It is b
a
sed
on
t
h
e i
n
tu
ition
th
at fo
r non
-p
oin
t
typ
e
s
o
f
fai
l
u
r
e
p
a
ttern
s, an
ev
en sp
read
o
f
test cases is
m
o
re lik
ely to
d
e
tect failures
u
s
i
ng
fewe
r t
e
st
cases t
h
an o
r
di
na
ry
ran
dom
t
e
sti
ng. T
h
e f
u
n
d
a
m
e
nt
al
i
d
ea
behi
nd
AR
T i
s
t
o
rewar
d
di
versi
t
y
a
m
ong sam
p
led test cases. If a test case does not detect
an
y failure, t
h
en
in
presence
o
f
con
tig
uou
s
fau
lty
regi
ons i
t
w
oul
d be
bet
t
e
r t
o
s
a
m
p
l
e
t
e
st
cases t
h
at
are far
fr
om
it
. The di
st
ance am
ong t
e
st
cases i
n
an i
n
p
u
t
dom
ai
n
D
de
p
e
nd
s o
n
t
h
e t
y
pe o
f
P
U
T.
I
n
t
h
e case o
f
n
u
m
eri
cal
i
nput
s,
t
h
e Eucl
i
d
ea
n
di
st
ance ca
n b
e
use
d
.
C
h
en
’s ap
p
r
oa
ch ge
nerat
e
s c
a
ndi
dat
e
i
n
p
u
t
s
ran
d
o
m
l
y
,
a
nd at eve
r
y step selects from the
m
the one that is
furth
e
st away fro
m
th
e alr
ead
y u
s
ed
inpu
ts. ART
was in
itiall
y in
trod
u
c
ed
for numerical v
a
lu
es an
d
it
calculates the
distance
betwe
e
n two s
u
ch
va
lues usi
ng t
h
e
Euclidea
n m
e
a
s
ure
.
T
h
e ba
sic algorithm
of ART is
depi
ct
ed i
n
Fi
gu
re 2
.
R
e
sul
t
s sho
w
t
h
at
ad
apt
i
v
e ra
nd
om
t
e
st
i
ng d
o
es o
u
t
p
e
r
f
o
rm
ordi
nary
ra
n
dom
test
i
n
g
sig
n
i
f
i
can
tly (
by u
p
t
o
as m
u
ch
as
5
0
%)
f
o
r
t
h
e set of
p
r
og
ra
m
s
u
n
d
e
r
st
udy. H
o
w
e
v
e
r, the r
e
str
i
ctio
ns of
th
is
m
e
t
hod are t
h
a
t
i
t
can generat
e
onl
y
n
u
m
e
ral
t
e
st
dat
a
,
req
u
i
r
es m
o
re m
e
m
o
ry
and com
put
at
i
o
n t
h
a
n
pu
rel
y
r
a
ndo
m
m
e
th
od
.
Z
= {}
Ad
d
ra
nd
om
t
e
st
case t
o
Z
and
ex
e
c
u
te
it
repeat until
st
o
p
p
i
ng
criterion
is satisfied
sam
p
le
set
W
of ra
n
dom
t
e
st
cases
f
o
r e
a
ch
w
of t
h
ese |
W
| test ca
ses
w.min
D
= min
(
dist(
w
,z
∈
Z
))
execute
and a
d
d to
Z
th
e
w
w
i
t
h
ma
x
i
mu
m
minD
Fi
gu
re
2.
Al
go
ri
t
h
m
of AR
T
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 5
,
O
c
tob
e
r
20
15
:
116
4
–
11
73
1
167
3.
2.
C
o
ns
trai
n
t
B
a
s
e
d T
e
s
t
Da
ta
Ge
nera
ti
on
Co
n
s
t
r
ain
t
Based
Testing
(C
BT) was th
e
fi
rst test case g
e
n
e
ration
m
e
th
od
u
s
ed fo
r m
u
tatio
n
testing
.
It was propo
sed
b
y
DeMillo
an
d Offu
tt [9
], b
a
sed
o
n
th
e
i
d
ea o
f
con
t
ro
l flow
an
alysis, sy
m
b
o
lic
ex
ecu
tio
n,
co
nstrain
t
on
m
u
tan
t
s an
d
pro
g
ram
ex
ecu
tio
n in ord
e
r t
o
g
e
n
e
rate test
data au
to
m
a
tica
lly. Accord
i
n
g to
t
h
e
authors, a m
u
tant shoul
d
be
killed if it satisfies
three conditions known
as
reac
hability, necessity and
sufficiency. The reacha
b
ility
condition e
x
presses that the m
u
ta
ted state
m
ent
m
u
st be reached
by test
data. If
test su
ites cann
o
t
ex
ecu
te t
h
e
m
u
tated
statemen
t, th
en
test
s h
a
ve no
ch
ance o
f
k
illin
g
the in
trodu
ced
m
u
tan
t
.
The necessity
condition
requi
r
es the ex
ec
ution
of the m
u
tated state
m
ent t
o
result in an error
in the program
’
s
state. This
m
e
ans t
h
at th
e
execution
outc
o
me of the
original and t
h
e m
u
t
a
t
e
d st
at
em
ent
s
m
u
st
be
di
ffe
rent
.
Oth
e
rwise, t
h
e syn
t
actical eq
u
a
lity o
f
th
e rest o
f
th
e
o
r
i
g
inal an
d
m
u
tan
t
p
r
og
ram
v
e
rsio
n
s
will n
e
ver
create
th
e d
i
fferen
t
co
m
p
u
t
atio
n
s
an
d
will
th
u
s
n
e
v
e
r resu
lt in
v
i
sib
l
e o
u
t
p
u
t
d
i
fferen
ces. Th
e
su
fficien
c
y con
d
ition
states that the
incorrect state
m
u
st
propa
ga
t
e
t
o
t
h
e l
a
st st
at
em
ent
and creat
e at
l
east
one
di
ffe
rent
out
put
bet
w
ee
n
ori
g
i
n
al
and
m
u
t
a
t
e
d pr
o
g
ram
.
The
C
B
T
m
e
t
hod
h
a
s bee
n
i
m
pl
em
ent
e
d i
n
a
t
o
ol
cal
l
e
d
God
z
illa
in
or
der t
o
t
e
st
t
h
e Fort
ra
n
pr
og
r
a
m
s
and has
be
en i
n
t
e
g
r
at
ed
wi
t
h
t
h
e
Mot
h
r
a
[1
0
]
m
u
tatio
n
testin
g
too
l
set. CBT
has em
pi
ri
cal
l
y
bee
n
s
h
o
w
n t
o
be a
n
e
ffect
i
v
e
ap
pr
oach;
h
o
w
eve
r
, i
t
has c
e
rt
ai
n
dra
w
bac
k
s
wi
t
h
re
spect
t
o
t
h
e
sym
bol
i
c
eval
uat
i
on
w
h
e
n
sol
v
i
ng t
h
e h
a
ndl
i
n
g
of a
r
r
a
y
s
, l
o
o
p
s,
n
o
n
-l
i
n
ea
r e
x
p
r
e
ssi
ons
an
d t
h
e pat
h
expl
osi
o
n pr
o
b
l
em
.
To
ov
erco
m
e
t
h
ese d
i
fficu
lties,
Offu
t
et al.
p
r
op
o
s
ed
D
yna
m
i
c
D
o
m
a
in
Red
u
c
tion
(
D
D
R
)
m
e
thod
wh
ich
was p
r
esen
ted
in
[11
]
. Th
is
ap
pro
ach still
retain
s reach
a
b
ility, n
ecessity an
d
sufficien
cy con
d
ition
s
b
u
t
im
pro
v
es h
o
w
t
o
han
d
l
e
t
h
ese
con
d
i
t
i
ons
. Th
i
s
m
e
t
hod gene
rat
e
s t
e
st
s by
basi
ng o
n
t
h
e re
duct
i
o
n o
f
t
h
e i
n
p
u
t
spaces of the
varia
b
les invol
v
ed. Th
e
DDR
approac
h
take
s in account the test data generation
problem as a
dy
nam
i
c pat
h
base
d p
r
obl
em
. It
uses t
h
e
p
r
og
ram
cont
r
o
l
fl
o
w
g
r
a
ph a
n
d
bases
o
n
a sea
r
ch
he
uri
s
t
i
c
m
e
t
h
o
d
ove
r t
h
e i
n
put
varia
b
les
dom
ain to ge
ne
ra
te
t
e
st data. Whe
n
it reaches a bran
ch
poi
nt in the path, the va
riables
use
d
wit
h
in t
h
at bra
n
c
h
pre
d
icate have
their
dom
ains
re
duc
ed in accordance with t
h
e e
x
e
c
ution of t
h
e
desired
bra
n
c
h
pat
h
.
I
f
t
h
ere
i
s
a c
h
oi
ce o
f
ho
w t
o
r
e
duce
t
h
e
d
o
m
a
i
n
, a
searc
h
p
r
oces
s i
s
m
a
de fr
om
t
h
e su
bs
eque
nt
pat
h
i
n
o
r
de
r
n
o
t
t
o
m
a
ke an
i
n
ap
pr
o
p
ri
at
e s
e
l
ect
i
on an
d r
e
st
ri
ct
subse
q
ue
nt
b
r
anc
h
out
p
u
t
s
.
Up
o
n
reac
hi
n
g
the target
node
, the rem
a
ining values
i
n
each varia
b
le's domain re
prese
n
t the val
u
es that
will cause exe
c
uti
on
of
t
h
e desi
re
d pat
h
,
i.e.
fo
r
m
u
ta
tio
n
testin
g
wh
ere t
h
e mu
tated
statem
e
n
t is th
e targ
et
n
o
d
e
; th
e rem
a
in
in
g
d
o
m
ain
s
rep
r
esen
t test v
a
l
u
es
th
at satisfy th
e reach
a
b
ility co
n
s
t
r
ain
t
.
In
some cases th
oug
h, a
do
m
a
in
will b
e
e
m
pty indicating that t
h
e
procedure
fa
iled e
ither
because t
h
e
path is infe
asible or it
wa
s too
diffic
u
lt to
find
v
a
lu
e to
ex
ecu
t
e it. Offu
tt
et al.
[1
1]
concl
u
d
e
d t
h
at
DDR
“
is less like
l
y to
fa
il to
f
i
n
d
a
test ca
se wh
en
a
test
ca
se exists, and
tha
t
imp
l
emen
ta
tion
s
can
be mo
re efficient
”. Com
p
ared t
o
the
CBT
ap
pr
o
a
ch
, th
is
p
r
oc
e
d
ur
e
wo
uld
seem
m
o
re
fa
vo
ra
ble.
3.
3. E
n
ha
nced
C
o
ntr
o
l
Fl
ow
Gra
p
h
Th
is app
r
o
ach
tran
sform
s
th
e
p
r
ob
lem
o
f
g
e
n
e
rating
test
data k
illin
g m
u
t
a
n
t
s to a cov
e
rin
g
bran
ch
es
altern
ativ
e. Thu
s
, effectiv
e heu
r
istic
s app
lied
fo
r bran
ch
testin
g
can b
e
ex
tend
ed to
mu
tan
t
s too
.
The
m
o
st
po
p
u
l
a
r m
e
t
h
o
d
s
fo
r
bra
n
c
h
t
e
st
i
ng a
r
e t
hos
e t
h
at
sel
ect
s
p
ecific p
a
th sets to
g
e
n
e
rate the soug
h
t
test
data. As
wi
t
h
al
l
pat
h
g
e
nerat
i
o
n m
e
t
hods
t
h
ei
r
m
a
jor de
fi
ci
ency
i
s
t
h
e ge
ne
rat
i
o
n
of i
n
fea
s
i
b
l
e
p
a
t
h
s, t
h
i
s
p
r
o
b
l
e
m
i
s
al
so i
nhe
ri
t
e
d
whe
n
em
pl
oy
i
ng
pat
h
gene
ra
t
i
on f
o
r
per
f
o
r
m
i
ng
m
u
t
a
t
i
o
n
t
e
st
i
ng t
oo.
In
[1
2]
, Papa
da
k
i
s and
M
a
l
e
vri
s
pr
o
p
o
se
d an effect
i
v
e pat
h
sel
ect
i
on st
rat
e
gy
in order to ac
hieve adequate coverage
. The pre
s
ented
strateg
y
targ
ets o
n
g
e
n
e
rating
m
u
ta
tio
n
ad
equ
a
te test sets
i
n
an
effecti
v
e way. Th
is is ach
iev
e
d
b
y
a strateg
y
th
at redu
ces the effects o
f
infeasib
le p
a
th
s an
d
eq
u
i
v
a
len
t
m
u
tan
t
s. Th
e au
tho
r
s
b
u
ilt an en
h
a
n
ced
g
r
ap
h
b
y
addi
ng a s
p
eci
al type of
vert
ex
for eac
h m
u
tant.
Eve
r
y
mu
tan
t
related
vertex
is con
n
ected
with its orig
in
al
corres
ponding node a
nd
repre
s
ents a special
m
u
tant rela
ted necessity const
r
aint [
12]. T
r
ea
ting each m
u
tant as
a b
r
an
ch
h
e
lp
s
fo
cu
s
on
sp
ecific
m
u
tan
t
s and
select can
d
i
d
a
te p
a
th
s in
o
r
d
e
r to
g
e
n
e
rate
data ai
m
i
n
g
at killin
g
t
h
e speci
fi
e
d
m
u
t
a
nt
s. Fi
gur
e 3 dem
onst
r
a
t
es t
h
e con
s
t
r
uct
i
on
of t
h
e
enha
nce
d
co
nt
rol
fl
ow
gra
p
h (t
he
o
r
i
g
in
al co
n
t
rol flow
g
r
ap
h on th
e left and
the enh
a
n
c
ed
one in
clud
ing
t
h
ree m
u
tan
t
s o
n
th
e
righ
t).
The m
a
i
n
i
d
ea of t
h
e p
r
o
p
o
s
e
d ap
p
r
oac
h
i
s
t
o
re
duce t
h
e
pr
obl
em
of f
u
l
f
i
l
l
m
e
nt
of t
h
e nece
ssi
t
y
conditions of m
u
tants to that of coveri
ng
bra
n
c
h
es.
T
h
is
is done
by transform
i
ng the require
d
nec
e
ssity
co
nstr
ain
t
s in
t
o
br
an
ch
pr
ed
i
cates an
d
r
e
p
r
esen
ting
th
em
i
n
th
e pr
og
r
a
m
co
n
t
r
o
l f
l
ow
grap
h. Th
e innov
ation
of t
h
i
s
ap
p
r
oac
h
i
s
t
h
e use of
an en
hance
d
c
ont
rol
fl
o
w
g
r
a
ph a
nd i
t
s
resp
ect
i
v
e bra
n
ch
pre
d
i
cat
es i
n
o
r
de
r t
o
pr
ocee
d wi
t
h
t
h
e sym
bol
i
c
eval
uat
i
o
n an
d t
e
st
dat
a
gener
a
t
i
on p
h
ase. T
h
e be
nefi
t
of t
h
e ap
pr
oac
h
i
s
t
h
e
in
corpo
r
ation
o
f
m
u
tatio
n
b
a
sed
criteria with
ex
isting
p
a
t
h
based t
e
st
dat
a
gene
rat
i
on m
e
t
h
o
d
s i
n
heri
t
i
ng al
l
th
eir m
e
rits. Th
e
d
e
tails o
f
this m
e
th
o
d
are presen
ted
in [12].
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Sur
vey on
Mut
a
tion-base
d
Te
st
Dat
a
Ge
ner
a
tion
(Le
Thi
My Hanh)
1
168
Fi
gu
re 3.T
h
e E
nha
nce
d
C
ont
r
o
l
Fl
o
w
G
r
ap
h [1
2]
En
hance
d
co
nt
rol
fl
ow
gra
p
h can be c
o
n
s
i
d
e
r
ed as a s
u
p
p
l
e
m
e
nt
t
o
t
h
e t
h
eory
o
f
t
e
st
dat
a
gene
rat
i
o
n
m
e
t
hods
base
d
on m
u
t
a
t
i
on cove
ra
ge. It
t
r
ansf
o
r
m
s
t
h
e test
dat
a
gene
r
a
t
i
on p
r
o
b
l
e
m
i
n
t
o
pat
h
s
e
l
ect
i
o
n
pr
o
b
l
e
m
i
n
gra
p
h
.
Ho
we
ver
,
f
o
r
a l
a
r
g
e
p
r
o
g
r
am
and t
h
ere
are m
a
ny
ge
ne
rat
e
d m
u
t
a
nt
s,
t
h
i
s
m
e
t
hod
wi
l
l
face
up t
o
t
h
e l
i
m
itat
i
on t
h
at
are t
h
e vast
n
u
m
b
er of
no
des i
n
t
h
e g
r
ap
h an
d t
h
e pat
h
ex
pl
os
i
on p
r
obl
em
.
Thes
e
red
u
ce t
h
e e
f
f
ect
i
v
eness
o
f
t
h
e m
e
t
hod
. T
h
ese
rest
ri
ct
i
o
ns
nee
d
t
o
be
f
u
rt
he
r
fi
g
u
re
d
out
t
o
i
m
pr
ove
t
h
e
effective
n
ess a
n
d
effici
en
cy
of th
e so
l
u
tio
n.
3.4. Dynamic Symb
o
lic Ex
ecution
Sym
bol
i
c
exec
ut
i
o
n
[
1
3]
i
s
an a
dva
nce
d
p
r
o
g
ram
anal
y
s
i
s
t
echni
que
i
n
whi
c
h i
n
p
u
t
s
an
d
ot
h
e
r
v
a
riab
les assume sy
m
b
o
lic rath
er th
an
p
a
rt
icu
l
ar v
a
l
u
es an
d
ou
tpu
t
is a
m
a
th
e
m
atica
l
ex
pressi
o
n
of th
ese
sym
bol
s. T
h
e s
y
m
bol
i
c
eval
u
a
t
i
on p
r
o
cess
of a prog
ram
co
n
s
ists of assignin
g
sym
b
o
lic valu
es to
v
a
riables in
or
der t
o
de
duc
e an abst
ract
a
l
geb
r
ai
c rep
r
es
ent
a
t
i
on
of t
h
e
pr
og
ram
’
s com
put
at
i
ons an
d re
prese
n
t
a
t
i
o
n. T
h
i
s
t
echni
q
u
e i
s
b
a
sed
o
n
t
h
e
se
l
ect
i
on o
f
pat
h
s fr
om
i
t
s
cont
rol
fl
o
w
gra
p
h
an
d t
h
e
com
put
at
i
on
o
f
sy
m
bol
i
c
st
at
es. The
sy
m
bol
i
c
st
at
e of a
pat
h
fo
rm
s a m
a
ppi
ng
fr
om
i
nput
vari
a
b
l
e
s t
o
sy
m
bol
i
c
val
u
es a
n
d
a set
o
f
con
s
t
r
ai
nt
s cal
l
e
d pat
h
c
o
ndi
t
i
ons
o
v
er t
h
ose
sym
bol
i
c
v
a
lu
es. Path
cond
itio
n
s
represen
t
a set o
f
con
s
train
t
s
cal
l
e
d sy
m
bol
ic ex
pres
si
o
n
s t
h
at
f
o
rm
t
h
e c
o
m
put
at
i
ons
p
e
rf
orm
e
d o
v
e
r
t
h
e sel
ect
ed
pa
t
h
. S
o
l
v
i
n
g
t
h
e
pat
h
co
nd
itio
ns resu
lts in
test
d
a
ta wh
ich
i
f
inpu
t to
t
h
e select
ed
p
a
th
, th
is
will b
e
ex
ecu
t
ed. If th
e p
a
t
h
con
d
ition
h
a
s no
so
lu
tion
th
e p
a
th
is in
feasi
b
le. Desp
ite o
f
th
e capab
ilities o
f
sy
m
b
o
lic ex
ecu
ti
o
n
, th
ere are sev
e
ral
p
r
ob
lem
s
asso
ciated
with
its ap
p
lication
:
p
a
th
selec
tion and the eval
uation of loops
,
m
o
dule calls, arra
ys and
co
nstrain
t
so
lvin
g
.
Th
e
p
r
ob
le
m
o
f
p
a
t
h
selectio
n
and
the ev
alu
a
tion
of loo
p
s
is that it will cau
se p
a
t
h
expl
osion.
Wh
en the size
of
program
increases, the size
of
const
r
ai
nt
ex
p
r
essi
o
n
s
obt
ai
n
e
d m
a
y
beco
m
e
ver
y
larg
e. Th
ese are th
e lim
i
t
atio
n
s
of th
is app
r
o
a
ch
an
d it is d
i
fficu
lt to
ap
p
l
y it fo
r a larg
e
p
r
og
ram
.
Dynam
i
c Symbolic E
x
ecution (DSE
) is a
m
o
re r
ecent innovation t
h
at
ove
rc
om
es
many of the
li
mitatio
n
s
of
trad
itio
n
a
l
sym
b
o
lic ex
ecu
tio
n. Th
is m
e
t
h
o
d
allo
ws au
tomatic g
e
n
e
ratio
n of test i
n
pu
ts th
at
achi
e
ve
hi
gh
c
ode
c
ove
rage
.
DSE
exec
ut
es
t
h
e
pr
og
ram
unde
r t
e
st
f
o
r
so
m
e
gi
ve
n t
e
st
i
n
p
u
t
s
(o
nes
ge
nerat
e
d
r
a
ndo
m
l
y)
, and
at t
h
e sam
e
ti
m
e
p
e
r
f
or
ms sym
b
o
lic
execu
tio
n in p
a
rallel to
co
llect sym
b
o
lic co
nstrain
t
s
obt
ai
ne
d
fr
om
pre
d
i
cat
es i
n
bra
n
c
h
st
at
em
ent
s
al
o
ng t
h
e
exec
ut
i
on t
r
a
ces. D
u
ri
ng
t
e
st
gene
rat
i
o
n,
DSE i
s
perform
e
d iteratively on the PUT to inc
r
eas
e code cove
rage. Initially, DSE random
ly choos
es one test input
from
the input
dom
a
in. Ne
xt, in t
h
e eac
h i
t
eration,
after
runn
ing
each
t
e
st in
pu
t,
DSE co
llects th
e p
a
t
h
co
nd
itio
n
o
f
t
h
e ex
ecu
tion
trace, and u
s
es a
search stra
teg
y
to
flip
a
branch
ing
n
o
d
e
in
th
e p
a
t
h
. Flippin
g
a
b
r
an
ch
ing
n
ode in
a path
constru
c
ts a
new path
th
at
sh
ares
th
e
p
r
efix
t
o
t
h
e no
de
with
t
h
e o
l
d
p
a
th
,
b
u
t
th
en
devi
at
es a
n
d t
a
kes a
di
ffe
rent
pat
h
.
Whet
her
suc
h
a
fl
i
p
pe
d
p
a
th is feasib
le is ch
eck
e
d
b
y
b
u
ild
i
n
g a con
s
train
t
syste
m
. If a co
n
s
t
r
ain
t
so
lv
er can
d
e
term
i
n
e th
at th
e con
s
train
t
system is satisfied
with
in
th
e av
ailab
l
e
reso
u
r
ces, D
S
E gene
rat
e
s a new t
e
st
i
n
p
u
t
t
h
at
wi
l
l
execut
e
al
on
g t
h
e f
l
i
pped
pat
h
an
d achi
e
ve a
ddi
t
i
onal
code c
o
vera
ge.
In t
h
i
s
w
a
y
,
DSE i
s
abl
e
t
o
ge
nerat
e
a s
e
t
of t
e
st
i
n
p
u
t
s t
h
at
achi
e
ve
hi
g
h
co
de co
vera
ge.
Using
DSE, no
n-lin
ear p
a
th
con
s
trai
n
t
s
are
sim
p
lifie
d
by th
e in
stan
tiatio
n
o
f
co
n
c
rete run
tim
e v
a
lu
es,
h
a
rv
ested fr
o
m
pr
og
r
a
m
ex
ecu
tio
n.
The aut
h
o
r
s i
n
[14]
p
r
o
p
o
se
d t
h
e appl
i
cat
i
on
of D
S
E fo
r
generat
i
ng t
e
s
t
dat
a
based o
n
m
u
t
a
ti
on
testin
g
.
After
gen
e
rating
m
u
tan
t
s fo
r t
h
e prog
ram
u
n
d
e
r test, th
ey will g
e
n
e
rate co
rrespo
nd
ing
weak-m
u
t
an
t-
killing c
o
nstra
i
nts for each m
u
tant. For exam
ple,
op1
>
op2
has a
co
rres
p
on
di
n
g
m
u
t
a
t
e
d ex
press
i
o
n
op
1
op2
th
en
co
nd
itio
n
to
k
ill th
is
m
u
tan
t
is ((
op
1
>
op
2
) &&
!(
op
1
op
2
)) || (!(
op1
>
op
2
) && (
op
1
op
2
)).
Nex
t
, all th
e g
e
n
e
rated
con
s
train
t
s are in
sert
ed
in
to
prop
er p
o
s
ition
s
of the o
r
ig
in
al pro
g
ram to
fo
rm
a
meta-
pr
o
g
ram
.
Aft
e
r t
r
ans
f
o
r
m
i
ng t
h
e ori
g
i
n
al
p
r
o
g
ram
unde
r t
e
st
i
n
t
o
a
m
e
t
a
-pr
o
gram
cont
ai
ni
n
g
al
l
m
u
t
a
nt
-
k
illin
g
con
s
train
t
s, th
e DSE
en
g
i
n
e
is
u
s
ed to
g
e
n
e
rate test in
pu
ts
for t
h
e m
e
ta-p
ro
gra
m
[10
]
. Th
e
au
tho
r
s
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 5
,
O
c
tob
e
r
20
15
:
116
4
–
11
73
1
169
b
u
ilt th
e
Pex
M
ut
at
or
t
o
ol
i
n
o
r
de
r t
o
s
u
p
p
o
r
t
f
o
r
gene
r
a
t
i
ng t
e
st
dat
a
fo
r C
#
p
r
og
ra
m
s
. Thei
r prel
im
i
n
ary
expe
ri
m
e
nt
al
st
udy
sh
o
w
s t
h
at
PexMutat
o
r
is ab
le to
stron
g
l
y k
ill m
o
re th
an
8
0
% of all th
e
m
u
tan
t
s fo
r the
st
udi
e
d
su
b
j
ec
t
s
. Ho
we
ver
,
t
h
i
s
m
e
t
hod fa
ces u
p
t
o
t
h
e
pr
o
b
l
e
m
i
s
t
o
i
n
t
r
o
d
u
ce as
m
a
ny
const
r
ai
nt
s as
m
u
t
a
nt
s of t
h
e pr
og
ram
unde
r t
e
st
i
n
t
o
t
h
e cor
r
es
po
n
d
i
n
g m
e
t
a
-pro
g
r
am
,
m
a
ki
ng i
t
exp
e
nsi
v
e f
o
r t
h
e
DS
E
engi
ne t
o
ge
ne
rat
e
t
e
st
i
n
p
u
t
s
fo
r a l
a
r
g
e n
u
m
ber of
bra
n
c
h
es.
O
n
e o
f
t
h
e way
s
t
o
ove
r
c
om
e t
h
i
s
pro
b
l
em
i
s
in
v
e
stig
ate tech
n
i
q
u
e
s th
at g
e
n
e
rate co
n
s
t
r
ain
t
s th
at en
ab
le
to
weak
ly k
ill m
u
l
tip
le
m
u
tan
t
s, th
u
s
redu
ci
ng
th
e
n
u
m
b
e
r
o
f
to
tal g
e
n
e
rated constrain
t
s
wh
ile
keep
ing
th
e same effecti
v
en
ess.
Papa
daki
s
et al
.
, [
15]
use
d
m
u
t
a
nt
schem
a
t
a
i
n
con
j
unct
i
o
n
wi
t
h
DSE t
o
g
e
nerat
e
t
e
st
dat
a
based
on
m
u
ta
tion anal
ysis to be satisfied
three
conditions na
me
d reacha
b
ility, necessity and sufficiency.
An
au
to
m
a
ted
framewo
rk
for testin
g
Ja
va
progra
m
s
according
to m
u
tation wa
s al
so propose
d
by a
u
thors
.
First
of
al
l
,
t
h
e sc
hem
a
t
i
c
m
e
t
a
-pro
g
r
am
versi
ons
of
t
h
e
p
r
o
g
ram
u
nde
r t
e
st
a
r
e
g
e
nerat
e
d.
Ne
xt
,
t
h
e m
u
t
a
nt
sc
hem
a
t
a
gene
rat
i
o
n co
m
ponent
p
r
o
d
u
ces a st
at
i
c
st
ruct
u
r
e o
f
t
h
e cal
l
and co
nt
r
o
l
fl
o
w
g
r
a
phs a
n
d a l
i
s
t
of t
h
e
in
trodu
ced
m
u
tan
t
s with
th
ei
r resp
ectiv
e pro
g
ram
state
m
e
n
ts. T
h
ese two artifacts are then
passe
d to t
h
e test
gene
rat
i
o
n m
o
dul
e. T
h
i
s
m
odul
e i
t
e
rat
i
v
el
y
sel
ect
s a
m
u
t
a
nt
as a t
a
rge
t
, per
f
o
r
m
s
DSE o
n
t
h
e sch
e
m
a
t
i
c
meta-
p
r
ogr
am
an
d pro
d
u
ces
so
m
e
te
st cases. T
h
ese test c
a
ses are
the
n
passe
d to the
test execut
o
r
whi
c
h
d
e
term
in
es th
eir ex
ecu
tion
path
, th
e m
u
tants th
at th
e test cases can infect an
d
t
h
e mu
tan
t
s th
at are k
illed.
After t
h
at the
process c
o
ntinue
s with t
h
e
next itera
tion. Finally, after reachi
n
g
a
predefi
n
ed number of
iteratio
n
s
o
r
time li
mit th
e p
r
o
cess end
s
an
d reports th
e
pro
d
u
c
ed
test cases and
th
e ach
i
e
v
e
d m
u
tatio
n
sco
r
e.
Th
e
p
r
op
osed
ap
pro
ach w
a
s
ex
p
e
r
i
m
e
n
t
ed
o
n
5 pr
ogr
am
s
w
ith
up
to 50
0 lin
es
o
f
cod
e
an
d th
e
ob
tain
ed
avera
g
e m
u
tation score is
63%. T
h
is approa
ch fa
ces
up
to
so
m
e
li
mita
tio
n
s
wh
en
app
l
yin
g
fo
r th
e large PUT.
Because m
a
ny m
u
tants are generate
d, t
h
e c
o
m
putational
c
o
st is e
xpe
nsive when speci
fying t
h
e
feasibl
e
pat
h
in
PUT as
well as so
lv
i
n
g th
e
co
nstrain
t
cond
itio
n
s
.
An im
p
r
ov
ed m
e
th
o
d
to
ov
erco
m
e
th
ese
restrictions is to
save t
h
e c
o
n
s
t
r
ai
nt
s chec
ke
d t
o
a
voi
d s
o
l
v
i
n
g t
h
e
sam
e
con
s
t
r
ai
nt
s m
a
ny
t
i
m
e
s. In
o
r
de
r
t
o
s
o
l
v
e t
h
e
pr
obl
em
of pat
h
ex
pl
o
s
i
on an
d eq
ui
val
e
nt
m
u
t
a
nts, i
t
i
s
possi
b
l
e t
o
use t
h
e appr
o
p
ri
at
e f
i
t
n
ess eval
uat
i
ons i
n
co
nju
n
c
tion wi
th
th
e effectiv
e h
e
u
r
istics.
3.
5. Searc
h
-B
ased
T
e
s
t
D
a
t
a
Ge
nera
ti
o
n
St
at
i
c
appr
oac
h
es
gene
rat
i
n
g
m
u
t
a
t
i
on-
base
d t
e
st
dat
a
rel
y
on s
o
l
v
i
ng t
h
e ent
i
r
e s
e
t
o
f
co
nst
r
ai
nt
s,
whe
r
eas
dy
na
m
i
c t
e
st
dat
a
gene
rat
i
o
n m
e
tho
d
s
det
ect
i
n
di
vi
d
u
al
faul
t
s
base
d
o
n
l
y
o
n
act
u
a
l
exec
u
t
i
ons
o
f
pr
o
g
ram
or desi
gn. Sea
r
ch
ba
sed t
e
st
dat
a
generat
i
o
n i
s
on
e of t
h
e dy
nam
i
c appr
oac
h
es.
It
m
odel
s
t
h
e test
i
n
g
t
a
sk as a search pr
o
b
l
e
m
t
h
at
i
s
gui
de
d by
a fi
t
n
ess fu
nct
i
on. T
h
e
n
, p
r
o
b
l
e
m
i
s
sol
v
ed
by
appl
y
i
ng
m
e
t
a
-
h
e
uristic tech
niq
u
e
s lik
e
g
e
netic alg
o
r
ith
m
s
, sim
u
lated
a
nnealing, clonal
selection
algorithm
or Tabu
search
to
op
tim
ize th
i
s
fu
n
c
ti
o
n
.
Howev
e
r, wh
atever m
a
y b
e
th
e
s
earch techniques em
ployed
fi
t
n
ess fu
nct
i
o
n or
cos
t
fun
c
tion
p
l
ays a m
a
j
o
r ro
le t
o
g
u
i
d
e
an
d seek in
pu
t test
d
a
ta
th
at ach
iev
e
the h
i
gh
est m
u
tatio
n
score.
Bo
ttaci [1
6
]
was th
e first to
su
gg
est using
search-b
ased
so
ft
ware en
g
i
n
eeri
n
g
t
o
k
ill
m
u
tan
t
s. Bo
ttaci
p
r
op
o
s
ed
a fitness fu
nctio
n for gen
e
tic algorith
m
s
b
a
sed
o
n
th
e con
s
trai
n
t
s
d
e
fi
n
e
d b
y
DeMillo
an
d Offutt in
or
der t
o
ge
ner
a
t
e
m
u
t
a
t
i
on-b
a
sed t
e
st
dat
a
. Ho
we
ver
,
t
h
i
s
pr
o
pose
d
ap
p
r
oach
rem
a
i
n
ed uni
m
p
l
e
m
e
nt
ed a
n
d
u
n
e
v
a
lu
ated
un
til
th
e
sub
s
equ
e
n
t
for work o
f
Ayari
et a
l
.,
[1
7]
. T
h
ey
us
ed B
o
t
t
aci
’s
pr
op
osal
t
o
de
fi
n
e
an
d
im
ple
m
ent a fitness function
that m
easures how close a test
case is to kil
l
a
m
u
tant. They used t
h
is fitness i
n
co
nju
n
c
tion
wi
th
th
e an
t co
lon
y
op
timizatio
n
(ACO) algo
rith
m
fo
r au
tomatic tes
t
d
a
ta g
e
n
e
ration
fo
r
Jav
a
pr
o
g
ram
s
. Tw
o
Jav
a
progra
m
s
were used to assess t
h
e
effec
tiv
en
ess o
f
t
h
e an
t co
lon
y
alg
o
rithm
.
Th
e
obt
ai
ne
d
res
u
l
t
s
i
ndi
cat
e t
h
at
AC
O a
p
pr
oac
h
per
f
o
rm
ed si
g
n
i
f
i
cant
l
y
bet
t
e
r t
h
a
n
genet
i
c
al
go
ri
t
h
m
and
Hi
l
l
Cli
m
b
i
n
g
in term
s o
f
attain
ed
m
u
tatio
n
score as well
as co
m
p
u
t
atio
n
a
l co
st.
Ho
wev
e
r,
th
e fitn
ess
functio
n
only im
plem
ents the reac
ha
bility co
m
ponent. The
neces
sity and s
u
fficiency com
pone
nts a
r
e still not s
o
lve
d
.
Ano
t
h
e
r m
e
ta-
h
euristic alg
o
rith
m
is u
s
u
a
lly u
s
ed
to
su
ppo
rt fo
r test d
a
ta
g
e
n
e
ration
th
at
is g
e
n
e
tic
alg
o
rith
m
(GA). Th
is alg
o
rith
m
is
p
r
ob
ab
ilistic search
alg
o
r
ith
m
in
sp
ired
fro
m
b
i
o
l
og
y. Th
e GA is an
iterativ
e p
r
o
c
ess to
find
t
h
e best so
lu
tion
in
th
e po
pu
lation
o
f
so
lu
tion
s
t
h
ro
ugh
m
a
n
y
g
e
n
e
ration
s
.
Algo
rith
m
is started
with
a set of so
lu
tion
s
(test d
a
ta) i
s
ran
d
o
m
l
y
generat
e
d cal
l
e
d
po
p
u
l
a
t
i
on.
T
h
en, t
h
e i
ndi
vi
d
u
al
s i
n
th
e po
pu
lation
are ev
al
u
a
ted
based
o
n
t
h
eir fi
tn
ess fun
c
tio
n.
Better indivi
duals are ha
ving
m
o
re chances t
o
be
sel
ect
ed as
par
e
nt
s i
n
o
r
de
r t
o
rep
r
o
d
u
ce
be
t
t
e
r of
fs
pri
n
g
b
y
usi
n
g c
r
o
sso
ver
an
d m
u
t
a
t
i
on
m
echani
s
m
s
. T
h
e
alg
o
rith
m
co
n
tin
u
e
s iteratin
g
u
n
til th
e
pre-defin
e
d
stopp
in
g
co
nd
itio
n is
met su
ch
as the m
a
x
i
m
a
l n
u
m
b
er o
f
gene
rations, all
m
u
tants are
killed or
pre
-
define
d m
u
ta
t
i
on score is reached. Afte
r a num
b
er of ge
ne
rations,
th
e alg
o
rith
m
will retu
rn
th
e
b
e
st in
d
i
v
i
du
al o
f
pop
u
l
ation
wh
ich
k
ills th
e
m
o
st
m
u
tan
t
s. Bau
d
r
y
et a
l
.,
[1
8]
applied the
GA to im
prove the random
l
y
gene
rated test
sets according to
m
u
tation for C#
programs. They
use
d
a p
o
pul
at
i
on
of
1
2
i
n
di
v
i
dual
s
c
onsi
s
t
i
ng
o
f
4 t
e
st
s e
ach.
Usi
n
g a
2
%
m
u
t
a
ti
on rat
e
ove
r
20
0 i
t
e
r
a
t
i
ons,
t
h
e m
u
t
a
t
i
on s
c
ore
reac
hes a
peak
o
f
8
0
%
,
w
ith
an
av
e
r
age
of
65-
70
%.
In
[19
]
,
w
e
p
r
op
o
s
e
d
to
app
l
y th
e
G
A
i
n
o
r
de
r t
o
ge
nerat
e
t
e
st
dat
a
fo
r
S
i
mu
lin
k
m
odel
s
.
W
e
al
so co
n
duct
e
d
t
h
e e
xpe
ri
m
e
nt
s an
d e
v
al
ua
t
e
d t
h
e
resu
lts on
5
S
i
mu
lin
k
m
o
d
e
ls an
d ob
tain
ed
the av
erag
e m
u
ta
tio
n
sc
ore
o
f
8
5
.
7
% wi
t
h
i
n
2
0
genet
i
c
ge
ner
a
t
i
ons
.
Bo
th
stud
ies, each
ind
i
v
i
du
al
is a set o
f
test d
a
ta and
th
e
fitn
ess calcu
lation
s
are
p
e
rfo
r
m
e
d
b
y
b
a
sing
on
th
e
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Sur
vey on
Mut
a
tion-base
d
Te
st
Dat
a
Ge
ner
a
tion
(Le
Thi
My Hanh)
1
170
ach
iev
e
d
m
u
ta
tio
n
score. The li
mitat
i
o
n
s
of th
is repr
ese
n
tation are to use
m
u
ch m
e
mory and inc
r
ea
se the
execution tim
e. More
ove
r, the fitness function
does
not
guide the search
m
e
thod by qua
n
titatively
m
e
asuri
ng
th
e clo
s
en
ess
o
f
k
illin
g sp
ecific m
u
tan
t
s
in
th
e each
g
e
n
e
tic g
e
n
e
rati
on
.
Fraser
et al
.,
[2
0]
p
r
op
os
ed a
n
effect
i
v
e
fi
t
n
e
ss fu
nct
i
o
n f
o
r t
h
e
GA
bas
e
d o
n
bra
n
c
h
di
st
ance a
nd a
p
p
r
oach l
e
vel
.
The a
p
p
r
oa
c
h
l
e
vel
descri
bes h
o
w
far a t
e
st
case i
s
from
t
h
e t
a
rget
i
n
t
h
e c
ont
rol
fl
ow
gr
aph
w
h
en i
t
h
a
s devi
at
e
d
fr
om
t
h
e
an
ticip
ated
cou
r
se. Th
is is usu
a
lly
m
easured as t
h
e n
u
m
b
er of
uns
at
i
s
fi
ed co
nt
r
o
l
depe
nde
nci
e
s bet
w
een t
h
e
p
o
i
n
t
o
f
d
e
v
i
atio
n and
t
h
e targ
et, and
th
e
v
a
lu
e
o
f
app
r
o
a
ch
lev
e
l will
b
e
0 if all con
t
rol d
e
p
e
nd
en
t bran
ch
es
are reache
d
[20]. The branc
h
dist
ance estimates how far the branc
h
at which execut
i
on di
verge
d
from
reachi
n
g the
m
u
ta
tion is from
evalua
ting to the
neces
s
a
ry outcom
e. Th
e a
u
thors a
pplied the
propos
ed
approach to a
s
e
t
o
f
10
op
en so
ur
ce
Ja
va
libraries and
ob
tain
ed th
e av
e
r
age m
u
tation score
of
72%.
In
[
18]
, B
a
u
d
r
y
al
so p
r
op
ose
d
a
ne
w al
g
o
ri
t
h
m
t
o
ove
rco
m
e t
h
e l
i
m
i
t
a
t
i
ons
o
f
t
h
e
GA
i
n
or
der
t
o
en
h
a
n
c
e th
e
quality o
f
test
d
a
ta. Th
at was Bacterio
log
i
cal
alg
o
rith
m
(BA).
In th
is al
g
o
rithm
,
an
in
d
i
v
i
dual is
an at
om
i
c
uni
t
- i
t
can
not
be
di
vi
ded
.
B
A
m
a
i
n
t
a
i
n
s a m
e
m
o
ry
set
co
n
s
i
s
t
i
ng
of t
h
e
best
i
n
di
vi
d
u
al
(s)
fr
om
each
ge
neration.
As
an atom
ic unit, indivi
duals
canno
t
be m
a
ted and s
o
variation ste
m
s purely from
the
rep
r
o
d
u
ct
i
on a
nd m
u
t
a
t
i
on p
r
oces
s. A m
e
m
o
ry
set
i
s
al
so m
a
i
n
t
a
i
n
ed, whe
r
e
new i
n
di
vi
d
u
al
s are a
dde
d i
f
their fitnes
s e
x
ceeds som
e
thres
hol
d.
Individual
f
itness
is based on a narrow
i
n
g search
space
, with
calcu
latio
n
s
b
a
sed
o
n
wh
at is left to
op
ti
m
i
z
e
. Fo
r ex
am
p
l
e, app
lied
t
o
mu
tatio
n
testing
,
fitn
ess is calcu
lated
by ba
sing
on how m
a
ny
m
u
tants a test
kills that the
m
e
m
o
ry set does
not
.
If t
h
is
fit
n
ess e
x
ceeds a t
h
reshol
d,
that indi
vidual is reproduced and
place
d into the m
e
m
o
ry. Resu
lts repo
rted
th
at using
a BA
g
e
n
e
rates a
me
m
o
ry set with
a m
u
tatio
n score
o
f
9
6
%
in
ju
st
3
0
gen
e
r
a
tion
s
.
A
n
av
er
ag
e m
u
tatio
n sco
r
e of
96% w
a
s
obt
ai
ne
d
by
ex
ecut
i
n
g
o
n
l
y
4
6
3
7
5
m
u
t
a
n
t
s, co
m
p
ared
with
th
e
GA attain
ing
an
a
v
er
age
mu
ta
t
i
o
n
s
c
or
e
of
8
5
% in
48
0000
m
u
tan
t
ex
ecu
tio
ns. Fo
r B
A
, th
e in
itial p
opu
latio
n
con
s
isted
of 30
tests an
d
a me
m
o
r
y
t
h
res
hol
d o
f
2
0
% (
i.e.
a test h
a
d
t
o
k
ill o
v
er 20
%
o
f
remain
in
g
m
u
ta
n
t
s to
b
e
add
e
d
to
th
e m
e
mo
ry set
)
.
Com
p
arisons
were als
o
m
a
de with a GA a
p
proach;
howe
v
e
r it is d
i
fficu
lt to
ascertai
n
th
e
fairn
e
ss
o
f
t
h
e
expe
rim
e
nts. For exam
ple, the GA consisted of 12 m
e
m
b
er
s of 4 tests each, equating to 48 tests, where
as the
BA used b
e
t
w
een
3
an
d 10
tests for its in
itial
main
p
opu
lation
.
An
ot
he
r e
vol
ut
i
ona
ry
ap
pr
oac
h
t
h
at
i
s
ve
ry
pr
om
i
s
i
ng t
o
s
u
p
p
o
rt
a
u
t
o
m
a
t
i
c
t
e
st
dat
a
generat
i
o
n i
s
clo
n
a
l selection
algorith
m
(CSA) – a sub
f
iel
d
in artificia
l immu
n
e
system.
Wh
en
app
l
yin
g
th
is m
e
th
o
d
in
the
co
n
t
ex
t o
f
m
u
tatio
n
testin
g
,
an
an
tig
en
is an in
trod
u
c
ed
m
u
tan
t
. An
an
tib
od
y is a test
d
a
ta wh
ich
is g
e
nerated
to
k
ill m
u
tan
t
s
.
Th
e affi
n
ity o
f
an
an
tib
od
y
(test d
a
ta) is
measu
r
ed
b
y
m
u
ta
tio
n
score. An
tib
od
y evo
l
u
tion
occurs t
h
rough the proc
ess
of
clonal selection, guided
by the
affinity
va
lues.
T
h
e
high affi
nity antibodies
g
e
n
e
rate m
o
re
clo
n
e
s th
an
low affin
ity on
es, bu
t m
u
tate le
ss. Th
is
p
r
o
cess aim
s
at refin
i
n
g
an
tibod
ies t
o
k
ill
as m
a
n
y
m
u
ta
n
t
s as
po
ssib
l
e. Th
e
b
e
st an
ti
b
o
d
i
es
will b
e
add
e
d to
a me
m
o
ry set fo
r
sav
i
ng
and
ret
u
rn
t
o
t
e
st
ers w
h
e
n
t
e
st
i
ng
pr
ocess
f
i
ni
shes.
M
a
y
[
21]
a
d
op
ted th
i
s
m
e
th
od
to
evo
l
v
e
test
d
a
ta fo
r
Fortr
a
n
pr
o
g
ram
s
basi
n
g
on m
u
t
a
t
i
on t
e
st
i
n
g
.
I
n
[
2
2]
, we al
s
o
pr
o
pose
d
t
o
a
ppl
y
C
S
A wi
t
h
som
e
m
odi
fi
cat
i
ons t
o
ge
ner
a
t
e
t
e
st
dat
a
f
o
r
Si
m
u
l
i
nk
m
odel
s
.
Tao Xie
et al.,
[
2
3]
de
fi
ne
d
a c
o
st
f
u
nct
i
o
n
f
r
om
fa
ul
t
sim
u
l
a
t
i
on t
r
a
ces t
o
real
val
u
es i
n
HD
L
pr
o
g
ram
s
. Thi
s
cost
fu
nct
i
o
n fo
r di
r
ect
i
n
g
search he
u
r
i
s
t
i
c
s has been
d
e
fi
ne
d o
n
t
h
e t
e
st
i
nput
spac
e. B
y
mapping a
pai
r
of
fa
ult simulation t
r
aces
ont
o the
Co
nt
rol a
n
d
Data
Flow Graph
(CDFG) st
ruct
ure
,
the
au
tho
r
s
were
ab
le to
an
alyze q
u
an
titativ
ely h
o
w far a
fau
lt effect
h
a
s b
e
en
prop
ag
ated
th
rou
g
h
bo
th
t
h
e
cont
rol
a
nd
da
t
a
fl
ows
.
The
m
acro p
r
o
p
a
g
at
i
on di
st
a
n
ce
and t
h
e l
o
cal
pr
opa
gat
i
o
n c
o
st
t
oget
h
er
f
o
rm
a
co
m
p
lete so
lu
tio
n
t
o
t
h
e
n
ecessity a
n
d
suffi
cien
cy sub
-
prob
lem
s
o
f
fau
lt
d
e
tectio
n.
On
e m
a
j
o
r limitati
o
n
of
th
e work is to
still lack
an
au
to
m
a
t
i
o
n
too
l
fo
r th
e co
n
s
tru
c
tio
n
o
f
CDFG
as well as test
d
a
ta g
e
n
e
ratio
n.
An
ot
he
r
wo
r
k
was
pr
op
ose
d
by
Zha
n
a
n
d C
l
ark
[2
4]
i
n
wh
i
c
h t
h
ey
ge
ne
r
a
t
e
d t
e
st
dat
a
i
n
t
h
e c
o
nt
ext
o
f
m
u
tatio
n
testin
g
fo
r
M
a
t
l
ab/
Si
m
u
l
i
n
k
m
odel
s
by
usi
n
g si
m
u
l
a
t
e
d anneal
i
n
g a
nd
ran
d
o
m
t
e
st
i
ng. T
h
e
m
e
thod
random
ly generates
a large set of t
e
st-data,
de
tects the m
u
tant-ki
lling ability, and t
h
en m
i
nim
i
zes the
test set wh
ilst retain
in
g
its overall
m
u
tan
t
-k
i
llin
g
ab
ility. Fo
r m
u
tan
t
s th
at
can
no
t
b
e
k
illed
b
y
the ran
d
o
m
test
set, th
e m
e
th
od
p
r
ov
id
es an
effectiv
e m
ean
s of au
to
m
a
tic
ally g
e
n
e
rati
n
g
in
d
i
v
i
du
al test d
a
ta for
fu
lfi
llin
g
in
d
i
v
i
d
u
a
l m
u
tan
t
-k
illin
g
aims. In
th
is
way, a targ
eted
test d
a
ta g
e
n
e
rati
o
n
is u
tilized
to
co
m
p
le
m
e
n
t
th
e
ran
d
o
m
t
e
st
generat
i
o
n i
n
or
d
e
r t
o
achi
e
ve m
u
t
a
t
i
on a
d
e
q
ua
cy
.
3.
6. Hybrid
T
echniques
As
prese
n
ted a
b
ove, m
e
ta-heuristic searc
h
a
nd
dy
nam
i
c sym
bol
i
c
execut
i
on t
e
c
h
ni
q
u
e h
a
ve em
erged
as two
su
ccessfu
l app
r
o
a
ch
es to
au
to
m
a
tica
l
ly g
e
n
e
rate
t
e
st
dat
a
t
h
at
achi
e
ve hi
g
h
m
u
t
a
ti
on co
ve
rage
. B
o
t
h
approaches
ha
ve their advant
ages, bu
t they also have s
p
eci
fic dra
w
bac
k
.
Searc
h
-base
d
testing scales well and
can ha
n
d
l
e
an
y
code an
d t
e
st
cri
t
e
ri
on
bu
t
i
t
get
s
st
uck i
n
l
o
cal
opt
i
m
a and de
gra
d
es w
h
e
n
t
h
e
searc
h
l
a
ndsca
pe o
ffe
rs n
o
g
u
i
d
a
n
ce
. DSE b
a
sed t
e
st
i
ng ex
pl
oi
t
s
t
h
e effi
ci
e
n
cy
o
f
m
odern co
nst
r
ai
nt
sol
v
e
r
s w
h
i
c
h
are n
o
t
d
e
p
e
n
d
en
t
on
search
heu
r
istics,
bu
t t
h
ere are li
m
its
to
bo
t
h
scalab
ility
an
d
th
e types of con
s
traints th
at
can be han
d
l
e
d
.
I
n
[2
5]
, Har
m
an
et al.
in
tro
d
u
c
ed
a
h
ybr
id
D
S
E
and
s
e
ar
c
h
-
b
as
e
d
s
o
f
t
w
a
r
e
te
s
ting
ap
pro
a
ch
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 5
,
O
c
tob
e
r
20
15
:
116
4
–
11
73
1
171
to
g
e
n
e
rate stro
ng
ly ad
eq
u
a
t
e
test d
a
ta to
kill first an
d
h
i
gh
er
o
r
d
e
r m
u
ta
n
t
s. Th
e au
t
h
ors im
p
l
e
m
en
ted
th
ei
r
app
r
oach
i
n
a t
ool
cal
l
e
d S
H
OM
. T
h
e
p
r
o
p
o
se
d a
p
p
r
oach
was e
v
al
uat
e
d
on
1
7
su
b
j
ect
pr
o
g
ram
s
, i
n
cl
udi
ng
7
real
w
o
rl
d
pr
o
g
ram
s
(fo
ur
fr
om
t
w
o di
ffe
r
e
nt
cl
ose
d
so
u
r
ce i
n
dust
r
i
a
l
sy
st
em
s and t
h
ree
fo
r w
h
i
c
h
sou
r
c
e
co
d
e
is
p
u
b
licly av
ailab
l
e). Th
ey repo
rted
t
h
e
resu
lts
of a
n
em
pirical evaluation
of SHOM’s
efficiency and
effectiv
en
ess fo
r stro
ng
first
o
r
d
e
r m
u
tatio
n
ad
equ
acy.
Th
e resu
lts sh
ow t
h
at SHOM
can k
ill up to
38
% of t
h
e
first o
r
d
e
r
m
u
tan
t
s
left un
k
illed
u
s
ing
reachab
ility
an
d
i
n
fectio
n
,
wh
ich
in
tu
rn
k
ills
up
to
3
6
% o
f
th
e m
u
tan
t
s
left u
n
k
illed
u
s
in
g
reach
a
b
ility alo
n
e
. Th
ey also
rep
o
rted
t
h
e resu
lts o
f
a
furth
e
r em
p
i
ric
a
l stu
d
y
o
f
SHOM’s
efficiency and effective
n
ess for st
r
o
n
g
sec
o
nd
or
der m
u
t
a
ti
on ade
q
uacy
. The res
u
l
t
s
sh
owe
d
t
h
at
SH
OM
ca
n
k
ill u
p
to
48
% o
f
th
e second
o
r
d
e
r m
u
tan
t
s left un
k
ille
d
u
s
in
g
reach
a
b
ility an
d in
fecti
o
n, wh
ich in
t
u
rn k
ills
u
p
to
41
%
o
f
th
e m
u
tan
t
s left
un
k
illed
u
s
i
n
g reach
a
b
ility alo
n
e
.
Thi
s
p
r
ese
n
t
e
d
seve
ral
t
ech
ni
q
u
es
use
d
t
o
gene
ra
te test d
a
ta b
a
sed on
m
u
tatio
n
testin
g
.
All
ap
pro
ach
es
are p
r
o
m
isin
g
to form
u
l
ate h
i
g
h
qu
ality test su
ites. So
m
e
well-kno
wn
tech
n
i
q
u
e
s and
related
pape
rsa
r
e bri
e
f
l
y
prese
n
t
e
d
i
n
Tabl
e 1.
Tabl
e
1. T
h
e
t
y
pi
cal
t
e
st
dat
a
gene
rat
i
o
n a
p
p
r
oac
h
es
Autho
r
[Ref
]
Year
Technique
Subjec
ts St
udied
Subjec
t
Language
Average
mu
ta
tio
n
sco
r
e
DeMillo and Offut [9]
1991
Constr
aint-
b
ased test data
gener
a
tion
5 s
m
all pr
ogr
a
m
s
For
t
r
a
n
98%
O
ffut
et a
l
.
[11]
1999
Dy
nam
i
c Do
m
a
in
Reduction
12 s
m
all pr
ogr
a
m
s
For
t
r
a
n
Not given
Chen
et a
l
.
[8]
2004
Adaptive Rando
m
T
e
sting
12 s
m
all pr
ogr
a
m
s
C++
Not given
Baudr
y
et al.
[18]
2005
Search-based test
data generation
using genetic algor
ith
m
An exam
ple in
E
i
ffel L
i
br
ary
C#
85%
Baudr
y
et al.
[18]
2005
Search-based test
data generation
using Bacter
iologi
cal algor
ith
m
An exam
ple in
E
i
ffel L
i
br
ary
C#
96%
Aya
r
i
et al.
[17]
2007
Search-based test
data generation
using ant colony
algor
ith
m
2 s
m
all pr
ogr
a
m
s
Java
88%
Papadakis
et a
l
.
[12]
2009
E
nhanced Contr
o
l Flow Gr
aph
8 s
m
all pr
ogr
a
m
s
Java
90.
2%
Z
h
ang
et a
l
.
[14]
2010
Dy
nam
i
c Sy
m
bolic E
x
ecution
5 s
m
all pr
ogr
a
m
s
C#
90%
Papadakis
et a
l
.
[15]
2010
Dy
nam
i
c Sy
m
bolic E
x
ecution
5 tiny
exam
ples
+
plus 3
s
m
all Si
e
m
ens suit
e exa
m
ples
C 63%
Frazer
et al.
[20]
2010
Search-based test
data generation
using genetic algor
ith
m
2 non-
tr
ivial exam
ples:
Co
m
m
ons M
a
th
&JodaT
i
m
e
Java 72%
Har
m
an
et al.
[25]
2011
Co
m
b
ine D
y
na
m
i
c
Sy
m
bolic
Execution and Search-based
testing
7 r
eal world pr
ogr
am
s and 10
s
m
all pr
ogr
a
m
s
C 71%
Hanh
et a
l
.
[19]
2014
Search-based test
data generation
using genetic algor
ith
m
5 Sim
u
link
m
odels
Sim
u
link
85.
7%
Hanh
et a
l
.
[22]
2014
Search-based test
data generation
using the clonal selection
algor
ith
m
2 Sim
u
link
m
odels
Sim
u
link
88.
1%
4.
FUTU
RE T
R
ENDS
It wou
l
d
no
t b
e
po
ssi
b
l
e to co
n
c
l
u
d
e
t
h
is su
rv
ey withou
t sp
en
d
i
n
g
a litt
le ti
me d
i
scu
ssing
the
pos
si
bl
e f
u
t
u
re
t
r
end
s
o
f
t
e
st
dat
a
ge
nerat
i
on
base
d
on
m
u
ta
tion anal
ysis. It can se
e that there are four
i
m
p
o
r
tan
t
d
i
rectio
n
s
fo
r research
: th
e scalab
i
lity
o
f
th
e cu
rren
t
tech
n
i
qu
es
for larg
e-scale so
ft
ware pro
j
ect, th
e
stu
d
y
for h
i
g
h
q
u
a
lity h
i
gh
er
o
r
d
e
r m
u
tan
t
s
to
red
u
c
e th
e size o
f
test set,
eli
m
in
atio
n
o
f
eq
u
i
v
a
len
t
m
u
tan
t
s
b
e
fo
re g
e
n
e
ratin
g
test
d
a
ta,
an
d au
t
o
m
a
tic
adj
u
stm
e
n
t
o
f
con
f
i
g
uration p
a
ram
e
ters for th
e m
e
ta-h
eu
ristic
alg
o
rith
m
s
in
search
-b
ased test d
a
ta g
e
n
e
ration
.
So
l
v
ing
th
ese con
c
ern
s
will i
m
p
r
o
v
e
sign
ificantly
th
e
effectiv
en
ess
of techn
i
qu
es and
th
e qu
ality o
f
test set.
The ap
pr
oac
h
e
s
prese
n
t
e
d i
n
t
h
i
s
paper
we
re ex
peri
m
e
nt
ed o
n
sm
all
prog
ram
s
. It
i
s
expect
e
d
t
o
conduct studie
s
about the s
calability of the curren
t techniques for large-s
cale industry s
o
ft
ware. Rece
nt work
h
a
s tend
ed
to
fo
cus on
m
o
re elab
orate form
s
o
f
m
u
tatio
n
th
an
on
th
e relatively si
m
p
l
y
fau
lts wh
ich
h
a
v
e
b
een
p
r
ev
iou
s
ly consid
ered
. Th
ere is an
in
terest
in
th
e se
m
a
n
tic effects of
m
u
ta
tio
n
,
rat
h
er th
an
th
e sy
n
t
actic
achi
e
vem
e
nt
of
a m
u
t
a
t
i
on [2
6
]
. It
i
s
desi
red
t
o
gene
rat
e
hi
g
h
er
o
r
de
r m
u
t
a
t
i
on a
n
d
resem
b
l
e
real
faul
t
s
.
In t
h
e
fu
t
u
re, th
erefo
r
e, stud
ies will fo
cu
s
o
n
g
e
n
e
ratin
g
test
d
a
ta
to
k
ill h
i
gh
er ord
e
r m
u
tatio
n
as well as reduce the
size o
f
test su
it
es. Th
e m
o
re
m
u
tan
t
s k
illed
b
y
a test
set, th
e b
e
tter th
e
m
e
asu
r
ed
ad
equ
a
cy o
f
th
e test s
e
t. Th
is
will en
h
a
n
c
e the qu
ality o
f
test d
a
ta and
d
e
crease ti
m
e
sp
en
t
on
so
ft
ware testin
g
p
r
o
cess.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Sur
vey on
Mut
a
tion-base
d
Te
st
Dat
a
Ge
ner
a
tion
(Le
Thi
My Hanh)
1
172
On
e of th
e mo
st ch
allen
g
e
s in
test d
a
ta
g
e
n
e
ration
u
s
in
g m
u
tatio
n
testin
g
is t
h
e
d
e
tectio
n
of
eq
u
i
v
a
len
t
m
u
tan
t
s.
It
wastes ti
m
e
to
g
e
n
e
rate test d
a
ta
bu
t we
n
e
v
e
r
fin
d
ou
t test t
h
at is ab
le t
o
k
ill th
ese
m
u
tan
t
s. In
t
h
e fu
t
u
re, m
u
tatio
n
testing
meth
od
s sho
u
l
d seek
t
o
avo
i
d
in
itial creatio
n or elimin
atio
n
o
f
eq
u
i
v
a
len
t
m
u
tan
t
s
b
e
fo
re gen
e
rating
tests.
Th
is
work
will red
u
c
e th
e time wasted
on
g
e
n
e
rating
test
su
ites
and
i
m
prove
t
h
e m
u
t
a
t
i
on co
v
e
rage
o
f
t
e
st
da
t
a
.
It can
also
b
e
seen
th
at search
-b
ased
test d
a
ta g
e
n
e
ration
tech
n
i
q
u
e
s d
e
pen
d
on
setting
p
a
ram
e
ters.
These
pa
ram
e
ters a
r
e
dra
w
n
fr
om
expe
ri
m
e
nt
s an
d
fi
x
e
d in the
searching
proces
s
.
They a
ffect
to the
effectiv
en
ess
of m
e
ta-h
eu
ristic alg
o
rith
m
s
a
n
d
ob
tain
ed
test
su
ites. It is exp
ected
t
h
at, in
th
e fu
tu
re, th
ere will
b
e
m
o
r
e
w
o
rk
co
n
c
en
tr
ating
o
n
self-
t
un
ingof
p
a
r
a
m
e
ter
s
to
seek
h
i
gh
qu
al
ity
test d
a
ta. Th
e in
fo
r
m
atio
n f
r
o
m
pre
v
i
o
us
ge
ner
a
t
i
ons ca
n
be
u
s
ed t
o
c
h
o
o
se t
h
e a
p
p
r
op
ri
at
e
param
e
t
e
rs fo
r
t
h
e cu
rre
nt
ge
n
e
rat
i
o
n
5.
CO
NCL
USI
O
N
Th
is
p
a
p
e
r presen
ts a co
m
p
reh
e
nsiv
e
surv
ey
of the m
o
st pro
m
in
en
t tech
n
i
q
u
e
s in au
to
m
a
tic test d
a
ta
gene
rat
i
o
n bas
e
d on
m
u
t
a
t
i
on. These
t
ech
n
i
ques
i
n
cl
u
d
e r
a
nd
om
t
e
st
dat
a
ge
nerat
i
o
n,
c
onst
r
ai
nt
-
b
ase
d
t
e
st
dat
a
gene
rat
i
o
n, en
ha
nced c
o
nt
r
o
l
fl
ow
gra
p
h, dy
nam
i
c sym
bol
i
c
execut
i
o
n
,
searc
h
-
b
as
ed t
echni
que
s and t
h
e
hy
b
r
i
d
a
p
pr
oac
h
es.
The res
u
lts of the survey indicate th
at there are quite a lot researc
h
es
i
n
t
h
e fi
el
d
of
gen
e
rat
i
ng t
e
s
t
d
a
ta b
a
sed
on
m
u
ta
tio
n
testing
,
and
m
o
st o
f
th
em
are p
o
s
itiv
e.
In th
e
work
th
at
we
d
e
p
l
o
y
ed
abo
u
t
t
h
e u
s
e
o
f
meta-h
eu
ristic alg
o
rith
m
s
to
search
for test d
a
ta th
at
is
ab
le to
k
ill man
y
m
u
tan
t
s, th
e in
itial resu
lts are
pr
om
i
s
i
ng.
For
t
h
e
t
e
st
da
t
a
gene
rat
i
o
n t
echni
que
s
bas
e
d
on
co
nst
r
ai
nt
an
d
dy
nam
i
c sy
m
bol
i
c
execut
i
o
n
,
t
h
e
o
b
t
ain
e
d
tests
can
k
ill m
u
tan
t
s with
a
h
i
gh p
r
op
ortion
.
Ho
wev
e
r, th
ey
can
en
coun
ter th
e p
a
t
h
exp
l
o
s
ion
pr
o
b
l
e
m
when
han
d
l
i
n
g l
a
r
g
e pr
o
g
ram
s
and desi
g
n
s. F
o
r
search
-
b
ased
m
e
t
hods
, t
e
st
dat
a
are
opt
i
m
i
zed
t
h
r
o
u
g
h
ge
ner
a
t
i
ons rel
y
on e
v
al
uat
i
n
g t
h
e c
o
st
f
unct
i
o
n.
I
n
t
h
e st
u
d
y
o
f
t
h
e ot
he
r aut
h
o
r
s
as wel
l
as o
u
r
wo
rk
,
th
e co
st fun
c
tio
n has
n
o
t
gu
i
d
ed for test
d
a
ta g
e
n
e
ration
t
o
ward
s the
h
i
gh
er m
u
tatio
n
sco
r
e yet. Th
us, in
t
h
e
fut
u
re, t
h
e c
o
st
fun
c
t
i
on m
a
y
be im
pro
v
e
d
b
y
gui
di
n
g
t
h
e
pr
ocess o
f
t
e
st
dat
a
gene
rat
i
o
n t
h
at
ori
e
nt
at
es t
h
e
speci
fi
c m
u
t
a
nt
s i
n
eac
h
ge
net
i
c
gene
rat
i
o
n.
REFERE
NC
ES
[1]
B. Beizer,
Softw
are Testing Tech
niques
, 2nd
ed
.:
Thomason Co
mputer Press, 199
0.
[2]
R. DeMillo
, R
.
Lipton
, and F
.
Sa
y
w
ard, "Hints
on Test
Da
ta Se
l
ect
ion: He
lp for
Practi
c
ing for
Program
m
e
r",
IE
EE
computer
, n
o
. 11
, pp
. 34-41
, 197
8.
[3]
A.J. Offutt and J.M. Voas, "Subsumption of Con
d
ition Coverag
e
Techn
i
ques b
y
Mutation Testin
g",
George Mason
University and
Reliable So
ftware Technologies
Co
rp.
,
Techn
i
cal R
e
port ISSE-TR-9
6
-01, 1996
.
[4]
S
a
s
w
at Anand
et al.
, "An Orches
trat
ed S
u
rve
y
on
Autom
a
ted Software Test
Cas
e
Generation",
Jo
urnal of Systems
and Software
, vo
l. 86
, no
. 8
,
pp
. 1
978-2001, 2013
.
[5]
S. Ali, L
.
C.
Bri
a
nd, H. Hem
m
a
ti,
a
nd R.K.
P
a
nes
a
r-W
alaweg
e
,
"A S
y
s
t
em
at
ic
Review of th
e
Applica
tion and
Empirical Investigation of Search-B
ased Test Case Generation"
, in
IEEE T
r
ansactions on Software Engineering
,
2010, pp
. 742
-
762.
[6]
M
.
Harm
an, "Autom
ated tes
t
d
a
ta gen
e
ration using search base
d software engineer
ing", in
2nd Workshop o
n
Automation of S
o
ftware Test (
A
S
T
07
)
at the 29t
h
International
Conference on Software Engineering
, USA, 2007
,
p. 2
.
[7]
P
.
M
c
M
i
nn, "S
earch-bas
ed s
o
ft
ware
test data genera
tion: A surv
ey
"
,
Software Testing, Verifica
tio
n and Reliability
,
vol. 14
, no
. 2
,
pp
. 105-156
, 2004
.
[8]
T.Y. Ch
en, H.
Leung,
and I
.
K.
Mark, "Adaptive Random Testin
g, Adva
nces in
Computer Scien
ce -
ASIAN 2004:
Higher-Lev
el Decision Mak
i
ng,”
in
9th
Asian
Computing Science Conferen
ce
, pp
. 320-329, 2004.
[9]
R.A. Dem
illo
a
nd A.J. Offutt
,
"Constraint
-Bas
ed Autom
a
tic
T
e
st Data
Gener
a
tion",
I
EEE Transaction Softwa
r
e
Engineering
, vol. 17
, pp
. 900-91
0, 1991
.
[10]
R.A. DeMillo
,
D.S. Guindi, W
.
M. McCr
ack
en,
A.J. Offutt and
K.N. King, "An extend
ed overvi
e
w of the Moth
r
a
software testing
environment", in
Proceedings of
Symposium on Software
Testing
,
Analysis
and Verificat
ion
, 1988
,
pp. 142-151
.
[11]
A.J. Offutt, Z. Jin and J. Pan,
"The d
y
n
a
mic domain reduction ap
proach to test data gener
a
tion"
,
Software: Practice
and Exp
e
rien
ce
,
vol. 29
, no
. 2
,
pp
. 167-193
, 1999
.
[12]
M. Papadakis an
d N. Malevris, "A
n Effectiv
e Pat
h
Selection Stra
t
e
g
y
for Muta
tion
Testing", in
Pr
o
ceed
ings
of As
ia
Pacific Software Engin
eering
Conference
, 2009,
pp. 422-429
.
[13]
J.C. King
, "S
y
m
bolic execut
ion and
program testing",
Communications of the AC
M
, vol. 19
, no
. 7
,
pp
. 38-94
, 197
6.
[14]
Lingming Zha
n
g,
Ta
o Xie,
Lu Zha
n
g,
N.
Tillma
nn,
J.
de Halle
ux,
Hong Mei,
"T
e
s
t ge
ne
ra
tion via
D
y
na
mic
S
y
m
bolic
Execu
tion for m
u
ta
tio
n testing",
in
Software Maintena
nce (
I
CSM)
, 2010 I
EEE International Conferen
ce
on
, Timisoara, 2
010, pp
. 1-10
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
JECE Vo
l. 5
,
N
o
. 5
,
O
c
tob
e
r
20
15
:
116
4
–
11
73
1
173
[15]
M. Papadakis
and N.
Malevr
is,
"Autom
atic
m
u
tation test case gene
ration via d
y
namic s
y
mbolic execution"
, in
Proceed
ings of t
h
e 21st Inter
nati
onal Symposium on Software Rel
i
abili
ty Eng
i
neer
ing (
I
SSRE’10)
, USA, 2010, pp.
121-130.
[16]
Leonardo Bottaci, "A Genetic Algorithm Fitn
ess Function for Mutation
Testing"
, in
Pr
oceed
ings
of the Softwar
e
Engineering using Metah
e
uristic
inova
tive
Algorithms workshop
,
April 2001, pp.
3-7.
[17]
K. A
y
ari, S. B
ouktif,
and G.
Antoniol, "Auto
m
atic muta
tion
test
input d
a
ta gener
a
tion v
i
a ant
colon
y
", in
Proceed
ings of t
h
e 9th annual
c
onferenc
e
on Ge
neti
c and e
v
olut
ionary computat
ion
, London
, En
gland, 2007
, pp
.
1074-1081.
[18]
B. Baudr
y,
F
.
F
l
eure
y,
J
.
M
.
J
é
zé
quel,
and Y.
L.
T
r
aon, "Gen
es
an
d Bact
eri
a
for A
u
tom
a
tic
Tes
t
Cas
e
s
Optim
iza
tio
n
in the NET E
nvironm
ent", in
Proceedings o
f
the 13th Inte
rnati
onal Symposium on Software Reliabi
lit
y
Engineering
, 20
02, pp
. 195-206
.
[19]
L.T.M. Hanh, K.T. Tung and
N.T. Binh, "Mutatio
n-based
Test Data Generation for
Si
mulink Models using Genetic
Algorithm and Simulated Annealing",
Internation
a
l Journal of Co
m
puter and Information Technology
, vol. 3, no
.
4, pp
. 763-771
, J
u
ly
2014.
[20]
G. Fraser and
A. Zeller
, "Mutation-dr
iven
generation of unit te
s
t
s
and or
acl
es
", in
Proceed
ings of the 19
th
International Symposium on Softwar
e Testing an
d Analysis (
I
SSTA’10)
, Tren
to, Italia, 2011, pp. 1
47-158.
[21]
P.
S.
May
,
"T
e
s
t Da
ta
Ge
ne
ra
tio
n: Two Evolutionar
y
Appro
ach
es to Mutation
Testing",
T
h
e U
n
iver
s
i
t
y
of Ken
t
,
PhD Thesis, 200
7.
[22]
Le
Thi M
y
H
a
n
h
, Ngu
y
en
Than
h Binh
and Khu
a
t
Thanh
T
ung,
"A Novel Test
Data Gen
e
ration
Approach
Based
Upon Mutation
Testing
b
y
Usin
g Artificial Im
mune S
y
stem f
o
r Simulink models", in
The S
i
xth In
ternationa
l
Conference on
K
nowledge and S
y
stems Engineering
, Hanoi, Vietn
a
m, 2014.
[23]
Tao Xie, W. Mueller and F.
Leto
mbe, "HDL-Mutation Ba
sed Sim
u
lation Data Generati
on b
y
Prop
agation Guided
S
earch",
in
14th Euromicro
Conference
on Dig
ita
l System Design
, Oulu, 2011, pp.
608-615.
[24]
Y. Zhan
and J
.
A. Clark
,
"S
ear
c
h
-bas
ed M
u
ta
tio
n Tes
ting
for S
i
m
u
link M
odels
",
ACM Dig
ital Library
, pp. 1061
-
1068, 2005
.
[25]
Mark Harman,
Yue Jia and William B.
Langdo
n, "Strong High
er Order Mutati
on-Based Test Data Gen
e
ration", in
Pr
oceed
ings
of
t
h
e 19th
ACM
SI
GSOFT
s
y
mpos
ium
, New York,
USA, 2011, pp.
212-222.
[26]
Yue J
i
a and M
a
rk Harm
an, "An Anal
y
s
is
and S
u
rv
ey
of the Development of Mutation Testing"
,
Crest Centre,
King’s college London, Techn
i
ca
l Report TR-09-
06
, 2011
.
BIOGRAP
HI
ES OF
AUTH
ORS
Le Thi My
Ha
nh
gained M. Sc from the University
of
Danan
g
in Computer Science in 2004.
She is currently
a PhD
student of the Information
Techno
log
y
Faculty
,
University
of Science and
Techno
log
y
, Danang, Vietnam. Her resear
ch
in
ter
e
st is abou
t softwa
re testing and more
generally
application
of h
e
ur
istic techniques
to pr
oblems in softw
a
re
engin
eering
.
Nguy
en Thanh Binh
graduated from The Un
iversity
of
Dan
a
ng, University
of Science and
Techno
log
y
in 1997, he got a
PhD degree in Com
puter Scien
ce from
Grenoble Institut
e
of
Techno
log
y
(Fr
a
nce) in 2004. He is currently
a
ssociate professor of the Information Technolo
g
y
Faculty
,
Th
e University
of
Danan
g
, University
of
Science and
Technolog
y
(V
ietnam). He is dean
of Information
Techno
log
y
Faculty
at Th
e Univ
ersity
of Dan
a
ng, Univ
ersity
of
Science and
Techno
log
y
since 2010.
He directs a research
team sin
ce 2009
. His r
e
s
earch
interests include
software testab
ility
,
software testing
and softwar
e
qu
ality
.
K
huat Thanh
Tung
com
p
lete
d the
B.E
.
d
e
gr
ee
in Softwa
r
e
Engineering f
r
o
m
University
of
Science and
Technolog
y
,
Danang,
Vietnam, in
2014. Curr
ently
,
he is par
ticipating
in th
e
res
earch t
eam
a
t
DATIC Laborator
y, Th
e Univ
ers
i
t
y
o
f
Danan
g
, Univers
i
t
y
of
S
c
ience and
Techno
log
y
, Vie
t
nam
.
His
res
ear
ch inter
e
s
t
s
incl
ude
software en
gineer
ing, softw
a
re testing, and
AI in softwar
e
engineer
ing.
Evaluation Warning : The document was created with Spire.PDF for Python.