I
AE
S
I
nte
rna
t
io
na
l J
o
urna
l o
f
Art
if
icia
l In
t
ellig
ence
(
I
J
-
AI
)
Vo
l.
5
,
No
.
3
,
Sep
tem
b
er
2
0
1
6
,
p
p
.
1
0
5
~
1
1
6
I
SS
N:
2252
-
8938
105
J
o
ur
na
l ho
m
ep
a
g
e
:
h
ttp
:
//ia
e
s
jo
u
r
n
a
l.c
o
m/o
n
lin
e/in
d
ex
.
p
h
p
/I
J
AI
Ano
m
a
lies
Dete
ct
io
n Ba
sed o
n t
he
RO
C
Analy
sis
usi
ng
Cla
ss
ifiers in
Ta
c
tical Co
g
nitive Ra
dio
Sys
te
m
s
:
A su
rv
ey
Ah
m
ed
M
o
u
m
e
na
El
e
c
tro
n
ic De
p
a
rtm
e
n
t,
S
a
a
d
Da
h
lab
-
Bli
d
a
Un
iv
e
rsit
y
,
A
l
g
e
ria
Art
icle
I
nf
o
AB
ST
RAC
T
A
r
ticle
his
to
r
y:
R
ec
eiv
ed
J
u
n
5
,
2
0
1
6
R
ev
i
s
ed
A
u
g
8
,
2
0
1
6
A
cc
ep
ted
A
u
g
u
s
t
2
4
,
2
0
1
6
Re
c
e
iv
e
r
o
p
e
ra
ti
n
g
c
h
a
ra
c
teristic
(ROC)
c
u
rv
e
is
a
n
i
m
p
o
rtan
t
tec
h
n
i
q
u
e
f
o
r
o
rg
a
n
izin
g
c
las
sif
iers
a
n
d
v
isu
a
li
z
in
g
th
e
ir
p
e
rf
o
rm
a
n
c
e
in
tac
ti
c
a
l
sy
ste
m
s
in
t
h
e
p
re
se
n
c
e
o
f
ja
m
m
in
g
sig
n
a
l.
ROC
c
u
rv
e
s
a
r
e
c
o
m
m
o
n
l
y
u
se
d
to
e
v
a
lu
a
te
th
e
p
e
rf
o
rm
a
n
c
e
o
f
c
las
sif
ier
s
fo
r
a
n
o
m
a
li
e
s
d
e
tec
ti
o
n
.
T
h
is
p
a
p
e
r
g
iv
e
s
a
su
rv
e
y
o
f
R
OC
a
n
a
l
y
sis
b
a
s
e
d
o
n
th
e
a
n
o
m
a
l
y
d
e
tec
ti
o
n
u
sin
g
c
las
sif
iers
f
o
r
u
sin
g
th
e
m
in
re
se
a
rc
h
.
In
re
c
e
n
t
y
e
a
rs
h
a
v
e
b
e
e
n
in
c
re
a
sin
g
l
y
a
d
o
p
ted
i
n
th
e
m
a
c
h
in
e
lea
rn
in
g
a
n
d
d
a
ta
m
in
in
g
re
se
a
rc
h
c
o
m
m
u
n
it
ies
.
T
h
is
su
rv
e
y
g
i
v
e
s
d
e
f
in
it
io
n
s
o
f
th
e
a
n
o
m
a
l
y
d
e
te
c
ti
o
n
t
h
e
o
ry
a
n
d
h
o
w
to
u
se
o
n
e
R
OC
c
u
rv
e
,
w
h
a
t
a
ROC c
u
rv
e
,
w
h
e
n
w
e
u
se
ROC c
u
rv
e
s.
K
ey
w
o
r
d
:
An
o
m
al
y
d
ete
ctio
n
t
h
eo
r
y
C
las
s
i
f
ier
E
v
alu
a
tio
n
J
am
m
i
n
g
s
i
g
n
al
R
OC
An
al
y
s
i
s
Co
p
y
rig
h
t
©
2
0
1
6
In
stit
u
te o
f
A
d
v
a
n
c
e
d
E
n
g
i
n
e
e
rin
g
a
n
d
S
c
ien
c
e
.
Al
l
rig
h
ts
re
se
rv
e
d
.
C
o
r
r
e
s
p
o
nd
ing
A
uth
o
r
:
Ah
m
ed
Mo
u
m
e
n
a
,
El
e
c
tro
n
ic De
p
a
rtm
e
n
t
,
S
a
a
d
Da
h
lab
-
Bli
d
a
Un
iv
e
rsity
,
A
l
g
e
ri
a
.
E
m
ail:
aa
m
o
u
m
e
n
a@
g
m
ail.
co
m
1.
I
NT
RO
D
UCT
I
O
N
T
h
is
s
u
r
v
e
y
p
r
esen
t
s
a
n
o
v
el
m
ac
h
in
e
lear
n
i
n
g
ap
p
r
o
ac
h
es
to
s
p
ec
tr
u
m
s
en
s
i
n
g
in
co
l
lab
o
r
ativ
e
co
g
n
iti
v
e
r
ad
io
s
y
s
te
m
s
in
th
e
p
r
esen
ce
o
f
s
o
u
r
ce
s
i
g
n
al,
j
am
m
in
g
s
i
g
n
a
l
a
n
d
n
o
i
s
e.
C
o
g
n
itiv
e
r
ad
io
(
C
R
)
i
s
a
n
o
v
el
tec
h
n
o
lo
g
y
t
h
at
allo
w
s
i
m
p
r
o
v
in
g
s
p
ec
tr
u
m
u
til
izatio
n
b
y
e
n
ab
lin
g
o
p
p
o
r
tu
n
i
s
tic
ac
c
ess
to
th
e
lice
n
s
ed
s
p
e
ctr
u
m
b
an
d
b
y
u
n
lice
n
s
ed
u
s
er
s
.
T
h
is
i
s
ac
co
m
p
lis
h
ed
t
h
r
o
u
g
h
h
eter
o
g
en
eo
u
s
ar
ch
itec
t
u
r
es a
n
d
tec
h
n
iq
u
e
s
o
f
d
y
n
a
m
ic
s
p
ec
tr
u
m
ac
ce
s
s
(
DSA
)
.
T
h
e
C
R
i
s
d
e
f
in
ed
a
s
a
s
m
ar
t
w
ir
eles
s
co
m
m
u
n
ica
tio
n
s
y
s
te
m
t
h
at
is
a
w
ar
e
o
f
i
ts
en
v
ir
o
n
m
e
n
t
a
n
d
is
ca
p
ab
le
to
lear
n
f
r
o
m
t
h
e
e
n
v
ir
o
n
m
en
t
a
n
d
ad
ap
t
it
s
tr
a
n
s
m
is
s
io
n
p
ar
a
m
eter
s
,
s
u
c
h
as
f
r
eq
u
en
c
y
,
m
o
d
u
latio
n
,
an
d
tr
an
s
m
is
s
io
n
p
o
w
er
an
d
co
m
m
u
n
icatio
n
p
r
o
to
co
ls
.
An
i
m
p
o
r
tan
t
asp
ec
t
o
f
a
C
R
is
s
p
ec
tr
u
m
s
e
n
s
i
n
g
(
SS
)
,
wh
ich
i
n
v
o
l
v
es
a
p
r
i
n
cip
al
ta
s
k
:
j
a
m
m
in
g
s
ig
n
al
d
etec
tio
n
.
J
a
m
m
i
n
g
s
ig
n
al
d
etec
tio
n
r
ef
er
s
to
th
e
d
ete
ctio
n
o
f
an
o
m
alies
cr
ea
ted
b
y
u
n
d
es
ir
ab
le
s
ig
n
al
s
th
at
d
is
r
u
p
t
o
r
j
eo
p
ar
d
ize
th
e
c
o
m
m
u
n
icatio
n
s
b
et
w
ee
n
p
r
i
m
ar
y
u
s
er
s
(
P
Us)
an
d
s
ec
o
n
d
ar
y
u
s
er
s
(
SUs
)
,
u
s
i
n
g
m
ac
h
in
e
lear
n
i
n
g
al
g
o
r
ith
m
s
an
d
p
atter
n
s
r
ec
o
g
n
itio
n
.
T
h
is
tas
k
is
i
m
p
o
r
tan
t
s
o
t
h
at
t
h
e
u
n
l
icen
s
ed
u
s
er
s
(
j
am
m
er
s
)
d
o
ca
u
s
e
in
ter
f
er
en
ce
to
lic
en
s
ed
u
s
er
s
(
P
Us,
SUs
)
[
L
ilian
,
1
2
]
.
T
h
e
r
ec
eiv
ed
s
ig
n
al
b
y
e
v
e
r
y
w
id
eb
an
d
C
R
r
ec
ei
v
er
is
u
s
ed
b
y
a
cla
s
s
i
f
ier
f
o
r
d
etec
tio
n
a
t
a
f
u
s
io
n
ce
n
ter
(
F
C
)
to
m
a
k
e
a
g
lo
b
al
d
ec
is
io
n
ab
o
u
t
t
h
e
a
v
ailab
il
it
y
o
f
a
n
o
m
alie
s
/o
u
t
lier
s
ca
u
s
e
d
b
y
t
h
e
e
f
f
ec
ts
o
f
j
am
m
er
s
.
I
n
t
h
e
la
s
t
f
e
w
y
ea
r
s
,
R
O
C
g
r
ap
h
is
ap
p
ea
r
in
g
m
o
r
e
in
ten
s
el
y
i
n
t
h
e
d
o
m
ai
n
o
f
m
ac
h
in
e
lear
n
in
g
.
A
ct
u
all
y
,
t
h
eir
u
s
e
as
a
m
etr
ic
to
ev
alu
a
te
m
ac
h
i
n
e
lear
n
in
g
alg
o
r
ith
m
s
h
a
s
b
ec
o
m
e
n
ec
ess
ar
y
.
I
n
t
h
is
s
u
r
v
e
y
,
th
e
ter
m
s
R
O
C
an
a
l
y
s
is
,
R
OC
g
r
ap
h
an
d
R
OC
c
u
r
v
e
ar
e
u
s
e
d
b
u
t th
e
m
o
s
t u
s
ed
is
R
O
C
an
al
y
s
i
s
.
On
e
o
f
t
h
e
ea
r
lies
t
ad
o
p
ter
s
o
f
R
OC
an
al
y
s
is
i
n
m
ac
h
i
n
e
lear
n
in
g
[
Sp
ac
k
m
a
n
,
8
9
]
w
a
s
w
h
o
d
em
o
n
s
tr
ated
th
e
v
al
u
e
o
f
R
OC
cu
r
v
es
in
e
v
al
u
ati
n
g
a
n
d
co
m
p
ar
i
n
g
al
g
o
r
ith
m
s
.
R
ec
en
t
y
ea
r
s
h
a
v
e
s
ee
n
a
n
in
cr
ea
s
e
i
n
th
e
u
s
e
o
f
R
O
C
an
al
y
s
i
s
in
t
h
e
m
ac
h
in
e
lear
n
in
g
co
m
m
u
n
it
y
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8938
IJ
-
AI
Vo
l.
5
,
No
.
3
,
Sep
tem
b
er
2
0
1
6
: 1
0
5
-
116
106
Mo
s
t
b
o
o
k
s
o
n
d
at
a
m
i
n
i
n
g
a
n
d
m
ac
h
in
e
lear
n
i
n
g
,
i
f
t
h
e
y
m
en
tio
n
R
OC
c
u
r
v
e
s
at
all,
h
av
e
o
n
l
y
a
b
r
ief
ex
p
la
n
atio
n
o
f
m
et
h
o
d
.
R
OC
cu
r
v
es a
r
e
co
n
ce
p
tu
all
y
ea
s
y
,
b
u
t t
h
er
e
ar
e
s
o
m
e
n
o
n
-
o
b
v
io
u
s
co
m
p
lex
itie
s
th
at
ar
is
e
w
h
en
t
h
e
y
ar
e
u
s
ed
i
n
r
esear
ch
a
n
d
d
ev
elo
p
m
en
t [
Fa
w
ce
tt,
0
3
]
.
2.
ANO
M
AL
Y
DE
T
E
CT
I
O
N
T
H
E
O
RY
Su
p
p
o
s
e
th
at
f
o
r
s
o
m
e
p
h
y
s
ical
m
ea
s
u
r
e
m
e
n
t
a
p
r
i
m
ar
y
u
s
er
(
P
U)
p
r
o
d
u
ce
s
an
o
u
t
p
u
t
s
ig
n
al
*
(
)
,
-
+
,
o
v
er
a
ti
m
e
i
n
ter
v
al
,
-
Su
p
p
o
s
e
th
a
t
t
h
e
s
i
g
n
al
m
a
y
h
a
v
e
b
ee
n
p
r
o
d
u
ce
d
b
y
j
am
m
i
n
g
s
i
g
n
al,
s
o
u
r
ce
s
ig
n
al
an
d
a
m
b
ie
n
t
n
o
is
e
o
f
t
y
p
e
AW
GN.
T
h
er
e
ar
e
t
w
o
p
o
s
s
ib
il
ities
ar
e
ca
lled
t
h
e
h
y
p
o
t
h
esi
s
an
d
th
e
h
y
p
o
t
h
esi
s
,
r
esp
ec
tiv
el
y
,
an
d
ar
e
co
m
m
o
n
l
y
w
r
i
tten
i
n
t
h
e
co
m
p
ac
t n
o
t
atio
n
:
First h
y
p
o
th
e
s
is
:
ev
e
n
ts
:
n
o
r
m
al
o
b
s
er
v
atio
n
s
.
Seco
n
d
h
y
p
o
t
h
esi
s
:
ev
en
ts
: a
n
o
m
a
l
y
/ab
n
o
r
m
al
o
b
s
er
v
at
io
n
s
.
T
o
d
ec
i
d
e
b
et
w
ee
n
t
h
e
f
ir
s
t
an
d
s
ec
o
n
d
h
y
p
o
th
e
s
es
o
n
e
m
i
g
h
t
ap
p
l
y
a
h
i
g
h
t
h
r
esh
o
ld
to
th
e
class
if
ie
r
o
u
tp
u
t
an
d
m
ak
e
a
d
ec
is
io
n
t
h
at
t
h
e
a
n
o
m
al
ies
ar
e
p
r
ese
n
t
if
a
n
d
o
n
l
y
if
th
e
t
h
r
esh
o
ld
is
e
x
ce
ed
ed
th
e
t
h
r
es
h
o
ld
v
alu
e
T
h
e
en
g
in
ee
r
is
th
e
n
f
ac
ed
w
it
h
th
e
p
r
ac
tica
l
q
u
esti
o
n
o
f
w
h
er
e
to
f
ix
t
h
e
th
r
es
h
o
ld
s
o
as
to
en
s
u
r
e
th
at
t
h
e
n
u
m
b
er
o
f
d
ec
is
io
n
er
r
o
r
s
is
s
m
all.
T
h
er
e
ar
e
t
w
o
t
y
p
e
s
o
f
er
r
o
r
p
o
s
s
ib
le:
th
e
er
r
o
r
o
f
m
is
s
in
g
(
d
ec
id
e
u
n
d
er
(
p
r
o
b
le
m
o
f
an
o
m
alie
s
is
p
r
esen
t)
)
an
d
th
e
er
r
o
r
o
f
f
alse
alar
m
(
d
ec
id
e
u
n
d
er
(
n
o
p
r
o
b
lem
o
f
an
o
m
alies
is
p
r
ese
n
t)
)
.
T
h
er
e
is
al
w
a
y
s
a
co
m
p
r
o
m
is
e
b
et
w
ee
n
c
h
o
o
s
in
g
a
h
i
g
h
t
h
r
es
h
o
ld
to
m
a
k
e
th
e
a
v
er
ag
e
n
u
m
b
er
o
f
f
a
ls
e
alar
m
s
s
m
all
v
er
s
u
s
c
h
o
o
s
in
g
a
lo
w
th
r
es
h
o
ld
to
m
a
k
e
t
h
e
av
er
a
g
e
n
u
m
b
er
o
f
m
is
s
es
s
m
all.
T
o
q
u
an
ti
f
y
t
h
i
s
co
m
p
r
o
m
i
s
e
it
b
ec
o
m
e
s
n
ec
ess
ar
y
to
s
p
ec
if
y
t
h
e
s
tati
s
tic
al
d
is
tr
ib
u
tio
n
o
f
u
n
d
er
ev
er
y
o
f
t
h
e
h
y
p
o
t
h
ese
s
an
d
.
Fig
u
r
e
1
.
Sh
o
w
Dec
is
io
n
C
r
ite
r
io
n
.
3
.
DE
F
I
NIT
I
O
N
o
f
RO
C
CURV
E
A
R
O
C
c
u
r
v
e
is
a
t
w
o
d
i
m
e
n
s
io
n
al
(
2
d
)
o
f
th
e
ac
cu
r
ac
y
o
f
a
clas
s
i
f
ier
f
o
r
an
o
m
al
y
d
ete
ctio
n
.
T
h
is
2
d
cu
r
v
e
s
h
o
w
,
h
o
w
th
e
tr
u
e
-
p
o
s
iti
v
e
r
ate
(
T
P
R
)
o
f
d
etec
tio
n
d
ec
r
ea
s
es
as
t
h
e
f
alse
-
p
o
s
itiv
e
r
ate
(
FP
R
)
in
cr
ea
s
es
[
Ma
tj
a,
1
1
]
.
A
R
O
C
p
lo
ts
T
P
R
o
f
d
etec
tio
n
a
g
ain
s
t
FP
R
.
T
h
ese
t
w
o
t
y
p
e
s
o
f
r
a
te
m
ea
n
s
d
etec
tio
n
th
r
es
h
o
ld
.
T
P
R
is
h
i
g
h
e
s
t
a
n
d
FP
R
is
lo
w
e
s
t.
T
h
is
p
r
in
ci
p
al
r
elatio
n
i
n
t
w
o
co
m
p
o
n
e
n
ts
o
f
ac
c
u
r
ac
y
w
ill
ch
an
g
e
f
r
o
m
o
n
e
cla
s
s
i
f
ier
to
th
e
n
e
x
t.
T
h
is
m
ak
e
s
t
h
e
f
o
r
m
o
f
ev
er
y
R
O
C
d
if
f
er
en
t
t
h
e
m
o
th
er
.
W
e
ca
n
u
s
e
R
OC
a
n
al
y
s
is
to
k
n
o
w
h
o
w
a
s
i
m
p
le
o
r
in
d
iv
id
u
al
clas
s
i
f
ier
is
b
eh
av
in
g
o
n
d
ataset?
,
o
r
to
co
m
p
ar
e
th
e
ac
cu
r
ac
y
o
f
t
w
o
o
r
th
r
ee
o
r
m
o
r
e
class
if
ier
s
o
n
th
e
d
atase
t.
R
OC
c
u
r
v
e
ex
p
lai
n
m
o
r
e
d
eta
iled
an
al
y
s
es
ab
o
u
t
th
e
e
x
p
ec
t
ed
ac
cu
r
ac
y
an
d
co
s
t
o
f
t
h
e
cl
ass
i
f
ier
.
I
f
w
e
k
n
o
w
h
o
w
ab
n
o
r
m
a
l
o
b
s
er
v
atio
n
s
e
v
e
n
ts
ar
e
i
n
r
elatio
n
to
n
o
r
m
al
o
b
s
er
v
atio
n
s
e
v
en
t
s
,
w
e
ca
n
es
ti
m
ate
th
e
r
atio
o
f
th
e
t
w
o
k
i
n
d
s
o
f
er
r
o
r
s
f
o
r
ev
er
y
t
h
r
esh
o
ld
lev
el,
th
e
s
a
m
e
ca
s
e
f
o
r
th
e
co
s
ts
.
W
e
ca
n
ch
o
ice
an
o
p
tim
a
l c
lass
i
f
ier
o
u
t o
f
m
a
n
y
ca
n
d
id
ates.
T
h
e
k
n
o
w
led
g
e
o
f
w
h
et
h
er
a
s
i
m
p
le
R
O
C
d
o
m
i
n
ate
s
all
o
th
er
s
is
i
m
p
o
r
ta
n
t.
I
t
is
i
m
p
o
r
tan
t
to
i
m
p
r
ec
is
e
e
v
en
t
-
cla
s
s
a
n
d
er
r
o
r
-
co
s
t.
I
n
ad
d
itio
n
,
w
h
e
n
th
er
e
is
n
o
s
i
m
p
le
d
o
m
in
ato
r
,
t
h
e
v
ar
io
u
s
R
O
C
w
h
ic
h
s
h
ar
e
d
o
m
i
n
atio
n
ca
n
b
e
u
s
e
d
to
b
u
ild
h
y
b
r
id
cla
s
s
i
f
ier
wh
ich
eq
u
al
s
o
r
o
u
tp
er
f
o
r
m
s
a
n
y
s
i
m
p
le
cla
s
s
i
f
ier
[
P
r
o
v
o
s
t,
0
1
.
a]
.
N
o
r
m
a
l
s
D
é
c
is
i
o
n
t
h
r
e
s
h
o
ld
N
o
Y
e
s
A
n
o
m
a
li
e
s
Evaluation Warning : The document was created with Spire.PDF for Python.
IJ
-
AI
I
SS
N:
2252
-
8938
A
n
o
ma
lies
Dete
ctio
n
B
a
s
ed
o
n
th
e
R
OC
A
n
a
lysi
s
u
s
in
g
C
la
s
s
ifie
r
s
in
Ta
ctica
l C
o
g
n
itive.
.
(
Ah
m
ed
Mo
u
m
e
n
a
)
107
A
m
a
tr
ix
is
u
s
ed
ca
lled
co
n
f
u
s
io
n
m
atr
i
x
(
r
ep
r
esen
t
s
th
e
co
n
f
u
s
io
n
b
et
w
ee
n
clas
s
es).
T
h
e
r
e
ar
e
f
o
u
r
o
u
tp
u
ts
f
o
r
cla
s
s
i
f
icat
io
n
o
f
ea
ch
i
n
s
ta
n
ce
/o
b
s
er
v
a
tio
n
o
r
p
att
er
n
.
I
f
th
e
in
s
ta
n
ce
is
p
o
s
iti
v
e
an
d
is
clas
s
if
ied
as
s
u
c
h
th
e
n
w
e
d
en
o
te
it
as
class
if
ier
s
in
(
)
an
d
(
)
ar
e
ca
lled
d
ef
a
u
lt
d
etec
to
r
s
.
T
h
e
p
er
f
ec
t
clas
s
if
ier
i
s
(
)
(
)
.
A
ll
cla
s
s
i
f
ier
s
ar
e
lo
ca
ted
o
n
th
e
d
iag
o
n
al
li
n
e
h
a
v
e
th
e
s
a
m
e
p
er
f
o
r
m
an
ce
.
I
t
is
s
aid
th
e
y
h
a
v
e
n
o
in
f
o
r
m
atio
n
ab
o
u
t
t
h
e
p
r
o
b
lem
.
A
ll
clas
s
i
f
ier
s
lo
ca
ted
ab
o
v
e
th
e
d
iag
o
n
al
ar
e
u
s
e
f
u
l.
C
o
n
f
u
s
io
n
m
atr
ix
ca
lcu
latio
n
: th
a
t sh
o
w
co
r
r
ec
t a
n
d
T
ab
le
1
.
C
o
n
f
u
s
io
n
Ma
tr
i
x
.
T
P
=tr
u
e
p
o
s
itiv
es:
a
n
a
n
o
m
a
l
y
o
b
s
er
v
atio
n
is
cla
s
s
i
f
ied
c
o
r
r
ec
tly
s
u
c
h
as
a
n
o
m
a
l
y
o
b
s
er
v
atio
n
,
w
h
ic
h
m
ea
n
s
p
r
esen
t
a
n
d
d
etec
ted
.
FP
=f
alse
p
o
s
itiv
es:
a
n
o
r
m
al
o
b
s
er
v
atio
n
is
clas
s
if
ie
d
s
u
ch
as
an
o
m
al
y
o
b
s
er
v
atio
n
,
w
h
ic
h
m
ea
n
s
n
o
t
p
r
esen
t
b
u
t
d
etec
ted
.
T
N=
tr
u
e
n
eg
a
tiv
e
s
:
a
n
o
r
m
al
o
b
s
er
v
atio
n
is
clas
s
i
f
ied
s
u
c
h
a
s
n
o
r
m
al
o
b
s
er
v
atio
n
,
w
h
ic
h
m
ea
n
s
n
o
t
p
r
ese
n
ted
a
n
d
n
o
t
d
etec
ted
.
FN=
f
alse
n
e
g
ati
v
es:
an
an
o
m
al
y
o
b
s
er
v
atio
n
is
f
au
lts
cla
s
s
i
f
ied
s
u
c
h
as n
o
r
m
al
o
b
s
er
v
atio
n
,
w
h
ic
h
m
ea
n
s
p
r
ese
n
t b
u
t
n
o
t d
etec
ted
.
T
r
u
e
p
o
s
itiv
e
r
ate:
: p
o
s
itiv
es
co
r
r
ec
tly
cla
s
s
i
f
ied
/to
tal
p
o
s
iti
v
es
Fals
e
p
o
s
iti
v
e
r
ate
(
also
ca
lled
f
alse
alar
m
r
ate)
:
:
n
e
g
ati
v
es
in
co
r
r
ec
tl
y
clas
s
i
f
ied
/
to
tal
class
i
f
ied
.
P
o
s
itiv
es p
r
e
d
ictio
n
r
ate
th
at
ar
e
co
r
r
ec
ts
.
T
r
u
e
n
eg
ativ
e
r
ate:
Fals
e
n
e
g
ati
v
e
r
ate:
A
d
d
itio
n
al
ter
m
s
a
s
s
o
ciate
d
with
R
OC
c
u
r
v
e
s
ar
e:
Sen
s
iti
v
it
y
=
r
ec
all
Sp
ec
if
icit
y
=
=1
-
f
alse p
o
s
it
iv
e
r
ate.
P
o
s
itiv
e
p
r
ed
ictiv
e
v
al
u
e=
p
r
ec
is
io
n
.
4
.
RO
C
SPAC
E
R
OC
g
r
ap
h
s
ar
e
t
w
o
-
d
i
m
e
n
s
i
o
n
al
cu
r
v
es
i
n
w
h
ic
h
is
p
lo
tt
ed
o
n
th
e
y
-
a
x
i
s
an
d
is
p
lo
tte
d
o
n
th
e
x
-
a
x
is
.
A
n
R
O
C
g
r
ap
h
d
ep
icts
r
elat
iv
e
tr
ad
eo
f
f
s
b
et
w
ee
n
b
e
n
ef
its
(
)
an
d
co
s
ts
(
)
Fig
u
r
e
.
2
,
b
elo
w
s
h
o
w
s
a
n
R
OC
g
r
ap
h
w
it
h
f
i
v
e
clas
s
if
ier
s
lab
eled
th
r
o
u
g
h
.
A
d
is
cr
ete
class
i
f
ier
is
o
n
e
th
at
o
u
tp
u
ts
o
n
l
y
a
cla
s
s
lab
el.
E
ac
h
s
i
m
p
le
clas
s
if
ier
d
etec
to
r
p
r
o
d
u
ce
s
an
(
FP
R
,
T
P
R
)
p
air
co
r
r
esp
o
n
d
in
g
to
a
s
i
m
p
le
p
o
in
t
i
n
R
OC
s
p
ac
e.
T
h
e
cla
s
s
if
ier
s
i
n
Fig
u
r
e
.
2
ar
e
all
d
is
c
r
ete
class
i
f
ier
s
.
Se
v
er
al
p
o
in
t
s
in
R
O
C
s
p
ac
e
ar
e
i
m
p
o
r
tan
t
to
n
o
te.
T
h
e
lo
w
er
l
ef
t
p
o
in
t
(
)
r
ep
r
esen
ts
t
h
e
s
tr
ate
g
y
o
f
n
e
v
er
is
s
u
in
g
a
p
o
s
i
tiv
e
class
i
f
icatio
n
;
s
u
c
h
a
clas
s
i
f
ier
co
m
m
it
s
n
o
er
r
o
r
s
b
u
t
also
g
ain
s
n
o
.
T
h
e
o
p
p
o
s
ite
s
tr
ateg
y
o
f
u
n
co
n
d
iti
o
n
all
y
i
s
s
u
i
n
g
Ev
e
n
t
s
D
e
c
i
si
o
n
A
n
o
mal
y
N
o
r
mal
Y
e
s
T
P
H
i
t
FP
F
a
l
s
e
-
a
l
a
r
m
N
o
FN
M
i
ss
i
n
c
o
r
r
e
c
t
p
r
e
d
i
c
t
i
o
n
s
T
N
C
o
r
r
e
c
t
-
r
e
j
e
c
t
i
o
n
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8938
IJ
-
AI
Vo
l.
5
,
No
.
3
,
Sep
tem
b
er
2
0
1
6
: 1
0
5
-
116
108
p
o
s
itiv
e
clas
s
i
f
icatio
n
s
,
is
r
e
p
r
esen
ted
b
y
th
e
u
p
p
er
r
ig
h
t
p
o
in
t
(
)
T
h
e
p
o
in
t
(
)
r
ep
r
esen
ts
p
er
f
ec
t
class
i
f
icatio
n
.
p
er
f
o
r
m
a
n
ce
i
s
p
er
f
ec
t a
s
s
h
o
w
n
i
n
th
e
F
ig
u
r
e
.
I
n
f
o
r
m
a
ll
y
,
o
n
e
p
o
in
t
i
n
R
O
C
s
p
ac
e
is
b
etter
th
an
an
o
th
er
i
f
it
is
to
th
e
n
o
r
t
h
w
est
(
is
h
ig
h
er
,
is
lo
w
er
)
.
C
las
s
i
f
ier
s
ap
p
ea
r
in
g
o
n
t
h
e
le
f
t
-
h
a
n
d
s
id
e
o
f
a
n
R
OC
g
r
ap
h
,
n
ea
r
th
e
x
-
ax
is
,
m
a
y
b
e
th
o
u
g
h
t
o
f
as
‘
‘
co
n
s
er
v
a
tiv
e
’
’
:
th
e
y
m
ak
e
p
o
s
iti
v
e
c
lass
if
icatio
n
s
o
n
l
y
w
i
th
s
tr
o
n
g
ev
id
e
n
ce
s
o
t
h
e
y
m
a
k
e
f
e
w
f
al
s
e
p
o
s
itiv
e
er
r
o
r
s
,
b
u
t
t
h
e
y
o
f
te
n
h
av
e
lo
w
as
w
e
ll.
C
lass
if
i
er
s
o
n
t
h
e
u
p
p
er
r
ig
h
t
-
h
an
d
s
id
e
o
f
an
R
O
C
g
r
ap
h
m
a
y
b
e
t
h
o
u
g
h
t
o
f
as
‘
‘
l
ib
er
al’
’
:
t
h
e
y
m
a
k
e
p
o
s
it
iv
e
cl
ass
i
f
icatio
n
s
w
i
th
w
ea
k
e
v
id
en
ce
s
o
th
e
y
cla
s
s
i
f
y
n
ea
r
l
y
all
p
o
s
iti
v
es
co
r
r
ec
tl
y
,
b
u
t
th
e
y
o
f
ten
h
av
e
h
ig
h
I
n
Fi
g
u
r
e
.
2
,
A
is
m
o
r
e
co
n
s
er
v
ati
v
e
th
a
n
.
Ma
n
y
r
ea
l
w
o
r
ld
d
o
m
ai
n
s
ar
e
d
o
m
i
n
ated
b
y
lar
g
e
n
u
m
b
er
s
o
f
n
e
g
ati
v
e
in
s
ta
n
ce
s
o
r
o
b
s
er
v
atio
n
s
,
s
o
p
er
f
o
r
m
a
n
ce
i
n
th
e
f
ar
lef
t
-
h
an
d
s
id
e
o
f
t
h
e
R
OC
g
r
ap
h
b
ec
o
m
e
s
m
o
r
e
i
n
ter
esti
n
g
.
Fig
u
r
e
2.
Sh
o
w
s
a
n
R
O
C
g
r
ap
h
w
it
h
f
i
v
e
d
is
cr
ete
clas
s
i
f
ier
s
lab
eled
A
th
r
o
u
g
h
E
[
Fa
w
ce
tt,
0
3
]
.
5.
USI
N
G
DA
T
AS
E
T
T
E
ST
T
O
B
UIL
D
RO
C
CURV
E
O
F
CL
ASS
I
F
I
E
RS F
O
R
D
E
T
E
CT
I
O
N
T
h
e
class
if
ier
o
u
t
p
u
t r
ep
r
esen
t
s
all
r
an
g
e
s
o
f
p
o
s
s
ib
le
s
co
r
es.
P
o
in
ts
ab
o
u
t d
ataset:
T
h
e
d
ef
in
i
tio
n
o
f
‘
’
ev
e
n
t
’
’
m
u
s
t
b
e
clea
r
a
n
d
m
u
s
t
co
v
er
a
ll
n
ec
e
s
s
ar
y
cir
c
u
m
s
ta
n
ce
s
.
E
v
er
y
e
v
e
n
t
‘
’
an
o
m
al
y
o
b
s
er
v
at
io
n
s
’
’
o
r
‘
’
n
o
r
m
a
l
o
b
s
er
v
atio
n
s
’
’
w
ill
b
e
r
an
k
ed
in
d
ep
en
d
en
tl
y
b
y
class
i
f
ier
d
etec
to
r
.
E
v
er
y
e
v
en
t
s
h
o
u
ld
b
e
lab
eled
as
‘
’
a
n
o
m
al
y
’
’
o
r
as
‘
’
n
o
r
m
al
’
’
.
I
t
is
n
ec
ess
ar
y
to
h
a
v
e
m
a
n
y
ex
a
m
p
les
o
f
b
o
th
an
o
m
alie
s
a
n
d
n
o
r
m
al.
E
v
er
y
ca
teg
o
r
y
o
f
an
o
m
alie
s
an
d
n
o
r
m
al,
t
h
er
e
m
u
s
t
b
e
r
ep
r
esen
tati
v
e
t
y
p
e
s
a
n
d
p
r
o
p
o
r
tio
n
s
o
f
ev
e
n
ts
.
Ho
w
to
g
r
o
u
p
d
ataset
s
co
r
es:
Dec
id
e
w
h
a
t
t
y
p
e
o
f
th
r
es
h
o
l
d
to
u
s
e.
W
e
ca
n
ch
o
o
s
e
v
alu
es
o
f
th
r
es
h
o
ld
s
th
at
co
r
r
esp
o
n
d
to
f
ix
ed
lev
els
o
f
FP
R
.
T
h
e
n
u
m
b
er
o
f
v
alu
e
s
o
f
th
r
e
s
h
o
ld
s
w
e
u
s
e
w
i
ll
b
e
t
h
e
n
u
m
b
er
o
f
R
OC
cu
r
v
e
p
o
in
ts
o
n
th
e
g
r
ap
h
.
R
u
n
t
h
e
cla
s
s
i
f
ier
d
ete
cto
r
o
n
th
e
e
v
al
u
atio
n
d
ataset.
C
o
m
p
ar
e
t
h
e
a
n
o
m
al
y
cla
s
s
i
f
ier
s
d
etec
tio
n
r
es
u
lt
to
th
e
g
r
o
u
n
d
tr
u
th
lab
el
o
v
e
r
all
ev
en
ts
‘
’
an
o
m
al
y
o
b
s
er
v
atio
n
s
:
e
v
en
t.1
:
’
’
o
r
‘
’
n
o
r
m
al
o
b
s
er
v
atio
n
s
:
ev
en
.
2
:
’’.
6.
I
NT
E
RP
R
E
T
A
T
I
O
N
O
F
R
O
C
CURV
E
T
h
er
e
ar
e
tw
o
t
y
p
e
s
o
f
e
v
en
ts
an
d
t
w
o
t
y
p
es o
f
ac
c
u
r
ac
ies p
o
s
s
ib
le.
R
O
C
cu
r
v
e
w
it
h
t
w
o
-
d
i
m
en
s
io
n
s
(
2
d
)
,
y
-
a
x
is
s
h
o
w
s
u
cc
e
s
s
r
a
te
(
ab
n
o
r
m
a
l
o
b
s
er
v
atio
n
s
:
e
v
en
t
s
)
o
f
d
etec
tio
n
a
n
d
x
-
a
x
is
s
h
o
w
er
r
o
r
r
ate
(
n
o
r
m
al
o
b
s
er
v
atio
n
s
:
ev
e
n
ts
)
.
Su
cc
ess
i
s
b
etter
an
d
er
r
o
r
is
n
o
t
g
o
o
d
.
I
d
e
a
l
R
OC
m
ea
n
s
i
n
y
-
ax
i
s
th
e
v
a
lu
e
s
g
r
o
w
s
at
a
q
u
ic
k
est
r
ate
an
d
in
x
-
ax
i
s
t
h
e
v
al
u
es
r
is
e
s
s
w
if
tl
y
u
p
w
ar
d
,
th
e
er
r
o
r
v
alu
e
s
f
o
r
(
n
o
r
m
a
l
o
b
s
er
v
atio
n
s
:
ev
e
n
t
s
)
x
-
a
x
is
m
u
s
t
r
i
s
e
lar
g
e.
T
h
e
p
er
f
ec
t
R
O
C
c
u
r
v
e
to
u
ch
e
s
t
h
e
p
o
i
n
t
(
0
,
1
)
.
T
h
er
e
is
d
if
f
er
e
n
t
f
o
r
m
o
f
R
OC
c
u
r
v
e
s
w
h
ic
h
m
ea
n
s
d
if
f
er
en
t
le
v
el
s
o
f
clas
s
if
ier
ac
cu
r
ac
y
.
A
p
e
r
f
ec
t
class
if
ier
w
i
ll
h
av
e
a
s
u
cc
es
s
r
ate
o
f
1
.
0
f
o
r
(
ab
n
o
r
m
al
o
b
s
er
v
at
io
n
s
:
e
v
e
n
ts
)
w
h
ile
h
a
v
i
n
g
an
er
r
o
r
r
at
e
o
f
0
.
0
f
o
r
(
n
o
r
m
al
o
b
s
er
v
atio
n
s
: e
v
e
n
ts
)
.
Un
f
o
r
tu
n
atel
y
,
t
h
i
s
r
esu
lt is
d
i
f
f
i
c
u
lt to
o
b
tain
.
E
ac
h
R
O
C
i
s
b
ased
o
n
t
h
e
m
ea
s
u
r
e
m
en
t
s
o
f
c
lass
if
ier
p
er
f
o
r
m
a
n
ce
at
d
i
f
f
er
e
n
t
d
ec
is
io
n
th
r
es
h
o
ld
v
alu
e
s
.
B
ased
o
n
th
e
R
OC
c
u
r
v
e,
th
e
s
tr
icter
th
r
es
h
o
ld
v
alu
e
clo
s
er
to
(
0
,
0
)
p
o
in
t
an
d
th
e
m
o
r
e
len
ien
t
th
r
es
h
o
ld
v
al
u
e
ap
p
ea
r
clo
s
er
to
(
1
,
0
)
p
o
in
t.
T
h
e
(
0
,
0
)
p
o
in
t
co
r
r
esp
o
n
d
s
to
tell
(
N
O)
an
d
(
1
,
0
)
p
o
in
t
co
r
r
esp
o
n
d
s
to
tell (
YE
S).
T
h
e
ai
m
is
to
m
in
i
m
ize
ex
p
ec
ted
co
s
t a
n
d
to
m
ax
i
m
ize
th
e
T
P
R
g
iv
e
n
a
f
ix
ed
FP
R
.
Evaluation Warning : The document was created with Spire.PDF for Python.
IJ
-
AI
I
SS
N:
2252
-
8938
A
n
o
ma
lies
Dete
ctio
n
B
a
s
ed
o
n
th
e
R
OC
A
n
a
lysi
s
u
s
in
g
C
la
s
s
ifie
r
s
in
Ta
ctica
l C
o
g
n
itive.
.
(
Ah
m
ed
Mo
u
m
e
n
a
)
109
Fin
all
y
,
w
e
w
i
ll
g
i
v
e
t
h
e
co
n
ce
p
t
o
f
d
etec
tio
n
w
h
ic
h
t
h
e
R
OC
c
u
r
v
e
w
ill
b
e
ea
s
ier
to
u
n
d
er
s
ta
n
d
.
T
ab
le
1
.
d
escr
ib
e
th
e
T
P
R
an
d
FP
R
f
o
r
f
o
u
r
p
o
s
s
ib
le
d
etec
tio
n
o
u
tp
u
t
s
.
T
h
er
e
ar
e
t
w
o
p
o
s
s
ib
le
tr
u
e
cla
s
s
es
:
(
an
o
m
al
y
o
b
s
er
v
atio
n
:
ev
e
n
ts
)
an
d
(
n
o
r
m
al
o
b
s
er
v
atio
n
:
ev
e
n
ts
)
an
d
p
o
s
s
ib
le
d
ec
is
io
n
cla
s
s
es
(
YE
S:
m
ea
n
s
an
o
m
al
y
a
n
d
NO:
it
is
n
o
r
m
al
.
T
w
o
o
f
t
h
e
s
e
o
u
tp
u
ts
ar
e
s
u
c
ce
s
s
f
u
l
w
h
e
n
t
h
e
d
ec
is
io
n
m
at
ch
es
tr
u
t
h
an
d
t
w
o
ar
e
er
r
o
n
eo
u
s
,
w
h
en
t
h
er
e
is
a
m
is
m
atc
h
b
et
w
ee
n
t
h
e
d
ec
is
io
n
an
d
th
e
tr
u
t
h
.
W
e
w
ill
u
s
e
t
h
e
ter
m
s
T
P
an
d
F
P
b
ec
au
s
e
th
e
y
ar
e
f
r
eq
u
e
n
tl
y
u
s
ed
in
f
o
r
m
u
las a
n
d
to
r
ep
r
esen
t th
e
ax
e
s
o
f
R
OC
c
u
r
v
e
s
.
7.
ADVA
N
T
A
G
E
S O
F
US
I
N
G
RO
C
ANALY
SI
S
a.
Vis
u
a
lize
ac
cu
r
ac
y
o
f
clas
s
i
f
ie
r
f
o
r
d
etec
tio
n
.
b.
Facilitate
t
h
e
co
m
p
ar
is
o
n
o
f
m
o
r
e
class
if
ier
s
.
c.
R
ec
o
g
n
ize
th
e
i
m
p
o
r
tan
ce
o
f
v
alu
e
th
r
e
s
h
o
ld
d
ec
is
io
n
.
8.
M
E
ASURE
S O
F
RO
C
ANA
L
YS
I
S F
O
R
ANO
M
AL
Y
DE
T
E
C
T
I
O
N
8
.
1
M
ea
s
ures o
f
Acc
ura
cy
Vis
u
a
lizatio
n
o
f
R
O
C
c
u
r
v
e
p
r
o
v
id
es
to
a
cla
s
s
i
f
ier
s
g
lo
b
al
ac
cu
r
ac
y
.
T
h
e
n
at
u
r
e
o
f
R
O
C
is
s
teep
er
w
h
ic
h
m
ea
n
s
a
n
o
m
a
l
y
o
b
s
er
v
atio
n
s
r
ate
is
g
r
ea
ter
.
T
h
e
n
a
tu
r
e
o
f
R
O
C
i
s
f
latter
w
h
ic
h
m
ea
n
s
t
h
e
n
o
r
m
a
l
o
b
s
er
v
atio
n
s
r
ate
is
g
r
ea
ter
.
W
e
ca
n
v
ie
w
t
h
at
R
O
C
cu
r
v
e
ap
p
r
o
ac
h
es
at
th
e
p
o
in
t
o
f
p
er
f
ec
tio
n
(
0
,
1
)
.
Ne
y
m
an
-
p
ea
r
s
o
n
cr
iter
io
n
w
h
i
ch
m
ea
n
s
T
P
R
at
f
ix
ed
FP
R
.
T
h
e
f
ir
s
t
m
ea
n
s
t
h
at
t
h
er
e
is
a
p
ar
ticu
lar
FP
R
,
an
d
th
e
s
ec
o
n
d
m
ea
n
s
a
s
i
m
p
le
m
e
asu
r
e
o
f
ac
c
u
r
ac
y
.
T
P
R
at
f
ix
e
d
FP
R
an
d
A
U
C
w
ill b
e
e
x
p
lai
n
ed
in
s
ec
t
io
n
.
.
8
.
1
.
1
Ney
m
a
n
-
P
ea
rso
n Cr
it
er
io
n
T
h
e
im
p
o
r
tan
ce
o
f
Ne
y
m
a
n
-
P
ea
r
s
o
n
cr
iter
io
n
o
f
an
o
m
al
y
d
etec
tio
n
is
to
m
ax
i
m
ize
th
e
r
ate
o
f
HI
T
(
T
P
)
at
a
f
ix
ed
r
ate
o
f
f
alse
-
al
ar
m
s
(
FP
)
.
A
f
ter
FP
R
is
f
i
x
ed
,
it
r
em
ai
n
s
to
k
n
o
w
w
h
at
th
e
b
est
T
P
R
ac
h
iev
ab
le
f
o
r
th
at
le
v
el
i
s
.
I
t
is
p
o
s
s
ib
le
to
s
ee
at
th
e
g
lo
b
al
f
i
g
u
r
e
o
f
t
h
e
R
O
C
c
u
r
v
e
,
a
n
d
d
ec
id
e
u
p
o
n
a
f
i
x
ed
FP
R
a
n
d
th
is
in
s
tatis
tical
h
y
p
o
th
e
s
is
m
a
y
n
o
t
b
e
co
r
r
ec
t.
T
h
e
R
OC
p
r
o
v
id
es
ess
en
tial
cl
u
e
i
f
th
e
R
OC
i
s
s
teep
i
n
t
h
e
r
eg
io
n
o
f
in
ter
e
s
t.
Si
m
ilar
l
y
,
i
f
t
h
e
R
O
C
i
s
v
er
y
f
lat
i
n
t
h
e
r
eg
io
n
o
f
in
ter
est,
t
h
an
a
lar
g
er
FP
R
w
ill
n
o
t
g
ai
n
m
u
c
h
.
U
s
i
n
g
m
ea
s
u
r
e
o
f
ac
cu
r
ac
y
,
co
m
p
ar
in
g
t
w
o
o
r
m
u
lti
p
le
R
OC
c
u
r
v
e
s
,
ea
s
y
f
i
n
d
th
e
R
OC
cu
r
v
e
w
it
h
g
r
ea
ter
T
P
R
f
o
r
a
g
iv
e
n
f
ix
ed
FP
R
.
8
.
1
.
2
Are
a
un
der
t
he
RO
C
curv
e
–
AU
C
T
o
co
m
p
ar
e
class
i
f
ier
s
f
o
r
d
etec
tio
n
w
e
r
ed
u
ce
R
O
C
p
er
f
o
r
m
a
n
ce
to
a
s
i
m
p
le
s
c
alar
v
al
u
e
r
ep
r
esen
tin
g
ex
p
ec
ted
p
er
f
o
r
m
an
ce
.
T
h
e
ar
ea
o
f
th
i
s
zo
n
e
is
ca
lled
th
e
"
A
r
ea
U
n
d
er
C
u
r
v
e
o
r
AU
C
[
B
r
ad
ley
,
9
7
]
,
[
Han
le
y
,
8
2
]
a
n
d
h
as
b
ec
o
m
e
a
b
etter
alter
n
ativ
e
o
f
ex
ac
tit
u
d
e
(
ac
cu
r
ac
y
)
o
r
er
r
o
r
to
ev
alu
at
e
th
e
clas
s
i
f
ier
s
.
Si
n
ce
th
e
AUC
is
a
p
o
r
tio
n
o
f
th
e
ar
ea
o
f
th
e
u
n
it
s
q
u
ar
e,
its
v
a
l
u
e
w
ill
al
w
a
y
s
b
e
b
et
w
ee
n
0
an
d
1
.
0
.
Ho
w
e
v
er
,
b
ec
au
s
e
r
a
n
d
o
m
g
u
e
s
s
i
n
g
p
r
o
d
u
ce
s
t
h
e
d
iag
o
n
al
li
n
e
b
et
w
ee
n
(
0
,
0
)
an
d
(
1
,
1
)
,
w
h
ic
h
h
as
an
ar
ea
o
f
0
.
5
,
n
o
r
ea
lis
tic
clas
s
if
ier
s
h
o
u
ld
h
a
v
e
an
AUC
le
s
s
th
a
n
0
.
5
[
Fa
w
ce
tt,
0
3
]
.
T
h
e
A
U
C
o
f
a
cla
s
s
i
f
ier
is
eq
u
i
v
a
le
n
t
to
t
h
e
p
r
o
b
ab
ilit
y
t
h
at
a
c
lass
if
ier
g
i
v
e
a
h
i
g
h
er
r
an
k
i
n
g
o
f
a
p
o
s
it
iv
e
e
le
m
en
t
to
a
n
eg
a
tiv
e
ele
m
e
n
t.
T
h
e
A
UC
is
al
s
o
v
er
y
clo
s
e
to
th
e
co
ef
f
icie
n
t
o
f
Gin
i
[
B
r
ei
m
an
,
8
4
]
w
h
ich
co
r
r
esp
o
n
d
to
th
e
ar
ea
b
et
w
ee
n
t
h
e
R
OC
cu
r
v
e
a
n
d
d
iag
o
n
al
s
p
ac
e.
I
n
[
Da
v
i
d
,
1
1
]
th
e
r
elatio
n
s
h
ip
b
et
w
ee
n
AUC
a
n
d
co
ef
f
icie
n
t
Gin
i
w
a
s
s
p
ec
if
ied
to
g
iv
e
(
1
)
Fig
u
r
e
3
.
w
it
h
t
h
e
ar
ea
u
n
d
er
t
w
o
R
O
C
c
u
r
v
e
s
,
A
a
n
d
B
.
I
n
Fig
u
r
e
.
3
a
,
th
e
clas
s
i
f
ier
B
h
a
s
a
lar
g
es
t
ar
ea
an
d
th
e
r
ef
o
r
e
b
est
av
er
ag
e
p
er
f
o
r
m
an
ce
.
I
n
Fi
g
u
r
e
3
b
s
h
o
w
s
t
h
e
AUC
o
f
a
b
in
ar
y
class
i
f
ier
A
,
an
d
a
s
co
r
in
g
clas
s
i
f
ier
B
.
class
if
ier
A
r
ep
r
esen
t
s
p
er
f
o
r
m
a
n
ce
o
f
class
i
f
ier
B
w
h
e
n
B
is
u
s
ed
w
it
h
a
n
in
d
i
v
id
u
al
f
i
x
ed
th
r
es
h
o
ld
.
T
h
r
o
u
g
h
th
e
p
er
f
o
r
m
a
n
ce
o
f
t
h
e
t
w
o
i
s
e
q
u
al
to
a
g
iv
e
n
p
o
in
t
(
A’
s
t
h
r
es
h
o
ld
)
,
A’
s
p
er
f
o
r
m
a
n
ce
b
ec
o
m
e
s
i
n
f
er
io
r
to
B
f
u
r
th
er
f
r
o
m
th
i
s
p
o
in
t.
I
t is p
o
s
s
ib
le
f
o
r
a
h
i
g
h
clas
s
i
f
ie
r
to
p
er
f
o
r
m
w
o
r
s
e
in
a
s
p
ec
if
ic
r
eg
io
n
o
f
R
O
C
s
p
ac
e
th
an
a
lo
w
A
U
C
clas
s
if
ier
.
Fig
u
r
e
3a
.
s
h
o
w
s
a
n
ex
a
m
p
le
o
f
th
i
s
:
cl
as
s
i
f
ier
B
is
g
e
n
er
all
y
b
est
t
h
an
A
e
x
ce
p
t
at
FP
R
>
0
.
6
,
w
h
er
e
A
h
as s
lig
h
t a
d
v
a
n
ta
g
e.
I
n
p
r
ac
tice
A
U
C
p
er
f
o
r
m
s
v
er
y
g
o
o
d
an
d
is
al
w
a
y
s
u
s
ed
a
g
e
n
er
al
m
ea
s
u
r
e
is
d
esire
d
.
B
ec
au
s
e
o
f
it
s
ex
tr
e
m
el
y
g
en
er
al
n
at
u
r
e,
th
e
A
U
C
m
e
asu
r
e
is
id
ea
ll
y
s
u
ited
f
o
r
h
ig
h
-
le
v
el
clas
s
i
f
ier
co
m
p
ar
is
o
n
s
,
s
u
c
h
as
i
n
ev
al
u
atin
g
co
r
e
an
o
m
al
y
d
etec
to
r
t
ec
h
n
o
lo
g
y
.
I
t
is
al
s
o
u
s
e
f
u
l
f
o
r
s
u
m
m
ar
izi
n
g
th
e
en
tire
f
ig
u
r
e
o
f
a
clas
s
i
f
ier
’
s
p
er
f
o
r
m
an
ce
.
I
f
y
o
u
h
a
v
e
m
o
r
e
s
p
ec
if
ic
n
ee
d
s
i
n
a
p
ar
ticu
lar
d
etec
tio
n
s
etti
n
g
,
it
m
a
y
b
e
p
r
ef
er
ab
le
to
u
s
e
t
h
e
p
ar
tial
-
A
U
C
o
r
e
v
en
an
is
o
p
er
f
o
r
m
a
n
ce
li
n
e
in
co
n
j
u
n
c
tio
n
with
a
n
R
O
C
co
n
v
e
x
h
u
l
l to
p
r
o
v
id
e
m
o
r
e
m
ea
n
i
n
g
f
u
l c
o
m
p
ar
is
o
n
s
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8938
IJ
-
AI
Vo
l.
5
,
No
.
3
,
Sep
tem
b
er
2
0
1
6
: 1
0
5
-
116
110
Fig
u
r
e
3a
.
T
h
e
cu
r
v
e
o
n
th
e
le
f
t s
h
o
w
s
t
h
e
ar
ea
u
n
d
er
t
w
o
R
OC
cu
r
v
es.
Fig
u
r
e.
3
b
.
T
h
is
cu
r
v
e
s
h
o
w
s
t
h
e
ar
ea
u
n
d
er
th
e
c
u
r
v
e
s
o
f
a
d
is
cr
ete
class
i
f
ier
(
A
)
an
d
a
p
r
o
b
ab
ilis
tic
class
i
f
ier
(
B
)
.
Fig
u
r
e.
3
: ROC
a
n
d
A
UC
c
u
r
v
es [
Fa
w
ce
tt,
0
7
]
.
8
.
1
.
3
P
a
rt
ia
l
-
AUC
P
ar
tial
ar
ea
u
n
d
er
t
h
e
c
u
r
v
e
is
j
u
s
t
li
k
e
th
e
g
lo
b
al
ar
ea
u
n
d
er
th
e
c
u
r
v
e,
e
x
ce
p
t
t
h
at
o
n
l
y
a
s
u
b
s
et
o
f
th
e
R
O
C
p
ictu
r
e
is
co
n
s
id
er
ed
.
Fo
r
in
s
tan
ce
,
if
it
i
s
k
n
o
w
n
th
a
t
a
FP
R
o
f
0
.
5
0
o
r
g
r
ea
ter
is
co
m
p
letel
y
u
n
ac
ce
p
tab
le,
th
e
n
o
n
l
y
t
h
e
lef
t
h
al
f
o
f
th
e
R
OC
cu
r
v
e
n
e
ed
b
e
co
n
s
id
er
ed
.
I
n
th
is
ca
s
e
,
p
ar
tial
a
r
ea
u
n
d
er
cu
r
v
e
w
ill
n
o
lo
n
g
er
r
an
g
e
b
et
w
ee
n
0
.
5
an
d
1
.
0
,
lik
e
th
e
A
U
C
m
ea
s
u
r
e
d
id
,
b
u
t
it
w
ill
h
a
v
e
a
n
e
w
m
i
n
i
m
u
m
an
d
m
ax
i
m
u
m
le
v
el
(
f
r
o
m
0
.
1
2
5
to
0
.
5
,
r
esp
ec
tiv
el
y
)
w
h
ic
h
al
w
a
y
s
d
ep
en
d
s
o
n
h
o
w
m
u
ch
o
f
t
h
e
p
ict
u
r
e
is
co
n
s
id
er
ed
.
I
n
o
r
d
er
to
n
ar
r
o
w
d
o
w
n
th
e
r
eg
io
n
o
f
t
h
e
cu
r
v
e
o
f
i
n
ter
est,
it
is
n
ec
ess
ar
y
to
h
a
v
e
in
m
i
n
d
a
f
ix
ed
m
ax
i
m
u
m
FP
R
,
a
f
i
x
ed
m
i
n
i
m
u
m
T
R
P
.
I
n
ter
m
s
o
f
f
o
cu
s
it
is
s
o
m
e
w
h
er
e
b
et
w
ee
n
g
lo
b
al
ar
ea
an
d
th
e
FP
R
f
i
x
ed
at
T
P
R
,
b
ec
au
s
e
it
co
n
s
id
er
s
ac
cu
r
ac
y
o
v
er
a
r
an
g
e
o
f
th
e
R
OC
g
r
ap
h
s
b
u
t
n
o
t
o
v
er
th
e
to
tal
c
u
r
v
e
p
ictu
r
e.
Ho
w
e
v
er
,
lik
e
th
e
g
l
o
b
al
m
ea
s
u
r
e
A
U
C
,
it
s
u
f
f
er
s
f
r
o
m
a
m
b
i
g
u
it
y
b
ec
au
s
e
if
t
h
e
cu
r
v
e
s
cr
o
s
s
o
n
e
an
o
th
er
w
it
h
i
n
t
h
e
r
eg
io
n
o
f
in
ter
est,
i
t
is
n
o
t
clea
r
t
h
at
o
n
e
o
f
t
h
e
cu
r
v
es
h
a
v
in
g
a
lar
g
er
ar
ea
w
ill
u
n
a
m
b
i
g
u
o
u
s
l
y
b
e
th
e
b
est
cl
ass
i
f
ier
to
u
s
e
u
n
d
er
d
ep
lo
y
m
en
t
co
n
d
itio
n
s
.
Ho
w
e
v
er
,
i
f
a
s
i
m
p
le
R
O
C
cu
r
v
e
d
o
m
i
n
ate
s
t
h
e
r
eg
io
n
o
f
i
n
te
r
est,
th
e
n
t
h
e
p
ar
tial
AUC
m
ea
s
u
r
e
b
ec
o
m
es
les
s
p
r
o
b
lem
atic
[
W
alter
,
0
5
]
,
[
Ma
n
,
1
3
].
9
.
E
VALUA
T
I
O
N
O
F
ANO
M
AL
I
E
S DE
T
E
CT
I
O
N
US
I
NG
(
RO
C)
O
R
(
AUC)
Stan
d
ar
d
m
ea
s
u
r
es
f
o
r
ev
alu
a
t
in
g
a
n
o
m
a
l
y
d
etec
tio
n
p
r
o
b
le
m
s
:
R
ec
all
(
Det
ec
tio
n
r
ate
o
r
tr
u
e
p
o
s
itiv
e
r
ate
(
T
P
R
)
)
r
atio
b
e
t
w
ee
n
t
h
e
n
u
m
b
er
o
f
co
r
r
ec
tly
d
etec
ted
an
o
m
alie
s
a
n
d
t
h
e
to
tal
n
u
m
b
er
o
f
an
o
m
alie
s
.
Fa
ls
e
a
lar
m
(
f
alse
p
o
s
iti
v
e
r
ate
(
FP
R
)
)
r
atio
b
et
w
ee
n
t
h
e
n
u
m
b
er
o
f
d
ata
r
ec
o
r
d
s
f
r
o
m
n
o
r
m
al
clas
s
th
at
ar
e
m
is
c
lass
if
ied
as
an
o
m
alie
s
an
d
th
e
to
t
al
n
u
m
b
er
o
f
d
ata
r
ec
o
r
d
s
f
r
o
m
n
o
r
m
al
clas
s
.
R
OC
C
u
r
v
e
i
s
a
tr
ad
eo
f
f
b
et
w
ee
n
d
etec
tio
n
r
ate
(
T
P
R
)
an
d
f
alse
alar
m
r
ate
(
FP
R
)
.
A
r
ea
u
n
d
er
th
e
R
OC
c
u
r
v
e
(
AUC)
is
ca
lc
u
lated
u
s
in
g
a
tr
ap
ez
o
id
r
u
le.
Evaluation Warning : The document was created with Spire.PDF for Python.
IJ
-
AI
I
SS
N:
2252
-
8938
A
n
o
ma
lies
Dete
ctio
n
B
a
s
ed
o
n
th
e
R
OC
A
n
a
lysi
s
u
s
in
g
C
la
s
s
ifie
r
s
in
Ta
ctica
l C
o
g
n
itive.
.
(
Ah
m
ed
Mo
u
m
e
n
a
)
111
Ma
in
id
ea
:
b
u
ild
a
cla
s
s
i
f
icat
i
o
n
m
o
d
el
f
o
r
n
o
r
m
al
(
an
d
a
n
o
m
alo
u
s
)
ev
e
n
t
s
b
ased
o
n
lab
el
ed
tr
ain
in
g
d
ata,
an
d
u
s
e
it
to
clas
s
i
f
y
e
ac
h
n
e
w
u
n
s
ee
n
e
v
e
n
t
C
la
s
s
i
f
icatio
n
m
o
d
els
m
u
s
t
b
e
ab
le
to
h
a
n
d
le
s
k
e
w
ed
(
i
m
b
alan
ce
d
)
clas
s
d
is
tr
ib
u
tio
n
s
.
U
s
e
m
o
d
if
ied
cla
s
s
i
f
icat
io
n
m
o
d
el
to
lear
n
th
e
n
o
r
m
a
l
b
eh
av
io
r
a
n
d
th
e
n
d
etec
t a
n
y
d
ev
ia
tio
n
s
f
r
o
m
n
o
r
m
al
b
eh
a
v
io
r
as a
n
o
m
alo
u
s
.
Fig
u
r
e
4
.
R
OC
c
u
r
v
es
f
o
r
d
if
f
er
en
t a
n
o
m
alies d
etec
tio
n
m
et
h
o
d
s
[
A
r
i
n
d
a
m
,
0
8
]
.
10.
P
RO
P
O
SI
T
I
O
N
S O
F
RO
C
ANALY
SI
S
Data
s
et
m
u
s
t
b
e
class
if
iab
le
i
n
th
e
f
ir
s
t
ca
teg
o
r
y
(
an
o
m
al
y
o
b
s
er
v
atio
n
)
an
d
in
th
e
s
ec
o
n
d
ca
teg
o
r
y
(
n
o
r
m
al
o
b
s
er
v
atio
n
)
.
W
h
a
t is
an
an
o
m
al
y
/o
u
tlier
o
r
ab
n
o
r
m
al
o
b
s
er
v
atio
n
an
d
w
h
at
is
a
n
o
r
m
al
o
b
s
er
v
a
tio
n
?
.
A
ll
c
lass
if
ier
s
h
a
v
i
n
g
th
r
e
s
h
o
l
d
v
alu
e
s
b
en
e
f
it
f
r
o
m
t
h
e
b
i
-
d
i
m
e
n
s
io
n
al
(
2
d
)
v
is
u
a
lizatio
n
o
f
th
e
R
O
C
cu
r
v
e.
E
v
alu
a
te
t
h
e
p
er
f
o
r
m
a
n
ce
o
f
a
class
i
f
icat
io
n
s
y
s
te
m
i
s
a
v
er
y
i
m
p
o
r
tan
t
is
s
u
e
b
ec
au
s
e
th
e
s
e
p
er
f
o
r
m
an
ce
s
ca
n
b
e
u
s
ed
f
o
r
lear
n
i
n
g
a
s
s
u
ch
o
r
to
o
p
ti
m
ize
th
e
v
al
u
es
o
f
t
h
e
h
y
p
er
-
p
ar
a
m
eter
s
o
f
t
h
e
cl
ass
i
f
ier
.
Fo
r
a
lo
n
g
ti
m
e,
t
h
e
cr
iter
io
n
u
s
ed
to
e
v
alu
ate
t
h
i
s
p
er
f
o
r
m
a
n
ce
w
a
s
th
e
co
r
r
ec
t
clas
s
i
f
icatio
n
r
ate,
th
at
is
to
s
a
y
t
h
e
n
u
m
b
er
o
f
ele
m
e
n
ts
i
n
a
te
s
t
d
atab
ase
co
r
r
ec
tly
cla
s
s
i
f
ied
.
T
h
e
p
r
o
b
lem
i
s
t
h
at
s
u
c
h
a
te
s
t
is
n
o
t
s
u
itab
le
f
o
r
ill
-
d
e
f
in
ed
e
n
v
ir
o
n
m
en
ts
.
I
n
m
an
y
s
it
u
atio
n
s
,
n
o
t
all
er
r
o
r
s
h
a
v
e
th
e
s
a
m
e
co
n
s
eq
u
e
n
ce
s
.
So
m
e
er
r
o
r
s
h
a
v
e
co
s
t
m
o
r
e
th
an
o
t
h
er
s
,
f
o
r
ex
a
m
p
le,
m
ed
ical
d
iag
n
o
s
tic
s
.
I
m
p
r
o
p
er
d
iag
n
o
s
i
s
o
r
tr
ea
t
m
en
t
ca
n
,
in
f
ac
t,
h
a
v
e
d
if
f
er
e
n
t
co
s
ts
o
r
d
an
g
er
s
ac
co
r
d
in
g
to
th
e
t
y
p
e
o
f
er
r
o
r
.
W
e
p
r
o
v
id
e
an
o
v
er
v
ie
w
o
f
th
e
ev
alu
a
tio
n
cr
iter
ia
o
f
class
i
f
icatio
n
s
y
s
te
m
s
in
t
w
o
class
es
as
d
is
c
u
s
s
ed
ab
o
v
e
a
n
d
m
o
r
e
g
e
n
er
all
y
m
u
lti
-
clas
s
s
y
s
te
m
s
t
h
at
w
e
w
i
l
l
d
is
cu
s
s
in
t
h
e
s
ec
t
io
n
s
.
1
1
.
G
E
N
E
RA
L
I
Z
AT
I
O
N
AND
D
E
C
I
SI
O
N
P
RO
B
L
E
M
S
O
F
T
H
E
RO
C
ANAL
YSI
S
T
O
M
UL
T
I
C
L
ASS P
RO
B
L
E
M
S
1
1
.
1
.
M
ulti
-
cla
s
s
RO
C
W
ith
m
o
r
e
th
a
n
t
w
o
class
e
s
th
e
s
it
u
atio
n
b
ec
o
m
e
s
v
er
y
co
m
p
le
x
if
t
h
e
g
lo
b
al
s
p
ac
e
is
to
b
e
m
an
a
g
ed
.
T
h
e
co
n
f
u
s
io
n
m
at
r
ix
w
it
h
class
e
s
b
ec
o
m
es
a
m
atr
i
x
w
it
h
a
d
i
m
en
s
io
n
(
)
T
h
e
co
r
r
ec
t
class
if
icatio
n
a
n
d
(
)
p
o
s
s
ib
le
er
r
o
r
s
.
Fo
r
ex
a
m
p
le
f
o
r
class
e
s
,
w
e
g
et
6
d
i
m
e
n
s
i
o
n
al
s
p
ac
es.
I
n
th
e
p
ap
er
[
Srin
v
a
s
an
,
9
9
]
h
as
d
escr
ib
ed
th
at
th
e
an
al
y
s
is
b
eh
i
n
d
th
e
R
O
C
C
H
ex
ten
d
s
to
m
u
ltip
l
e
class
es a
n
d
m
u
ltid
i
m
e
n
s
io
n
al
co
n
v
e
x
h
u
ll
s
.
I
n
[
P
r
o
v
o
s
t,
0
1
.
b
]
,
[
Fa
w
ce
tt,
0
6
]
an
d
[
L
an
d
g
r
eb
e,
0
6
]
p
r
o
p
o
s
es to
m
a
n
ip
u
late
class
es b
y
g
e
n
er
atin
g
R
OC
c
u
r
v
es,
o
n
e
f
o
r
ea
ch
clas
s
.
On
th
e
s
et
o
f
all
clas
s
es,
th
e
(
*
+
)
R
OC
cu
r
v
e
co
r
r
esp
o
n
d
s
to
th
e
e
v
alu
a
tio
n
o
f
p
er
f
o
r
m
a
n
ce
s
u
s
in
g
t
h
e
cla
s
s
as
p
o
s
it
iv
e
clas
s
a
n
d
all
o
th
er
cla
s
s
es
as
n
e
g
ati
v
e,
d
en
o
ted
:
⋃
(
2
)
W
ith
*
+
an
d
C
is
t
h
e
s
e
t o
f
all
cl
ass
es.
T
h
e
co
s
t
o
f
m
is
cla
s
s
i
f
icatio
n
is
,
f
o
r
th
i
s
ap
p
r
o
ac
h
,
f
ix
ed
f
o
r
ea
ch
class
b
ec
au
s
e
w
e
d
o
n
o
t
s
ee
k
to
d
if
f
er
e
n
tiate
th
e
er
r
o
r
s
.
Un
d
er
th
ese
co
n
d
itio
n
s
,
s
p
ac
e
p
er
f
o
r
m
an
ce
ev
a
lu
atio
n
i
s
d
im
en
s
io
n
s
,
w
h
ic
h
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8938
IJ
-
AI
Vo
l.
5
,
No
.
3
,
Sep
tem
b
er
2
0
1
6
: 1
0
5
-
116
112
a
m
o
u
n
ts
to
u
s
e
o
n
l
y
t
h
e
ele
m
en
ts
o
f
th
e
p
r
in
cip
al
d
ia
g
o
n
al
o
f
t
h
e
co
n
f
u
s
io
n
m
atr
i
x
.
Fo
r
ex
a
m
p
le,
f
o
r
th
r
ee
class
es
(
)
w
e
o
b
tain
a
t
h
r
ee
-
d
i
m
en
s
io
n
al
s
p
ac
e
ea
s
il
y
r
ep
r
esen
tab
le.
No
w
,
w
e
w
i
ll
p
o
s
itio
n
in
t
h
e
co
n
tex
t
o
f
co
m
p
ar
in
g
th
e
p
er
f
o
r
m
an
ce
o
f
clas
s
i
f
ier
s
.
W
e
n
ee
d
it
to
co
m
p
ar
e
t
w
o
h
y
p
er
p
lan
e
s
.
T
h
e
p
r
o
b
lem
is
t
h
at
ac
co
r
d
in
g
to
th
e
ar
ea
s
o
f
t
h
e
s
p
ac
e
p
er
f
o
r
m
an
ce
o
f
t
h
e
class
i
f
ier
s
m
a
y
v
ar
y
.
W
e
ca
n
h
av
e
o
n
o
n
e
ar
ea
a
h
y
p
er
p
lan
e
is
b
etter
t
h
an
o
th
er
a
n
d
i
n
o
th
er
ar
ea
t
h
e
s
ec
o
n
d
h
y
p
er
p
lan
e
w
h
ic
h
is
b
etter
t
h
an
t
h
e
f
ir
s
t.
T
h
is
is
w
h
y
in
th
e
liter
at
u
r
e
w
h
e
n
tr
y
i
n
g
t
o
co
m
p
ar
e
d
if
f
er
en
t
class
i
f
icatio
n
s
y
s
te
m
s
;
w
e
r
ed
u
ce
h
y
p
er
p
lan
e
s
in
to
s
ca
lar
v
alu
es.
I
n
th
e
g
en
er
al
ca
s
e,
th
e
s
ca
lar
v
alu
e
t
h
at
is
u
s
ed
to
ch
ar
ac
ter
ize
th
e
p
er
f
o
r
m
an
ce
o
f
R
O
C
m
u
l
ti
-
c
lass
i
s
Vo
lu
m
e
Un
d
er
th
e
R
OC
h
y
p
er
-
s
u
r
f
ac
e
(
VU
S).
1
1
.
2
.
M
ul
t
i
-
cla
s
s
AUC
T
h
e
A
r
ea
U
n
d
er
C
u
r
v
e
i
s
a
m
ea
s
u
r
e
o
f
th
e
d
i
s
cr
i
m
in
ab
ili
t
y
o
f
a
p
air
o
f
cla
s
s
e
s
.
I
n
a
t
w
o
-
clas
s
p
r
o
b
lem
,
t
h
e
A
U
C
is
a
s
i
m
p
le
s
ca
lar
v
a
lu
e,
b
u
t
a
m
u
lti
-
c
lass
p
r
o
b
le
m
i
n
tr
o
d
u
ce
s
th
e
i
s
s
u
e
o
f
co
m
b
in
i
n
g
m
u
ltip
le
p
air
w
is
e
d
is
cr
i
m
i
n
ab
ilit
y
v
alu
e
s
[
Dav
id
,
1
1
].
On
e
ap
p
r
o
ac
h
to
ca
lcu
la
tin
g
m
u
lti
-
clas
s
A
U
C
s
w
as
ta
k
e
n
b
y
[
P
r
o
v
o
s
t,
0
1
.
b
]
in
th
ei
r
w
o
r
k
o
n
p
r
o
b
a
b
ilit
y
esti
m
atio
n
tr
ee
s
.
T
h
ey
ca
lc
u
lated
A
U
C
s
f
o
r
m
u
lti
-
clas
s
p
r
o
b
lem
s
b
y
g
e
n
er
atin
g
e
v
er
y
clas
s
r
ef
er
en
ce
R
O
C
c
u
r
v
e
i
n
tu
r
n
,
m
ea
s
u
r
in
g
t
h
e
A
UC
,
a
n
d
th
e
n
s
u
m
m
i
n
g
th
e
AUC
s
w
eig
h
t
ed
b
y
th
e
r
e
f
er
en
ce
class
s
p
r
ev
ale
n
ce
i
n
th
e
d
ata
s
e
t.
Mo
r
e
p
r
ec
is
ely
,
th
e
y
d
ef
in
e
∑
(
)
(
)
(
3
)
W
h
er
e
(
)
is
th
e
ar
ea
u
n
d
er
t
h
e
class
r
e
f
er
en
ce
R
O
C
c
u
r
v
e
f
o
r
,
as
i
n
eq
u
a
tio
n
ab
o
v
e.
T
h
is
d
ef
in
i
tio
n
r
eq
u
ir
es
o
n
l
y
A
U
C
ca
lc
u
latio
n
s
,
s
o
i
ts
o
v
er
all
co
m
p
le
x
it
y
i
s
(
)
.
T
h
e
ad
v
an
ta
g
e
o
f
A
U
C
f
o
r
m
u
latio
n
is
th
at
is
g
e
n
er
ated
d
ir
ec
tly
f
r
o
m
clas
s
r
ef
er
en
ce
R
O
C
cu
r
v
e
s
,
an
d
th
es
e
cu
r
v
es
ca
n
b
e
g
en
er
ated
a
n
d
v
is
u
aliz
ed
ea
s
il
y
.
T
h
e
d
is
ad
v
an
tag
e
i
s
th
at
t
h
e
class
r
e
f
er
en
ce
R
OC
is
s
e
n
s
iti
v
e
to
clas
s
d
is
tr
ib
u
tio
n
s
an
d
er
r
o
r
co
s
ts
,
s
o
th
is
f
o
r
m
u
lat
io
n
o
f
is
as
w
ell.
T
h
e
p
ap
e
r
[
Dav
id
,
1
1
]
tak
es
a
d
if
f
er
e
n
t te
c
h
n
iq
u
e
i
n
t
h
eir
d
er
iv
atio
n
o
f
a
m
u
lti
-
clas
s
g
e
n
er
a
lizatio
n
o
f
th
e
A
r
ea
U
n
d
er
C
u
r
v
e.
T
h
e
y
d
esire
d
a
m
ea
s
u
r
e
th
at
i
s
in
s
e
n
s
iti
v
e
to
class
d
is
tr
ib
u
t
io
n
an
d
er
r
o
r
co
s
ts
.
T
h
e
d
er
iv
atio
n
is
to
o
d
etailed
to
s
u
m
m
ar
ize
h
er
e,
b
u
t
it
is
b
ased
u
p
o
n
t
h
e
f
ac
t
th
at
th
e
A
r
ea
Un
d
er
C
u
r
v
e
is
eq
u
i
v
ale
n
t
to
t
h
e
p
r
o
b
ab
ilit
y
t
h
at
t
h
e
clas
s
i
f
ier
w
il
l
r
an
k
a
r
a
n
d
o
m
l
y
c
h
o
s
en
p
o
s
itiv
e
i
n
s
tan
ce
h
i
g
h
er
th
a
n
a
r
an
d
o
m
l
y
ch
o
s
en
n
e
g
ati
v
e
i
n
s
ta
n
ce
.
Fro
m
t
h
is
p
r
o
b
a
b
ilis
tic
f
o
r
m
,
th
e
y
d
er
iv
e
a
f
o
r
m
u
latio
n
t
h
at
m
ea
s
u
r
es
th
e
u
n
w
ei
g
h
ted
p
air
w
is
e
d
is
cr
i
m
in
ab
il
it
y
o
f
class
es.
T
h
eir
m
ea
s
u
r
e,
w
h
ich
th
e
y
ca
l
l
,
is
eq
u
i
v
alen
t to
:
(
)
∑
(
)
(
)
(
4
)
W
h
er
e
n
is
th
e
n
u
m
b
er
o
f
class
es
a
n
d
(
)
is
th
e
ar
ea
u
n
d
e
r
th
e
t
w
o
-
clas
s
R
O
C
cu
r
v
e
in
v
o
l
v
i
n
g
c
lass
e
s
a
n
d
.
T
h
e
s
u
m
m
atio
n
i
s
co
m
p
u
ted
o
v
e
r
all
p
air
s
o
f
d
is
tin
c
t
clas
s
es,
ir
r
esp
ec
tiv
e
o
f
o
r
d
er
.
T
h
er
e
a
r
e
(
)
s
u
c
h
p
air
s
,
s
o
th
e
ti
m
e
co
m
p
lex
i
t
y
o
f
th
eir
m
ea
s
u
r
e
is
(
)
W
h
ile
Han
d
an
d
T
ills
f
o
r
m
u
latio
n
i
s
w
e
ll d
escr
ib
ed
an
d
i
s
i
n
s
e
n
s
iti
v
e
to
c
h
an
g
es
i
n
cla
s
s
d
is
tr
ib
u
tio
n
,
th
er
e
is
n
o
ea
s
y
w
a
y
to
v
is
u
alize
t
h
e
s
u
r
f
ac
e
w
h
o
s
e
ar
ea
is
b
ein
g
ca
lc
u
lated
.
12
.
CO
M
P
ARING
M
ANY
CL
AS
SI
F
I
E
R
S F
O
R
ANO
M
AL
I
E
S DE
T
E
CT
I
O
N
W
h
en
m
u
ltip
le
cla
s
s
i
f
ier
s
ar
e
u
s
ed
o
n
th
e
s
a
m
e
d
ataset,
w
e
ca
n
p
lo
t
t
h
eir
R
OC
o
n
t
h
e
s
a
m
e
f
i
g
u
r
e.
T
h
is
f
ac
ilit
ate
s
t
h
e
co
n
cl
u
s
io
n
s
ab
o
u
t d
o
m
in
a
n
ce
.
1
2
.
1
Do
m
ina
nce
o
f
RO
C
cur
v
e
I
n
F
ig
u
r
e
.
5
,
w
e
r
e
m
ar
k
th
at
c
u
r
v
e
A
d
o
m
i
n
ates
c
u
r
v
e
s
B
,
C
an
d
D
co
m
p
letel
y
;
m
ea
n
s
t
h
e
class
if
ie
r
A
o
u
tp
er
f
o
r
m
th
e
o
t
h
er
s
cla
s
s
if
ier
s
B
,
C
an
d
D.
cu
r
v
es
A
,
B
,
C
d
o
m
in
a
te
o
v
er
a
s
elec
t
r
eg
io
n
o
f
t
h
e
R
O
C
cu
r
v
e.
T
P
R
1
A
B
C
D
Evaluation Warning : The document was created with Spire.PDF for Python.
IJ
-
AI
I
SS
N:
2252
-
8938
A
n
o
ma
lies
Dete
ctio
n
B
a
s
ed
o
n
th
e
R
OC
A
n
a
lysi
s
u
s
in
g
C
la
s
s
ifie
r
s
in
Ta
ctica
l C
o
g
n
itive.
.
(
Ah
m
ed
Mo
u
m
e
n
a
)
113
Fig
u
r
e
5
.
Fo
u
r
R
OC
c
u
r
v
e
s
w
i
th
d
if
f
er
en
t v
al
u
es o
f
th
e
ar
ea
u
n
d
er
cu
r
v
e.
1
2
.
2
RO
C
c
o
nv
ex
hu
ll (
RO
CCH
)
T
h
e
R
OC
C
H
s
h
o
w
s
th
e
b
est
p
o
s
s
ib
le
p
er
f
o
r
m
a
n
ce
o
f
a
s
et
o
f
cla
s
s
i
f
ier
s
,
i
f
y
o
u
tak
e
t
h
e
m
ax
i
m
u
m
o
f
ac
cu
r
ac
y
o
f
e
v
er
y
clas
s
i
f
ier
a
n
d
i
n
ter
p
o
late
b
et
w
ee
n
d
i
f
f
er
en
t
cla
s
s
i
f
ier
s
w
h
en
e
v
er
n
ec
e
s
s
ar
y
to
co
r
r
ec
t
f
o
r
an
y
h
u
ll
s
.
I
n
th
e
co
r
r
ec
tio
n
,
t
o
j
o
in
t
w
o
o
r
m
o
r
e
cla
s
s
if
ier
s
b
y
s
tr
aig
h
t
li
n
e
is
an
in
ter
p
o
latio
n
.
T
h
e
p
o
in
ts
(
0
,
0
)
an
d
(
1
,
1
)
ca
n
also
b
e
u
s
ed
in
b
u
ild
in
g
th
e
R
OC
C
H.
I
f
th
er
e
ar
e
m
an
y
R
O
C
cu
r
v
es,
th
e
b
est
m
eth
o
d
to
co
m
p
ar
e
t
h
e
m
i
s
to
co
n
s
tr
u
ct
t
h
e
R
O
C
C
H
an
d
lo
o
k
w
h
ich
c
u
r
v
e
s
d
o
m
in
a
te
o
v
er
w
h
ic
h
r
e
g
io
n
s
o
f
t
h
e
f
i
g
u
r
e.
W
e
ca
n
u
s
e
t
h
e
R
OC
C
H
to
g
u
id
e
th
e
co
n
s
tr
u
c
tio
n
o
f
a
h
y
b
r
i
d
class
i
f
ier
w
h
ic
h
is
g
o
o
d
w
h
e
n
w
e
co
m
p
ar
e
w
it
h
a
s
i
m
p
le
o
r
in
d
iv
id
u
al
cla
s
s
i
f
ie
r
.
Fig
u
r
e
6
.
L
in
e
s
a
n
d
s
h
o
w
th
e
o
p
ti
m
al
cla
s
s
i
f
ier
u
n
d
er
d
if
f
er
en
t
s
ets o
f
co
n
d
itio
n
s
[
Fa
wce
tt,
0
7
]
.
I
f
y
o
u
ai
m
to
co
v
er
j
u
s
t
4
0
%
o
f
th
e
tr
u
e
p
o
s
it
iv
e
s
y
o
u
s
h
o
u
l
d
ch
o
o
s
e
m
eth
o
d
A
,
w
h
ic
h
g
i
v
es
a
f
a
ls
e
p
o
s
itiv
e
r
ate
o
f
5
%.
I
f
y
o
u
ai
m
to
co
v
er
8
0
%
o
f
th
e
tr
u
e
p
o
s
itiv
e
s
y
o
u
s
h
o
u
ld
ch
o
o
s
e
m
e
th
o
d
B
,
w
h
ic
h
g
i
v
es
a
f
al
s
e
p
o
s
itiv
e
r
ate
o
f
6
0
% a
s
co
m
p
a
r
ed
w
ith
A’
s
8
0
%.
I
f
y
o
u
ai
m
to
co
v
er
6
0
% o
f
th
e
tr
u
e
p
o
s
itiv
e
s
th
e
n
y
o
u
s
h
o
u
l
d
co
m
b
in
e
A
a
n
d
B
.
F
P
R
T
P
R
0
1
1
0
.
5
0
.
5
α
β
T
P
R
1
0
.
5
A
B
C
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8938
IJ
-
AI
Vo
l.
5
,
No
.
3
,
Sep
tem
b
er
2
0
1
6
: 1
0
5
-
116
114
Fig
u
r
e
7
.
T
h
e
R
OC
co
n
v
e
x
h
u
ll id
en
ti
f
ies p
o
ten
t
ial
l
y
o
p
ti
m
a
l c
lass
i
f
ier
s
[
Fa
w
ce
t
t,
0
7
]
.
1
2
.
3
I
s
o
-
perf
o
rm
a
nce
lin
e
s
B
y
s
ep
ar
ati
n
g
clas
s
i
f
icatio
n
p
er
f
o
r
m
an
ce
f
r
o
m
cla
s
s
an
d
co
s
t
d
is
tr
ib
u
tio
n
as
s
u
m
p
tio
n
s
,
t
h
e
d
ec
is
io
n
g
o
al
ca
n
b
e
p
r
o
j
ec
ted
o
n
to
R
OC
s
p
ac
e
f
o
r
a
n
ea
t
v
i
s
u
al
izatio
n
.
Fo
r
m
al
l
y
,
let
th
e
p
r
io
r
p
r
o
b
ab
ilit
y
o
f
a
p
o
s
iti
v
e
ex
a
m
p
le
b
e
(
)
s
o
t
h
e
p
r
io
r
p
r
o
b
ab
ilit
y
o
f
a
n
eg
at
iv
e
e
x
a
m
p
le
is
(
)
(
)
C
o
s
ts
o
f
f
a
ls
e
p
o
s
iti
v
e
an
d
f
alse
n
e
g
ati
v
e
er
r
o
r
s
ar
e
g
iv
en
b
y
(
)
an
d
(
)
r
esp
ec
tiv
el
y
.
T
h
e
ex
p
ec
ted
co
s
t
o
f
a
class
if
ica
tio
n
b
y
t
h
e
clas
s
i
f
ier
r
ep
r
esen
ted
b
y
a
p
o
in
t
(
)
in
R
OC
s
p
ac
e
is
:
(
)
(
)
(
)
(
)
(
)
(
)
(
6
)
T
h
er
ef
o
r
e,
tw
o
p
o
in
t
s
(
)
an
d
(
)
h
av
e
th
e
s
a
m
e
p
er
f
o
r
m
an
ce
i
f
:
(
)
(
)
(
)
(
)
(
7
)
T
h
is
eq
u
atio
n
d
e
f
i
n
es
t
h
e
s
lo
p
e
o
f
an
i
s
o
-
p
er
f
o
r
m
an
ce
lin
e,
i.g
.
,
all
clas
s
i
f
ier
s
co
r
r
esp
o
n
d
in
g
to
p
o
in
ts
o
n
t
h
e
li
n
e
h
a
v
e
t
h
e
s
a
m
e
e
x
p
ec
ted
co
s
t.
E
ac
h
s
et
o
f
cla
s
s
an
d
co
s
t
d
is
tr
ib
u
tio
n
s
d
ef
in
e
s
a
f
a
m
il
y
o
f
is
o
-
p
er
f
o
r
m
a
n
ce
lin
e
s
.
L
i
n
es
‘
’
m
o
r
e
n
o
r
th
w
est
’
’
h
av
in
g
a
lar
g
er
in
ter
ce
p
t
ar
e
b
ette
r
b
ec
au
s
e
th
e
y
co
r
r
esp
o
n
d
to
class
if
ier
s
w
it
h
l
o
w
er
ex
p
ec
ted
co
s
t
[
P
r
o
v
o
s
t,
9
7
]
.
13.
CO
M
B
I
NING
C
L
AS
SI
F
I
E
RS
Su
p
p
o
s
e
w
e
h
a
v
e
g
e
n
er
ated
tw
o
clas
s
i
f
ier
s
,
an
d
,
w
h
ic
h
s
c
o
r
e
clien
ts
b
y
t
h
e
p
r
o
b
ab
ilit
y
t
h
e
y
w
i
ll
b
u
y
t
h
e
p
o
lic
y
.
I
n
R
O
C
s
p
ac
e,
b
est p
o
in
t lies a
t
(
)
an
d
b
est p
o
in
t lies a
t
(
)
W
e
w
an
t to
m
ar
k
et
to
e
x
ac
tl
y
8
0
0
p
e
o
p
le
s
o
o
u
r
s
o
lu
tio
n
co
n
s
tr
ai
n
t is
:
I
f
w
e
u
s
e
,
w
e
e
x
p
ec
t:
C
an
d
id
ates
w
h
ic
h
is
to
o
f
e
w
.
I
f
w
e
u
s
e
w
e
e
x
p
ec
t:
C
an
d
id
ates
w
h
ic
h
is
to
o
m
a
n
y
.
W
e
w
an
t a
cla
s
s
i
f
ier
b
et
w
ee
n
an
d
.
T
h
e
s
o
lu
tio
n
co
n
s
tr
ain
t i
s
s
h
o
w
n
as a
d
as
h
ed
lin
e.
I
t in
ter
s
ec
ts
t
h
e
li
n
e
b
et
w
ee
n
an
d
at
ap
p
r
o
x
im
ate
l
y
(
)
A
clas
s
i
f
ier
at
p
o
in
t
w
o
u
ld
g
iv
e
th
e
p
er
f
o
r
m
a
n
ce
w
e
d
esir
e
an
d
w
e
ca
n
ac
h
iev
e
it
u
s
i
n
g
lin
ea
r
in
ter
p
o
latio
n
.
C
alcu
late
as th
e
p
r
o
p
o
r
tio
n
al
d
is
tan
ce
t
h
at
lies
o
n
th
e
li
n
e
b
et
w
ee
n
an
d
(
)
(
)
T
h
er
ef
o
r
e,
if
w
e
s
a
m
p
le
d
ec
i
s
io
n
s
at
a
r
ate
o
f
an
d
d
ec
is
io
n
s
at
a
r
ate
o
f
;
w
e
s
h
o
u
ld
attain
p
er
f
o
r
m
a
n
c
e
[
Fa
w
ce
tt,
0
7
].
Evaluation Warning : The document was created with Spire.PDF for Python.