Internati
o
nal
Journal of Ele
c
trical
and Computer
Engineering
(IJE
CE)
V
o
l.
5, N
o
. 4
,
A
ugu
st
2015
, pp
. 72
0
~
72
8
I
S
SN
: 208
8-8
7
0
8
7
20
Jo
urn
a
l
h
o
me
pa
ge
: h
ttp
://iaesjo
u
r
na
l.com/
o
n
lin
e/ind
e
x.ph
p
/
IJECE
Lip Image Feature Extraction Ut
ilizing Snake’s Control Points
for Lip Reading Applications
Farid
a
h,
Balz
a
Achm
ad,
Binar Lis
t
yan
a
S
Departement of
Engineering Ph
ysics, Gadjah Mada University
, Yog
y
ak
arta, Indon
esia
Article Info
A
B
STRAC
T
Article histo
r
y:
Received
Ja
n 17, 2015
Rev
i
sed
May
4, 201
5
Accepted
May 26, 2015
Snake is an active contour model that
c
a
tch
e
s
a
nd locks
im
age
edges
,
th
en
loca
liz
es
them
a
ccura
tel
y
.
Th
e s
i
m
p
les
t
S
n
ake
c
ons
is
ts
of a s
e
t
of control
points
that
ar
e c
onnect
ed b
y
s
t
ra
ight lin
es to for
m
a closed loop
. This paper
discusses the application of Snake to fi
nd the vi
s
u
al featur
e of li
p s
h
apes
. In
most previous p
a
pers, visual feature of lip shapes is represented
b
y
Snake’
s
contour. In
this
paper, th
e f
eatur
e of
lip sh
apes is represented b
y
six con
t
ro
l
points on lip
Snake’s contou
rs. B
y
sim
p
l
y
utili
zing six
co
ntrol points
representing one lip Snake’s contour, it
is expected to reduce the burden on
pattern recognition stage.
To demonstrate the p
e
rfo
rmance of
this method,
s
o
m
e
anal
ys
is
has
been
cond
ucted
on th
e effect of lip co
nditions an
d
illum
i
nat
i
on.
Th
e results shows that
th
e ov
eral
l
lip fe
atur
e extr
a
c
tion using
the proposed
method is better for lips that
have more con
t
rast to th
e
surrounding skin, optim
um
roo
m
illum
i
nation t
h
at gives the best result is in
the r
a
nge of 330
-340 lux.
Keyword:
Feature
Ext
r
action
Li
p Feat
ure
Lip Readi
ng
Li
p Se
gm
ent
a
t
i
on
Sn
ak
e’
s Coun
to
ur
Copyright ©
201
5 Institut
e
o
f
Ad
vanced
Engin
eer
ing and S
c
i
e
nce.
All rights re
se
rve
d
.
Co
rresp
ond
i
ng
Autho
r
:
Fari
da
h,
Depa
rt
em
ent
of E
ngi
neeri
n
g
Phy
s
i
c
s,
Gad
j
a
h
M
a
da
Uni
v
ersity
,
Jal
a
n Gra
f
i
k
a 2
Y
ogy
a
k
arta, Indonesia
Em
a
il: farid
a
h@ug
m
.
ac.id
1.
INTRODUCTION
Co
mm
u
n
i
catio
n
is v
e
ry
im
p
o
r
tan
t
in
ou
r life. W
ithou
t co
mm
u
n
i
catio
n
,
hu
m
a
n
b
e
in
g
s
will
no
t
d
e
v
e
l
o
p
as ad
van
ced
as at presen
t, and
a lo
t o
f
inform
atio
n
will n
o
t
co
nv
eyed
p
r
op
erly as well. As on
e
form
of c
o
m
m
uni
cati
on,
vi
sual
c
o
m
m
uni
cat
i
on b
ecom
e
s im
port
a
nt
w
h
en a
u
di
o com
m
uni
cat
ion i
s
not
possi
bl
e, f
o
r
exam
ple in an
envi
ronm
ent with a larg
e
noise,
or whe
n
the audio can not
be easily
hea
r
d, s
u
ch as
for dea
f
peo
p
l
e
. F
o
r s
u
ch cases, vi
s
u
al
co
m
m
uni
cat
i
on can
be per
f
o
r
m
e
d by
readi
n
g t
h
e spea
k
e
r'
s l
i
p
m
ovem
e
nt
s.
Each
syllab
l
e prono
un
ced
b
y
a
p
e
rson
will
fo
rm
a p
a
ttern
of
d
i
fferen
t
li
p
sh
ap
e [1
].
C
u
r
r
ent
de
vel
o
pm
ent
of di
gi
t
a
l
im
age proce
ssi
ng a
nd
pat
t
e
rn rec
o
g
n
i
t
i
on
t
echn
o
l
o
gi
es al
l
o
ws
us t
o
recogn
ize cert
a
in
obj
ects
u
tilizin
g
v
i
su
al
d
a
ta to
b
e
tran
slated
in
to co
rresp
ond
ing
info
rmatio
n
to
un
derstand
the
cha
r
acter of
t
h
e object, suc
h
as
in
au
t
o
m
a
t
i
c lip
-read
ing
system
[2
]-[6
]. Ex
tract
io
n
o
f
i
n
form
a
tio
n
o
r
v
i
su
al t
r
aits that ex
ist in lip
i
m
ag
es is an
i
m
p
o
r
tan
t
p
a
rt
wh
ich
still b
e
co
m
e
s a
m
a
j
o
r
fo
cu
s
of
research
an
d
d
e
v
e
l
o
p
m
en
t in
au
to
m
a
tic l
i
p
read
ing
sy
ste
m
. Th
e ex
tracted
v
i
su
al
in
fo
rm
atio
n
sh
ou
ld rep
r
esen
t lip
m
ovem
e
nt
pat
t
erns c
o
r
r
es
po
n
d
i
n
g t
o
t
h
e w
o
rds s
p
o
k
e
n
by
t
h
e spea
ker. T
h
e chal
l
e
n
g
e i
n
de
vel
o
pi
n
g
f
eat
ure
ex
traction
m
e
t
h
od
on
th
is syste
m
is th
e d
i
fficu
lty th
at arise
s
fro
m
ex
tern
al d
i
stu
r
b
a
n
ces, su
ch
as lig
h
ting
,
li
p
co
nd
itio
n, and
th
e way t
h
e speak
er prono
unce wo
rd
s.
Tw
o ap
p
r
oac
h
es we
re
used
i
n
e
x
t
r
act
i
n
g
vi
sual
feat
ure
o
f
l
i
p
m
ovem
e
nt pat
t
e
r
n
s,
nam
e
l
y
im
age-
base
d a
nd m
o
d
e
l
-
base
d a
p
p
r
o
aches
[7]
.
Im
age-
base
d a
p
p
r
oach typically uses im
age inform
ation s
u
ch a
s
size,
col
o
r and c
o
o
r
di
nat
e
s o
f
pi
xe
l
s
, and use
d
as
a feat
ure
vector. The adva
ntages of th
is
meth
od
are th
is meth
od
is relatively quick and it is easy to obtain fe
ature vect
or
s
of an im
age. Howeve
r, the
dimension of the feature
v
ector is u
s
u
a
l
l
y larg
e, an
d
co
n
s
equ
e
n
tly, it
b
eco
m
e
s a h
u
g
e
bu
rd
en
in
th
e reco
gn
itio
n p
r
o
cess.
As fo
r the
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 4
,
Au
gu
st 2
015
:
72
0
–
72
8
7
21
second a
p
proa
ch, m
odel-base
d m
e
thod is
a
m
e
thod t
h
at utilizes a
m
odel of
lip patterns i
n
pronouncing
words.
In th
is m
e
th
o
d
, a m
o
d
e
l is con
s
tru
c
ted of sev
e
ral m
o
d
e
l
p
a
ram
e
ters, u
s
u
a
lly in
th
e
fo
rm
o
f
p
a
ram
e
ter sp
ace.
The m
odel
nee
d
s t
o
be
abl
e
t
o
pr
ovi
de c
o
m
p
l
e
t
e
pi
ct
u
r
e
of
t
h
e act
ual
l
i
p
sha
p
es as
wel
l
as l
i
p
s
h
ape
ch
ange
s
whi
l
e
t
a
l
k
i
n
g
o
n
l
y
usi
n
g
as
fe
w
param
e
t
e
rs
as p
o
ssi
bl
e. S
o
m
e
exam
pl
e of
m
odel
-
base
d
m
e
t
hod
i
s
S
n
a
k
e a
n
d
Act
i
v
e C
ont
ou
r M
o
del
s
[8]
,
[
9
]
,
A
c
t
i
v
e S
h
a
p
e M
odel
s
[
10]
, [
1
1]
, an
d
De
f
o
rm
abl
e
M
odel
s
[
12]
,
[
13]
.
Sna
k
e i
s
a si
m
p
l
e
m
odel
-
bas
e
d m
e
t
hod t
o
obt
ai
n
t
h
e c
o
n
t
ou
rs o
f
a
n
ob
j
ect
. Thi
s
m
e
t
hod
was
fi
rst
devel
ope
d
by
Kaas et
al
[8]
whi
c
h ha
d bee
n
ap
pl
i
e
d t
o
e
x
t
r
act
i
o
n o
f
vi
sual
cha
r
act
eri
s
t
i
c
s of l
i
p
s [
8
]
,
[14]
.
Sna
k
e is an ac
tive contour m
odel t
h
at catches and lo
c
k
s i
m
age edges, t
h
en l
o
calizes them
accurately. The
si
m
p
lest Sn
ak
e co
n
s
ists of a set o
f
co
n
t
ro
l po
in
ts th
at
are connected
by straight lin
es to form
a
closed loop.
Th
is
p
a
p
e
r
will d
i
scu
ss t
h
e app
licatio
n
o
f
Snak
e to fi
n
d
v
i
su
al feat
u
r
e of l
i
p
sh
ap
es. Th
e
featu
r
e
o
f
lip sh
ap
es
is n
o
t
th
e same as lip
co
n
t
o
u
r
m
o
d
e
ls m
e
n
t
io
n
e
d
in
p
r
ev
i
o
u
s
p
a
p
e
rs,
b
u
t will b
e
rep
r
esen
ted
b
y
six
co
n
t
ro
l
poi
nts on lip cont
ours. T
h
is
pape
r will give an overview
to the rea
d
er,
a sim
p
le
m
e
thod that can be
use
d
to
o
b
t
ain
v
i
su
al featu
r
es of lip
sh
ap
es. By si
mp
ly u
tilizin
g
six
con
t
ro
l p
o
i
n
t
s rep
r
esen
ting
o
n
e
lip
con
t
our, it is
expecte
d
to re
duce t
h
e burde
n
on pa
t
t
e
r
n
re
cog
n
i
t
i
on st
a
g
e
.
To
dem
onst
r
a
t
e th
e p
e
rforman
ce of th
is meth
od,
so
m
e
an
alysis will b
e
co
ndu
cted
on
t
h
e effect o
f
illu
min
a
ti
o
n
and
lip cond
itio
n
s
.
2.
R
E
SEARC
H M
ETHOD
The propose
d m
e
thod is illustrated in Figure
1. Th
e m
e
thod consists of three steps, nam
e
ly
lip
segm
ent
a
t
i
on, cont
ou
r
e
x
t
r
act
i
on, and
feature extraction.
Fi
gu
re
1.
Sc
he
m
a
t
i
c
di
agram
of
t
h
e
pr
o
p
o
s
e
d
m
e
t
hod
2.
1. I
m
a
g
e
Se
gmen
ta
ti
o
n
Im
age
segm
ent
a
t
i
o
n
m
e
t
hod need
s
t
o
be per
f
o
r
m
e
d
be
fo
re
t
h
e ext
r
act
i
o
n pr
o
cess. Im
age
segm
ent
a
t
i
on aim
s
t
o
separat
e
t
h
e ob
ject
(l
i
p
, i
n
t
h
i
s
case) fr
om
t
h
e back
gr
o
u
n
d
(s
ki
n
)
. The m
e
t
hod us
ed i
n
t
h
i
s
pa
per
i
s
a
com
b
i
n
at
i
on
of
H
u
l
b
e
r
t
an
d
P
o
g
g
i
o
col
o
r
t
r
a
n
sf
orm
a
t
i
on a
n
d
Ot
su t
h
re
sh
ol
di
n
g
.
Hul
b
ert
an
d P
o
g
g
i
o
c
o
l
o
r t
r
ansf
o
r
m
a
ti
on i
s
based
on
di
f
f
ere
n
ces i
n
col
o
r c
o
m
posi
t
i
on bet
w
ee
n l
i
p
as t
h
e
object and s
k
in as the bac
k
ground.
Ski
n
col
o
rs are m
a
rked
m
o
re on c
o
lo
r
com
posi
t
i
on c
o
m
p
are t
o
b
r
i
g
ht
ness
,
even
on
di
ffe
re
nt
peo
p
l
e
. C
o
l
o
r com
posi
t
i
o
n
s
of ski
n
s are r
e
m
a
rkabl
y
co
n
s
t
a
nt
even
whe
n
ex
pos
ed by
a
l
o
t
of
illu
m
i
n
a
tio
n
.
An ex
am
p
l
e of h
i
stog
ram
s
d
e
p
i
ctin
g RGB co
lor co
m
p
o
s
itio
n
s
o
f
li
p
s
and
sk
in can b
e
seen
i
n
Fi
gu
re
2 [
1
5]
.
It
can
be see
n
t
h
at
t
h
e
di
ffe
re
nce
bet
w
ee
n re
d a
nd
g
r
een
f
o
r l
i
p
s i
s
great
e
r
t
h
an t
h
at
f
o
r
s
k
i
n
s
.
Hu
l
b
ert and
Pog
g
i
o
[16
]
d
e
fined
th
e v
a
l
u
e
o
f
th
e
p
s
eu
do
hue to
illu
strate t
h
is
d
i
fferen
c
e,
as fo
llo
ws.
,
,
,
,
(1
)
Ot
su t
h
resh
ol
d
i
ng [
17]
, com
m
onl
y
referred
t
o
as adapt
i
v
e
t
h
resh
ol
d
,
i
s
an aut
o
m
a
t
i
c
thres
h
ol
di
n
g
t
echni
q
u
e.
Ot
s
u
t
h
re
sh
ol
di
ng
i
s
neede
d
t
o
pe
rf
orm
im
age bi
nari
zat
i
o
n t
o
t
h
e im
age resul
t
e
d f
r
om
Hul
b
e
r
t
an
d
Po
ggi
o c
o
l
o
r t
r
ans
f
orm
a
t
i
ons. Ot
s
u
m
e
t
hod
cal
cul
a
t
e
s t
h
e
val
u
e
of t
h
res
hol
d
(
T
)
f
o
r
se
gm
ent
a
t
i
on ba
sed
o
n
the input im
age. The process
e
d
im
ag
e is u
s
u
a
lly in
g
r
ayscale fo
rm
at co
n
s
ists o
f
two
imp
o
rtan
t p
a
rts, na
m
e
ly
ob
ject
an
d
bac
k
g
r
ou
n
d
. Ot
s
u
t
h
res
hol
di
n
g
t
echni
que s
eek
s t
h
e o
p
t
i
m
al
t
h
reshol
d val
u
e t
o
separat
e
ob
ject
fr
om
b
ackgr
oun
d
by
m
a
x
i
m
i
z
i
n
g
th
e v
a
r
i
an
ce
b
e
tw
een
classes (
obj
ect and b
ackgr
oun
d)
w
h
ile m
i
n
i
miz
i
n
g
th
e
varia
n
ce
within classes
.
T
h
e
maxim
u
m
value of the
va
riance betwee
n
classes is
defi
ned
i
n
eq
uat
i
o
n (2
).
m
a
x
(2
)
Whe
r
e
and
are th
e
p
r
o
b
ab
ility o
f
p
i
x
e
ls in
each
class, wh
ile
and
are t
h
e
mean grayscale of eac
h
class. Th
e
p
r
o
b
ab
ilities an
d
m
ean
s
for
each
class are
up
d
a
ted
iterativ
ely.
Hu
lb
er
t and
Po
gg
io
co
lo
r
T
r
an
s
f
or
m
a
t
i
on
I
n
put
Li
p I
m
a
g
e
Ot
s
u
Th
re
sh
ol
din
g
IM
AGE
SEGM
E
N
T
A
T
I
ON
LIP
CO
NT
OU
R
EX
TRA
C
T
I
O
N
F
E
AT
UR
E
EXT
R
AC
T
I
ON
S
l
op
e
N
o
rm
al
i
z
at
io
n
Si
x
-
p
o
i
n
t
Fe
at
ur
e
Det
e
ct
i
o
n
Six
-
l
i
p
ke
y
poi
nt
in
fo
rm
at
io
n
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Lip
Ima
g
e Featu
r
e Extra
c
tio
n Utilizin
g
Sn
a
k
e’s
C
o
n
t
ro
l Poin
ts fo
r
Lip
Rea
d
i
n
g
Ap
p
lica
t
io
n
s
(
F
ari
d
ah)
72
2
Fi
gu
re
2.
C
o
m
p
ari
s
on
o
f
s
k
i
n
an
d l
i
p
R
G
B
h
i
st
ogram
s [1
5]
2.
2. L
i
p
C
o
n
t
o
u
r E
x
trac
ti
o
n
Sna
k
es,
fi
rst
d
e
vel
o
ped
by
Kass et
al
[8]
,
i
s
a m
e
t
hod t
h
at
u
s
es act
i
v
e co
nt
o
u
r
m
odel
s
t
o
det
ect
certain
feature
s
in a
n
im
age
.
T
h
e
features
are
flexi
b
le s
u
rface c
u
rves
that can adapt
dynam
i
ca
lly to t
h
e
bo
u
nda
ry
ed
ge
of an
o
b
ject
.
Thi
s
sy
st
em
consi
s
t
s
of a set
of
poi
nt
s t
h
at
are i
n
t
e
rc
on
nect
ed an
d co
nt
r
o
l
l
e
d by
sp
lin
es, as shown
in
Figure 3
.
Th
e
d
e
termin
atio
n
of th
e object in an image thr
ough
active contour is an
in
teractiv
e p
r
ocess. Th
e u
s
er
m
u
st
esti
mate
t
h
e in
itial c
o
n
t
ou
r wh
ich
is u
s
ually se
t n
early
si
m
ilar
to
th
e o
b
j
ect
featu
r
es. Fu
rtherm
o
r
e, th
e con
t
ou
r
will b
e
p
u
lled
to
ward
s
th
e features in
th
e i
m
ag
e d
u
e
t
o
th
e in
fl
u
e
n
c
e o
f
the
in
tern
al en
erg
y
th
at con
s
tru
c
t
th
e im
ag
e.
Fi
gu
re
3.
B
a
si
c f
o
rm
of
a
n
act
ive contour [18]
Activ
e co
n
t
ou
r is a set
o
f
con
t
ro
l
po
in
ts in a
co
n
t
o
u
r, wh
ich p
a
ram
e
ter are d
e
fi
n
e
d as
[9
],
,
(3
)
w
h
er
e
and
a
r
e co
o
r
di
nat
e
s
of
t
h
e
c
ont
rol
poi
nt
s i
n
t
h
e
c
ont
ou
r,
a
n
d
s
i
s
t
h
e
i
n
de
x
o
f
t
h
e c
ont
rol
poi
nt
s (Fi
g
u
r
e 3)
. The S
n
ake
s
param
e
t
e
r can be ex
p
r
esse
d
as a funct
i
o
n of e
n
er
gy
, w
h
i
c
h co
nsi
s
t
s
of
t
h
re
e
ki
n
d
of
ene
r
gi
es,
nam
e
ly
i
n
t
e
rnal
e
n
e
r
gy
(
), im
age energy (
), a
n
d c
onst
r
ai
nt
ene
r
gy
(
)
[9]
,
∗
(4
)
Int
e
r
n
al
e
n
er
gy
i
s
d
u
e t
o
t
h
e
el
ast
i
c
i
t
y
and t
h
e
be
ndi
ng
o
f
s
p
l
i
nes c
onst
r
uct
i
ng
t
h
e
Sna
k
e.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 4
,
Au
gu
st 2
015
:
72
0
–
72
8
7
23
(5
)
whe
r
e
α
is an
elasticit
y co
n
s
tan
t
and
β
i
s
a bendi
ng c
o
nst
a
n
t
of t
h
e co
nt
o
u
r
. T
h
e val
u
e
of
α
m
a
k
e
s th
e sp
lin
es
act as
m
e
m
b
ra
nes,
whe
r
eas
β
d
e
term
in
es th
e stiffn
ess
o
f
t
h
e sp
lin
es. Set
tin
g
zero
to
α
makes the sna
k
e does
not
ca
re
of
t
h
e
l
e
ngt
h
o
f
eac
h
spl
i
n
e,
w
h
i
l
e
s
e
t
t
i
ng zer
o t
o
β
pe
rm
its the spl
i
nes to
f
o
rm
straight c
o
r
n
er
s.
(6
)
whe
r
e
w
is t
h
e
weight
of ea
c
h
fe
ature
of t
h
e object im
ag
e. The s
n
ake
wi
ll stuck
on the
s
e features
which a
r
e
tip
ically th
e actu
a
l con
t
ou
r of
th
e
object. T
h
e
effect
of e
x
ternal ene
r
gy is c
ont
rolled by a
param
e
ter,
γ
.
2.
3.
Fea
t
ure E
x
tr
acti
on L
i
p
Ima
g
e
In th
is
p
a
per,
we tak
e
six po
i
n
ts fro
m
o
u
t
er
lip
-bo
r
d
e
r as the feature, as illu
strated in Figu
re 4. Th
ese
p
o
i
n
t
s are basically th
e leftmo
st,
righ
tm
o
s
t, u
p
m
o
s
t, an
d bo
tto
mm
o
s
t p
o
i
n
t
s
o
f
t
h
e lip.
Th
e
feature
p
o
in
ts are
t
a
ken
fr
om
t
h
e Sna
k
e
poi
nt
s
obt
ai
ne
d
fr
om
t
h
e c
ont
ou
r e
x
t
r
act
i
o
n st
ep
.
W
e
use
4
0
c
ont
rol
poi
nt
s f
o
r t
h
e
Sna
k
e, w
h
i
c
h g
i
ves pr
ope
r
t
h
e
si
x-
p
o
i
n
t
feat
u
r
e.
Fig
u
re
4
.
Six-po
in
t
featu
r
e represen
ting
li
p
pattern
In
th
e case
o
f
lip
th
at is n
o
t
u
p
righ
t, it is n
ecessary to
untilt
th
e i
m
ag
e.
Th
is tilt n
o
r
malizatio
n
is d
o
n
e
b
y
calculating t
h
e
slope
,
, bet
w
ee
n t
h
e
l
e
ft
m
o
st
(poi
nt
6
)
a
n
d t
h
e ri
g
h
t
m
ost
(p
o
i
nt
3
)
poi
nt
s,
ar
ct
an
(7
)
an
d th
en
ro
tatin
g th
e im
ag
e acco
rd
ing
t
o
th
e slop
e as
shown
in Figure
5
.
Fi
gu
re
5.
Sl
o
p
e
N
o
rm
al
i
zati
o
n
2.
4.
T
e
s
t
i
n
g a
nd An
al
ysi
s
The pe
rf
orm
a
nce of t
h
e p
r
o
p
o
se
d feat
u
r
e ext
r
act
i
o
n i
s
express
e
d by
t
h
e
devi
at
i
on
of t
h
e si
x-
p
o
i
n
t
feature
to t
h
e
actual res
p
ec
tive coordinat
e
s, which is
ca
lle
d
a
s
e
x
tr
ac
tio
n
er
ro
r.
Th
e ex
traction
erro
r is
(
a
)
1
2
3
4
5
6
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Lip
Ima
g
e Featu
r
e Extra
c
tio
n Utilizin
g
Sn
a
k
e’s
C
o
n
t
ro
l Poin
ts fo
r
Lip
Rea
d
i
n
g
Ap
p
lica
t
io
n
s
(
F
ari
d
ah)
72
4
cal
cul
a
t
e
d f
r
o
m
t
h
e vert
i
cal
devi
at
i
o
n o
f
p
o
i
n
t
1
,
2
,
4 a
n
d 5
,
as w
e
l
l
as ho
ri
zo
nt
al
de
vi
at
i
on o
f
poi
nt
5 an
d
6
.
The m
easurem
ent
m
e
t
hod i
s
sho
w
n i
n
Fi
gu
re 6. T
h
e e
x
t
r
a
c
t
i
on er
ro
r i
s
r
e
prese
n
t
e
d
by
t
h
e avera
g
e
de
vi
at
i
o
n
of
t
h
e si
x
poi
nt
s.
Fi
gu
re 6.
The
m
e
t
hod o
f
cal
c
u
l
a
t
i
ng feat
u
r
e ext
r
act
i
o
n
e
r
r
o
r
Th
e an
alysis
will fo
cu
s
on
t
h
e ex
traction
erro
r
un
der
v
a
ried
ligh
t
illu
m
i
n
a
tio
n and
lip
co
nd
itio
ns.
Th
e lip
co
nd
itio
n
s
in
clud
es
red
lip
s
(lip
s th
at
co
n
t
rast w
ith
t
h
e
co
l
o
r of
th
e sk
in)
and
p
a
le lip
s
(lip
s with
co
lor
closes to the c
o
lor of surroundi
ng
sk
i
n
).
An exam
ple of
each lip c
o
ndition
is
give
n in Figure
7. The
room
lig
h
tin
g
is
v
a
r
i
ed
as fo
llow
,
(a)
100
-1
10
lux, (
b
)
1
8
0
-
190
lu
x, (
c
)
230
-240
lux
,
(
d
)
330
-3
40
lux
,
and
(
e
)
380
-
39
0 l
u
x
.
(a)
(
b
)
Fig
u
re 7
.
Two
co
nd
itio
ns o
f
li
p
s
, red
(a)
and
p
a
le
(b
)
3.
R
E
SU
LTS AN
D ANA
LY
SIS
3.
1. Resul
t
s of
L
i
p
C
o
n
t
o
u
r E
x
tr
acti
on
Li
p co
nt
o
u
r e
x
t
r
act
i
on res
u
l
t
s
are sh
ow
n i
n
Fi
gu
re 8
.
Ext
r
a
c
t
i
on o
f
t
h
e c
o
nt
o
u
r i
s
d
o
n
e u
s
i
ng S
n
a
k
es
wi
t
h
40
co
nt
r
o
l
poi
nt
s. T
h
e s
u
ccess
o
f
fi
ndi
ng
t
h
e e
x
act
c
ont
ou
rs i
n
t
h
i
s
m
e
t
hod
de
pen
d
s
on
t
h
e
sel
ect
i
o
n
o
f
in
itial co
n
t
o
u
r an
d
co
n
t
ro
l
p
a
ram
e
ters, n
a
mely
α
,
β
and
γ
. In
th
is
pap
e
r, we
u
s
e ellip
ses as con
t
ou
r
in
itializat
io
n
,
wh
ich
is cl
o
s
e to
t
h
e sh
ap
e
o
f
lip
s. Meanwh
ile, th
e contro
l
p
a
ram
e
ters
α
,
β
and
γ
a
r
e va
ried
bet
w
ee
n
0.
3 a
n
d
1
,
t
o
fi
n
d
t
h
e
opt
i
m
al
param
e
t
e
r val
u
es. The
res
u
lts indicate that
contour ext
r
action can
be
d
o
n
e
prop
erly
b
y
th
e
Sn
ak
e
with
th
e sam
e
v
a
lu
es for all contro
l
p
a
ram
e
ters.
(a)
(b
)
(c)
Fi
gu
re 8.
C
o
nt
ou
r
e
x
t
r
act
i
o
n resul
t
by
va
ry
i
n
g
t
h
e val
u
e
s
o
f
α
,
β
, and
γ
, (a
) 0.
3;
0
.
3;
0.
3 p
r
o
p
erl
y
e
x
t
r
act
ed,
(b
) 0.
3;
0
.
7;
0.
3 not
p
r
o
p
erl
y
ext
r
act
ed
, (c) 0
.
3;
0.
3;
0
.
7
n
o
t
pr
o
p
erl
y
ext
r
ac
t
e
d
1
2
3
4
5
6
V
erti
cal
d
evi
ati
o
n
H
o
r
i
zonta
l
d
evi
at
i
o
n
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 4
,
Au
gu
st 2
015
:
72
0
–
72
8
7
25
3.
2. Resul
t
s of
L
i
p
Fea
t
ure
E
x
tr
acti
on
Vi
sual
feat
u
r
e ext
r
act
i
o
n i
s
done
wi
t
h
t
h
e i
n
f
o
rm
at
i
on obt
ai
ned f
r
om
t
h
e l
i
p
cont
o
u
r ex
t
r
act
i
on b
y
u
tilizin
g
Sn
ak
e co
n
t
ro
l
p
o
i
n
t
s. Visu
al feat
u
r
e ex
traction
st
arts with
slop
e no
rm
alizat
io
n
,
with
resu
lts as
sh
own
in
Figur
e
9
.
(a)
(b)
Fig
u
re
9
.
(a) Ti
lted
lip
,
(b)
up
tilted
o
r
slop
e
no
rm
alized
lip
The
n
u
m
b
er o
f
s
n
ake
poi
nt
s use
d
i
n
t
h
i
s
s
t
udy
was a
s
m
a
ny
as 4
0
p
o
i
n
t
s
.
O
u
t
of
t
h
e
40
S
n
a
k
e
cont
rol
p
o
i
n
t
s
, si
x
c
ont
r
o
l
poi
nt
s
are
sel
ect
ed
as lip
feature,
as
show
n
b
y
Fig
u
r
e
10
.
(a) un
tilted
o
r
i
g
in
al im
ag
e
(b)
ex
traction
result
Fi
gu
re
1
0
. E
x
a
m
pl
e of si
x
p
o
i
n
t
s
l
i
p
feat
u
r
e t
a
ken
f
r
om
Sna
k
e c
ont
rol
p
o
i
n
t
s
3.3. Tes
t
Resu
lts and
Discus
sion
Li
p feat
u
r
e e
x
t
r
act
i
on
has
b
een ap
pl
i
e
d t
o
red a
n
d pal
e
l
i
p
s u
nde
r di
ffe
rent
r
o
om
li
ght
i
n
g. T
h
e
exam
pl
es of s
u
ch co
n
d
i
t
i
ons c
a
n be
seen i
n
T
a
bl
e 1.
Tabl
e
1
sho
w
s t
h
at
t
h
e
pr
o
pose
d
feat
ure e
x
t
r
act
i
o
n
wo
rk
s
b
e
tter for red
lip
d
e
sp
ite th
e i
llu
m
i
n
a
tio
n
o
f
th
e roo
m
. Seg
m
en
tatio
n
p
r
o
c
ess is ab
le to
p
r
op
erly d
i
fferen
tiat
e
th
e lip
fro
m
th
e sk
in
,
h
e
n
ce en
ab
le th
e Sn
akes to
d
e
tect th
e ed
g
e
o
f
th
e lip an
d
p
r
o
v
i
d
e
co
n
t
ro
l po
in
ts t
h
at are
g
ood
rep
r
esen
t
a
tio
n
of th
e lip featu
r
e. Fo
r
p
a
le lip
s,
th
e segmen
tatio
n
pro
c
ess still can
n
o
t p
r
ov
ide go
od
resu
lt
for so
m
e
roo
m
illu
m
i
n
a
tio
n
.
Howev
e
r, th
e
Sn
ak
e is s
till a
b
le to
fi
n
d
fairly g
o
o
d
resu
lt
in
18
0-1
9
0
and
330
-
3
4
0
l
u
x roo
m
i
llu
m
i
n
a
tio
n
,
alt
h
oug
h th
e
ob
tain
ed
six
-
po
in
t
featu
r
es are
no
t
o
p
tim
al o
n
e
s.
Evaluation Warning : The document was created with Spire.PDF for Python.
IJECE
ISS
N
:
2088-8708
Lip
Ima
g
e Featu
r
e Extra
c
tio
n Utilizin
g
Sn
a
k
e’s
C
o
n
t
ro
l Poin
ts fo
r
Lip
Rea
d
i
n
g
Ap
p
lica
t
io
n
s
(
F
ari
d
ah)
72
6
Tab
e
l
1
.
Examp
l
e of Li
p
Featu
r
e Ex
t
r
actio
n
u
n
d
e
r
Different Lip
an
d Room
Lig
h
tin
g
C
o
n
d
ition
s
Lip
co
nditio
n
Roo
m
illumina
ti
o
n
(lux)
Red
Pale
1
00-
110
1
80-
190
2
30-
240
3
30-
340
3
80-
390
Qu
an
titativ
ely, th
e p
e
rfo
r
m
a
nce o
f
lip
feature ex
trac
tio
n
rep
r
esen
ted
b
y
ex
traction
erro
r
can
b
e
seen
in Table
2.
As
expecte
d
, t
h
e
extraction errors of re
d
lip
s
are lower t
h
an th
o
s
e
o
f
p
a
le
lip
s, am
o
u
n
ting
5.4
p
i
x
e
ls co
m
p
are to
2
5
p
i
x
e
ls for
p
a
le lip
s. R
oom
li
ghting also has
less effect
on the ca
se
of
red lips
.
Tabel
2.
Ext
r
a
c
t
i
on E
r
r
o
r
U
n
der
Di
f
f
ere
n
t
R
oom
Li
ght
i
n
g
C
o
n
d
i
t
i
ons
(
I
n
Pi
xel
U
n
i
t
)
Roo
m
illu
m
i
nation (lux)
Lip co
ndition
Extraction error
due to
roo
m
illu
m
ina
tion
Red Pale
100-
11
0
7.
0
34
36.
3 ± 30.
6
180-
19
0
5.
9
21
24.
3 ± 20.
3
230-
24
0
5.
7
40
28.
2 ± 19.
5
330-
34
0
3.
2
8.
1
20.
4 ± 25.
7
380-
39
0
5.
3
22
21.
8 ± 16.
4
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE Vo
l. 5
,
N
o
. 4
,
Au
gu
st 2
015
:
72
0
–
72
8
7
27
4.
CO
NCL
USI
O
N
Ove
r
al
l
l
i
p
feat
ure e
x
t
r
act
i
o
n
usi
n
g t
h
e p
r
op
ose
d
m
e
t
hod i
s
bet
t
e
r
fo
r l
i
p
s
t
h
at
ha
ve m
o
re
co
nt
rast
t
o
t
h
e
sur
r
o
u
ndi
n
g
sk
i
n
, wi
t
h
ext
r
ac
t
i
on err
o
r
of 5
.
4 pi
xel
s
,
co
mp
are to
25
p
i
xels fo
r p
a
le lip
s. Op
tim
u
m
ro
o
m
illu
m
i
n
a
tio
n
that g
i
v
e
s t
h
e best resu
lt is in th
e rang
e
of
3
30-340
lux
with
ex
traction
erro
r
o
f
20
.4
p
i
x
e
ls.
M
a
nual
qua
nt
i
f
i
cat
i
on m
e
t
h
o
d
f
o
r
g
r
o
u
n
d
t
r
ut
h has
u
n
ce
r
t
ai
nt
ues o
f
0.
6
i
n
h
o
ri
z
o
nt
al
di
rect
i
o
n an
d
2.
4 i
n
vertical di
rection.
ACKNOWLE
DGE
M
ENTS
Thi
s
pa
per a i
s
part
o
f
resea
r
c
h
g
r
ant
on Li
ps
M
o
tio
n
Pattern
Sim
i
larity A
n
alysis o
n
Portab
le Dev
i
ce
fo
r
Deaf
Ki
ds
Speec
h T
h
era
p
y
i
n
I
n
do
nesi
a
n
La
n
gua
ge
fu
nde
d
by
Uni
v
e
r
si
t
a
s Ga
d
j
ah
M
a
da (
2
01
3
)
.
REFERE
NC
ES
[1]
A.
Balza,
et
.
a
l
.
, “Lip Motion Recognition for In
donesian S
y
llab
i
e Pronounciat
i
o
n
Utilizing Hidd
en Markov Model
Method”,
Te
lk
omnik
a
, 13(1
)
, pp
. 173-180
, 2015
.
[2]
E.D. P
e
t
a
j
a
n, “
A
utom
atic L
i
pre
a
ding to Enh
a
nc
e S
p
eech R
ecog
n
ition”
,
Proc
. of
IEEE Conf
eren
ce on Compute
r
Vision and
Pat
t
e
r
n Recogn
ition
,
pp. 40–47
, 1985
.
[3]
U.
Meier,
et
.al
.
,
“
A
daptive Bim
odal S
e
ns
or F
u
s
i
on for Automati
c S
p
eechr
ead
i
ng”,
Proc. of the Internationa
l
Conference on
Acoustics, Speech
, and Signa
l
Processing ICASSP’9
6
, 1996
.
[4]
K.
Ma
se
,
et
.
a
l
.,
“
A
utom
atic Lip
r
eading
b
y
Opt
i
c
a
l Flow Ana
l
y
s
is”,
S
y
stems and
C
o
mputer
, 22(6)
,
pp. 67-76
, 1991
.
[5]
C. Breg
ler
,
et.a
l
., “Improving C
onnected Letter
Recognition b
y
Lipreading”,
Pr
oceed
ings
of
IE
EE In
ter
nationa
l
Conference on
Acoustics,
Speech
and Signal Processing
, pp
. 557–
560, 1993
.
[6]
R.
Musta
f
a,
et
.
a
l
.
, “An Efficient Lip-reading M
e
thod
Using K-Nearest Neighbor
Algorithm”,
Telkomnika
, 13(1)
,
pp. 180-186
, 20
15.
[7]
A.
W.
C.
Liew
,
et
.al
.
,
“
L
ip Conto
u
r Extrac
tion fro
m
Color
Im
ages
us
ing a Deform
able M
odel”
,
Pattern Recognition
,
35, pp
. 2949-29
62, 2002
.
[8]
M.
Ka
ss,
et
.
a
l
.
, “
S
nakes: Active
Contour Model”,
Int.
Journal Co
mputer Vision
, 4
,
pp
. 321–331
, 1
988.
[9]
R. Kaucic,
et
.al
.,
“
A
ccurate, Re
al
-tim
e,
Unadorned Lip Track
ing”,
Proceedings of the 6th Internati
onal Conferen
c
e
on Computer Vision
, pp
. 370
–37
5, 1998
.
[10]
J. Lue
ttin
,
et.a
l
.
, “Visual Speech Recognition
Using Activ
e
Shape Models and Hidd
en
Markov Models
”,
Pr
oceed
ings
of
I
EEE
Inter
nat
ion
a
l Conf
er
ence
o
n
Acous
t
i
cs
,
Spe
ech and
Signa
l P
r
oces
s
i
ng
, pp
. 8
17–820, 1996
.
[11]
T.F. Cootes,
et
.
a
l.
, “
U
s
e
of Ac
tive S
h
ape M
o
d
e
ls
for Loc
a
ting
S
t
ructures
in
M
e
dica
l Im
ages
”,
Image Vision
Comput.,
12
(6),
pp. 355–365
, 19
94.
[12]
A.
W.
C.
Lie
w
,
et
.al.
, “Region-
based Approach
to
Robust Lip Co
ntour Ex
traction
”
,
I
E
E
El
ec
tron.
Lett
.
, 36(15), p
p
.
1272–1274, 200
0.
[13]
A.
L.
Yuille,
et
.
a
l
.
, “
F
ea
ture
Extr
act
ion from
Fac
e
s Using Deform
able Tem
p
l
a
te
s”,
Int. Journal
Computer Visio
n
,
8(2), pp
. 99–111
, 1992
.
[14]
P.
De
lma
s
,
et
.
a
l.
, “Automatic Snakes for Robus
t Lip
Boundaries Extraction
”
,
I
EEE In
ternat
ion
a
l Conferen
ce
o
n
Acoustic, Speech
,
and
Signal Pro
cessing (
I
CCASP’99)
, 1999.
[15]
N. Eveno,
et
.
a
l.
, “A New Color
Transformation
For Lips Seg
m
entation”,
IEE
E
Workshop on M
u
ltimedia Sign
a
l
Processing (
MMSP’01)
, 2001.
[16]
A.
Hulbert,
et
.al
.
,
“
S
y
n
th
esizing
a
Colour Algor
ith
from
Exam
ples
”,
S
c
ience
, 239
,
pp. 482-485
, 19
98.
[17]
N. Otsu, “A Threshold Selection
Method from Gray
-Lev
el Histo
g
ram”,
IEEE Transaction On
Systems, Man, and
Cybernet
ics.
Vol
.
SMC-9
,
Pon
tif
i
c
ia Uni
versidan
ce Ca
toli
ca Do
Rio De
Jan
e
iro
, 1979.
[18]
D. Cremers, et.al., “Diffusion Snakes: Introducing St
atistical Shape Knowledge into th
e Mumford-Sha
h
Functional”,
Int
Journal of Computer Vision
, 50(3
)
, pp
. 295-313
, 2
002.
BIOGRAP
HI
ES OF
AUTH
ORS
Farid
a
h
, is senior lectur
er at th
e Department of
Engineer
ing Phy
s
ics
,
Faculty
of
Engineer
ing,
Universitas Gadjah Mad
a
, Yog
y
ak
arta, Indon
esia. She r
eceived her
Bach
elor degr
ee
in
Engineering Phy
s
ics from
Institut Tekno
logi
Sepuluh Nopem
b
er, Surabay
a
, I
ndonesia,
and
Master degree in Microelectron
i
cs from Nany
ang Technolog
ical University
, S
i
ngapore. Her
research and teaching inter
e
sts are in Instrume
ntations and Visual Se
nsors. She
has published
research
pap
e
rs in various
journals and Conferences.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Lip
Ima
g
e Featu
r
e Extra
c
tio
n Utilizin
g
Sn
a
k
e’s
C
o
n
t
ro
l Poin
ts fo
r
Lip
Rea
d
i
n
g
Ap
p
lica
t
io
n
s
(
F
ari
d
ah)
72
8
Balz
a Achma
d
, is sen
i
or lecturer at th
e Department
of
En
gineer
ing Ph
y
s
ics, Facu
lty
o
f
Engineering, U
n
iversitas Gadjah
Mada, Yog
y
akarta, Indonesia. His resear
ch interests ar
e in
Instrumentations
, Robotics and
Visu
al S
e
ns
ors
.
He is
a m
e
m
b
er at the C
e
nter fo
r Robotics
and
Autom
a
tion (Ce
n
tRA) as wel
l
as the
Int
e
grat
e
d
and Sm
art
Green Bu
ilding
(
I
NSGREEB),
Universitas Gad
j
ah Mada. H
e
has
published
resear
ch
pap
e
rs in
var
i
ous journals
and
Conferences.
Binar List
y
a
na
S
, is an
engin
eer
at Technolog
y
Centre,
PT Dirg
antar
a
Indon
esia. She r
e
ceived
her Ba
chelor
d
e
gree
in
Engin
eering
P
h
y
s
i
c
s
from
Univers
ita
s
Gadjah M
a
d
a
, Yog
y
akar
ta
,
Indonesia.
His
research interests are in
Instrumen
t
ations
and Visu
al Sensors.
Evaluation Warning : The document was created with Spire.PDF for Python.