Internati
o
nal
Journal of Ele
c
trical
and Computer
Engineering
(IJE
CE)
V
o
l.
6, N
o
. 5
,
O
c
tob
e
r
201
6, p
p
. 2
478
~248
7
I
S
SN
: 208
8-8
7
0
8
,
D
O
I
:
10.115
91
/ij
ece.v6
i
5.1
213
5
2
478
Jo
urn
a
l
h
o
me
pa
ge
: h
ttp
://iaesjo
u
r
na
l.com/
o
n
lin
e/ind
e
x.ph
p
/
IJECE
Emotion
a
l F
a
ci
al
Expres
s
i
on Based on Act
i
on Unit
s and
Facial Muscle
Ahm
a
d
Hoir
u
l
Basori,
H
a
ni
Moai
teq
Ab
d
u
llah
AlJahd
ali
Faculty
of Computing
and Inform
ation
Technolog
y
R
a
bigh
, King
Abdulaziz Univ
ersity
, Jedd
ah,
Kingdom of Saudi Arabia
Article Info
A
B
STRAC
T
Article histo
r
y:
Received
J
u
n 20, 2016
Rev
i
sed
Au
g
18
, 20
16
Accepted
Sep 28, 2016
The virtu
a
l human play
vital roles in vi
rtual reality
and game. The process of
Enrich
ing the vir
t
ual human thro
ugh their
expr
ession is one of the aspect th
at
m
o
s
t
res
earch
er
s
t
udied
and
im
proved.
This
s
t
u
d
y
aim
s
to d
e
m
ons
trate
th
e
combination of
facial action units (F
ACS
)
and
faci
al m
u
s
c
le
to
produce
a
real
istic f
aci
al e
xpression. The
result of exper
i
m
e
nt succeed o
n
producing
particular
expres
sion such as ang
e
r, ha
pp
y
,
sad w
h
ich ar
e able to
convey
th
e
emotional state of the virtual hu
man.
This achie
vem
e
nt is believ
e
d to bring
full m
e
nt
al
im
mersion towards v
i
rtual
hum
an
and
audi
ence
.
The
f
u
ture works
will abl
e
to ge
nerat
e
a com
p
le
x virtual hum
an
expression that
com
b
ines
ph
y
s
ic
al
fa
ctos
s
u
ch as
wr
inkle
,
f
l
uid d
y
n
a
mics fo
r tears or sweating.
Keyword:
Em
o
tio
n
a
l ex
pr
ession
Facial anim
ation
Facial m
u
scle
Vi
rt
ual
h
u
m
a
n
Copyright ©
201
6 Institut
e
o
f
Ad
vanced
Engin
eer
ing and S
c
i
e
nce.
All rights re
se
rve
d
.
Co
rresp
ond
i
ng
Autho
r
:
Ahm
a
d H
o
i
r
ul
B
a
sori
Facul
t
y
o
f
C
o
m
put
i
ng an
d
I
n
fo
rm
ati
on Tec
h
n
o
l
o
gy
R
a
bi
g
h
,
King
Abdu
laziz Un
iv
ersity,
Jedd
ah,
K
i
ng
do
m
o
f
Saud
i Ar
ab
ia
Em
a
il: u
c
h
i
h
a
.h
o
i
ru
l@g
m
ail.
co
m
1.
INTRODUCTION
Gam
e
and virtual reality application built with
som
e
core c
o
m
pone
nts such as: building, the
characte
r
(human
or anim
al),
envi
ronm
ent, grass, trees etc
.
All these
el
ements a
r
e a
ffect
the realism
of
virtual
reality g
a
m
e
it
self. So
m
e
research
er st
u
d
i
ed
th
e
h
u
m
an
ch
aracter t
o
b
e
realistic as th
e real world
b
y
enri
ch
ing
w
ith
f
acial expr
ession
,
vo
ice/sp
eech
,
an
d
h
a
p
tic tech
no
logy [
1
]-
[3
].
W
h
il
e th
er
e is r
e
sear
ch
er
th
at fo
cu
s on
con
v
e
r
sat
i
onal
di
al
og
ue t
o
gi
ve cha
r
act
er m
o
re l
i
v
e a
n
d
at
tractive. It ca
n help
hum
a
n int
e
ract
o
r
m
a
ke som
e
di
al
og
ue t
h
e
n
got
res
p
o
n
se
f
r
o
m
t
h
e charact
er t
h
ro
u
g
h
t
h
ei
r A
I
be
havi
or
[
3
]
.
Fabri et.al. (1
99
9) ex
p
l
ain th
at v
i
rtu
a
l reality
th
at
h
a
s
co
llab
o
rativ
e
features facilitae its u
s
er to
do
comm
unication usi
ng thei
r facil expressi
on,eye gaze,
body and hand gesture. Th
ere two m
a
in obstacle on
pr
o
duci
ng t
h
e
real
i
s
t
i
c
vi
rt
ual
hum
an:
fi
rst
,
ho
w t
o
b
u
i
l
d
e
m
oti
on
dat
a
ba
se, t
h
e
n
sec
o
n
d
o
b
st
acl
e i
s
h
o
w t
o
apply the em
otion t
h
at store
d
in data
base to the cha
r
act
er
(hum
an or a
n
i
m
al) in orde
r to ac
hieve thei
r goal
.
The ot
her
rese
arche
r
argue
d
that hum
an character not on
ly d
e
termin
ed
b
y
th
e ph
ys
ical appeara
n
ce
, it also
n
eed so
m
e
b
e
li
ev
ab
ility p
e
rspectiv
e [4
],[5
].
Th
erefo
r
e, so
me au
thor
p
r
o
duced
an
ind
i
v
i
du
al ch
aracter th
at h
a
s
beha
vi
o
r
, em
ot
i
on as
pect
an
d ev
en
ge
nde
r
.
It
ha
s bee
n
cont
i
n
ue
d
by
ot
he
r resea
r
c
h
er by
c
o
n
s
i
d
e
r
i
ng t
h
e
en
v
i
ron
m
en
t el
e
m
en
ts: sh
adow, lig
h
t
co
m
p
o
s
itio
n
an
d
filter to
exp
r
ess th
e em
o
tio
n
con
t
en
t [6
]. Zag
a
lo
an
d
Torres (2008), has dem
onstra
t
e a char
act
er t
h
at
ex
pres
s t
h
e
i
r em
ot
i
on by
dem
onst
r
at
i
n
g t
h
e t
ouc
hi
n
g
e
f
fect
s
a
m
o
n
g
two
char
acter
s, th
is t
o
u
c
h
i
ng r
e
f
e
r
to
an
ex
pr
essi
o
n
lik
e
hu
g,
han
d
sh
ak
ing
etc. A
l
t
h
oug
h touch
i
ng
activ
ities o
ccurred
o
n
v
i
rt
u
a
l env
i
ron
m
en
t, it can
affect
the em
o
tio
n
a
l con
d
ition
o
f
u
s
er, th
erefo
r
e it can
b
e
use as
m
e
di
um
t
o
st
rengt
h
e
n
t
h
e em
ot
i
onal
exp
r
essi
on o
f
vi
rt
ual
cha
r
act
ers. Thi
s
st
u
d
y
pro
p
o
se
d a di
ffe
rent
app
r
oach
on f
aci
al
anim
ati
on usi
n
g m
u
scl
e
based an
d
actio
n
un
its, wh
ere th
e m
u
scle ele
m
en
t is
u
s
ed
to
st
ren
g
t
h
e
n
t
h
e
em
ot
i
onal
se
ns
at
i
on
of
h
u
m
a
n cha
r
act
ers.
Evaluation Warning : The document was created with Spire.PDF for Python.
IJECE
ISS
N
:
2088-8708
Emotional F
a
c
i
al Expre
ssion
Base
d
on Action
Un
its
and Facial Muscle (A
hmad
Hoir
ul B
a
sori)
2
479
2.
RELATED WORKS
The resea
r
ch
o
f
vi
rt
ual
h
u
m
a
n has re
veal
n
u
m
e
rous m
e
t
h
od t
o
e
x
p
r
ess
em
ot
i
on t
h
r
u
g
h
t
h
e avat
ar
.
Tabl
e
1
prese
n
t
st
at
e of t
h
e a
r
t
of
di
verse
m
e
th
od
to conv
ey th
e em
o
tio
n
a
l
ex
pressi
o
n
.
Tabl
e 1. O
v
er
v
i
ew
o
f
vi
rt
ual
h
u
m
a
n
st
udy
Avatar or vi
rtual w
o
rld f
eatures
Author
An avatar that
i
m
it
ate hu
m
a
n behavior according to its
r
o
le and em
otional context.
I
t
also pr
ovide avatar with
rich individuality
that can
m
a
ke
autorespon to its
sur
r
oundin
g
[5]
,
[7]
,
[8]
Am
avatar
with r
i
ch em
otional featur
es: body
contact
such as huggi
ng an
d handshake
[2]
T
h
e em
otional exp
r
ession of vir
t
ual h
u
m
a
n thr
ough faci
al
expression that sup
ported by lighting,
filter and shadow
[6]
Vir
t
ual hu
m
a
n with visual and
soun
d
effect to convey
the
em
otion
[
9
]-[
1
0
]
T
h
e other
author
o
ffer
a r
i
ch inter
a
ction with visual and
haptic featur
es
[
1
1
]-[
12
]
Pr
oduce haptic pajam
a
that has
visual and haptic featur
es
to enrich the intera
ction
[13]
Co
m
b
ine visual,
acoustic and haptic to im
p
r
ove r
ealis
m
of vir
t
ual hu
m
a
n expr
ession
[14]
,
[
1
5
]
Genrae extr
e
m
e
ex
pression
by
co
m
b
ining T
ear
s and
sweat with e
m
otio
nal f
acial expression
[16]
,
[
1
7
]
Calculate the oxy
g
e
nation absor
p
tion
of facial hu
m
a
n
tissue to render re
a
listic facial skin
[18]
I
n
tegr
ate the hu
m
a
n pose walking with facial expr
ession
to enrich the avata
r
e
m
otional expres
sion
[19]
2.
1.
Pseud
omuscle
-
based Tec
hni
que for
Faci
al Anim
ati
o
n
Faci
al
anim
at
ion t
ech
ni
q
u
es
con
s
i
s
t
of som
e
di
ffe
rent
ap
pr
oac
h
es suc
h
as key
fram
i
ng
base
d o
n
sha
p
e i
n
t
e
r
pol
at
i
on,
per
f
o
r
m
a
nce-
dri
v
en a
n
i
m
at
i
on, pa
ra
m
e
t
e
ri
zed
m
odel
s
, pse
u
dom
uscl
e-
base
d,
m
u
scl
e
-
base
d
a
n
d
l
i
n
g
u
i
s
t
i
c
focuse
d
sim
u
l
a
t
i
on [2
0
]
. B
a
sed on t
h
e p
o
p
u
l
arity in
terpo
l
atio
n, it s
e
e
m
s th
at k
e
y f
r
am
in
g
tech
n
i
qu
e attracts th
e an
i
m
ato
r
to
p
r
o
d
u
ce con
v
e
n
tion
a
l an
imatio
n
.
W
h
ile in
p
a
ram
e
teriz
e
d
m
o
d
e
l, th
e m
o
d
e
l
is created from
a group of
param
e
te
rs. It
al
so i
n
v
o
l
v
es
geom
et
ri
c t
r
an
sfo
r
m
a
t
i
on, re
gi
o
n
i
n
t
e
rp
ol
at
i
on
a
n
d
mapping a
p
proach to ge
ne
rate face
plana
r
.
Pse
u
dom
u
scle-base
d
tec
hni
que
has a
m
u
scle ele
m
ent whi
c
h is
co
n
t
ro
lled
b
y
math
e
m
atica
l
o
p
e
rator calcu
l
a
tio
n
fo
r d
e
fo
rmatio
n
p
u
rpo
s
es. Mu
scle-b
ased
tech
n
i
qu
e is b
u
ilt
fro
m
mass an
d
sp
ring
to
produ
ce certain
m
u
scle an
i
m
atio
n
s
(see Fig
u
re 1 fo
r th
e illu
stratio
n
of m
u
scle
b
a
sed
r
e
nd
er
i
n
g)
.
Figure.
1
M
u
scl
e
base
d
ren
d
e
r
i
n
g
exam
pl
e pi
c
t
ure t
a
ken
f
r
o
m
http://doi.ieeec
s.org/cm
s/Co
mpute
r
.org
/dl/mags/cg/
2
001/01/
figures/g10187.gif
The well-known
researc
h
e
r
who creates m
u
scle-bas
e
d
fa
ci
al
expres
si
o
n
i
s
Parke i
n
1
9
72 a
n
d 1
9
74
[2
1]
-[
2
2
]
whi
c
h has
been e
x
t
e
nde
d by
Kei
t
h
W
a
t
e
rs [
2
3]
.
W
a
t
e
rs e
nha
nces m
u
scl
e
faci
al
expressi
o
n
s by
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE
Vo
l. 6
,
N
o
. 5
,
O
c
tob
e
r
20
16
:
247
8
–
24
87
2
480
usi
n
g
param
e
teri
zed m
e
t
hod
.
W
a
t
e
rs
’
wo
r
k
i
s
base
d
on
F
A
C
S
t
h
e
o
ry
w
h
i
c
h
uses
act
i
o
n
uni
t
f
o
r t
h
e
m
u
scl
e
m
ovem
e
nt
(see
Fi
g
u
re
2
)
.
Fi
gu
re 2.
Act
i
o
n uni
t
AU
1 o
n
param
e
t
e
ri
zed-
m
uscl
e
m
odel
[2
3]
Th
e co
m
b
in
atio
n
of th
e
Action
Un
its is in
tend
ed to
c
r
eat
e si
x f
u
ndam
e
nt
al
em
ot
i
ons s
u
c
h
as:
neut
ral
,
h
a
pp
y,
d
i
sgu
s
t, fear su
rp
rise an
d
ang
e
r.
Th
e
six
em
o
ti
o
n
s
th
at
W
a
ters
p
r
op
o
s
ed
are well
illu
strated
in
Fig
u
re
3.
All
of t
h
e
fa
cial
m
odels are created by
c
h
anging
t
h
e m
u
scl
e
m
odel
u
s
i
ng
pa
ram
e
t
e
r fo
r eac
h
fu
n
d
a
m
ent
a
l
em
ot
i
on.
Fi
gu
re 3.
Para
m
e
t
e
ri
zed-m
uscl
e
m
odel
s
on
exp
r
essi
ng
si
x basi
c
em
ot
i
ons
[
23]
2.
2.
Interp
olation Technique for
Faci
al
Animation
The m
e
t
hod
o
f
s
h
ape
i
n
t
e
rp
ol
at
i
o
n
o
n
fac
i
al
ani
m
at
i
on
i
s
br
oa
dl
y
use
d
by
faci
al
a
n
im
at
ors t
o
construct polygonal
s
u
rface on 3D
face
.
Interpolation
c
once
p
t in a
one-dim
e
nsion
area can
be c
o
m
puted
th
ro
ugh
two
v
a
lu
e and
i
n
terpolation coefficient
α
,
refe
r to
e
quatio
n.
(1
)
This val
u
e wi
ll be use
d
as
a cont
rol
poi
nt to
cha
n
ge
the facial expression int
o
a
not
her
face
exp
r
essi
on
. I
n
t
e
rp
ol
at
i
o
n
t
ech
ni
q
u
es ca
n
be c
a
t
e
go
r
i
zed
i
n
to sev
e
r
a
l w
a
ys as show
n in
Table 2
.
Hap
p
y
An
ger
Neutral
Fear
Disgu
s
t
Surprise
Value=
α
(value
1
) + (1.0-
α
) (value
2
) 0.0<
α
<1.0
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Em
ot
i
o
n
a
l
F
a
c
i
al
Expre
ssi
o
n
Base
d
on
Act
i
o
n
Uni
t
s
a
n
d
Fa
ci
al
Muscl
e
(
A
hm
a
d
Hoi
r
ul
B
a
so
ri
)
2
481
Tabl
e 2. Int
e
rp
ol
at
i
on
t
e
c
h
ni
q
u
es [2
4]
Interpolation Details
Key
E
xpr
ession
I
n
ter
polation
The transf
or
m
a
tion of
f
ace is
utilizing control
point
whi
c
h is
m
oved in
a straight f
r
a
m
e [
21]. If
surf
ace
m
odel is static
,
m
odif
i
cation of
m
odel
is only
possible in ver
t
ex level.
Bilinear Expressio
n
I
n
ter
polation
Bilinear concept has several wa
y
s
to
express poly
gonal
surface on 3D
f
ace.
Bilinear ble
n
d
o
f
fo
ur
key
p
o
ses can be
obtai
ned fr
om
four
expr
essions gener
a
ted using tw
o inter
polation par
a
m
e
ters.
N-
Di
m
e
nsional
E
xpr
ession I
n
ter
polation
T
echnique for
N-
Dim
e
nsional uses four
i
n
ter
polatio
n factor
s and
sixteen key
expr
essions to gener
a
te four
-
d
im
ension inter
polation
ro
o
m
.
Pair
wise E
xpr
ession
I
n
ter
polation
Advanced techni
que for
ex
pr
essi
on inter
polation
uses nu
m
e
r
ous
expression poses
which are sel
ected
and built by interpolation factor
to
m
e
r
g
e with chosen p
o
ses.
T
h
e
contr
o
l value
for
this
m
e
thod is
star
ting pose,
end pose and inter
polat
ion value.
Facial Region
I
n
ter
polation
This technique is t
o
split f
ace are
a
int
o
so
m
e
independent regions. For
exa
m
pl
e, Kleise
r splits f
ace area into
two regions i.e. upper and lower
region which ena
b
le users to
m
a
nipulate e
m
otion and speech at the
s
a
me
t
i
me
[
2
5
]
.
Th
e m
o
st p
opular
m
u
scle b
a
sed
an
d
action
un
its f
acial an
i
m
atio
n
tch
n
i
q
u
es is sh
ap
e
n
t
erp
o
l
ation
th
at
allo
wed
u
s
er t
o
co
m
b
in
e th
e
m
u
ltip
le sh
ape to
p
e
rfo
r
m
p
a
rticu
l
ar sh
ap
e. Sh
ap
e
In
terpo
l
atio
n
allows
u
s
er
t
o
co
m
b
in
e nu
m
e
r
o
u
s
p
r
e-
scu
l
pted
expr
essions to
p
r
od
u
c
e
desire
d
facial expressi
on. T
h
e avatar 3D
m
odel
co
nsists o
f
a lo
t o
f
v
e
rtices that b
u
ild
th
e m
e
sh
. Verti
ces carry a lo
t o
f
in
form
at
io
n
su
ch
as p
o
s
ition
o
f
vertices
(x
, y
,
z) a
n
d i
n
dex
.
Ge
ne
ral
l
y
, t
h
ere
are t
w
o
di
ff
ere
n
t
su
b
-
t
y
pes o
f
vert
e
x
ani
m
at
i
on cal
l
e
d M
o
rp
h a
n
i
m
at
i
on
and pose
animation.
2.
3.
Mo
rph Animatio
n
Mo
rp
h an
im
ati
o
n is th
e
b
a
sic
an
d sim
p
le an
imatio
n
th
at m
a
n
i
pu
lates k
e
y
fram
e ti
me lin
e with
m
e
sh
sna
p
sh
ot
i
n
t
e
rp
ol
at
i
on.
T
h
e t
e
chni
que
i
s
c
o
n
s
i
d
ere
d
as
ol
d t
echni
que
whi
c
h i
s
use
d
pri
o
r
t
o
s
k
el
et
al
ani
m
at
i
on
[2
6]
. M
o
rp
h a
n
im
ati
on i
s
a va
ri
et
y
of ap
p
r
oa
ch t
h
at
i
n
vol
ve
s ent
i
r
e seri
es
of
vert
e
x
dat
a
t
o
be i
n
t
e
r
pol
at
ed e.
g.
an
im
a
tio
n
for
runn
ing
.
Th
is
meth
o
d
is qu
ite fast
wh
en ani
m
at
io
n
i
n
vo
lves th
e
wh
o
l
e
mesh
du
e t
o
t
h
e easy
m
odi
fi
cat
i
on a
m
ong t
h
e
key
fram
e
s. Ne
ver
t
hel
e
ss, t
h
i
s
a
p
p
r
oach
i
s
n
o
t
sui
t
a
bl
e
fo
r t
h
i
s
faci
al
a
n
i
m
at
i
o
n
because it cannot s
u
pport m
u
ltiple m
o
rph ani
m
ation.
2.
4.
Pose Anim
ati
o
n
Po
se an
im
atio
n
is th
e latest tech
n
i
q
u
e
wi
d
e
ly u
s
ed
for com
p
lex
an
i
m
ati
o
n su
ch
as faci
al an
im
a
tio
n
or
skel
et
al
ani
m
at
i
on. T
h
i
s
t
echni
que
co
nce
r
n
on
bl
e
ndi
ng
m
u
lt
i
p
l
e
di
scre
t
e
pose
s
, e
x
pre
ssi
ng
, art
i
c
ul
at
i
n
g
as
an of
fset
t
o
fo
un
dat
i
o
n ve
rt
ex by
con
s
i
d
e
r
i
ng
di
ffe
rent
w
e
i
ght
t
o
pr
o
d
u
ce t
h
e fi
nal
resul
t
.
In t
h
i
s
ca
se, t
h
i
s
study uses pos
e
anim
a
tion for the a
v
atar fa
cial expr
es
si
o
n
.
Pose a
n
i
m
at
ion i
s
a
bl
en
d
sha
p
e i
n
t
e
r
pol
at
i
on
technique c
o
mmonly used in facial
ani
m
ation beca
us
e each facial
expressi
on i
s
placed i
n
differe
nt
an
im
a
tio
n
s
. To
produ
ce facial ex
p
r
ession co
m
p
letel
y
, p
o
s
e an
im
atio
n
techn
i
qu
e utilize a co
llec
tio
n
of
ori
e
nt
at
i
on
p
o
s
e
t
h
at
p
r
e-
de
fi
n
e
d i
n
si
de m
e
sh w
h
i
c
h i
s
cal
l
e
d as
an
o
f
f
set
t
o
wa
r
d
s
ori
g
i
n
a
l
vert
e
x
dat
a
.
3.
R
E
SEARC
H M
ETHOD
Dev
e
l
o
p
m
en
t o
f
i
n
terp
o
l
ation
is in
itiated
after n
e
u
t
ral appearan
ce
wh
ich
n
a
m
e
d
as facial ex
pressi
on
base.
N
u
m
e
rous of
si
n
gul
a
r
t
a
rget
s fo
r part
i
c
ul
ar
a
ppea
r
a
n
ce
a
r
e pr
od
uc
ed by
reassi
g
n
i
ng ve
rt
i
ces
fr
om
t
h
e
ori
g
i
n
al
m
e
sh. Thi
s
t
echni
qu
e needs
paral
l
el
i
s
m
of
m
e
sh st
ruct
u
r
e am
ong
goal
a
nd
o
r
i
g
i
n
m
e
sh (i
d
e
nt
i
cal
am
ount
of
nu
m
e
rous
vert
e
x
and t
r
i
a
ngl
e
)
.
Goal
i
s
m
easure
d
by
Deg
r
e
e
of F
r
eed
om
(D
OF
)
i
wh
ich h
a
s
cost
0 an
d 1. T
h
e D
O
F res
p
o
n
si
bl
e o
n
i
n
t
e
r
pol
at
i
o
n p
r
oce
ss. M
e
rge
d
ve
r
t
i
ces l
o
cat
i
on i
s
a spot
t
h
at
ha
s DOF
> 0. D
O
F=
0 c
o
r
r
es
po
n
d
i
n
g t
o
OF
F o
r
N
o
EFFEC
T
. T
h
e
eq
u
a
tion
to
cal
cu
late
m
e
rg
ed
v
e
rtices lo
catio
n
an
d
w
e
igh
t
ed
co
m
b
in
atio
n
is exposed
i
n
eq
u
a
tion 2
and
3
.
'
ba
s
e
i
i
i
i
bas
e
vv
v
Wh
e
r
e
v
v
v
(2
)
'
0
ii
i
x
wx
(3
)
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE
Vo
l. 6
,
N
o
. 5
,
O
c
tob
e
r
20
16
:
247
8
–
24
87
2
482
Th
e m
i
x
e
d
lo
catio
n
is attain
ed
fro
m
th
e o
r
ig
in
al lo
catio
n
p
l
u
s
go
al spo
t
with
DOF > 0
.
Th
e en
tire
post
u
re
si
m
u
l
a
t
i
on s
h
o
w
n as
a di
ag
ram
as sho
w
n i
n
Fi
g
u
r
e
4.
Em
ot
i
on
as an i
n
p
u
t
c
o
nt
r
o
l
s
m
e
rger
st
ance
pr
o
g
ressi
on
an
d st
a
n
ces ass
o
r
t
m
e
nt
.
Fi
gu
re 4.
Fl
owch
art Po
se
An
im
a
tio
n
Th
e
p
r
o
c
ed
ure
o
f
fillin
g
t
h
e
face
m
a
sk
m
e
sh
is expo
sed in
C
#
co
d
e
frag
m
e
n
t
(refer to Figu
re 5).
Fi
gu
re 5.
C
#
fr
agm
e
nt
code
f
o
r
l
o
a
d
i
n
g t
h
e
3D
m
odel
As sh
o
w
n i
n
e
quat
i
o
n 2 a
nd
3, bl
e
nd s
h
a
p
e
i
n
t
e
rp
ol
at
i
on i
s
sui
t
a
bl
e fo
r m
u
scl
e
and Act
i
on U
n
i
t
s
approach. Faci
al Action Codi
ng
Sy
st
em
has becom
e
a st
andar
d
t
o
c
ont
rol
faci
al
exp
r
essi
on
of
h
u
m
a
n si
nce i
t
has re
veal
ed
com
p
l
e
t
e
cl
assi
fi
cat
i
on o
f
hum
an faci
al
regi
on
[1
6]
.
Thi
s
ap
pr
oac
h
gi
ves a
dva
nt
ages t
o
co
m
p
u
t
er
g
r
aph
i
cs an
im
a
t
o
r
to
p
e
rform
e
m
o
tio
n
a
l
faci
al
ex
pressi
o
n
on
v
i
rtu
a
l
hu
m
a
n
.
Th
e in
itial FACS is
created
from
hum
an facial pictures
by
di
st
r
i
but
i
n
g t
h
e
fac
e
i
n
t
o
s
p
eci
fi
c
sect
i
ons.
Eac
h
sect
i
o
n d
o
n
a
t
es on
creating precis
e
appeara
n
ce, refer t
o
Fi
gure
6.
Sta
r
t
End
Creat
e pos
e
for
e
ach
emotion
Blending
the
m
u
ltipl
e
pose
Pose
S
t
owage
Saved
Pose
in
differen
t
loca
tion
Render the
anim
ation
Read the
input of
emotion
Evaluation Warning : The document was created with Spire.PDF for Python.
I
J
ECE
I
S
SN
:
208
8-8
7
0
8
Em
ot
i
o
n
a
l
F
a
c
i
al
Expre
ssi
o
n
Base
d
on
Act
i
o
n
Uni
t
s
a
n
d
Fa
ci
al
Muscl
e
(
A
hm
a
d
Hoi
r
ul
B
a
so
ri
)
2
483
Fi
gu
re 6.
Facial Actio
n Cod
i
ng
System
s (FACS) an
d its Act
i
o
n
Un
its-1
Fi
gu
re
6 a
n
d
7 s
h
ow
sp
ot
o
f
Act
i
o
n
U
n
i
t
s
(
AUs
);
th
e detailed
m
ean
in
g
of ev
ery
Actio
n un
its is
structurized as
follow: AU1(
In
ner Br
ow
Rai
s
e
), AU
2
(
Outer Brow Rais
e)
,
AU4(
Br
ow
Lower
),
AU
9(
No
se
Wrinkler
)
,
AU1
5(
Li
p Corner
Depr
essor
),
A
U
1
7
(
C
h
in Ra
iser
).
Fi
gu
re
7.
Faci
a
l
Act
i
o
n
C
o
di
n
g
Sy
st
em
s (FA
C
S) a
n
d
i
t
s
Act
i
on
U
n
i
t
s
-2
No
te:
Red
an
d green
lin
e is po
rt
rayin
g
th
e m
u
scle
sh
ap
e an
d po
si
tio
n
for th
e face m
o
d
e
l.
4.
RESULT AND DIS
C
USSI
ON
The em
ot
i
on exp
r
essi
on
of a
v
at
ar i
s
ge
nera
t
e
d usi
n
g c
o
m
b
i
n
at
i
o
n o
f
Ac
t
i
on U
n
i
t
s
o
n
f
aci
al
m
u
scl
e
.
Each action units will contribute each
othe
r
to pe
rform
realistic facial ex
pression. Blend
sha
p
e interpol
ation
tech
n
i
qu
e is u
tilized
b
a
se sh
ap
e as a referen
ce for o
t
her
e
m
o
tio
n
exp
r
essio
n
. Figu
re
8
-
13
are th
e resu
lt o
f
faci
al
re
nde
ri
n
g
fo
r se
veral
e
m
oti
on e
x
p
r
ess
i
ons
usi
n
g
a c
o
m
b
i
n
at
i
on o
f
a
c
t
i
on
uni
t
s
a
n
d
faci
al
m
u
scl
e
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE
Vo
l. 6
,
N
o
. 5
,
O
c
tob
e
r
20
16
:
247
8
–
24
87
2
484
Fi
gu
re
8.
Di
s
g
ust
e
x
p
r
essi
o
n
Fi
gu
re 8 s
h
o
w
t
h
at
m
u
scl
e
n
ear m
out
h i
s
cont
ract
ed t
o
p
r
od
uce di
s
g
ust
exp
r
essi
on
. A
U
1 a
nd
AU
2
resp
o
n
si
bl
e t
o
co
nt
r
o
l
ey
eb
ro
w st
r
e
n
g
t
h
,
i
f
c
o
m
b
i
n
e wi
t
h
AU
4,
c
a
n
per
f
o
r
m
happi
ness l
o
o
k
.
Li
p
co
n
t
r
o
l/m
u
s
cle an
d
AU1
5
h
a
n
d
l
e lip
pr
og
r
e
ssio
n
altho
ugh e
m
o
tio
n
is p
r
o
d
u
c
ed
. A
ll f
r
ag
m
e
n
t
s ar
e w
o
rk
ing
or
ga
ni
sed t
o
p
r
o
d
u
ce ha
p
p
y
l
o
o
k
i
n
t
h
e c
h
a
r
act
er. T
h
e i
n
t
e
nsi
t
y
of t
h
e
h
a
ppi
ness ca
n
b
e
co
nt
rol
l
e
d b
y
l
e
vel
p
o
w
e
r
(r
ef
er
t
o
Figu
r
e
9A
and 9
B
).
(A
)
(B)
Fi
gu
re 9.
H
a
ppy Exp
r
essi
o
n
Hap
p
y
i
s
an exp
r
essi
on t
h
at
have m
o
re foc
u
s o
n
m
a
nage t
h
e l
i
p
sy
nchr
o
n
i
zat
i
on t
h
at
i
s
why
faci
al
m
u
scle n
ear lip is v
e
ry im
p
o
r
tan
t
to
produ
ce
certain
ex
pressio
n
.
On t
h
e ot
her
h
a
nd
, an
ge
r i
s
an o
p
posi
t
e
fel
l
i
ng
of
ha
p
p
y
t
h
at
al
so de
pen
d
on l
i
p
c
o
nt
r
o
l
as wel
l
.
A
U
1
and
AU
2 c
o
rre
spo
n
d
e
d
t
o
pr
o
duce
a l
o
o
k
ne
arby
t
h
e ey
e
b
row, while AU4 ge
nerate crinkle and inc
r
eas
e the
area
near
f
o
re
head
.
AU
9
f
o
r
t
i
f
i
e
s an
ger
l
o
ok
by
b
u
i
l
d
i
n
g c
r
i
n
kl
es
nea
r
by
n
o
se.
A
U
1
5
&
A
U
1
7
ar
r
a
nge
l
i
p
angl
e a
n
d c
h
i
n
appea
r
a
n
ce
(re
fer t
o
Fi
gu
re
1
0
A
an
d
1
0
B
)
.
Evaluation Warning : The document was created with Spire.PDF for Python.
IJECE
ISS
N
:
2088-8708
Emotional F
a
c
i
al Expre
ssion
Base
d
on Action
Un
its
and Facial Muscle (A
hmad
Hoir
ul B
a
sori)
2
485
(A
)
(B)
Figure 10.
An
ger
ex
pr
ession
Sadn
ess ex
pressio
n
is an
exp
r
essi
on
th
at pu
ll d
o
wn
AU2 th
at
m
a
k
e
eyes lo
ok
s lik
e tired
and
th
e
A
U
15
is
p
u
ll
do
wn
as w
e
ll as
sh
own
in Fi
g
u
re 14
an
d 15
.
Figure 11.
Sadness e
x
pres
sion
Whi
l
s
t
,
fear
e
x
p
r
esi
o
n
i
s
ki
nd
o
f
e
x
pres
si
on
bet
w
ee
n
ha
ppy
a
n
d a
n
ger
t
h
at
foc
u
s
o
n
ey
es an
d
l
i
p
man
i
p
u
l
ation
.
t
h
e eyes are
wid
e
op
en
ho
ev
er th
e AU2
still
p
u
ll down
and
m
o
u
t
h
is litt
le
b
it o
p
e
n
e
d
as sh
own
in
Figur
e 1
2
.
Figure 12. Fear
expres
sion
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
SN
:
2
088
-87
08
I
J
ECE
Vo
l. 6
,
N
o
. 5
,
O
c
tob
e
r
20
16
:
247
8
–
24
87
2
486
5.
CO
NCL
USI
O
N
Facial anim
a
t
ion is
ve
ry cha
llenging re
search in
c
o
m
puter gra
phics a
r
ea
. Action
Units
and Facial
m
u
scl
e
approa
ch i
s
ve
ry
at
t
r
act
i
v
e and a
d
a
p
t
a
bl
e t
o
be
use
d
i
n
m
ovi
e or t
r
ai
ni
ng a
p
pl
i
cat
i
on.
In t
h
i
s
pa
p
e
r we
h
a
v
e
sh
own
the cap
ab
ility o
f
actio
n
un
its an
d facial m
u
scle to
p
e
rform
p
a
rticu
l
ar em
o
tio
n
ex
pressi
on
.
Th
e
i
m
ple
m
entatio
n result portray
that
re
nd
eri
n
g
artificial face i
s
looks realistic
. The fut
u
re
resear
ch will focus
on
devel
opi
ng
fac
i
al m
u
scle m
e
thod
with
detail
pa
ram
e
ter that can
pe
rform
wrinkle on
face
area.
ACKNOWLE
DGE
M
ENTS
A
u
t
h
or
s ar
e
g
r
atef
u
l
to Facu
lty o
f
C
o
m
p
u
ting
an
d In
for
m
at
io
n
Techno
logy Rab
i
gh
,
K
i
ng
A
bdu
laziz
U
n
i
v
er
sity,
K
i
n
gdo
m
o
f
Saudi A
r
ab
ia
REFERE
NC
ES
[1]
A.
Garc,
et al.
, “Emotional face expression pr
of
iles supported
b
y
v
i
rtual human ont
olog
y
:
Research Articles,”
Comput. An
imat. Virtual Worlds
,
vol/issue:
17(3-4
)
, pp
. 259-269
, 2
006.
[2]
N. Zag
a
lo
and
A. Torres
,
“
C
ha
ract
er em
otion
e
xperien
ce
in vir
t
ual env
i
ronm
ent
s
,”
Vis. Comput.Springer-Verlag
,
vol/issue:
24(11)
, pp
. 981-986
, 2
008.
[3]
D. Traum, “Talking to Virtual Hu
mans:
Dialogue Models and Methodol
ogies for Embodied Conversational
Agents,
”
Modeling Communication with
Robots a
nd Virtual Humans,
Springer
Berlin, Heidelberg
, 2
008.
[4]
M. Fabri,
et al.
, “The
Emotional Avatar:
Non-ve
r
b
al Com
m
unicat
ion Between Inh
a
bi
tants of Collaborativ
e Virtual
Environments,” in
Proceed
ing
s
of the International Gesture
Workshop
on Gesture-Based Communication
in
Human-Comput
er Interaction
,
S
p
ringer-Verlag,
1999.
[5]
A.
G.
Rojas,
et a
l
.
, “
I
ndividu
ali
z
e
d
react
ion m
ovem
e
nts
for virtual
hum
ans
,
” in
Pr
oceed
ings
of the
4th inter
nationa
l
conferen
ce on
Computer grap
hics and
intera
ctive techniqu
es in Austra
lasia and Southeast Asia, ACM: Ku
ala
Lumpur, Malaysia,
2006
.
[6]
C. Melo
and A. Paiva, “Expres
sion of Emotion
s
in
Virtual Hu
mans Using Lights,
Shadows, C
o
mposition and
Filters,
”
in
Proceedings o
f
th
e 2
nd int
e
rnational
conf
erence
on
Affect
ive Computing and
Intellig
ent In
teract
io
n
,
Springer-Verlag
: Lisbon, Portug
al,
2007
.
[7]
Z. Wang,
et al.
, “Construction of Virtual Assistan
t Based on Basic Emotions Theory
,” Springer Ber
lin, Heid
elberg
,
2005.
[8]
A.
G.
Rojas,
et al.
, “Simulation of individual spont
aneous r
eactive b
e
havio
r
,” in
Pr
oceed
i
ngs
of the 7th
internationa
l jo
int con
f
eren
ce
on Autonomous agents
and m
u
ltiagen
t
syst
ems, Internat
ional
Foundation
fo
r
Autonomous Agents and Multiagent
Systems: Esto
ril,
Portugal,
vol. 1
,
2008
.
[9]
K.
Balc
ı
,
et
al
.,
“
X
face Open S
ource P
r
oj
ect
an
d S
M
IL-Agent
S
c
ripting
Langu
age for
Crea
ting
and Anim
atin
g
Embodied Conv
ersation
a
l Ag
ents,”
in
MM’07
,
September 23–28
,
2007, ACM:
Aug
s
burg, Bavaria
,
Germany,
2007.
[10]
N.
Bee,
et al.
, “Simplified f
a
cial
animation
control utilizing
novel input devices: a comparative stud
y
,
”
in
Proceed
ings of
the 13th in
terna
tional con
f
eren
c
e
on In
te
llig
ent
user interfac
es, ACM: Sanibe
l
Island, Florida
,
USA,
2009
.
[11]
C. Sandor
,
et
al
.
, “Exploring
Visuo-Haptic Mixe
d Reality
,
” 2007
[cited 2010]
.
[12]
K.
Sato,
et al
.,
“
H
aptic tel
e
xis
t
enc
e
,
”
in
ACM
SIGGRAPH 2007 emerging technolog
ies, A
C
M: San Diego,
California,
2007
.
[13]
J. Teh
and A. D. Cheok, “Intern
e
t
and Int
e
rne
t
Pa
jam
a
: Novel
S
y
s
t
em
s for Rem
o
te
Haptic In
ter
act
i
on,” in
The 5th
Workshop on Network
&
System Support for Games 2006 — N
E
TGAMES 2006
, ACM: Singapore
,
pp. 1-2
,
2006
.
[14]
A.
H.
Basori,
et al
.
,
“
T
he fe
asibili
t
y
o
f
hum
an haptic em
otion as a fea
t
ure to enhan
c
e
intera
ctiv
it
y a
nd
i
mme
rsi
v
e
n
e
ss o
n
vi
rt
ua
l
rea
l
ity
ga
me
,
”
i
n
Proc
e
e
dings of T
h
e 7th ACM SIGG
RAPH Internationa
l Conferenc
e
on
Virtual-Reality
Continuum and I
t
s Applications in Industry,
ACM, Singapore, p
p
. 1-2
,
2008
.
[15]
A.
H. Basori,
et al.
, “An
Integr
ation
Framewor
k For H
a
ptic Feedb
ack
To
Improve Facial Expr
ession On
Virtual
Human,
”
International Journal of
Innova
tive
Co
mputing, In
formation and
Contr
o
l
, 2012
.
[16]
A.
H.
Basori and Qasim A.
Z., “Extreme expr
e
ssion of sweating in 3D virtual human,”
Com
puters in Human
Behavior
. Vol. 3
5
, pp
. 307-314
,
2014. Availab
l
e
from, DOI: 10.1
016/j.chb.2014
.0
3.013.
[17]
Alkawaz M.
H.
,
et al
., “
R
ea
lis
ti
c faci
al expr
es
s
i
on of virtual hu
m
a
n bas
e
d on color, s
w
eat
, and tears
eff
ects
,
”
Sc
i
World J
, vol. 9
,
2014.
[18]
Alkawaz M
.
H.,
et al.
, “Ox
y
genation
absorptio
n and
light scattering
driven
fa
cia
l
an
im
ation
o
f
natur
a
l
virtu
a
l
hum
an,” Mult
im
edia
Tools
a
nd
Applications, pp
. 1-37
.
DOI: 10.1007/s11042-016-3564-2.
[19]
A. H. B
a
sori, “Emotion walk
in
g for hum
anoid
ava
t
ars
us
ing
b
r
ain s
i
gn
als
,
”
In
ternational Jour
nal of Advan
ced
Robotic Systems, InTech
, pp. 1–1
1, 2013
. doi: 10
.5772/54764.
[20]
F.
I.
Pa
rke,
“T
ec
hni
que
s for Facial Animation,New Trends in Anima
tion and
Visualiz
ation
,
”
edit
ed b
y
N
.
M.
Thallman
and
D. Thalmann, N
e
w York
, John Wiley
and
Sons, pp
. 229-241, 1991.
[21]
F. I. Parke
,
“
C
o
m
puter Gener
a
te
d Anim
ation of
Faces,”
Un
iversi
t
y
of Ut
ah, Mast
er Th
esis, Sa
lt
L
a
ke C
i
t
y
, 1972.
[22]
F. I. Parke, “A Parametr
ic model
for Human Faces,” University
of
Utah, PhD Th
esis, Salt Lake City
, 1974
.
[23]
K. W
a
ters, “
A
Muscle Model f
o
r Anim
ati
ng Three-Dimension
a
l Facial Expres
sion,”
ACM
,
Co
mputer Graphics
,
vol/issue: 21(4), 1987.
[24]
F. Parke and
K.
Waters, “Computer
Facial Animation,”
Massachu
s
etts, AK Peters
, Ltd
Wellesley
,
2008.
Evaluation Warning : The document was created with Spire.PDF for Python.
IJECE
ISS
N
:
2088-8708
Emotional F
a
c
i
al Expre
ssion
Base
d
on Action
Un
its
and Facial Muscle (A
hmad
Hoir
ul B
a
sori)
2
487
[25]
J
.
A. Kleis
e
r, “
F
as
t, Effi
cien
t,
Accurat
e
W
a
y
t
o
Repres
ent th
e
Hum
a
n F
ace,”
in
State of
the Art in Facia
l
Animation
,
SIG
G
RAPH '89 T
u
t
o
rials,
New Yor
k
, ACM, 1989
.
[26]
P
.
Ekm
a
n and
W
.
V. F
r
ies
e
n
,
“
F
acial
Action
Coding S
y
s
t
e
m
: A Techn
i
qu
e for
The M
e
as
urem
ent of F
a
ci
al
Movement,” Palo Alto, California, Cons
ulting
Psy
c
ho
logists Pres
s, Inc., 1978.
BIOGRAP
HI
ES OF
AUTH
ORS
Ahmad H
o
ir
ul Basor
i
, was bor
n in Wirowongso Villag
e
,
jem
b
er Cit
y
, Indon
esia, in 1982
. He
received
the B
.
Sc (Software
Engineering)
fr
om
Institut T
e
knologi Sepulu
h
Nopem
b
er
Surabay
a
in 20
04 and th
e Ph.D (Computer Gr
aphics) from U
n
iversiti Teknologi Mala
y
s
ia,
Johor Bahru, Johor, in 2011. Fr
om 2004 to present,
h
e
was a lecturer with the
Department of
Informatics, Faculty
of
Infor
m
ation Tekno
logi, Institut
Teknologi
Sepu
luh Nopember
Surabay
a
, Indon
esia. In 2011, he has appointed as
Assistant Prof
essor with the Department of
Com
puter Graphics and Multim
edia, Universiti T
e
knologi Mal
a
y
s
ia. Curr
entl
y
,
he
is appointed
as Associate Professor of Faculty
of Com
putin
g and Information Technolog
y
Rabigh, King
Abdulaziz Univ
ersity
. He is
the
member of Edito
ri
al b
o
ard of
some intern
ation
a
l
journal, more
than 32
art
i
cl
es
,
and a
l
s
o
a m
e
m
b
er of profes
s
i
onal m
e
m
b
ers
h
ip IEE
E
,
ACM
S
I
GGRAP
H,
IAENG and Senior Member of
IACS
IT. His
res
earch
int
e
res
t
s
includ
e Com
puter Graphi
cs
,
Facial Animation, Cloth Simula
tion, Medic
a
l Visualiz
ation
,
Haptic Inter
a
c
tion,
Man Machin
e
Intera
ction
and
Robotics.
Hani Moaite
q
Ab
dullah
AlJahd
ali
was born in Jedd
ah, Saudi Ar
abia, in 1983
. He
received th
e BSc (Computer Science)
, in 2005 fr
om King Abdul
Aziz University
in Jeddah. In
2009, He receiv
ed the MSc in
(Information Te
chnolog
y
)
and the PhD (Comp
u
ter Scien
ce)
from the Univer
sity
of Glasgow in 2015. Fr
om 2005 to 2007,
he was working at the Saud
i
Electricity
Compan
y
as
a Budg
et and
S
y
stem an
al
y
s
t. In 2011
, h
e
has
appointed
as a
lecturer
with the depar
t
ment of Information S
y
stems,
King AbdulAziz University
. C
u
rrently
, h
e
is
appointed as an
Assistant Professor of Facult
y
o
f
Computing an
d Information Technolog
y
in
Rabigh, King AbdulAziz Universi
ty
. His research in
terests
includ
e Information Security
,
Human Computer Interaction
,
G
r
aphica
l Password Authentication
and Robo
tics
.
Evaluation Warning : The document was created with Spire.PDF for Python.