Indonesi
an
Journa
l
of El
ect
ri
cal Engineer
ing
an
d
Comp
ut
er
Scie
nce
Vo
l.
10
,
No.
3
,
June
201
8
,
pp.
1203
~
1211
IS
S
N: 25
02
-
4752, DO
I: 10
.11
591/ijeecs
.v1
0.i
3
.pp
1203
-
1211
1203
Journ
al h
om
e
page
:
http:
//
ia
es
core.c
om/j
ourn
als/i
ndex.
ph
p/ij
eecs
Lessons
Learned
from th
e Qualit
y of Exp
er
i
ence (QoE
)
Assessm
ent of
4G M
obile Techn
ology in
Indonesi
a
Muhamm
ad S
uryane
ga
r
a
1
,
Fery
Andri
ya
nt
o
2
,
Aj
ib
Set
yo A
ri
fin
3
1,3
Gradua
te
Prog
ram i
n
Te
l
ec
om
m
unic
at
ions Ma
nage
m
ent
,
Depa
rte
m
ent
of
E
le
c
tr
ic
a
l
Eng
ineeri
ng
,
Univer
si
ta
s Ind
onesia
,
Indone
sia
2
Ministr
y
of
Co
m
m
unic
at
ion
an
d
Inform
at
ion
T
ec
hnolog
y
,
Indo
nesia
Art
ic
le
In
f
o
ABSTR
A
CT
Art
ic
le
history:
Re
cei
ved
Dec
14
, 201
7
Re
vised
Feb
10
, 2
01
8
Accepte
d
Ma
r
2
1
, 201
8
The
purpose
of
thi
s
rese
arc
h
is
t
o
assess
the
qual
ity
o
f
expe
ri
ence
(QoE)
of
4G
m
obil
e
te
ch
nolog
y
tha
t
is
used
in
the
Ind
onesia
n
m
ark
et.
The
stud
y
ana
l
y
z
ed
th
e
result
s
obta
in
ed
and
discussed
the
le
ss
on
le
arn
ed
f
or
stake
hold
ers
of
the
te
l
ec
om
m
unic
ations
industr
y
.
Prim
ar
y
data
was
obta
ined
b
y
conducting
a
m
ark
et
surve
y
in
Ma
y
2017,
with
a
conf
ide
n
ce
l
e
vel
of
95%,
rep
rese
nt
ing
the
4G
subs
cri
ber
s
in
the
Indone
s
ia
m
ark
et.
The
r
e
are
th
re
e
m
ai
n
subjec
ti
v
e
aspe
ct
s
of
Qo
E
:
service
qual
i
t
y
,
data
spee
d,
and
net
work
qual
ity
(
indi
c
at
e
d
b
y
stabilit
y
of
signal
).
Th
e
anal
y
ses
were
condu
ct
ed
base
d
on
ACR
-
HR
score
,
compari
ng
u
sers’
expe
ri
enc
e
s
of
4G
with
th
ei
r
pre
v
ious
3G
subs
cri
pti
ons.
It
was
found
tha
t
,
in
gen
e
ral
,
the
Indon
e
sian
m
ark
et
per
ceive
d
4G
as
offe
ring
a
bet
t
e
r
expe
rie
n
ce
th
a
n
3G.
How
eve
r,
one
of
the
important
le
ss
ons
tha
t
ca
n
be
learne
d
is
th
at
net
work
op
erators
should
provide
a
m
ore
stable
signa
l, a
s
t
he
m
ark
et ga
v
e a
lower
s
cor
e
for this a
spe
ct.
Ke
yw
or
d
s
:
4G
Ind
on
esi
a
Mob
il
e tec
hn
olo
gy
QoE
Qu
al
it
y o
f
e
xperience
Copyright
©
201
8
Instit
ut
e
o
f Ad
vanc
ed
Engi
n
ee
r
ing
and
S
cienc
e
.
Al
l
rights re
serv
ed
.
Corres
pond
in
g
Aut
h
or
:
Muh
am
m
ad
Surya
ne
gar
a,
Gr
a
duat
e Pro
gra
m
in
Tel
ecom
m
un
ic
at
ion
s Manag
em
ent
Dep
a
rtm
ent o
f El
ect
rical
En
gi
neer
i
ng, Uni
ve
rsita
s Indo
nesi
a,
Faculty
of E
ngineerin
g, Kam
pu
s
UI
De
pok 164
24, In
done
sia
.
Em
a
il
:
m
.su
ryaneg
a
ra@
ui.ac
.id,
s
ury
ane
gar
a
@g
m
ai
l.co
m
1.
INTROD
U
CTION
The
In
te
r
natio
nal
Tel
ecom
m
un
ic
at
io
n
Un
i
on
(
IT
U)
has
de
fine
d
qual
it
y
of
ex
pe
rience
(
QoE)
as
t
he
“degree
of
delight
or
a
nnoyan
ce
of
the
use
r
of
an
ap
plica
ti
on
or
se
r
vice”
[1
]
.
I
n
c
on
tr
ast
to
qu
al
it
y
of
s
erv
ic
e
(QoS
)
assessm
ent,
Q
oE
asse
ssm
ent
is
m
ain
ly
based
on
us
er
hab
it
s
(i.
e.
it
is
us
er
centere
d)
,
wh
il
e
Qo
S
assessm
ent
is
based
on
te
chn
ic
al
pe
rform
ance
(technolo
gy
centere
d)
[
2].
I
n
th
e
con
te
xt
of
m
ob
il
e
com
m
un
ic
at
ion
s
netw
orks
,
QoE
asses
ses
the
use
rs
’
per
c
eptions
of
t
he
serv
ic
e,
a
nd
th
ese
pe
rce
ptions
m
ay
relat
e
to
al
l
as
pects
of
t
he
te
chn
ic
al
syst
em
,
inclu
ding
te
r
m
inals,
ne
tw
orks,
inf
rastr
uctu
re,
et
c.
He
nce,
Q
oE
assessm
ent
is
essenti
al
,
since
exp
e
rience
ga
ined
f
r
om
pr
ev
iou
s
se
rv
ic
es
,
as
well
as
expect
at
ion
s
f
or
th
e
new
serv
ic
e,
infl
ue
nce
f
uture
stra
te
gic
plans
.
F
or
e
xam
ple,
Q
oE
asses
sm
ent
data
can
be
use
d
to
overc
om
e
the
pro
blem
of
ch
urn
rates
an
d
the
exclu
sio
n
of
new
se
rv
ic
e
plans
[
2],
aspe
ct
s
of
strat
egic
capab
il
it
y
that
cannot
be
sat
isfie
d
m
e
rely
b
y
util
iz
ing
QoS
data [
3].
Ind
on
esi
a,
t
he
four
t
h
m
o
st
popu
l
ou
s
country
in
t
he
w
or
l
d,
co
m
m
enced
it
s
com
m
ercial
i
m
ple
m
entat
io
n
of
4G
m
ob
il
e
te
chnolo
gy
in
20
15.
H
owe
ver,
pu
blic
QoE
assessm
ent
of
4G
has
not
bee
n
offici
al
ly
rep
ort
ed
since
it
s
com
m
encem
en
t,
desp
it
e
the
f
act
that
Qo
E
data
is
essenti
al
fo
r
s
uppo
rting
t
he
con
ti
nuin
g
st
r
at
egic
plan
s
of
m
ob
il
e
netw
ork
op
e
rato
rs.
Lesso
ns
t
hat
can
be
dra
wn
from
cur
re
nt
m
ark
et
exp
e
rience
a
re
no
le
ss
im
po
rtant
for
the
suc
cess
of
f
uture
te
chnolo
gy
r
ol
lou
ts
tha
n
the
y
are
f
or
th
e
c
urren
t
4G ser
vice.
This
researc
h
aim
s
to
assess
the
qual
it
y
of
QoE
of
4G
m
ob
il
e
te
ch
no
l
ogy
in
the
I
ndon
esi
an
m
ark
et
.
Su
bs
eq
ue
ntly
,
le
sso
ns
le
ar
ned
for
t
he
m
ob
il
e
netw
ork
opera
tors
a
re
der
i
ve
d.
A
m
ark
et
s
urvey
was
c
onduct
ed
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2502
-
4752
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci,
Vo
l.
10
, N
o.
3
,
June
2018
:
1203
–
1211
1204
in
Ma
y
2017,
with
a
m
ini
m
um
sa
m
p
le
com
plyi
ng
with
Sl
ov
i
n’
s
f
or
m
ula,
gi
ving
ra
ndom
sa
m
p
li
ng
w
it
h
a
confide
nce
le
ve
l
of
95
%
.
Th
e
responde
nts
wer
e
re
quest
e
d
to
giv
e
their
su
bject
ive
pe
r
cepti
on
of
24
it
e
m
s
relat
ing
to
t
hr
e
e
assessm
ent
a
sp
ect
s,
thes
e
be
ing
se
rv
ic
e
qual
it
y,
data
sp
eed,
a
nd
sta
bili
ty
of
sig
nal
f
or
t
he
4G
netw
ork.
T
o
pro
vid
e
a
com
pr
ehe
ns
ive
a
naly
sis,
the
ACR
-
HR
m
et
ho
d
w
as
us
ed
to
a
na
ly
ze
the
su
rv
e
y
data,
represe
nted
by
scor
es
rangin
g
from
-
5
(po
orest
)
to
+5
(b
e
st).
Su
c
h
a
sco
rin
g
syst
e
m
pr
ov
i
des
a
com
par
is
on
betwee
n
us
er
s’
cu
rr
e
nt
ex
pe
rience
of
4G
se
r
vices
a
nd
their
previ
ou
s
ex
pe
rience
of
3G.
The
resear
ch
them
e
pr
ese
nted
i
n
th
is
pap
er
is
pa
rt
of
o
ngoi
ng
w
ork
in
vestigat
in
g
the
sh
i
fting
of
the
I
ndonesi
an
m
ark
et
fr
om
3G
t
o
4G [4
]
,
[
5].
The
rem
ai
nd
e
r
of
the
pa
pe
r
is
structu
re
d
by
rev
ie
wing
under
ly
in
g
theo
ries
in
Se
ct
ion
2
a
nd
pr
ese
ntin
g
th
e
m
e
tho
d
of
m
ark
et
s
urvey
unde
rtake
n
i
n
Sect
ion
3.
T
he
res
ults
a
nd
discuss
i
on
of
le
sso
ns
le
arn
e
d
ar
e
pr
e
sented
in Sec
ti
on 4 an
d
t
he
c
oncl
us
i
on is pre
sented
in Sec
ti
on 5.
2.
THE
ORY O
F
Qo
E
ASSESS
MENT
The
def
i
niti
on
of
Q
oE
giv
e
n
by
the
E
uro
pe
an
Tel
ec
omm
u
nicat
ion
Stan
da
rd
s
I
ns
ti
tute
is
"a
m
easur
e
of
use
r
perfor
m
ance
based
o
n
both
ob
j
ect
iv
e
and
s
ubj
ect
iv
e
ps
yc
holo
gical
m
easur
es
of
us
in
g
ICT
ser
vi
ce
or
pro
du
ct
”
[
2].
I
n
li
ne
with
thi
s,
Re
com
m
end
at
ion
IT
U
-
T
G
.1011
set
s
out
two
ov
e
rall
m
e
thods
to
be
use
d
f
or
assessi
ng
Q
oE
:
su
bject
ive
assessm
ent
t
hro
ugh
pe
rce
pt
ion
sur
veys,
and
obj
ect
ive
assessm
ent
t
hro
ugh
m
on
it
or
ing,
plann
i
ng, a
nd
/
or
la
b
te
sti
ng.
2.1. Ob
jectiv
e
Assessme
nt
Objecti
ve
asse
ssm
ent
of
Q
oE
in
vo
l
ves
t
he
us
e
of
an
e
stim
ation
m
odel
m
a
tc
hed
t
o
the
act
ual
te
chnolo
gical
serv
ic
e
or
pro
du
ct
bei
ng
re
vi
ewed
.
T
he
m
od
el
sho
uld
c
om
pl
y
with
th
e
ser
vice
or
pro
duct
obj
ect
ives
,
pro
vid
e
the
new
app
li
cat
io
n
fea
tures,
a
nd
ha
ve
the
new
net
work
c
onfi
gur
at
ion
s.
In
reg
a
rd
to
com
m
un
ic
at
ion
s
te
ch
nolo
gy,
obj
ect
ive
QoE
assessm
ent
can
be
ca
rr
ie
d
ou
t
in
t
hr
ee
ways:
intrusi
ve
m
od
e,
non
-
intr
us
ive
m
od
e,
and
plann
i
ng
m
od
e
[
6]
.
In
intr
us
ive
m
od
e,
a
sign
al
is
inj
ect
ed
int
o
the
syst
e
m
un
der
te
st,
with the aim
o
f
m
easur
in
g
the
o
ut
pu
t si
gnal
. No
n
-
i
ntr
us
ive
m
od
e assessm
e
nts m
easur
e traff
ic
d
i
rectl
y wit
hout
add
i
ng
a
si
gn
a
l
to
the
syst
em
,
wh
il
e
assessm
ent
in
plan
n
in
g
m
od
e
is
no
t
perform
ed
in
a
real
syst
e
m
and
us
ua
ll
y
serv
es
as
a
too
l
f
or
desig
ning
f
uture
netw
ork
sy
stem
s.
Su
ch
a
m
od
e
re
qu
i
re
s
a
plan
ning
-
s
pecific
assum
ption
as
well
as
detai
le
d
kn
ow
le
dg
e
ab
ou
t
t
he
se
rv
ic
e
or
pr
oduct
bei
ng
ass
essed
[6
]
.
O
bj
ect
ive
asses
sm
ent
is
c
heap
e
r
a
nd
faster
than
sub
j
ect
ive
assessm
ent,
but
the
accu
r
acy
of
the
resul
ts
ob
ta
ine
d
de
pend
s
on
the
s
uitabil
it
y
of
the
m
od
el
.
Hen
ce,
obje
ct
ive
assessm
ent
is
con
sid
er
ed
to
pe
rfor
m
with
lim
it
ed
validit
y
and
reli
abili
ty
.
Si
m
il
ar
to
Qo
S
assessm
ent
[7
]
,
s
uch
a
n
obj
ect
ive
ass
essm
ent
wo
ul
d
f
ocus
on
th
e
Qo
E
te
chn
ic
al
m
et
ri
cs
of
a
serv
ic
e
app
li
cat
ion
.
T
he
exam
ple
of
obj
ect
ive
Q
oE
assessm
ent
wo
uld
be
a
la
bora
tory
-
base
d
assessm
ent
of
a
ne
w
vi
deo
se
rv
ic
e
be
f
or
e
it
is
co
m
m
ercial
ly
la
un
ch
ed
to
the
m
ark
et
[8
]
and
the
PSN
R
evaluati
on
of
vi
deo
ap
plica
ti
on
i
n
wireless
m
ul
tim
edia
sensor
netw
ork
[
9].
I
n
relat
io
n
t
o
t
he
4G
ne
tw
ork,
the
Office
of
C
omm
un
ic
at
ion
(
O
FCOM)
i
n
the
U
K
c
onduct
ed
a
data
s
pee
d
te
st
an
d
re
por
te
d
that
the
U
K
4G
netw
ork per
f
orm
ed
at
a f
ast
e
r data
s
peed tha
n
the
3G
netw
ork
[10
]
.
2.2. Sub
jecti
ve
A
sses
smen
t
This
rese
arc
h
is
de
velo
ped
ba
sed
on
s
ub
j
ect
ive
assessm
ent.
Subj
ect
i
ve
ass
essm
ent
of
Q
oE
m
easur
es
exp
e
rience
,
pe
rcep
ti
on,
a
nd
s
at
isfact
ion
as
t
hey
are
def
in
e
d
by
us
ers
.
U
s
ers
are
as
ke
d
to
pr
ov
i
de
opin
ion
s
about
t
he
qu
al
it
y
of
var
i
ous
aspects
of
a
se
rv
ic
e
or
te
ch
nolo
gy,
e
xpress
ed
us
in
g
a
rati
ng
scal
e
know
n
as
a
m
ean
opinio
n
scor
e
,
or
M
OS.
IT
U
def
i
nes
MOS
as
“t
he
va
lue
on
a
pred
efine
d
scal
e
th
at
a
subj
ect
as
s
ign
s
t
o
his
opi
nion
of
the
perform
a
nce
of
the
te
l
epho
ne
tra
ns
m
issi
on
syst
em
us
e
d
ei
ther
f
or
co
nv
e
rsati
on
or
f
or
li
ste
nin
g
to
s
poke
n
m
at
erial
”
[11].
Use
rs
re
sp
on
d
to
qu
e
sti
on
s
a
bout
thei
r
le
v
el
s
of
sat
isfact
ion
with
s
erv
ic
e
op
e
rati
ons
th
r
ough
sur
vey
te
chn
iq
ues
s
uc
h
as
inter
vie
ws,
discussi
on
gro
ups,
a
nd
quest
io
nnai
r
es.
I
n
com
par
ison
with
obj
ect
ive
m
et
hods
,
s
ubj
ect
ive
QoE
assess
m
ent
req
uire
s
higher
budget
s
and
m
or
e
extensiv
e
resou
rces.
S
ubj
ect
iv
e
a
ssess
m
ent
of
Qo
E
can
al
so
m
e
asur
e
ov
e
rall
syst
e
m
per
f
orm
ance
by
ta
ki
ng
int
o
account
te
c
hn
i
cal
par
am
et
ers
(
QoS).
Hen
c
e,
Qo
E
data
can
com
bin
e
knowle
dge
of
the
us
e
rs’
s
ubje
ct
ive
exp
e
riences
wi
th
the
syst
e
m
’s
te
chn
ic
al
par
a
m
et
ers.
An
e
xam
ple
of
this
m
et
ho
d
is
the
QoE
m
easur
em
ent
of
the
3G
netw
ork
in
the
U
K
[
12]
cond
ucted
by
OF
COM,
on
e
i
m
po
rtant
fin
ding
of
w
hich
was
that
us
e
rs
were
m
os
t
con
cern
e
d
ab
out
the
av
ai
la
bili
ty
of
vo
ic
e
-
cal
l
ser
vices
an
d
cl
arit
y
of
s
ound.
A
nothe
r
exam
ple
of
t
he
QoE s
ubj
ect
ive
assessm
e
nt includ
es
the e
val
uation o
f use
rs
’ perce
ption o
f c
om
pu
te
r gam
es ap
plica
ti
on
[
13
]
.
Evaluation Warning : The document was created with Spire.PDF for Python.
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci
IS
S
N:
25
02
-
4752
Lessons
Learn
ed
fr
om the
Q
uali
ty
o
f Ex
peri
ence (Q
oE)
Assessm
e
nt o
f
4G
… (
Muha
mma
d Su
r
ya
ne
ga
r
a
)
1205
3.
METHO
D
3.1. Pri
mar
y Data o
f QoE
S
ubj
ec
tive Asse
ssment
A
s
urvey
is
a
m
uch
-
us
e
d
t
ool
for
obta
inin
g
pri
m
ary
data
for
s
ubj
ect
ive
QoE
asses
sm
e
nt.
F
or
this
stud
y,
a
m
ark
et
su
rv
ey
was
cond
ucted
in
M
ay
20
17,
quest
ion
in
g
res
po
ndents
us
in
g
a
ra
ndom
face
-
to
-
f
ace
m
et
ho
d.
The
num
ber
of
res
ponde
nts
in
the
sam
ple
co
m
pl
ie
s
within
Sl
ovin
’s
f
or
m
ula
to
re
pr
ese
nt
a
gr
eat
er
than
1
m
il
l
ion
popula
ti
on
wi
th
a
m
arg
in
of
error
of
5%
.
T
he
m
ai
n
requir
e
m
ents
for
the
ch
os
en
res
pondent
s
wer
e
as
fo
ll
ow
s:
a.
Alrea
dy
be
s
ubscri
bed
to
4G
m
ob
il
e
serv
ic
e
(eit
he
r
as
pre
paid
or
postpai
d
us
er
s)
f
or
a
m
ini
m
u
m
of
one
day.
b.
Be
reside
nt
in
Jakar
ta
(the
ca
pital
ci
ty
of
I
ndonesi
a)
,
with
his/her
m
ai
n
routine
act
ivit
ie
s
bein
g
locat
e
d
in
the cit
y.
c.
Hav
e
s
ub
sc
ribe
d
to
a
3G ser
vi
ce be
fore
us
in
g 4
G
te
ch
nolo
gy
.
d.
Use
a
m
ob
il
e
han
dse
t
as
the
te
rm
inal
to
acc
ess
4G
ser
vice
(in
s
om
e
cou
nt
ries,
inclu
ding
Ind
on
esi
a,
4G
is
al
so
us
ed
for fi
xed
-
data
com
m
un
ic
at
ion
servic
es).
The
sam
ple
ar
ea
was
Jaka
rt
a,
a
ci
ty
with
a
po
pu
la
ti
on
of
ar
ound
12
m
illi
on
.
P
re
vio
us
res
earc
h
carried
out
by
the
auth
ors
ha
s
argue
d
that
Jakar
ta
can
be
us
e
d
as
a
sam
pling
a
rea
re
presenti
ng
the
na
ti
on
al
m
ark
et
b
eca
use
[4]:
a.
Jakar
ta
is
the
capit
al
ci
t
y,
the
‘life
sty
le
’
center
of
the
co
un
try
,
the
m
os
t
m
ul
ti
cultural
ci
ty
in
In
do
nesi
a,
and the ce
nter
for
m
od
er
n
tre
nd
s
that a
re su
bs
e
qu
e
ntly
f
oll
ow
e
d by
oth
e
r parts
of the
coun
t
ry.
b.
Ty
pical
ly
,
the
national
op
e
rat
or
s
la
un
c
h
thei
r
la
te
st
serv
ic
e
s,
inclu
ding
4G,
in
Ja
kar
ta
,
and
pe
netrate
into
oth
e
r
se
rv
ic
e a
reas a
fterw
a
r
ds.
c.
Desp
it
e
the
het
eroge
neity
of
t
he
c
ountry’s
dem
og
ra
phic
pr
of
il
e,
a
rou
nd
66%
of
t
he
popula
ti
on
of
Ja
ka
rta
is Java
nese.
Thi
s p
r
oport
io
n
is
sim
il
ar to
the
ov
e
rall
d
ist
ri
buti
on
o
f
po
pu
la
t
ion
i
n Ind
on
esi
a.
d.
So
m
e
60
%
of
the
Ind
on
e
sia
n
m
on
ey
supp
ly
is
locat
ed
in
Jakar
ta
,
30%
in
oth
er
ci
ti
es
an
d
10%
in
rural
areas.
T
her
e
f
ore,
al
l
m
ob
il
e
network
op
e
rato
rs
ha
ve
treat
e
d
m
ob
il
e
su
bsc
r
ibers
in
Ja
ka
rta
as
their
m
ai
n
rev
e
nue
s
ource
s,
ta
king
i
nto
a
ccount
bo
t
h
th
ei
r
dyna
m
ic
and
r
outi
ne
act
iv
it
ie
s
(the
te
rm
‘m
on
ey
supp
ly
’
ref
e
rs
to
the a
m
ou
nt of m
on
e
y avail
able in
a
n
ec
onom
y).
3.2. Ques
tionn
aire De
sign
Fact
or
s
that
in
flue
nce
QoE
are
any
char
act
erist
ic
of
a
us
er,
syst
e
m
,
servi
ce,
app
li
cat
ion
,
or
co
ntext
whose
act
ual
s
ta
te
or
set
ti
ng
m
ay
hav
e
an
infl
uen
ce
on
Q
oE
f
or
t
he
us
e
r
[
3].
In
t
his
r
esearch
,
m
ob
il
e
us
er
s
wer
e
re
quest
ed
to
respo
nd
t
o
24
sta
tem
ents
gr
ou
ped
int
o
three
assess
m
ent
aspects:
serv
ic
e
qual
it
y,
data
sp
ee
d,
a
nd
st
abili
ty
of
sig
nal.
T
hes
e
as
sessm
ent
aspects
represe
nt
factors
in
flue
nc
ing
Q
oE.
For
eac
h
sta
tem
ent,
responde
nts
wer
e
aske
d
to
giv
e
a
scor
e
re
pr
e
s
enting
thei
r
subj
ect
iv
e
evalu
at
ion
s
of
4G
and
3G
m
ob
il
e
su
bs
c
riptions.
T
he
s
c
al
e
us
e
d
w
as
a
n
a
bs
ol
ute
cat
e
gory
rati
ng
(AC
R)
of
5
po
i
nts
,
gra
ded
as
fol
lows
:
5
=
ve
ry
good,
4
=
go
od,
3
=
norm
al/or
dina
ry,
2
=
poor,
a
nd
1
=
ver
y
po
or
.
Table
1
gi
ve
s
a
sel
ect
io
n
of
the
sta
tem
ents includ
e
d
i
n
the
que
sti
onnaire.
Table
1.
E
xam
ple Questi
on
na
ire Stat
em
ents
STA
TE
ME
N
T
3
G services
(ever
u
sed
)
4
G service
(curr
en
tly
us
ed
)
SERVIC
E
QU
AL
I
TY
AS
PECT
S
1
The so
u
n
d
qu
ality
when
m
ak
in
g
tr
ad
itio
n
al
phone
calls
(T
D
M
-
b
ased
)
1
2
3
4
5
1
2
3
4
5
2
Qu
ality
of
sen
d
in
g
and
r
eceivin
g
SMS a
n
d
M
MS
1
2
3
4
5
1
2
3
4
5
3
The q
u
ality
of
v
o
i
ce
co
m
m
u
n
ic
atio
n
/v
o
ice
u
sin
g
m
e
ss
en
g
er
ap
p
licat
io
n
s
(su
ch
as W
h
at
sAp
p
,
LI
N
E,
etc.)
1
2
3
4
5
1
2
3
4
5
DAT
A SP
EE
D
A
SPECT
9
Data sp
eeds
while
d
o
wn
lo
ad
in
g
/
d
o
wn
lo
a
d
f
iles
f
ro
m
e
m
ails,
webs
ites
1
2
3
4
5
1
2
3
4
5
10
Data sp
eeds
when
u
p
lo
ad
in
g
/
u
p
l
o
a
d
f
iles
f
ro
m
e
m
ails,
webs
ites
1
2
3
4
5
1
2
3
4
5
11
Deliv
er
y
sp
eed o
f
tex
t
u
sin
g
m
ess
en
g
er
ap
p
licatio
n
s
(su
ch
as W
h
atsAp
p
,
LI
N
E,
etc.
)
1
2
3
4
5
1
2
3
4
5
STABI
LI
T
Y O
F S
IGNA
L
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2502
-
4752
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci,
Vo
l.
10
, N
o.
3
,
June
2018
:
1203
–
1211
1206
Table
1.
E
xam
ple Questi
on
na
ire Stat
em
ents
STA
TE
ME
N
T
3
G services
(ever
u
sed
)
4
G service
(curr
en
tly
us
ed
)
22
Sig
n
al stab
ility w
h
ile
d
o
wn
lo
ad
in
g
/
d
o
wn
lo
a
d
f
iles
f
ro
m
e
m
a
ils, w
eb
sites
1
2
3
4
5
1
2
3
4
5
23
Sig
n
al stab
ility w
h
ile
u
p
lo
ad
in
g
/
u
p
lo
a
d
f
iles
f
rom
e
m
ails,
webs
ites
1
2
3
4
5
1
2
3
4
5
24
Sig
n
al stab
ility w
h
ile
Wi
-
Fi tethe
ring
(
h
an
d
set as a
m
o
d
e
m
/
perso
n
al
h
o
tsp
o
t)
1
2
3
4
5
1
2
3
4
5
In
t
he
quest
io
nnai
re,
res
pond
ents
are
reques
te
d
to
ex
press
wh
at
they
per
c
ei
ve
ind
i
vidual
ly
fo
r
bo
t
h
3G
a
nd
4G
ser
vice.
Using
the
responde
nts’
s
cor
es
,
abs
olu
te
cat
ego
ry
rati
ng
with
hi
dd
e
n
r
efere
nce
(
ACR
-
HR
)
was
cal
c
ulate
d
by
s
ub
t
racti
ng
the
3G
sco
res
from
t
he
4G
s
cor
es
.
T
he
difference
bet
wee
n
the
MOS
val
ue
of
the
4G
ser
vice
and
t
he
3G
s
erv
ic
e
is
an
ad
aptat
ion
of
t
he
Re
com
m
end
at
ion
ITU
-
T
P
.910.
IT
U
def
i
ne
s
the
ACR
-
HR
m
eth
od
as
a
cat
ego
ry
judgm
ent
wh
e
re
te
st
sequ
ence
s
are
pr
e
sented
one
at
a
tim
e
and
are
rate
d
ind
e
pende
ntly
on
a
cat
eg
or
y
s
cal
e
[14].
Si
nc
e
the
m
ai
n
QoE
assessm
ent
be
ing
s
ought
is
for
4G
t
he
ref
e
ren
ce
con
ta
ine
d
i
n
t
he
pro
ce
dure is
3G ser
vice.
The ACR
-
HR
scal
e u
se
d
in
th
is researc
h
t
o
tr
anslat
e the
qua
li
ta
ti
ve
com
par
ison val
ues
is a
s foll
ow
s:
a.
ACR
-
HR
r
a
ng
e from
-
5
t
o
-
2:
4G se
rv
ic
e is
per
cei
ved as
m
uch
w
orse
tha
n 3G
b.
ACR
-
HR
r
a
ng
e from
-
2
t
o
-
1:
4G se
rv
ic
e is
per
cei
ved as
w
orse
t
han 3G
c.
ACR
-
HR
equa
l t
o
0:
4G se
rv
i
ce is pe
rceive
d as
j
ust the
s
am
e
as
3G
d.
ACR
-
HR
r
a
ng
e from
1
to 2:
4G ser
vice is
pe
rcei
ve
d
as
bet
ter
tha
n 3
G
e.
ACR
-
HR
r
a
ng
e from
2
to 5:
4G ser
vice is
pe
rceive
d
as
mu
ch be
tt
er
t
han
3G
4.
RESU
LT
S
AND DI
SCUS
S
ION
4.1.
Basic Re
sp
ondents
’ Dat
a
Table
2
pr
ese
nt
s
the
ba
sic
dat
a
for
t
he
re
spo
nd
e
nts.
It
s
hows
that
t
he
m
ajorit
y
of
res
po
nd
e
nts
wer
e
m
al
e.
The
age
of
res
pondents
cl
os
el
y
fo
ll
ow
s
a
norm
al
distrib
ution,
an
d
s
a
m
pling
is
pr
e
do
m
inantly
of
tho
s
e
aged
19
a
nd
20.
The
data
obta
ined
show
s
that
86
%
of
res
pondents
ha
d
us
e
d
4G
f
or
m
or
e
tha
n
three
m
on
ths
wh
il
e
71.09%
of r
es
ponde
nts
us
e
d
the
sam
e
m
ob
il
e p
hone
b
ra
nds
for bo
t
h 3
G
a
nd 4G s
er
vices.
Table
2.
Basi
c
Inform
at
ion
f
or Res
pondents
No
Data
Percentag
e
1
Gen
d
er
5
1
.42
%
m
ale
4
8
.58
% f
e
m
ale
2
Ag
e
Ag
e r
an
g
e f
ro
m
9 t
o
50
,
p
redo
m
in
an
tl
y
19
and
20
3
Leng
th
of
ti
m
e
as
a 4G su
b
scrib
er
8
6
%
m
o
r
e than
3
m
o
n
th
s
1
4
% less th
an
3
m
o
n
th
s
4
Bran
d
of
4G h
an
d
s
et
7
1
.09
% u
sing th
e sa
m
e br
an
d
f
o
r
b
o
th
4G an
d
3G
5
Edu
catio
n
al level (
in
clu
d
in
g
o
n
g
o
i
n
g
stu
d
y
or alr
e
ad
y
gr
ad
u
ated
)
4
9
.76
% bach
elo
r
d
eg
ree
1
8
.25
% dip
lo
m
a/v
o
catio
n
al deg
ree
2
7
.01
% hig
h
sch
o
o
l
6
Occup
atio
n
6
1
.37
% stu
d
e
n
t
2
3
.70
% p
rivate e
m
p
lo
y
ee
4
.7 % state e
m
p
lo
y
ee
0
.47
% oth
er
7
4
G op
erators su
b
scrib
ed
to
3
6
.26
% T
elk
o
m
se
l
3
1
% I
n
d
o
sat
2
5
.83
% XL
4
.7%
T
h
ree
1
.6 % S
m
a
rtf
ren
0
.47
% Axis
8
Locatio
n
s
m
o
st
f
re
q
u
en
tly
us
ed
to
access 4
G se
rvices
3
8
.5 % at ho
m
e
2
7
.2 % at scho
o
l or of
f
ice
2
2
%
in
pu
b
lic f
aci
lities
1
2
.1 % in
veh
icles (while tr
av
elin
g
)
Evaluation Warning : The document was created with Spire.PDF for Python.
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci
IS
S
N:
25
02
-
4752
Lessons
Learn
ed
fr
om the
Q
uali
ty
o
f Ex
peri
ence (Q
oE)
Assessm
e
nt o
f
4G
… (
Muha
mma
d Su
r
ya
ne
ga
r
a
)
1207
The
s
urvey
f
ound
that
st
ud
e
nts
an
d
pr
ivate
com
pan
y
em
p
loye
es
(incl
ud
i
ng
sta
te
-
owne
d
com
pan
ie
s
and
f
or
ei
gn
e
m
plo
ye
es)
const
it
uted
the
m
ajorit
y
of
4G
se
rv
ic
e
res
ponde
nts
by
pro
fessi
on.
T
his
s
ugge
sts
that
stud
e
nts
a
re
th
e
res
pondents
who
m
os
t
easi
ly
acce
pt
ne
w
te
chnol
og
y.
I
n
te
rm
s
of
net
work
op
e
rato
r
s,
the
resu
lt
s
m
irro
re
d
the
act
ual
m
ark
et
of
six
m
ai
n
operat
or
s
,
in
wh
ic
h
Tel
ko
m
sel
is
a
dom
ina
nt
natio
nal
op
e
rato
r
with a m
ark
et
s
har
e
of m
or
e th
an on
e
thir
d.
The
basic
dat
a
al
so
sho
ws
the
locat
io
ns
m
os
t
fr
eq
uen
tl
y
us
e
d
to
acc
ess
4G
ser
vic
es.
T
he
m
os
t
com
m
on
ly
us
ed
locat
io
n
for
acce
ssing
4G
was
at
ho
m
e.
Mob
il
e
ph
on
e
s
are
te
le
com
m
un
ic
at
io
n
te
ch
no
l
og
y
innov
at
io
ns
de
velo
ped
for
m
ob
il
e
us
e
bu
t
a
re
curre
ntly
us
ed
to
acce
ss
in
form
ation
an
d
te
le
com
m
un
ic
a
ti
on
s
m
os
t of
te
n w
he
n
m
ob
il
e ph
one
us
er
s ar
e
at
ho
m
e.
4.
2.
Results
of
th
e
Questi
on
n
aire
Qo
E
fo
r
Ser
vice Qu
ality As
pects
Figure
1
li
sts
detai
ls
of
se
r
vice
qu
al
it
y
aspe
ct
s
with
their
ACR
-
HR
sc
or
e
resu
lt
s.
O
veral
l,
it
can
be
seen
that
none
of
the
sc
or
es
e
xceed
1.5.
T
he
resu
lt
s
sign
ify
that
4G
ser
vice
qu
al
it
ie
s
are
per
cei
ved
as
“
bette
r”
than
3G,
bu
t
w
it
h
the
lo
west
ACR
-
HR
sco
r
es
giv
e
n
for
tr
aditi
on
al
vo
ic
e
-
cal
l
sound
qual
it
y
and
SMS
serv
ic
e
qu
al
it
y.
These r
esults
are
not
su
r
pr
isi
ng
sin
c
e
vo
ic
e
cal
ls
in
the
In
do
nesian
m
ob
il
e
networ
k
are
st
il
l
carried
on
the
2G
netw
ork,
4G
-
base
d
vo
ic
e
te
chn
ology
(su
c
h
as
VoL
TE)
ha
ving
not
ye
t
been
i
m
pl
e
m
ented.
The
current
vo
ic
e
-
cal
l
platfo
rm
is
ci
rcu
it
-
switc
he
d
(CS
)
fall
bac
k,
w
hi
ch
ena
b
le
s
the
prov
isi
on
of
vo
ic
e
a
nd
tra
di
ti
on
al
ci
rcu
it
-
s
witc
he
d
ser
vices.
Fig
ur
e
1
al
s
o
in
di
cat
es
that
us
e
r
s
ap
pr
eci
at
e
th
e
sign
i
ficant
s
erv
ic
e
im
pr
ove
m
ents
pro
vid
e
d
by
4G
f
or
vi
deo
cal
ls
and
m
ultim
e
dia
stream
ing
(
su
c
h
as
YouT
ube,
J
O
OX,
S
poti
fy,
et
c.)
s
ugge
sti
ng
that
sign
ific
an
t
te
chn
ic
al
adv
ance
has
pro
vid
e
d
a
bette
r
exp
erie
nce
f
or
4G
s
ub
sc
rib
ers.
A
dd
it
io
nal
ly
,
the
Ind
on
esi
a
n
m
ark
et
al
s
o
a
ppr
eci
at
es
the
im
pro
vem
ent
of
the
s
ound
qu
al
it
y
fo
r
voic
e
c
al
ls
us
in
g
m
essen
ger
app
li
cat
io
ns
.
H
ow
e
ve
r,
t
he
a
ve
rag
e
sco
re
fo
r
this
as
pect
is
qu
it
e
lo
w
(
1.0
ou
t
of
5.0)
,
in
di
cat
ing
a
sm
al
l
le
vel
of im
pr
ov
e
d Q
oE w
hen com
par
ed
w
it
h 3
G.
Figure
1. The
QoE r
e
su
lt
s
for
servic
e quali
ty
asp
ect
s
4.2.2. QoE
for
Data S
peed
A
spects
Figure
2
prov
i
des
detai
ls
of
t
he
data
sp
ee
d
aspects
a
n
d
t
he
ir
ACR
-
HR
s
cor
e
res
ults.
O
ver
al
l,
it
ca
n
be
seen
that
th
e
scor
es
ra
nge
from
1.
0
to
1.5
.
All
the
4G
da
ta
sp
eed
as
pects
are
per
cei
ved
as
“bett
er”
than
3G
data
sp
ee
d,
w
hi
le
no
si
ng
le
e
xp
e
rience
is
pe
rceive
d
as
“
worse”.
T
he
m
axim
u
m
ACR
-
HR
sco
res
are
gi
ve
n
for
the
ex
per
ie
nce
of
uploa
ding
file
s,
do
wn
l
oa
ding
file
s
a
nd
W
i
-
Fi
te
the
rin
g
(
us
in
g
ha
ndset
s
as
pe
rsonal
ho
ts
pots).
The
resu
lt
s
i
nd
ic
at
e
that
cu
rr
e
nt
i
m
ple
m
entat
ion
has
f
ulfill
ed
t
he
basic
purpo
se
of
4G
te
ch
nolo
gy,
wh
ic
h
is
t
o
e
nh
a
nce
data
c
omm
un
ic
at
ion
s
over
previ
ou
s
m
ob
il
e
te
chn
ol
og
y
e
voluti
on
s
.
Tec
hnic
al
ly
,
4G
sh
oul
d
accom
plish
data
c
om
m
un
ic
at
ion
s
of
up
to
1
G
bps
in
sta
ti
on
a
r
y
sta
te
s
and
100
M
bp
s
i
n
m
ov
ing
conditi
ons.
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2502
-
4752
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci,
Vo
l.
10
, N
o.
3
,
June
2018
:
1203
–
1211
1208
Figure
2. The
QoE r
e
su
lt
s
for
d
at
a s
pee
d
as
pe
ct
s
Howe
ver,
we
cannot
judge
the
cu
rr
e
nt
sub
j
ect
ive
Q
oE
a
s
infer
i
or
,
bec
ause
the
res
ults
ind
ic
at
e
a
scor
e
of
“
bette
r”,
al
thou
gh
t
he
scor
e
f
or
“be
tt
er”
is
on
ly
j
ust
beyond
the
“
sam
e
as
bef
ore
”
range.
I
n
fact
,
m
os
t
of
t
he
I
ndonesi
an
m
ark
et
has
enjoyed
3.5
G
te
chnolo
gy
(s
uc
h
as
hi
gh
-
s
pee
d
pac
ket
acce
s
s,
or
H
SP
A)
,
wh
ic
h
can,
i
n
s
om
e cases,
giv
e
data
sp
ee
d of
hund
r
eds of M
bps.
Qo
E
fo
r
N
e
tw
ork Qu
ality A
spects
Figure
3
prov
i
des
detai
ls
of
t
he
netw
ork
qual
it
y
aspects
and
thei
r
ACR
-
HR
sco
re
res
ults.
Ov
e
rall
,
it
can
be
seen
t
ha
t
the
sco
res
do
not
ex
ceed
1.2.
The
s
ub
j
ect
ive
ind
ic
at
io
n
of
netw
ork
qua
li
ty
is
rep
re
sen
te
d
by
the
te
rm
“st
abi
li
ty
of
sign
al
”.
Mob
il
e
subsc
r
ibers
ca
n
easi
ly
j
udge
t
his
ba
sed
on
te
c
hn
ic
al
ind
ic
at
ors
in
their
handsets. T
he
r
esults sh
ow
tha
t t
he
sta
bili
ty o
f
4G sign
al
s is
per
cei
ved
as “
bette
r” th
an 3
G
ser
vices. Ho
wev
e
r,
the
net
work
qu
al
it
y
fo
r
tra
diti
on
al
voic
e
cal
ls
has
a
sco
re
of
le
ss
tha
n
1.0.
Su
c
h
a
near
-
to
-
zer
o
sc
or
e
i
ndic
at
es
that
us
e
rs’
e
xp
erience
of
4G
is
alm
os
t
the
sa
m
e
as
their
e
xp
e
rience
of
3G.
T
his
is
c
onsist
ent
with
th
e
Q
oE
resu
lt
s
for
the
serv
ic
e
qual
it
y
aspect,
si
nce
t
he
vo
ic
e
-
cal
l
ne
twork
is
sti
ll
run
ning
on
t
he
2G
net
wor
k
(
us
in
g
t
he
unde
rly
ing
CS
fall
back
m
echan
ism
).
Anothe
r
intere
sti
ng
fi
nd
i
ng
i
s
that
the
Ind
onesi
an
m
ark
et
reg
a
rds
data ser
vice as
bein
g
m
or
e sta
ble tha
n vo
ic
e
serv
ic
e.
T
his is, th
ere
f
or
e
, a
re
li
able ind
ic
at
or
f
or m
ob
il
e n
et
wor
k
op
e
rato
rs
to
c
onside
r
the
im
pl
e
m
entat
i
on
of
vo
ic
e
c
omm
un
ic
at
ion
s
ser
vic
es,
su
c
h
as
VoLTE,
wh
ic
h
ar
e
fu
ll
y
delivere
d o
ver
the 4G
data
net
work.
Evaluation Warning : The document was created with Spire.PDF for Python.
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci
IS
S
N:
25
02
-
4752
Lessons
Learn
ed
fr
om the
Q
uali
ty
o
f Ex
peri
ence (Q
oE)
Assessm
e
nt o
f
4G
… (
Muha
mma
d Su
r
ya
ne
ga
r
a
)
1209
Figure
3. Q
oE results
for net
work qual
it
y aspect
s
5.
LE
SSONS LE
ARNED
In
the
fi
nal
par
t
of
the
quest
ionnaire
,
the
re
sp
on
de
nts
were
asked
to
gi
ve
gen
e
ral
Q
oE
con
cl
us
i
ons
for
their
c
urre
nt
4G
subsc
rip
ti
on
.
T
he
re
su
l
ts
ind
ic
at
ed
th
at
59
.
48%
of
r
esp
onde
nts
pe
r
cei
ved
4G
ser
vice
as
“bett
er”
than
3G,
w
hile
34.36
%
of
res
ponde
nts
per
cei
ved
4G
ser
vice
is
“
m
uch
bette
r”
t
han
thei
r
pr
e
vi
ou
s
3G
serv
ic
e.
He
nce,
I
ndon
esi
a
n
op
erators can
be
seen to
hav
e
de
li
ver
ed
si
gn
ifi
cant p
er
f
or
m
ance i
m
pr
ov
em
ents f
or
4G
ser
vices,
in
that
us
e
rs
noti
ce
posit
ive
diff
e
re
nces
a
nd
per
cei
ve
the
m
as
being
be
tt
er
than
3G.
Fo
r
eac
h
assessm
ent
po
i
nt
in
vestigat
ed
,
the
data
s
peed
an
d
netw
ork
qual
it
y
aspects
pro
vid
e
d
po
sit
i
ve
ACR
-
HR
sc
or
es
.
Su
c
h
data
sho
uld
be
easi
ly
unde
rstoo
d
an
d
translat
ed
int
o
strat
egic
ref
e
re
nces
f
or
m
ob
il
e
networ
k
oper
at
or
s.
QoE
data
is
al
so
us
e
f
ul
for
optim
iz
ing
products
an
d/or
ser
vices
[2
]
,
beca
us
e
te
ch
nical
te
a
m
s
m
a
y
no
t
h
ave
had
s
uffici
ent
knowle
dge
a
bout
t
he
s
pecifi
c
te
chn
ic
al
para
m
et
ers
that
a
ff
ect
us
er
ex
pe
rience.
S
om
e
of
the
m
ai
n
po
i
nts th
at
can be
furth
er elab
or
at
e
d
a
s strategic
ref
e
ren
ces
are
as
f
ollows:
a.
Op
e
rato
rs
s
houl
d
pay
s
pecial
at
te
ntion
to
t
he
aspect
of
network
qual
it
y,
since
sta
bili
ty
of
sig
nal
recei
ve
s
a
lowe
r
ACR
-
HR
sco
re.
T
hi
s
su
r
vey
re
su
lt
m
ay
al
so
ind
ic
at
e
that
su
bsc
ribers
ha
ve
e
xperie
nced
s
om
e
blan
k
spots
as
well
as
un
sta
bl
e
cov
e
rag
e
.
O
ne
possible
ca
us
e
w
ou
l
d
be
that,
as
the
devel
op
m
ent
of
4G
sta
rted
on
ly
a
few
ye
ar
s
ag
o,
4G
co
ver
a
ge
i
s
not
as
exte
nsi
ve
as
3G
c
ove
rag
e
.
It
is
the
r
efore
s
uggeste
d
that I
ndonesi
an
operat
or
s
sho
ul
d
acce
le
rate i
nfrastr
uctu
re
ro
l
lou
t.
b.
QoE
data
is
al
so
us
e
f
ul
for
optim
iz
ing
and
i
m
pr
ov
in
g
cu
rrent
data
ser
vices.
Tech
nical
te
a
m
s
of
te
n
f
oc
us
on
the
pr
oduct,
bu
t
do
no
t
ha
ve
m
uch
know
l
edg
e
a
bout
the
sp
eci
fic
te
chn
i
cal
par
am
et
ers
that
aff
ect
end
-
us
er
ex
per
ie
nc
e.
It
is
perha
ps
belie
ve
d
that
us
e
r
ex
pe
rien
ce
is
a
sub
j
ect
ive
m
easur
e,
but
it
can
al
s
o
be
analy
zed
obj
e
ct
ively
.
In
ot
he
r
w
ords
,
Qo
E,
w
hic
h
re
fle
ct
s
the
m
easur
em
ent
of
us
e
r'
s
ps
yc
holo
gical
feeli
ng
s
,
s
houl
d
al
s
o
be
e
xpr
essed
i
n
te
rm
s
of
QoS
c
har
a
ct
erist
ic
s.
I
n
t
his
way,
data
for
QoE
ca
n
be
com
bin
ed
with
te
chn
ic
al
QoS
pa
ram
et
ers,
pro
vid
in
g
valu
able
inf
orm
ati
on
ab
out
us
e
r
exp
e
rience
.
F
or
exam
ple,
this
Qo
E
as
sess
m
ent
has
ind
i
cat
ed
that
m
o
st
su
bs
c
ribe
rs
are
act
ively
us
in
g
m
esse
ng
e
r
app
li
cat
io
ns
suc
h
as
Wh
at
sA
pp
for
voic
e
-
c
al
l
co
m
m
un
ic
at
ion
s.
An
y
po
orer
te
ch
nical
pe
rfor
m
ance,
suc
h
as
lo
ng
e
r
delay
s
an
d
lo
wer
powe
r
sig
nals,
will
cause
a
degra
d
ed
sou
nd
qu
al
it
y
that
is
sub
j
ect
ivel
y
per
cei
ved
as
i
ncon
ven
ie
nt
by
us
ers
.
Hen
ce
,
te
ch
nical
qua
li
ty
enh
ancem
ents
are
nee
de
d
to
sat
isfy
us
ers’
su
bject
ive
exp
eriences i
f vo
ic
e comm
un
ic
at
i
on
s
servic
es a
r
e to
be pre
ferre
d ov
e
r
m
essen
ger ap
plica
ti
ons.
c.
QoE
data
can
help
in
ov
e
rcom
ing
the
pro
bl
e
m
of
hig
h
c
hur
n
rates.
Profits
an
d
bran
d
i
m
age
will
be
neg
at
ively
a
ff
e
ct
ed
if
se
rv
ic
e
consi
ste
ntly
fail
s
to
m
eet
con
su
m
er
exp
ect
at
ion
s
.
Opp
or
t
un
i
ti
es
to
de
vel
op
custom
ers’
loya
lt
y
will
be
increased
by
de
velo
ping
im
pr
ov
em
ent
s
in
current
ser
vices
with
the
help
of
QoE
data.
F
or
exam
ple,
the
su
r
vey
sho
ws
that
custom
ers
value
data
spe
ed
f
or
m
ob
il
e
te
therin
g
(
us
i
ng
m
ob
il
e
handse
t
as
W
i
-
Fi
ho
t
sp
ots
).
He
nce,
operat
or
s
s
hould
pay
at
te
nti
on
to
su
c
h
as
pects
of
m
ark
et
beh
a
vior
i
n
I
ndone
sia
,
w
her
e
co
ns
ist
ent
de
velo
pm
ent
of
t
he
releva
nt
ser
vice
will
be
li
kely
to
inc
reas
e
br
a
nd
loya
lt
y.
O
n
the
ot
her
ha
nd,
a
c
hai
n
of
ne
gative
reacti
on
s
m
ay
easi
ly
occur
wh
e
n
t
he
m
obil
e
netw
ork o
per
at
or
s
, as wel
l as
serv
ic
e
de
velo
per
s
, a
re
un
a
w
are
of
su
c
h p
he
n
om
ena.
d.
QoE
can
be
use
d
as
a
ref
ere
nc
e
fo
r
f
uture
s
erv
ic
e
ap
plica
ti
on
s
.
Fu
tu
re
se
rv
ic
es
will
be
adopted
by
the
m
ark
et
,
or
on
the
oth
e
r
hand
,
re
j
ect
ed
by
the
m
ark
et
.
O
ne
reason
for
r
ejecti
on
is
that
consum
ers
can
exp
e
rience
a
l
ow
e
r
ser
vice
exp
e
rience
wh
e
n
t
hey
sta
rt
to
util
iz
e
a
new
serv
ic
e.
S
uc
h
conditi
on
s
m
ay
occur
e
ve
n
if
the
s
er
vice
has
receive
d
po
sit
ive
fe
edb
ac
k
duri
ng
m
ark
et
te
sti
ng
a
nd
/
or
pre
-
com
m
ercial
iz
a
ti
on
ad
ver
ti
sin
g.
S
uch
a
re
j
e
ct
ion
can
be
predict
ed
a
nd
a
vo
i
ded
if
Q
oE
data
is
analy
z
ed
accuratel
y an
d i
n
detai
l t
hro
ug
h user t
est
ing b
efore the
pr
oduc
t i
s laun
c
he
d.
e.
QoE
assessm
ent
has
al
s
o
s
hown
t
hat
subsc
r
ibers
a
ppreciat
e
faster
data,
s
uggestin
g
that
op
e
rato
rs
s
houl
d
pr
i
or
it
iz
ing
relevan
t
data
co
m
m
un
ic
at
ion
app
li
cat
io
ns
.
Howe
ver,
m
ob
il
e
netw
ork
op
e
rato
rs
need
to
su
r
viv
e
in
t
he
era
of
O
TT
(
ov
er
-
the
-
to
p)
c
on
te
nt.
This
pres
ents
a
chall
enge,
as
m
ob
il
e
operators
ha
ve
ha
d
to
inv
est
to
buil
d
inf
rastr
uct
ur
e
,
but
do
no
t
enjoy
sign
ifi
cant
rev
e
nue
f
ro
m
the
co
m
m
ercial
iz
ation
of
app
li
cat
io
ns
, b
ecause s
ubscri
ber
s
are
p
ay
in
g f
or
t
he data
ne
twork
, not t
he se
rv
ic
e a
ppli
cat
ion
s.
6.
CONCL
US
I
O
N
This
pa
pe
r
ha
s
pr
ese
nted
Q
oE
assessm
ent
resu
lt
s
f
or
t
he
4G
m
ob
il
e
te
chn
ol
og
y
that
ha
s
bee
n
r
olled
ou
t
in
the
I
ndonesi
a
n
m
ark
et
.
A
m
ark
et
su
r
vey
has
be
en
co
nducte
d
with
a
confid
en
ce
le
vel
of
95
%
,
inv
est
igati
ng
t
he
4G
s
ub
sc
ri
ber
in
t
he
Ind
on
e
sia
m
ark
et
.
Th
ree
m
ai
n
su
bject
ive
aspe
ct
s
of
Q
oE
ass
essm
ent
hav
e
been
in
ve
sti
gated:
serv
ic
e
qu
al
it
y,
data
sp
ee
d
an
d
net
work
qu
al
it
y
(ind
ic
at
ed
by
sta
bili
ty
of
sign
al
).
The
analy
ses
wer
e
condu
ct
e
d
ba
sed
on
ACR
-
HR
sco
re,
co
m
par
ing
us
e
rs
’
ex
per
ie
nces
betwee
n
4G
a
nd
the
i
r
pr
e
vious
3G
s
ub
s
cripti
ons
.
Fo
r
the
se
rv
ic
e
qu
al
it
y
aspect,
the
ACR
-
HR
scor
e
s
we
re
posit
ive
but
nev
e
r
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2502
-
4752
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci,
Vo
l.
10
, N
o.
3
,
June
2018
:
1203
–
1211
1210
exceede
d
1.5.
This
res
ult
sig
nifies
that
4G
serv
ic
e
qu
al
it
ie
s
are
pe
rceive
d
as
“
bette
r”
than
3G,
bu
t
w
it
h
the
lowest
rati
ng
s
in
this
as
pect
f
or
tra
diti
onal
voic
e
cal
ls
an
d
SMS.
F
or
t
he
data
sp
ee
d
as
pe
ct
,
the
sc
or
es
range
d
from
1.
0
to
1.
5,
i
nd
ic
at
in
g
t
he
se
rv
ic
es
a
r
e
per
cei
ve
d
as
“bett
er”
tha
n
3G
data
s
pee
d.
In
this
as
pe
ct
,
the
m
axi
m
u
m
AC
R
-
HR
sco
res
wer
e
giv
e
n
f
or
the
exp
e
rienc
e
of
uploa
ding
file
s,
dow
nloa
ding
file
s
an
d
W
i
-
Fi
te
therin
g
(
us
in
g
ha
ndset
s
as
per
s
onal
hotsp
ots).
For
the
ne
twork
qual
it
y
aspect,
the
Q
oE
asses
sm
ent
had
no
scor
e
s
that
exc
eed
1.2,
with
the
netw
ork
qu
al
it
y
of
trad
it
io
nal
vo
ic
e
cal
ls
hav
i
ng
a
sco
re
of
bel
ow
1.0.
Su
c
h
a
near
-
to
-
zer
o
sc
or
e
i
nd
ic
at
es
t
hat
the
c
urren
t
ex
per
ie
nc
e
f
or
m
aking
tra
di
ti
on
al
voic
e
c
al
ls
is
felt
to
be
no
diff
e
re
nt
to
the
pr
e
vious
3G
s
erv
ic
e.
To
co
nc
lud
e,
base
d
on
Q
oE
assessm
ent
res
ults
f
or
t
hes
e
th
ree
as
pe
ct
s
of
us
er
e
xperie
nc
e,
the
I
ndonesi
an
m
ark
et
per
c
ei
ves
4G
as
pr
ov
i
ding
a
bette
r
ex
per
ie
nce
th
an
3G
ser
vice.
The
QoE
assessm
ent
res
ults
can
be
f
ur
t
her
de
ve
lop
e
d
int
o
str
at
egic
le
ssons
for
m
ob
il
e
network
operat
or
s
.
From
these
five
le
ss
on
s
,
a
par
ti
cul
arly
i
m
po
rtant
issue
hi
gh
li
ght
ed
is
that
I
ndonesi
an
operat
ors
sho
uld
pay
sp
eci
al
at
te
ntion
t
o
the
asp
ect
of n
et
w
ork q
ualit
y, since the
ACR
-
H
R sco
re fo
r
sta
bili
ty
o
f
sig
nal
is l
ow.
REFERE
NCE
S
[1]
Inte
rna
ti
ona
l
Te
l
ec
om
m
unic
at
ion
s
Union
(IT
U).
Am
endment
5:
New
def
ini
ti
ons
for
inc
lusion
in
Rec
om
m
enda
tio
n
ITU
-
T
P.10G.1
00.
ITU
-
T
P.1
0/G.
100.
T
elec
om
m
unic
at
ion
Standa
rdi
za
t
ion
sec
tor
of
IT
U.
2016.
[Onli
ne]
.
Avail
ab
le
f
rom
: w
ww
.
it
u.
int
[2]
Europe
an
Telecom
m
unic
at
ions
Standa
rds
Institute
(E
TSI).
Hu
m
an
Fact
ors
(HF
):
Quali
t
y
of
Expe
ri
enc
e
(QoE
)
req
uire
m
ent
s
fo
r
rea
l
-
t
ime
comm
unic
at
ion
servic
es.
E
TSI
-
T
R
102
643.
2
010.
[Online
]
.
Avail
ab
le
from
:
ww
w.e
tsi.
org
[3]
Brunnström
,
K.,
et
al.
Quali
n
et
W
hit
e
Paper
on
Defi
nit
ions o
f
Quali
t
y
of
Exp
erie
nce
.
Outpu
t
from
the
fift
h
Qualine
t
m
ee
ti
ng,
Marc
h
12,
2013
.
[Onlin
e]
.
Available:
h
ttps
:/
/ha
l
.
ar
chi
v
esouve
rte
s.fr
/hal
-
0
0977812/doc
um
ent
[4]
Sur
y
ane
g
ara,
M.
and
As
via
l
,
M.
“
In
Sear
chi
ng
f
or
4G
Mobile
Service
Appli
ca
t
i
ons:
The
Case
o
f
the
Indon
esia
n
Marke
t,”
Te
le
co
mm
unic
ati
ons J
o
urnal
of Australi
a
,
vol
.
63
,
pp
.
39
.
1
–
39.
16
,
2013
.
[5]
Sur
y
ane
g
ara,
M.
,
Andri
y
a
nto,
F.
and
W
ina
rko,
B.
“
W
hat
Chang
es
aft
er
sw
it
chi
n
g
to
4G
-
LT
E
?
F
indi
ngs
from
the
Indone
sian
Mark
et
,
”
I
EE
E
A
ccess
,
vol
.
5
,
pp
.
1707
0
-
17076,
2017
.
[6]
Inte
rna
ti
ona
l
T
el
e
comm
unic
at
i
ons
Union
(IT
U).
Ref
e
ren
ce
guide
to
qu
al
ity
of
expe
ri
enc
e
assess
m
en
t
m
et
hodologi
es.
I
TU
-
T
G.1011.
T
el
e
comm
unic
at
i
on
Standa
rdi
za
t
i
on
sec
tor
of
ITU
.
2016.
[Onlin
e]
.
Avail
ab
le
from
:
ww
w.i
tu.
int
[7]
Mobiu,
Aim
an,
and
Hart
ant
o
,
R
ud
y
.
“
Anal
y
s
is
q
ual
ity
of
serv
ice
from
Inte
rne
t
pr
otoc
ol
televi
sion
(IPTV)
service,”
Inte
rnational
Jo
urnal
of
In
formatic
s and
Comm
unic
ati
on
Techno
l
ogy
(
IJ
-
ICT)
,
Vol
1,
No.
2,
pp.
10
0
-
108,
2012
.
[8]
Inte
rna
ti
ona
l
T
e
le
comm
unic
atio
ns
Union
(IT
U).
O
bje
c
ti
ve
per
c
ept
ua
l
m
ult
imed
ia
vid
eo
qu
al
i
t
y
m
ea
surem
ent
o
f
HD
TV
for
dig
i
ta
l
ca
b
le
televis
ion
in
th
e
pr
ese
nce
of
a
fu
ll
r
efe
ren
ce.
I
T
U
-
T
J.341.
Tel
ec
om
m
unic
at
ion
Standa
rdi
za
t
ion s
ec
tor
of
ITU
.
2
016.
[Onlin
e]
.
A
vai
l
abl
e
from
:
w
ww
.
it
u.
in
t
[9]
Shakir,
Mus
ta
fa
,
Obaid
Ur
Rehman,
Z
ee
shan
Abb
as,
Abdulla
h
M
a
sood,
and
W
aj
e
e
ha
Shahid.
“
Ev
a
l
uat
ion
of
Vid
eo
Quali
t
y
in
W
irel
ess
Multi
m
edi
a
Sensor
Networks,”
Int
ernati
onal
Journal
of
Elec
tric
al
and
Computer
Engi
n
ee
rin
g
(
IJE
CE)
,
vol. 6,
no.
1
,
pp
.
223,
20
16.
[10]
The
Offic
e
of
Com
m
unic
at
ions
(OF
C
OM
).
M
ea
suring
m
obil
e
broa
dband
per
f
orm
anc
e
in
the
UK
:
4G
and
3
G
net
work pe
r
form
anc
e
.
[Onl
ine
]
.
Avail
ab
le
f
rom
: w
ww
.
ofc
om
.
org.
uk
[11]
Inte
rna
ti
ona
l
Telec
om
m
unic
atio
ns
Union
(IT
U
).
Mea
n
Opinio
n
Score
(MO
S)
Te
rm
inol
og
y
.
ITU
-
T
P.800.
1.
Te
l
ec
om
m
unic
ation
Stand
ard
ization sec
tor
of
I
TU
.
2016
.
[Onlin
e]
.
Available from:
ww
w.
it
u.
int
[12]
The
Offi
ce
of
C
om
m
unic
at
ions
(OF
COM
).
Mea
s
uring
m
obil
e
vo
i
ce
and
da
ta
qu
al
i
t
y
of
exp
erienc
e
.
2013.
[Onlin
e]
.
Avail
ab
le
f
rom
: w
ww
.
ofc
om
.
org.
uk
[13]
Mali
novski,
To
ni,
Marina
Vas
il
ev
a,
and
Vla
dimir
Tra
jkovi
k
.
“
Student
s’
Perc
eptions
during
Inte
gra
t
ion
o
f
Com
pute
r
Gam
es
in
Prim
ar
y
Ed
uca
t
ion:
QoE
Anal
y
s
is,”
Inte
rna
ti
onal
Journal
o
f
Informatic
s
an
d
Comm
unic
ati
o
n
Technol
ogy
,
Vol
.
3
,
No.
1,
pp.
13
-
22,
2014
.
[14]
Inte
rna
ti
ona
l
Telec
om
m
unic
atio
ns
Union
(IT
U)
.
Subje
ct
iv
e
v
id
eo
qua
li
t
y
assess
m
ent
m
et
hods
for
m
ult
imedia
a
pplications.
IT
U
-
T
P.910.
Te
l
ec
om
m
unic
at
ion
Standa
rdizat
ion
sec
tor
of
ITU.
2008.
[Online
]
.
Avail
abl
e
from
:
ww
w.i
tu.
int
Evaluation Warning : The document was created with Spire.PDF for Python.
Ind
on
esi
a
n
J
E
le
c Eng &
Co
m
p
Sci
IS
S
N:
25
02
-
4752
Lessons
Learn
ed
fr
om the
Q
uali
ty
o
f Ex
peri
ence (Q
oE)
Assessm
e
nt o
f
4G
… (
Muha
mma
d Su
r
ya
ne
ga
r
a
)
1211
BIOGR
AP
HI
ES OF
A
UTH
ORS
Muham
m
ad
Sur
y
an
ega
r
a
(Senio
r
Mem
ber
IEEE)
e
arn
ed
h
is
bac
he
lor’s
degr
ee
in
elec
tri
c
al
engi
ne
eri
ng
fro
m
Univer
sita
s
I
ndonesia
in
2003,
his
m
aste
r’s
degr
ee
from
Univer
sit
y
Coll
ege,
London,
UK
,
i
n
2004
and
h
is
Ph.D.
from
T
ok
y
o
Inst
it
ut
e
o
f
Technol
og
y
,
Japa
n,
in
2011
.
Curre
ntly
,
he
is
a
lectur
e
r
in
th
e
Depa
rtment
of
E
le
c
tri
c
al
En
ginee
ring,
Univer
si
ta
s
Indone
sia
.
H
e
has
publi
shed
m
an
y
a
ca
demi
c
pa
per
s
in
in
te
rn
at
i
onal
journ
al
s
an
d
proc
e
edi
ngs
as
a
m
ai
n
aut
hor
or
co
-
aut
hor
.
Hi
s
int
ere
sts
are
m
obil
e
and
wir
ele
ss
com
m
unic
at
io
n
te
chnol
og
y
,
bo
th
conc
e
rning
te
chn
ic
a
l
r
ese
ar
c
h
and
r
egulator
y
poli
c
y
m
an
age
m
ent
.
Fer
y
Andri
y
an
to
ea
rne
d
his
ba
ch
el
or’s
degr
e
e
in
industri
al
engi
n
e
eri
ng
from
Univer
sita
s
Gadj
ah
Mada
in
2007
a
nd
Master
degr
e
e
in
Te
l
ec
om
m
unic
a
ti
ons
Mana
g
ement,
Fa
cul
t
y
of
Engi
ne
eri
ng
Univer
sita
s Indo
nesia
in
2017.
C
urre
ntly
he is a sta
t
e
-
emplo
y
ee w
it
h
Mini
str
y
of
I
nform
at
ion
and
Com
m
unic
at
ions Re
public
of
In
donesia
.
Ajib
Set
y
o
Arif
in
(Mem
ber
IE
EE
)
e
arn
ed
his
bac
he
lor’s
degr
e
e
in
e
le
c
trica
l
e
ngine
er
ing
and
m
aste
r’s
degr
ee
in
T
el
e
communic
a
ti
ons
from
Univer
sita
s
In
donesia
in
200
9
and
201
1
respe
ctively
.
He
ea
rne
d
Ph.D.
fro
m
Keio
Univer
si
t
y
,
Jap
an,
in
201
5.
Curre
n
tly
,
he
is
a
l
ec
tu
rer
in
the
Dep
art
m
ent
of
El
e
ct
r
ic
a
l
En
gine
er
ing,
Univ
e
rsita
s
Indone
si
a.
His
m
ai
n
rese
ar
ch
intere
sts
is
W
ire
le
ss
Sensor
Network
as
we
ll
as
it
s
technical
a
ppli
c
at
io
ns on
I
CT
industr
ia
l
se
ct
or.
Evaluation Warning : The document was created with Spire.PDF for Python.