In
te
r
n
ation
a
l Jou
rn
al
o
f E
v
al
u
a
t
i
on
a
n
d
R
e
se
arc
h
in
Ed
u
c
ation
(
IJERE
)
V
o
l.7,
N
o.1,
M
a
r
ch
2018,
pp.
57~
6
4
ISSN
: 2252-
88
22,
D
O
I
:
10.11
59
1
/ijer
e
.
v
7
.
i
1
.11
0
4
5
57
Jou
rn
a
l
h
o
me
pa
ge
:
ht
tp:
//i
a
e
score
.
com
/
j
o
u
r
na
l
s
/
i
n
d
e
x
.
p
hp/IJ
ERE
Stude
nt Achievement in Turke
y
,
Accordin
g to Question Types
Used in PISA 2003-2012 Mat
hematic Literacy
T
ests
Yeşi
m Özer
Ö
z
k
a
n
1
, Nesrin
Özas
l
a
n
2
1
Dep
artm
en
t
o
f
Edu
cati
onal M
easu
r
em
ent
and Evaluat
i
o
n
, G
aziante
p
Univers
i
t
y
, Tu
rk
ey
2
M
i
n
ist
r
y of
Na
tional Educat
i
o
n,
T
urkey
Art
i
cl
e In
fo
ABSTRACT
A
r
tic
le hist
o
r
y
:
Re
ce
i
v
e
d
Jan
2
, 2018
Re
vise
d F
e
b 13,
201
8
A
c
c
e
pte
d
F
eb 27,
2
0
1
8
Th
e
ai
m
of
t
h
i
s
st
ud
y
i
s
t
o
det
e
rm
in
e
th
e
l
e
vel
of
achiev
e
m
e
nt
of
s
t
uden
t
s
part
ici
p
ati
n
g
i
n
P
rogramme
f
or
I
nternat
i
ona
l
S
t
udent
Asses
s
m
e
nt
(
P
I
S
A
)
20
03
a
nd
P
ISA
20
12
t
es
ts
i
n
Turk
ey
acc
o
r
din
g
t
o
q
u
estio
n
s
i
n
th
e
m
a
th
ematical
l
iteracy
t
es
t.
T
h
i
s
s
t
u
d
y
is
a
d
es
crip
tiv
e
s
urvey
.
W
i
t
hin
the
scope
of
t
he
s
tudy,
the
mat
h
emati
cal
literacy
t
est
i
t
e
m
s
were
c
l
a
ssified
a
s
m
u
lti
p
le-ch
o
i
ce,
c
o
m
pl
ex
m
ult
i
p
l
e-ch
oice
and
co
ns
truct
e
d
resp
on
se
ite
ms
accord
ing
t
o
t
he
d
iff
e
ren
t
q
u
e
s
t
ion
t
ypes
.
T
he
r
atio
o
f
correct
a
nd
p
artially
correct
a
n
d
i
nc
orrect
r
espo
ns
e
giv
e
n
t
o
eac
h
q
u
est
i
o
n
t
yp
e
h
a
s
b
een
det
e
rm
ined
.
F
i
n
d
ing
s
sho
w that
t
he ach
ievem
e
n
t
s
of
s
tu
dents
d
i
f
f
e
r accordi
n
g
t
o
d
i
f
f
e
r
e
n
t
t
y
p
e
s
o
f
q
u
e
s
t
i
o
n
s
.
W
h
i
le
t
he
q
ues
t
io
n
typ
e
w
i
t
h
t
he
h
ig
he
st
success
aver
age
in
t
he
P
ISA
2003
t
e
s
t
w
as
m
u
ltipl
e
-c
hoi
ce,
s
t
u
d
e
n
t
s
g
o
t
t
h
e
hi
ghest
scores
from
complex
multip
le-choice
questi
on
s
in
t
he
P
I
S
A
2
0
12
tes
t
.
The
ques
t
ion
n
ai
re
w
it
h
th
e
low
e
st
s
u
ccess
averag
e
was
fo
u
nd
t
o
be
complex
multipl
e
-choice
qu
est
i
ons
i
n
t
h
e
PI
SA
2
003
t
est
whi
l
e
s
tud
e
nt
s
go
t
the
low
e
st
s
cores
from
constructed
response
i
tems
i
n
the
PI
S
A
2
0
12
t
e
st
.
Acco
rdin
g
to
t
he
c
on
st
ruct
iv
i
s
t
ed
ucat
io
n
appro
a
ch
e
f
f
e
c
t
uat
e
d
in
2
00
5-2
006
acad
em
ic
y
ear,
i
t
i
s
exp
ected
t
o
ob
serve
a
ris
e
i
n
co
nst
r
uct
e
d
respons
e
qu
est
i
on
t
yp
e;
h
o
w
ever,
fin
d
i
n
gs
o
f
t
h
e
stu
dy
rev
e
al
t
hat
th
e
s
ucces
s
o
f
con
s
t
r
ucted
resp
ons
e
qu
estion
s
is
d
ecreas
ed
acco
rd
in
g
t
o
t
h
e
app
l
i
catio
n
years.
K
eyw
ord
:
Cons
truc
te
d resp
o
n
se
q
ues
t
ions
Cons
truc
t
i
v
i
st
t
e
a
c
h
in
g
Mu
lt
ip
l
e
c
h
o
i
c
e que
s
t
i
o
ns
PISA
S
t
ude
n
t
a
ch
iev
e
me
nt
s
Co
pyri
gh
t © 2
018 In
stit
u
t
e
of Advanced
En
gi
neeri
n
g
an
d
Scien
ce.
All
rights
res
e
rv
ed.
Corres
pon
d
i
n
g
Au
th
or:
Yeşim
Özer Özka
n
,
D
e
pa
rtme
nt
o
f
Educ
at
i
ona
l M
e
a
s
ur
em
ent a
n
d Eva
l
ua
t
i
o
n
,
G
a
z
i
a
n
tep U
n
i
v
er
sity,
Ü
n
i
v
ersi
te B
ul
varı 2
73
1
0
,
Ş
e
hi
tka
m
i
l
,
G
azia
n
t
e
p,
T
ürkiye
.
Em
ail:
yes
i
mo
zka
n
@
g
an
te
p.
edu.tr
1.
I
N
TR
OD
U
C
TI
O
N
Ba
se
d
on
t
h
e
l
e
a
rni
n
g
ou
tc
ome
s
i
n
t
h
e
curri
c
u
lu
m,
m
ea
su
re
men
t
a
nd
e
v
a
lu
ati
o
n
are
a
ppl
i
e
d
to
deter
m
i
n
e
t
h
e
leve
l
o
f
l
e
a
r
n
i
n
g
rea
l
i
z
e
d
dur
ing
t
h
e
ed
uc
at
ion
p
ro
ce
ss.
M
e
a
s
u
r
emen
t
a
n
d
e
v
al
u
a
ti
on
a
re
ca
rried
o
u
t
w
ith
t
he
a
im
o
f
assess
i
n
g
t
h
e
effec
t
ive
n
es
s
o
f
t
he
cu
rricu
l
u
m
and
the
t
e
a
c
h
i
ng
m
e
t
hod
s,
ide
n
tif
yin
g
a
nd
c
omple
t
i
n
g
le
ar
ning
de
fici
t
s
,
determ
in
ing
t
h
e
i
nter
es
ts
a
n
d
a
b
ili
ti
es
o
f
t
h
e
st
u
d
en
ts,
di
rec
ting
them
t
o
r
e
le
v
a
n
t
f
ie
ld
s
a
n
d
dete
rm
ini
n
g
st
u
d
en
t
ac
h
i
eve
m
e
n
t
[1]
.
Ma
ny
nat
i
o
nal
a
n
d
i
n
terna
t
i
ona
l
exa
m
ina
t
i
ons
t
ake
n
by
n
u
m
e
rous
s
tu
de
n
t
s
ar
e
hel
d
w
i
t
h
t
he
a
im
o
f
identifying
the
effi
c
i
e
n
cy
o
f
the
implem
e
n
te
d
e
d
u
c
a
t
io
n
sy
ste
m
a
nd
ma
ki
n
g
educ
a
tio
na
l
d
e
cisio
n
s
a
c
c
or
di
n
g
ly,
com
p
a
r
in
g
t
h
e
s
y
s
t
e
m
w
i
t
h
ot
her co
u
n
tr
ies
a
n
d
pre
p
ari
ng ne
w
e
d
uca
t
i
o
n po
l
i
cie
s
a
cc
ord
i
n
g
t
o
t
he o
bt
a
i
ne
d
re
su
l
t
s. La
r
ge-sca
le
e
xam
s
a
re
app
l
ied
w
h
e
n
t
he
n
um
b
e
r
o
f
c
a
n
dida
tes
e
n
ter
i
n
g
t
he
e
x
a
m
i
nat
i
o
n
i
s
h
i
gh
w
i
t
h
r
eg
a
r
ds
t
o
na
t
i
o
n
a
l
a
n
d
in
t
e
rna
t
io
na
l
e
x
am
i
n
a
t
i
ons.
L
a
rge
-
sca
l
e
e
x
a
m
s
cons
is
t
of
m
or
e
t
ha
n
o
n
e
su
b
-
t
e
s
t
o
r
dim
e
nsi
on,
a
nd
a
re
t
he
ty
pes o
f
e
xa
mi
nat
i
o
n
s
a
pp
l
i
ed
on
a la
rge
group o
f
s
tude
n
t
s
in
o
rder
to m
e
a
s
ur
e the
i
r
kn
o
w
led
g
e
and s
k
i
lls
[
2].
I
n
Turke
y, la
r
g
e
-sc
a
le exam
s
a
re implem
e
n
t
ed b
y t
h
e
Min
i
stry o
f
N
at
io
na
l Ed
uc
a
t
i
on (M
EB) a
n
d t
h
e
Assessment
S
e
l
e
ction
and
P
l
acement
c
e
n
t
e
r
(
Ö
SYM).
The
large-s
c
a
l
e
e
xa
mi
nat
i
o
n
s
pre
p
are
d
a
n
d
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SSN: 2252-
8822
IJERE
V
ol
.
7,
N
o.
1,
Ma
r
ch 201
8
: 57
– 64
58
implem
e
n
te
d
by
t
h
e
M
i
n
ist
r
y
of
N
a
tio
na
l
Ed
uca
t
i
o
n
in
c
l
ude
e
xam
i
na
tio
ns
s
uc
h
as
t
he
E
xa
min
a
ti
on
f
or
De
t
e
rmi
n
ing
St
ud
en
t
Su
cc
ess
(ÖB
B
S)
a
nd
th
e
Ex
a
m
in
a
t
i
o
n
fo
r
Tran
s
iti
o
n
f
rom
Pr
i
m
ar
y
to
S
ec
o
ndar
y
Ed
uca
t
i
on
(TE
O
G
)
.
A
m
ong
t
h
e
la
rge
sc
a
l
e
e
x
am
s
app
lie
d
by
t
h
e
Ö
S
Y
M
a
re
t
h
e
P
ubl
ic
P
ersonne
l
S
e
lec
tio
n
Exam
i
n
at
ion
(
K
PS
S),
the
Trans
iti
on
t
o
H
ig
her
Ed
ucat
i
on
Exam
i
n
at
ion
(
Y
G
S
),
t
he
A
ca
dem
i
c
P
e
rson
n
e
l
a
n
d
P
o
st-G
r
a
dua
te
E
d
u
cat
i
on E
n
tr
a
n
c
e
E
xam
i
na
tio
n
(A
LES
) a
n
d
the F
o
r
e
i
gn
Lan
gua
ge
E
xa
m
i
nat
i
on
(Y
D
S
).
Bes
i
de
s na
t
i
o
n
a
l
exa
m
i
n
a
t
io
n
s
,
Tur
k
ey a
lso par
tici
p
a
t
e
s
i
n in
te
rna
tio
na
l e
x
am
s
to d
e
t
erm
i
ne its p
lac
e
o
n
t
h
e
i
n
t
e
rn
ati
o
n
a
l
st
ag
e
a
nd
t
o
s
ee
i
t
s
p
ro
gress
i
n
r
ef
e
r
e
n
c
e
t
o
a
spe
c
i
fic
re
fer
e
nce
poin
t
a
n
d
t
o
ide
n
tify
t
h
e
defic
i
e
n
c
i
es
t
h
a
t
nee
d
t
o
be
a
d
d
r
e
ssed.
T
he
se
e
x
a
ms
i
nc
lu
de
t
he
P
r
o
g
r
am
me
f
or
I
nterna
ti
o
n
a
l
S
tude
nt
A
ssessme
n
t
(P
I
S
A
)
,
the
P
r
ogress
i
n
I
nter
nat
i
ona
l
Rea
d
ing
L
i
ter
a
cy
S
tu
dy
(
P
IRLS
),
a
nd
the
Tre
n
d
s
i
n
Interna
tio
na
l
Ma
them
at
ics
a
nd
S
c
ie
nce
S
t
ud
y
(TI
M
M
S
).
T
IMS
S
i
s
a
s
urvey
stu
dy
c
o
n
d
u
cte
d
by
t
h
e
Interna
tio
na
l
A
s
soc
i
a
t
i
o
n
fo
r
th
e
Eva
l
ua
ti
on
o
f
E
d
uca
t
iona
l
A
c
h
ie
ve
me
nt
(IEA
)
a
t
fo
ur-
y
ear
i
n
t
e
r
vals
t
o
assess
th
e
kn
o
w
led
g
e
a
n
d
s
k
ill
s
of
s
tu
de
nts
i
n
t
he
f
o
u
rth
a
n
d
e
ig
h
t
h
gra
d
e
s
i
n
t
h
e
fiel
ds
o
f
m
a
t
h
em
at
ic
s
an
d
sc
i
e
n
c
e
[
3
].
P
I
R
LS
i
s
a
l
a
rg
e
sc
al
e
e
x
amin
at
ion
c
o
n
t
aini
ng
su
bj
e
c
t
s
suc
h
a
s
rea
d
i
n
g
ski
l
l
s
an
d
ha
bits
o
f
4
t
h
grade
stu
d
e
n
ts
(
9-yea
r-a
ge
g
rou
p
),
a
nd
w
h
e
t
her
teac
h
i
ng
m
ater
i
a
ls
a
n
d
m
e
tho
d
s
use
d
by
teac
hers
t
o
te
ac
h
rea
d
i
n
g
s
k
il
ls
t
o
s
t
ude
nt
s
a
r
e
ade
qua
te
a
s
w
e
ll
a
s
t
he
c
ontri
bu
t
i
ons
o
f
pa
ren
t
s
to
t
he
ir
r
eadi
n
g
ski
l
l
s
deter
m
i
n
ed
t
hr
ou
g
h
n
a
t
i
o
nal
and
inte
rna
t
io
n
a
l
t
e
s
t
s
a
nd
s
u
rve
y
s
a
n
d
c
om
p
a
re
d
w
i
t
h
o
the
r
c
ountrie
s
t
o
re
vea
l
the s
i
m
i
la
r
ities
and di
ffere
n
ce
s
[4].
Th
e
PISA
i
s
t
h
e
b
i
gg
est
e
d
u
c
a
t
ion
a
l
st
ud
y
su
rv
ey
o
rg
ani
z
ed
b
y
t
he
O
r
g
a
n
izat
i
on
for
Ec
o
nomic
C
o-
opera
tio
n
an
d
D
e
vel
opm
en
t
(
O
ECD
)
,
aimi
n
g
a
t
co
l
l
e
c
t
i
n
g
i
n
form
at
i
o
n
o
n
t
he
m
at
h
e
m
a
tics,
s
cie
n
c
e
a
nd
rea
d
i
n
g
s
k
i
l
l
s
of
s
tu
de
nts
in
t
he
1
5
ye
ar
a
ge
g
rou
p
a
n
d
t
he
ir
l
e
v
e
l
o
f
mot
i
v
a
tio
n
t
o
l
e
arn
t
h
ese
subj
ect
s,
t
h
e
ir
o
p
i
n
ion
s
a
bou
t
t
h
emsel
v
es,
th
e
i
r
l
e
arni
n
g
me
t
h
od
s,
t
h
e
s
c
h
ool
e
n
v
ir
on
m
e
nts
an
d
t
h
e
fa
mil
i
e
s
.
H
e
l
d
e
ver
y
three
year
s
s
i
n
c
e
20
0
0
,
the
P
I
S
A
e
xam
seeks
t
o
a
ssess
how
m
uc
h
of
t
he
b
a
s
i
c
k
n
o
w
l
e
d
ge
a
n
d
s
k
ills
t
he
stude
n
t
s
have
.
P
I
S
A
r
esults
a
r
e
u
se
d
by
po
li
cy
m
aker
s
a
r
o
u
n
d
t
h
e
w
o
rl
d
t
o
c
om
par
e
t
he
k
now
led
g
e
a
n
d
s
k
i
l
l
leve
ls
o
f
s
t
u
d
e
n
ts
i
n
t
h
e
i
r
c
o
u
n
t
r
i
e
s
w
i
t
h
th
ose
in
o
ther
c
ou
n
t
ries
p
ar
t
i
ci
pa
t
i
n
g
i
n
t
h
e
p
r
oj
e
c
t
a
nd
to
s
e
t
st
a
ndard
s
t
o
r
a
i
se
qua
l
i
t
y
i
n
e
duca
t
io
n
a
n
d
i
d
en
t
i
fy
t
he
s
tre
n
g
th
s
a
n
d
we
a
k
n
e
ss
es
o
f
the
i
r
edu
c
at
ion
sy
st
ems
[5].
F
irst
i
nt
r
o
d
u
c
e
d
i
n
2
000,
t
he
P
I
S
A
t
e
st
c
o
n
t
a
i
n
e
d
que
stio
n
s
from
thr
e
e
fields,
wi
th
a
p
rimary
f
ocus
o
n
que
st
ions
m
eas
uri
ng
lite
rac
y
s
ki
l
l
s
b
u
t
T
urke
y
w
a
s
no
t
a
pa
rt
o
f
the
pr
ogr
am
.
Tur
k
ey
j
o
i
ned
the
pro
g
r
a
m
for
the
firs
t
t
i
m
e
i
n
2
0
0
3
,
w
h
en
t
he
f
oc
us
o
f
t
h
e
test
s
h
i
f
t
e
d
t
o
a
ssessing
s
t
ud
ent
s
’
m
a
them
atica
l
s
ki
lls.
In
2
0
0
6
,
the
foc
u
s
w
a
s
on nat
u
ra
l
sc
ie
nc
es
q
ue
sti
o
ns.
A
l
l
the
s
e
t
h
ree
te
sts
r
e
pre
s
en
t
e
d
t
h
e
fir
s
t
eva
l
ua
t
i
o
n
p
e
r
io
d
of
t
he
P
I
S
A
test. The
s
e
c
on
d
e
v
a
l
u
a
ti
on per
i
o
d
of t
h
e
P
I
S
A
w
as c
om
ple
t
e
d
in 2
0
09
w
i
t
h
a
r
e
a
di
ng tes
t
, in
2
01
2 w
i
t
h
a
mat
h
emati
cal
t
est
and
wit
h
s
c
i
e
n
c
e
t
e
st
s
in
201
5
[5
].
R
eg
u
l
ar
r
e
v
i
e
w
o
f
P
I
S
A
d
a
t
a
b
y
y
e
a
r
s
i
s
i
m
p
o
r
t
a
n
t
bo
th
f
or
m
oni
t
o
ri
ng t
h
e
pro
g
ress in
e
d
u
ca
ti
o
n
a
nd a
s
se
ss
i
n
g the
e
ffic
i
e
n
c
y
o
f ne
w
l
y i
n
t
r
oduc
e
d
pol
ic
ie
s
[6].
V
a
r
i
o
u
s
m
e
a
s
u
r
e
m
e
n
t
t
o
o
l
s
a
r
e
i
n
c
l
u
d
e
d
i
n
t
h
e
P
I
S
A
t
e
s
t
.
O
n
e
o
f
t
hese
i
s
the
stu
d
e
n
t
q
u
es
ti
o
nna
ire
w
h
ic
h
co
nta
i
ns
m
any
a
s
pe
ct
s
gau
g
i
n
g
t
h
e
i
n
t
e
r
est
of
t
he
s
tu
de
nt
s
i
n
t
he
w
e
i
gh
te
d
fie
l
d
te
ste
d
a
nd
the
i
r
se
lf-
bel
i
e
f
,
a
s
w
e
l
l
as
t
he
i
r
a
tt
it
u
d
e
tow
a
rd
t
he
s
ch
oo
l
an
d
the
i
r
s
ense
o
f
be
l
o
n
g
i
ng.
A
n
o
t
h
er
i
s
the
o
p
e
n
-
e
nde
d
sc
ho
ol
s
u
r
v
e
y
,
w
h
i
c
h
p
ro
vi
des
i
n
fo
rma
t
i
o
n
ab
out
t
h
e
s
t
r
u
c
t
u
re
a
n
d
r
es
ource
s
of
t
he
s
ch
oo
l,
e
va
lua
t
es
t
h
e
sch
o
o
l
'
s
c
urric
u
l
u
m,
a
nd
de
te
rmine
s
t
he
s
c
h
o
o
l
'
s
e
duca
tio
n
a
l
p
o
l
i
c
i
es.
Anot
her
me
asurem
ent
t
o
o
l
u
se
d
in
t
h
e
P
I
S
A
t
e
s
t
a
r
e
c
o
g
n
i
t
i
v
e
t
e
s
t
s
a
s
s
e
s
s
i
n
g
r
e
a
d
i
n
g
,
s
c
i
e
n
c
e
a
n
d
m
a
t
h
l
i
t
era
c
y
to
m
ea
su
re
t
h
e
a
ca
d
e
mi
c
su
c
ces
s
of
the
stu
d
e
n
t.
T
h
e
q
ue
s
t
i
o
nna
ire
s
a
nd
c
ogn
i
t
i
v
e
t
e
s
t
s
ar
e
inte
n
d
e
d
t
o
e
va
l
u
a
t
e
t
h
e
s
t
ude
n
t
i
n
a
m
u
lti
d
i
me
nsi
ona
l
w
a
y.
T
he
q
u
e
s
t
i
o
n
type
s
use
d
i
n
PISA
c
ogni
tive
tes
t
s
a
r
e
m
u
ltip
le
c
ho
i
c
e
,
c
om
pl
e
x
m
ul
t
i
p
l
e
c
h
o
i
c
e
,
op
e
n
ende
d,
se
m
i struc
t
ur
ed a
n
d
sh
o
rt a
nsw
e
r
qu
e
s
tio
ns [
5].
Whe
n
t
he
r
e
l
a
t
ed
l
i
t
er
at
ure
was
exam
ine
d
,
we
f
oun
d
st
ud
ie
s
exa
m
i
n
i
ng
the
perf
orm
a
nce
leve
ls
o
f
stude
n
t
s
w
ith
r
e
g
ards
t
o
d
i
ffe
r
e
nt
t
y
p
e
s
of qu
est
i
o
n
s
o
n
t
he ba
s
i
s
of
g
ender
[3],[7]-[
9]
.
We
a
lso
fo
un
d
o
u
t
tha
t
perform
ance
l
e
v
e
l
s
w
i
t
h
r
e
g
a
r
ds
t
o
d
i
ffe
ren
t
q
ue
s
tio
n
ty
pe
s
w
e
r
e
exam
in
ed
b
y
di
ffe
r
ent
disc
ip
li
nes
a
n
d
b
y
ac
adem
ics
fro
m
d
i
f
fer
e
nt
f
ie
l
d
s
[10]
-[2
1
].
T
hese
s
t
udie
s
s
h
o
w
t
h
a
t
stud
ent
s
u
cc
es
s
rat
e
s
wi
th
m
u
l
t
i
p
l
e
ch
oi
ce
que
st
ions
a
re
f
our
time
s
h
i
g
h
e
r
in
c
om
paris
on
t
o
o
t
h
er
q
u
e
st
io
n
t
y
p
es.
It
i
s
a
l
so
s
een
t
ha
t
t
h
ere
a
r
e
st
u
d
i
es
exa
m
in
i
ng
t
h
e
ps
ych
o
m
e
tric
a
spe
c
t
s
of
d
iffere
n
t
q
ues
t
ion
type
s
[22]
-
[
24]
a
nd
s
t
u
d
ie
s
o
n
t
he
p
os
i
t
i
v
e
con
t
ri
b
u
ti
ons
o
f
the
c
o
ns
tru
c
ti
v
i
s
t
t
ea
c
h
i
n
g
a
ppr
oa
ch
t
o
a
cade
m
ic
a
c
h
ie
ve
me
n
t
[
1
1
]
,
[
25],[26].
P
r
e
v
io
us
rese
arc
h
er
[
27
]
exam
i
n
ed
P
I
S
A
20
03
a
n
d
P
I
S
A
200
6
c
o
g
n
i
t
i
ve
t
es
t
s
f
o
cu
si
ng
o
n
dif
f
e
r
en
t
wei
g
h
t
ed
a
rea
s
o
n
the
basi
s
of
t
h
e
q
ue
s
t
i
o
n
t
y
p
e
s
and
fo
un
d
tha
t
t
he
p
erc
e
n
t
age
o
f
s
u
cc
ess
wit
h
m
u
l
t
i
ple
cho
i
ce
q
u
e
s
t
i
o
n
s
was
hi
ghe
r
t
h
a
n
o
t
h
er
que
st
i
on
t
ype
s.
T
his
st
u
dy
c
o
n
c
e
r
ns
i
t
s
elf
w
i
th
t
he
d
i
s
trib
u
tio
n
o
f
que
sti
ons
i
n
t
h
e
P
I
S
A
20
0
3
a
n
d
P
ISA
201
2
c
o
g
n
i
t
i
ve
t
e
s
t
s
w
i
t
h
a
foc
u
s
o
n
m
a
t
he
ma
tic
a
l
l
i
t
e
r
ac
y
a
c
c
o
rd
i
n
g
to
d
i
ffe
re
nt
q
ues
tio
n
ty
pes. In o
t
her
w
o
rds,
t
he stu
d
y
a
ims at
de
t
e
r
mini
n
g
the
ty
p
e
s
o
f
ques
t
ions
w
here
st
ude
n
t
s ha
d th
e
h
i
g
h
e
s
t
a
n
d
the l
o
w
e
st
s
uc
c
e
ss
perc
enta
ge
s
i
n
t
he
P
ISA
m
a
them
at
ical
liter
a
c
y
t
e
s
t
s.
The
first
r
eas
o
n
f
or
t
he
s
e
l
e
c
t
i
on
of
t
he
P
IS
A
2003
a
n
d
P
I
S
A
2
0
12
te
sts
w
a
s
the
i
n
t
r
o
duc
t
i
on
o
f
t
he
con
s
truc
t
i
v
i
s
t
t
e
a
c
h
in
g
ap
proa
ch
a
s
an
e
d
u
ca
tio
na
l
ph
i
l
o
s
o
p
hy
i
n
our
s
ch
o
o
l
s
dur
in
g
the
20
0
5
-2
00
6
ac
a
d
em
ic
yea
r
a
n
d
t
he
s
t
ude
n
t
s
w
h
o
rec
e
ive
d
e
d
u
ca
t
i
o
n
a
c
c
o
rd
in
g
to
t
h
i
s
p
h
ilos
o
p
h
y
t
oo
k
the
i
r
fir
s
t
PIS
A
t
est
in
2
01
2.
It
i
s
ex
pe
cte
d
t
ha
t
s
t
ude
n
t
s
w
ho
re
ce
ive
d
e
duc
at
i
on
a
c
c
o
r
d
i
n
g
t
o
t
h
is
p
h
i
l
o
so
p
hy
w
i
ll
ha
ve
m
ore
succ
ess
w
i
t
h
st
r
u
c
t
ure
d
r
e
s
p
ons
e
que
st
i
o
n
s
i
n
na
tio
na
l
an
d
inte
r
n
at
i
ona
l
e
x
a
m
i
na
ti
ons.
A
n
o
t
her
re
aso
n
w
as
t
ha
t
b
o
t
h
o
f
the
Evaluation Warning : The document was created with Spire.PDF for Python.
IJERE
I
S
S
N
:
2252-
88
22
T
h
e
Stu
d
e
n
t
s
Achiev
em
e
n
t i
n
T
u
rke
y
,
Acc
o
rdin
g t
o
Q
u
es
t
i
o
n T
y
pes
U
s
e
d
in
PISA ..
.
(Ye
ş
i
m
Özer
Özka
n
)
59
selec
t
e
d
t
es
ts
w
ere
m
a
th
s
k
ills
w
e
i
g
h
t
ed
t
e
s
t
s
.
H
o
w
e
ve
r,
i
t
sh
o
uld
be
n
ot
e
d
t
h
a
t
th
e
r
e
we
re
d
i
ffe
ren
t
p
a
r
t
i
c
i
p
a
n
t
s
i
n
b
o
t
h
t
e
s
t
s
s
i
n
c
e
t
h
e
P
I
S
A
i
s
a
t
e
s
t
c
o
n
d
u
c
t
e
d
o
n
st
u
d
en
ts
i
n
the
1
5
-yea
r-a
ge
g
ro
u
p
.
H
o
w
e
ver
,
w
ith
t
he
P
I
S
A
20
0
3
a
nd
P
ISA
20
12
m
a
t
h
e
m
atics
liter
a
c
y
t
es
ts,
i
t
i
s
p
o
s
s
i
b
l
e
t
o
re
veal
t
he
d
i
f
fere
nce
be
t
w
e
e
n
t
h
e
s
u
cc
ess
ra
te
o
f
stud
e
n
t
s
i
n
Tu
rk
ey
i
n
re
l
a
t
i
on
t
o
di
ff
ere
n
t
que
s
tio
n
type
s.
I
n
li
ne
w
ith
t
he
s
t
u
d
y
o
b
j
e
c
t
ive
s
,
the
b
a
sic
var
i
a
b
le
s
the
‘exa
m
yea
r
s’
a
n
d
t
he
‘
ques
tio
ns
t
y
p
e
s’
a
r
e
di
scus
s
e
d
i
n
t
he
p
a
p
er
.
Mul
t
i
p
l
e
c
h
o
ic
e
an
d
com
p
le
x
m
u
l
t
i
p
le
c
h
o
ic
e
a
n
d
struc
t
ur
ed
r
e
s
ponse
ques
t
ion
s
w
e
r
e
exam
ined
f
o
r
t
he
v
a
r
i
a
ble
‘q
ue
s
tio
n
ty
pes’
w
h
i
l
e
the
P
I
S
A
2
00
3
an
d 20
1
2
w
ere
take
n as
a
b
a
s
i
s
f
or
t
he
va
r
i
able the ‘exam
years’.
2.
RESEARCH
M
ETH
O
D
2.1.
Rese
arch
Des
i
g
n
A
s
t
he
s
tud
y
w
as
r
e
p
e
a
te
d
base
d
on
q
ua
n
tita
t
i
ve
a
n
d
O
E
CD
d
a
t
a
a
s
t
h
e
stu
d
y
an
alyz
e
s
s
t
a
ti
st
i
cal
data
w
i
t
h
re
gards
to
m
athem
a
tica
l
l
i
t
e
r
ac
y
te
st
r
esu
l
t
s
o
f
stu
d
e
n
ts
i
n
T
u
rke
y
i
n
t
h
e
P
I
S
A
2
0
0
3
a
nd
P
I
S
A
2
0
1
2
t
e
st
s
,
t
h
e
s
tu
dy
i
s
a
n
e
x
a
m
pl
e
of
a
s
ec
ond
a
r
y
an
a
l
y
s
i
s
s
t
udy
.
T
he
s
uc
cess
r
a
te
o
f
t
h
e
st
u
d
en
ts
i
n
t
h
e
sa
m
p
le
w
a
s
i
nve
st
iga
t
e
d
i
n de
p
t
h t
h
ro
ug
h te
sts.
H
enc
e
t
hi
s stu
d
y
is a
desc
ripti
v
e
stud
y.
2.2.
Po
pu
la
t
i
o
n
a
nd Sa
mp
le
The
st
u
d
y
i
n
c
l
ude
s
s
t
u
d
e
n
t
s
i
n
t
h
e
15-
yea
r-age
g
r
o
up
w
h
o
r
e
c
e
iv
ed
f
o
r
ma
l
s
c
hoo
l
t
r
a
i
ning
i
n
Tu
rk
ey
in t
he ye
a
rs 2
0
03 a
n
d 2
0
12. T
he
P
ISA
20
03 c
o
g
n
it
i
v
e te
st
s a
nd q
ues
t
i
o
nna
ires w
ere
a
p
p
l
i
e
d o
n
4,
855 s
t
u
d
e
n
t
s
from
12
pr
i
m
a
r
y
sch
o
o
l
s
a
n
d
14
7
hi
g
h
s
c
h
oo
ls
r
an
d
o
m
l
y
pic
k
e
d
i
n
M
a
y
from
se
ve
n
ge
o
g
rap
h
i
ca
l
r
e
gio
n
s
[2
8].
T
h
e
P
I
S
A
2
0
1
2
c
og
ni
t
i
ve
t
e
s
t
s
a
nd
q
ues
t
i
onna
ires
w
ere
app
li
e
d
o
n
4,
84
8
s
t
ude
nt
s
fr
om
1
70
s
c
h
oo
ls
rand
om
ly
c
ho
se
n
b
y
t
he
P
IS
A
Interna
t
i
o
nal
Ce
nte
r
a
ft
er
c
lassifi
ca
t
i
o
n
a
ccor
d
ing
to
s
ch
o
o
l
ty
pe
s
[5]
.
Cog
n
i
t
i
ve
tes
t r
e
sul
t
s
obta
i
ne
d fr
om
t
he
se te
s
ts w
e
r
e
used in
t
he s
tudy.
2.3.
D
a
ta
C
olle
ctio
n
The
da
t
a
c
o
llec
tio
n to
ols of th
i
s st
u
d
y ar
e the
P
I
SA
200
3
a
n
d
P
ISA
201
2
m
a
the
m
a
tica
l
l
it
e
r
ac
y
tes
t
s.
Th
e
st
udy
d
ata
h
a
v
e
b
een
ob
t
a
in
ed
fro
m
a
d
a
t
a
b
a
se
(
http
:
/
/
www.
o
e
c
d.
org
/
pisa
/)
a
va
i
l
ab
le
t
o
inv
e
stiga
t
ors
pu
b
l
i
s
he
d
b
y
t
he
O
ECD
.
I
n
the
P
I
S
A
200
3
tes
t
s,
t
her
e
a
r
e
84
que
st
i
o
n
s
o
n
m
a
them
atica
l
l
i
t
era
c
y,
28
o
n
rea
d
i
n
g
s
k
i
l
l
s
,
34
o
n
s
c
i
enc
e
l
i
t
e
r
ac
y,
a
nd
1
9
on
pr
o
b
l
e
m
-
s
ol
v
i
n
g
s
k
i
l
l
s
w
i
t
h
v
a
r
i
o
u
s
l
e
v
e
l
s
o
f
d
i
f
f
i
c
u
l
t
y
[
2
1
]
.
In
t
he
P
IS
A
2
0
1
2
t
est
s
,
ther
e
are
84
q
u
es
tio
ns
on
ma
the
m
a
t
i
c
a
l
liter
a
c
y
,
44
on
r
ead
i
ng
s
k
i
l
ls,
a
n
d
52
o
n
scie
nc
e
lite
rac
y
[
2
9
].
B
ase
d
o
n
va
r
i
ab
les
t
o
b
e
use
d
f
or
t
h
e
s
t
ud
y
from
P
I
S
A
20
03
a
nd
P
I
S
A
201
2
c
o
gn
iti
ve
tes
t
d
a
t
a
,
d
ata
on
sc
i
e
nce
l
i
te
rac
y
a
nd
re
adi
n
g
a
b
i
l
i
t
y
tes
t
s
w
e
r
e
exc
l
ude
d
a
n
d
d
a
ta
c
o
n
t
a
ini
n
g
on
ly
ma
them
at
i
c
al
l
itera
c
y
item
s
w
e
r
e
selec
t
e
d
.
The
que
st
i
o
n
t
ype
s
u
s
e
d
i
n
P
I
S
A
2
0
0
3
a
n
d
2
0
1
2
c
o
g
n
i
t
i
v
e
t
e
s
t
s
are
mult
iple
c
h
o
i
c
e,
c
omp
l
e
x
m
ul
t
i
pl
e
ch
oi
c
e
a
n
d
s
truc
t
u
re
d
a
n
d
s
hor
t
re
spo
n
se
que
st
i
o
n
s
.
Thi
s
c
la
ss
ifi
c
a
tio
n
take
s
i
n
to
c
o
n
s
i
d
era
t
i
o
n
t
h
e
P
I
S
A
200
3
Co
gn
i
tive
Te
st
C
ode
B
o
o
k
p
ub
l
i
s
h
e
d
b
y
t
h
e
O
E
CD
a
n
d
t
he
P
IS
A
20
1
2
T
e
c
h
n
i
ca
l
Re
por
t
[3
0],
[
31].
The
d
i
s
t
ribu
t
i
o
n
o
f
ques
t
i
o
ns
i
n
P
I
S
A
2
0
0
3
a
n
d
P
I
S
A
2
0
1
2
m
a
t
h
e
m
a
t
i
c
s
lite
rac
y
t
e
s
ts
b
y item
ty
pes
is
g
i
v
e
n
i
n Ta
ble
1.
Tab
l
e
1. D
istri
b
u
tio
n
of P
ISA
Mat
h
em
atica
l
L
ite
rac
y
Test
Q
u
esti
ons b
y
It
em
Types
Item
Ty
p
e
s
PISA
2003
P
I
S
A
2012
Nu
m
b
e
r
o
f
i
t
em
P
e
r
cen
t
a
g
e
(
%)
N
u
m
b
e
r
o
f
i
t
e
m
s
P
er
cen
t
a
g
e
(
%
)
M
u
lti
ple
C
hoice
1
7
20.
2
21
2
5
C
o
m
p
le
x Multipl
e
C
hoic
e
1
1
13.1
12
14.29
St
r
u
c
t
ure
d
Re
s
ponse
56
66.
7
51
60.
71
T
o
ta
l
84
100.
0
84
100.
0
Whe
n
T
a
b
le
1
i
s
exa
m
i
n
e
d
,
it
i
s
s
ee
n
t
h
at
s
t
r
uct
u
red
r
e
sp
onse
q
uest
i
ons
f
or
m
the
ma
jo
r
ity
o
f
que
st
ions
i
n
b
o
t
h
te
st
s,
fo
l
low
e
d
by m
u
lt
i
p
le
c
ho
i
c
e
que
s
t
i
o
ns
a
nd
com
p
le
x m
u
lti
pl
e
cho
i
ce
ques
tio
ns.
2.4.
Analys
is of
th
e Data
The
P
I
SA
200
3
a
nd
P
I
SA
2012
ma
t
h
em
atic
s
li
t
e
rac
y
t
es
t
item
s
u
se
d
a
s
d
at
a
co
ll
e
c
t
i
o
n
too
l
s
were
c
l
a
s
si
fi
ed
a
s
mu
lti
pl
e
cho
i
c
e
,
c
o
m
p
l
ex
m
ul
t
i
p
l
e
c
hoi
c
e
a
n
d
s
t
r
u
c
t
ure
d
r
es
po
nse
q
u
e
s
ti
o
n
s
o
n
the
bas
i
s
of
t
he
ty
pes
of
ques
t
ions
t
o
be
e
xa
m
i
ne
d.
A
d
ata
b
ase
co
n
t
ai
n
i
n
g
c
o
g
n
i
t
ive
tes
t
i
t
e
ms
h
as
b
e
e
n
a
c
c
e
sse
d
us
in
g
th
e
in
t
e
rna
t
io
na
l
d
a
taba
se
p
ub
lis
hed
b
y
t
he
O
ECD
.
F
irst
o
f
a
ll,
d
at
a
fr
om
o
t
h
er
c
ou
n
t
ries
h
ave
bee
n
d
e
l
e
t
e
d
a
n
d
exp
u
nge
d
fro
m
t
h
i
s
da
ta
bas
e
.
The
o
b
t
ai
ne
d
tex
t
f
ile
h
as
b
een
c
o
n
v
e
r
te
d
in
t
o
E
xc
el
f
orm
a
t.
D
urin
g
t
h
e
fi
le
con
v
er
sio
n
,
a
t
t
e
nt
i
o
n
has bee
n
pa
i
d
t
o
w
ha
t the
data
i
n ea
c
h
c
o
lumn
refers to b
y
c
onsi
d
e
r
i
n
g t
h
e
c
o
g
n
it
i
v
e te
st
c
o
d
e
boo
k
publ
ish
e
d
b
y
th
e
OECD.
As
t
h
e
v
a
r
i
a
bl
e
s
m
ade
me
an
i
n
gf
u
l
by
t
h
e
c
o
l
u
m
n
s
d
i
ffe
r
ed,
l
i
n
e
s
wer
e
draw
n
be
tw
ee
n
t
h
e
da
ta
a
n
d
t
h
e
c
o
nve
rs
io
n
p
r
ocess
w
a
s
com
p
le
te
d
.
D
a
ta
o
the
r
t
han
thos
e
o
n
m
at
he
m
a
tica
l
lite
rac
y
w
e
r
e
d
e
l
e
te
d
an
d
rem
ove
d
from
the
col
u
m
n
s
i
n
t
he
E
xce
l
f
ile
t
o
g
i
ve
t
he
d
a
t
a
b
as
e
i
t
s
fina
l
sha
p
e
5
1
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SSN: 2252-
8822
IJERE
V
ol
.
7,
N
o.
1,
Ma
r
ch 201
8
: 57
– 64
60
Th
e
s
e
st
ep
s
were
i
mp
l
e
me
nt
ed
s
e
p
a
r
at
el
y
fo
r
PI
SA
2
00
3
an
d
PIS
A
20
12
te
sts.
T
he
refor
e
,
tw
o
se
para
te
E
xce
l
fi
les w
e
re
obta
i
ne
d from
the
da
taba
se
.
The
f
ile
c
r
eat
e
d
f
or
t
he
P
I
S
A
200
3
te
st
c
on
ta
in
s
answ
e
r
s
by
4,8
5
5
s
tude
n
t
s
in
r
e
s
po
nse
t
o
8
4
ma
them
at
i
c
al
que
st
ions
a
n
d
the
fi
le
c
re
at
ed
f
or
P
ISA
20
1
2
t
es
t
c
o
n
t
a
i
ns
a
nsw
e
rs
by
4,
84
8
s
t
u
d
e
nt
s
i
n
re
sp
on
se
t
o
84
mat
h
emat
i
cal
q
u
e
st
i
o
n
s
.
The
c
o
lu
mn
w
it
h
t
h
e
q
u
e
s
ti
on
code
s
w
a
s
a
d
d
e
d
t
o
t
he
t
o
p
o
f
th
e
stude
n
t
a
nsw
e
rs
by
t
h
e
in
ve
st
i
g
at
ors
acc
ord
i
ng
to
t
he
o
r
d
e
r
i
n
t
h
e
c
o
de
b
o
o
k
.
F
or
e
a
c
h
que
s
t
i
o
n,
f
ul
l
y
c
orre
ct,
part
ial
l
y
c
or
rec
t
,
inc
o
rr
ect
a
nd
una
nsw
e
red
ques
tio
ns
h
a
v
e
bee
n
c
a
l
cul
a
ted
by
u
sin
g
t
h
e
Ex
cel
"
COUNTI
F
"
com
m
a
nd.
I
n
order
to
f
in
d
o
u
t
t
h
e
num
ber
of
q
ues
tio
ns
a
nsw
e
re
d
by
ea
c
h
s
t
u
de
nt
,
dat
a
o
n
t
h
e
ful
l
y
c
o
rre
ct,
part
ial
l
y
c
orre
ct
a
n
d
e
mp
t
y
c
olum
ns
w
ere
c
o
l
l
ec
t
e
d.
T
he
f
o
l
l
o
w
i
ng
f
o
r
mu
l
a
h
as
b
een
u
se
d
to
cal
cu
la
t
e
t
h
e
t
o
ta
l s
c
o
r
e
.
ot
al Sco
r
e
=
(F
ully
c
orr
ect×
2)
+
(
P
a
r
t
ia
ll
y
cor
r
e
c
t
×
1) +
(Inc
o
rre
ct×
0).
A
s
c
lea
r
l
y
s
ta
te
d
i
n
t
he
form
ula,
t
he
p
oi
n
t
f
or
e
ac
h
c
o
m
p
le
te
a
nsw
e
r
w
a
s
2,
t
he
s
c
o
r
e
f
o
r
ea
ch
p
a
r
tia
ll
y
c
o
rre
ct
a
nsw
e
r
w
a
s
1,
a
nd
t
h
e
po
int
for
fa
lse
answer
s
wa
s
0.
T
he
a
nswe
rs
w
ere
sc
ored
b
a
s
ed
o
n
t
he
2
-
1
-
0
s
ys
tem
use
d
i
n
the
P
I
S
A
s
y
s
tem
.
The
t
o
t
a
l
scor
e
w
a
s
d
i
vi
de
d
b
y
t
he
n
um
ber
o
f
a
tte
n
d
i
n
g
s
t
u
d
en
ts
t
o
fin
d
t
h
e
a
ve
rage
s
c
o
re.
S
i
n
c
e
ea
c
h
c
orr
ect
a
n
swer
w
as
s
c
o
re
d
a
s
2
,
t
h
e
av
e
r
ag
e
s
c
o
r
e
s
o
bt
ain
e
d
are
b
a
se
d
o
n
corr
ect
a
nsw
e
r
s
b
e
i
ng
sc
ored
a
s
2
.
T
he
ave
r
age
scor
es
obta
i
ne
d
w
h
e
n
c
alc
u
la
t
i
n
g
t
he
s
ucc
e
ss
perc
enta
ge
s
have
b
e
e
n
mu
lt
ipl
i
e
d
b
y
50.
S
ubse
que
n
t
l
y
,
sam
e
ques
tio
n
type
s
w
e
r
e
g
r
o
u
p
e
d
a
m
ong
t
hem
s
e
l
v
e
s
a
nd
a
v
e
r
age
s
c
o
r
e
s
a
n
d
ge
ne
ral
aver
age
score
s
w
er
e
ca
l
c
u
l
a
t
e
d
o
n
the
ba
sis of q
ue
st
io
n ty
pes. Ta
b
le
4
.1 a
nd A
n
ne
x
Tab
l
e
1 w
e
re cr
eated fol
l
o
w
i
ng
the
a
p
pl
ica
t
i
on
of
t
he
a
fore
me
nti
one
d
s
t
e
p
s
t
o
P
IS
A
200
3
d
a
ta
w
he
rea
s
T
a
b
l
e
4
.
2
a
nd
A
n
ne
x
Ta
ble
2
w
e
r
e
c
re
ated
f
ol
low
i
ng
the a
p
p
l
i
c
a
t
ion
of the
a
f
o
r
e
m
e
nt
i
o
n
e
d ste
p
s t
o
P
IS
A 201
2.
3.
RESULT
S
A
N
D
ANALY
S
IS
I
n
t
h
i
s
pa
rt
o
f
the
s
t
u
d
y
,
P
I
SA
2
0
0
3
a
n
d
P
IS
A
2012
T
ur
k
e
y
da
ta
w
e
re
a
na
lyz
e
d
b
y
qu
e
s
t
i
o
n
t
ype
s.
Usi
n
g
th
e
d
a
t
a
s
o
u
r
c
e
s
obt
ai
ned
,
w
e
g
o
t
r
es
ul
ts
o
n
t
h
e
st
udy
p
r
ob
lem
and the
su
b-pro
b
le
m
s
. Inter
p
r
etat
i
ons o
f
the f
i
nd
i
ngs a
re
b
elow
.
To
f
i
n
d
an
a
n
s
w
e
r
to
t
h
e
f
i
r
st
s
u
b
-pro
b
l
e
m
o
f
‘Wha
t
i
s
t
he
s
ucc
e
s
s
p
e
r
c
e
n
ta
g
e
d
is
t
r
i
b
u
t
io
n
o
f
stude
n
t
s
in
t
he
P
IS
A
2003
m
a
the
m
a
t
ic
s
l
i
te
r
acy
t
es
t
b
y
t
h
e
que
s
t
i
o
n
t
ype
s
?
’,
t
he
p
erc
e
n
ta
ge
a
nd
fre
q
uen
c
y
di
stri
b
u
t
i
on
s
o
f
s
tude
n
t
r
espo
nses
w
e
r
e
exam
ined
by
type
s
i
n
t
h
e
PISA
2
003
t
est
.
A
d
et
ail
e
d
a
n
a
l
y
s
is
o
f
a
l
l
it
e
m
t
y
p
es i
s gi
ven
in
A
nne
x Ta
b
l
e 1
and
th
e
i
r
summ
ar
y illus
t
ra
t
i
o
n
is pr
esen
ted
in
T
ab
le
2
.
Tab
l
e
2.
S
tude
nt S
ucc
e
ss Ra
t
e
s by
Q
u
est
i
o
n
T
ypes i
n
t
he
P
ISA
20
0
3
Tu
rkey
M
at
h
e
ma
ti
cal
Li
t
e
ra
cy
Te
s
t
I
t
em T
yp
es
Multiple
C
hoi
c
e
C
o
m
ple
x
Multipl
e
C
hoic
e
S
t
r
uc
tur
e
d Re
spon
s
e
G
e
n
er
al
N
%
N
%
n
%
n
%
I
t
em
1
7
20.
2
11
13.
1
56
66.7
84
100
.
0
Re
ac
h
e
d
1427.4
29.
4
1388.
1
28.
6
1188.
4
24.5
1262.
9
26.0
Full
C
r
e
dit
641
.
3
44.
9
440.
5
31.
7
459.
4
38.6
493.
7
39.1
Pa
r
tia
l C
r
e
d
it
0
0
.
0
0
0
.
0
29.
1
2.
4
19.
7
1.
4
No
C
r
e
dit
740
.
4
51.
9
913.
4
65.
8
660.
4
55.6
709.
7
56.2
Not
Rea
c
h
e
d
45.7
3.
2
34.
2
2.
5
39.
5
3.
3
40.
1
3.
2
I
nva
lid
0
0
.
0
7
.
2
0
.
1
35.
2
0.
7
24.
4
0.
5
M
i
ssing
72.1
1.
5
102
2
.
1
591.
3
%
12.
8
212
%
4.
4
T
o
ta
l
Sc
ore
128
2.
6
881.
1
947
.
9
1006.
9
Ave
r
a
g
e
Sc
ore
.89
.
63
.
7
7
.
88
Suc
c
e
ss
pe
rce
n
t
a
g
e
(
%)
4
4
.
6
3
31.
49
38.70
38.
95
Whe
n
T
a
b
l
e
2
i
s
exam
in
ed,
i
t
i
s
see
n
t
hat
t
h
e
item
t
y
pe
i
n
w
h
ic
h
the
p
e
r
centa
ge
o
f
s
u
cc
ess
is
t
he
hi
ghe
s
t
(44.
63
%
)
i
s "mul
tip
le
cho
ice
"
q
ue
st
i
ons
w
he
rea
s
t
h
e
que
s
tio
n ty
pe
in
w
h
ic
h t
h
e pe
rce
n
ta
ge o
f su
c
cess
is
t
he
l
ow
e
s
t
(
31.
4
9
%)
i
s
"co
m
p
l
e
x
m
u
l
t
i
p
l
e
c
h
o
i
c
e
"
que
st
i
ons.
T
h
e
s
t
ude
nts
scor
e
d
t
he
h
i
g
hes
t
o
n
"m
ul
tip
le
cho
i
ce
"
que
st
i
ons
(
29.
4%)
a
n
d
the
low
e
s
t
o
n
"struc
t
ure
d
r
e
s
pons
e"
q
ue
stion
s
(
24
.5
%).
The
h
i
gh
e
s
t
nu
mbe
r
o
f
com
p
le
te
l
y
c
orre
ct
a
n
s
w
e
rs
(
44.9
%
)
w
e
re
o
bta
i
ned
w
i
th
"
m
u
l
t
i
p
le
c
ho
ic
e"
q
ues
tio
ns
a
nd
t
he
l
e
a
s
t
(
3
1
.
7
%)
w
ith
"
c
o
mp
lex
m
u
l
tip
le
c
h
o
i
c
e"
q
ues
t
i
ons.
T
h
e
h
i
g
h
es
t
n
u
m
b
er
o
f
i
n
c
o
r
r
e
c
t
a
n
s
w
e
r
s
(
6
5
.
8
%
)
w
e
r
e
g
i
v
e
n
f
o
r
"com
ple
x
m
u
ltip
le
c
ho
i
c
e"
que
s
t
i
o
ns
a
n
d
t
he
l
ea
st
(
5
1
.9
%)
f
or
"
mult
iple
c
h
o
ice
"
q
u
e
st
ions.
The
type
o
f
que
st
ion
m
o
st
l
e
f
t
u
n
a
n
sw
ere
d
(
3.
3%)
w
e
r
e
"
struc
t
ur
e
d
r
esp
ons
e"
q
ues
tio
n
s
a
n
d
t
he
t
y
p
e of
q
ue
sti
o
n
le
a
s
t
left
una
nsw
e
r
e
d (
2
.5%)
w
er
e
"com
pl
e
x
m
ult
i
p
le
c
hoi
c
e
"
q
u
es
ti
ons.
To
f
in
d
a
n
a
n
s
w
e
r
to
t
he
s
e
c
on
d
s
u
b-pro
b
l
em
o
f
‘Wha
t
is
t
he
s
uc
ce
ss
perc
en
tage
d
is
tri
b
u
tio
n
of
stude
n
t
s
in
t
he
P
IS
A
2012
m
a
the
m
a
t
ic
s
l
i
te
r
acy
t
es
t
b
y
t
h
e
que
s
t
i
o
n
t
y
p
e
s
?
’.
t
he
p
er
ce
nta
g
e
a
nd
fre
q
uen
c
y
di
stri
b
u
t
i
on
s
o
f
s
t
u
den
t
r
e
s
p
onse
s
w
er
e
ex
am
ined
a
c
c
or
d
i
n
g
t
o
q
ues
tio
n
ty
pe
s
in
t
he
P
IS
A
2012
t
e
st.
A
Evaluation Warning : The document was created with Spire.PDF for Python.
IJERE
I
S
S
N
:
2252-
88
22
T
h
e
Stu
d
e
n
t
s
Achiev
em
e
n
t i
n
T
u
rke
y
,
Acc
o
rdin
g t
o
Q
u
es
t
i
o
n T
y
pes
U
s
e
d
in
PISA ..
.
(Ye
ş
i
m
Özer
Özka
n
)
61
deta
i
l
e
d
a
na
l
y
sis
of
a
ll
que
st
i
o
n
ty
pe
s
is
g
i
v
en
i
n
A
nne
x
T
a
bl
e
2
a
n
d
t
he
i
r
s
um
ma
ry
illu
strat
i
o
n
i
s
pr
es
ente
d
in Ta
b
le
3.
Tab
l
e 3.
S
t
ude
nt
S
uc
cess
rate
a
s
by Q
u
e
s
ti
on
Type
s i
n
the
P
IS
A
2
012
T
u
rk
ey
M
a
t
h
e
ma
t
i
c
a
l Li
tera
cy
T
es
t
It
e
m
T
y
p
e
s
Multiple
C
hoi
c
e
C
o
m
ple
x
Multipl
e
C
hoic
e
S
truc
ture
d Re
sponse
G
e
n
er
a
l
N
%
n
%
N
%
N
%
Item
2
1
25
1
2
14.
29
5
1
60.
71
8
4
1
00.
0
Re
ac
h
e
d
1451.
4
29.
93
1453.
1
29.
97
133
0.
7
27.
44
1378.
4
2
8
.
4
3
F
u
l
l
C
r
e
dit
729.
7
51.
3
619.
8
42.
65
4
36.
4
32.
79
535.
9
3
8
.
8
7
P
a
rti
a
l
C
r
e
d
it
0
0
.
0
0
0
.
0
22.9
1.
72
13.
9
1.
01
N
o
C
r
e
dit
715.
6
49.
3
827.
3
56.9
8
57.
6
65.
1
817.
8
59.
3
N
o
t
Re
ac
h
e
d
6
0.
413
5
.
9
0
.
406
13.6
1.
02
10.
6
0.
76
Inva
li
d
0.
5
0.
01
8
.
8
0
.
2
7
.
0
0
.
1
5
.
7
0
.
1
M
i
ssi
ng
42.
76
0
.
9
24.
8
0
.
5
1
52.
1
3.
1
106.
6
2.
2
T
o
ta
l
S
c
or
e
1459.
4
1239.
6
895.
8
108
5.
8
A
v
e
r
a
g
e
S
c
or
e
1
.
001
.
85
.
65
.
77
S
u
c
c
e
ss
p
e
r
cen
t
a
g
e
(
%
)
5
0
.
0
5
42.
71
32.
35
3
8.
62
Whe
n
T
a
b
le 3
i
s
e
x
am
ine
d
,
i
t
i
s
see
n
t
ha
t
th
e
que
st
io
n
t
y
pe
i
n
w
h
ic
h
t
h
e
pe
rce
n
ta
ge
o
f
suc
c
e
s
s
is
t
he
hi
ghe
s
t
(
50.
0
5
%
)
i
s
"m
ult
i
p
l
e
c
h
o
i
ce"
q
ue
st
i
o
ns.
w
h
e
r
ea
s
t
h
e
q
u
es
ti
on
t
y
p
e
i
n
w
h
i
c
h
t
he
p
e
r
ce
n
t
a
g
e
o
f
success
i
s
the
lowest
(
32.35%)
is
"struc
t
ure
d
r
espo
nse
”
ques
tio
ns.
The
stude
nts
sc
ore
d
t
he
h
igh
e
st
o
n
"com
ple
x
m
ult
i
p
l
e
ch
o
i
ce
"
q
u
es
ti
ons
(
29.9
7
%)
a
nd
the
lo
w
e
st
on
"s
truc
t
u
re
d
respo
n
se"
que
st
io
ns
(
27.
44
%)
.
T
h
e
h
i
g
h
e
s
t
n
u
m
b
e
r
o
f
c
o
m
p
l
e
t
e
l
y
c
o
r
r
e
c
t
a
n
s
w
e
r
s
(
5
1
.
3
3
%
)
w
e
r
e
ob
tai
n
e
d
w
i
t
h
"mul
t
i
ple
c
h
oi
c
e
"
q
u
e
st
ion
s
and
t
h
e
leas
t
(32.
79%
)
w
ith
"struc
t
ure
d
r
esp
onse"
q
ues
t
i
o
ns.
Th
e
h
i
g
h
e
s
t
n
u
m
b
er
o
f
i
n
c
o
r
r
ec
t
a
n
swers
(65.1%
) w
e
re
g
i
v
e
n
for "stru
c
t
ur
e
d
r
espo
ns
e" que
s
t
i
o
ns a
n
d
t
he
l
e
as
t (49.
3%) for "mult
i
p
l
e
c
hoic
e
" q
u
e
stio
ns.
The
t
y
pe
o
f
q
u
es
ti
on
m
o
st
l
eft
una
nsw
e
red
(
1
.02%)
w
e
re
"struct
u
re
d
resp
onse"
ques
t
ions
a
nd
t
h
e
t
ype
o
f
que
st
ion
leas
t l
e
ft
u
na
nsw
e
r
e
d
(
0
.406%)
w
e
re "c
o
mple
x m
u
l
t
i
p
l
e
c
h
o
i
ce
" qu
est
i
on
s.
I
n
o
rde
r
t
o
f
i
n
d
a
n
a
n
sw
er
t
o
the
th
ird
su
b-pro
b
lem
of
‘
H
o
w
d
o
s
t
ude
n
t
s
ucc
e
ss
r
a
t
es
i
n
t
h
e
P
I
S
A
20
1
2
m
athem
a
tica
l
l
i
t
e
r
ac
y
te
st.
a
n
o
u
t
p
u
t
o
f
c
onstruc
tiv
is
m
th
at
f
irs
t
b
e
g
a
n
t
o
ta
ke
a
d
o
p
t
e
d
d
uri
n
g
t
h
e
2
0
05-
20
0
6
aca
dem
i
c
year
.
com
p
ar
e
to
t
he
P
ISA
20
03
m
a
t
h
e
m
a
t
ica
l
l
i
t
er
a
c
y
tes
t
acc
ord
i
n
g
t
o
que
st
ion
ty
pes
?
’
.
perc
en
tage
(
%
)
v
al
ues
o
f
t
he
d
a
t
a
ge
ne
rate
d
by
c
o
mpa
r
i
n
g
t
h
e
da
ta
f
ro
m
the
first
tw
o
tab
l
es
w
e
r
e
pr
esen
te
d
in Ta
b
le
4.
Ta
b
l
e
4.
S
ummary
o
f F
i
nd
ing
s
from
the Com
p
aris
on o
f
P
ISA
200
3-2
012
Te
s
t
s
Ye
a
r
s
I
t
e
m
T
y
p
e
s
Multi
ple C
hoice
C
o
m
ple
x
Multipl
e
C
hoic
e
S
t
r
uc
ture
d Re
sponse
G
e
n
er
a
l
2003
2012
2003
2012
2003
2012
2003
2
012
Item
s
(
%
)
20.
2
25
13.
1
14.
29
66.
7
60.
71
100
.
0
100.
0
Re
ac
h
e
d
(%)
29.
4
29.
93
28.
6
29.
97
24.
5
27.
44
26.
0
28.
43
Ful
l
C
r
e
dit(%
)
44.
9
51.
33
31.
7
42.
65
38.
6
32.
79
39.
1
38.
87
P
a
rti
a
l
Cr
ed
it(%)
0
.0
0
.0
0
.0
0
.0
2
.4
1
.7
2
1
.
5
1
.
0
1
N
o
C
r
e
dit(%
)
51.
8
49.
3
65.
8
56.
93
55.
6
64.
44
56.
2
23.
05
N
o
t
Re
ac
h
e
d(%)
3
.
2
0
.
4
1
2.
5
0.
41
3
.
3
1
.
0
2
3.
2
0
.
76
Suc
c
e
ss
(%)
44.63
50.
05
31.
49
42.
71
38.
70
32.
35
38.95
38.
62
Ta
b
l
e
4
re
ve
al
s
va
ry
in
g
s
t
ud
ent
res
p
o
n
ses
i
n
T
urkey
t
o
t
h
e
P
IS
A
200
3
a
nd
P
I
S
A
201
2
m
a
the
m
a
tic
s
lite
rac
y
t
es
t
s
.
Ther
e
a
r
e
four
c
omm
on
st
u
d
e
nt
r
e
a
c
t
i
o
ns
t
hat
a
r
e
no
te
w
o
rthy
in
t
he
se
t
ests
:
(1)
In
b
o
t
h
tes
t
s.
stude
n
t
s
g
a
ve
t
he
h
i
g
h
e
st
n
u
m
be
r
of
c
om
pl
etel
y
c
o
rr
ect
a
nsw
e
rs
in
t
he
c
ateg
ory
"m
u
lti
ple
ch
o
i
ce
"
q
u
e
stio
ns.
F
i
ft
y-si
x
(2)
In
b
ot
h
tes
t
s.
t
he
l
eas
t
inc
o
r
r
ec
tly
a
nsw
e
re
d
q
u
e
st
i
o
n
s
w
e
r
e
"mu
l
tip
l
e
c
hoic
e
"
qu
e
s
tio
ns
.
(3
)
In
bo
th
t
e
s
ts.
the
q
u
e
s
t
i
on
t
y
p
e
le
ft
b
lan
k
t
he
m
ost
w
e
r
e
"
s
t
ruc
t
u
re
d
re
spons
e
"
q
ues
t
i
o
n
s
.
(4)
In
b
o
t
h
tes
t
s.
stude
n
t
s
go
t
th
e
h
i
g
h
es
t
sc
or
es
on
“
m
ult
i
ple
cho
i
c
e”
q
u
es
tions.
A
s
a
gene
ral
re
sul
t
o
f
t
h
e
st
ud
y.
it
is
s
e
e
n
t
h
a
t
stude
n
t
s
uc
ce
ss
is
h
i
g
her
i
n
m
ult
i
pl
e
ch
o
i
c
e
ques
t
ions
i
n
bo
t
h
P
I
S
A
t
e
s
t
s
.
O
v
e
r
t
h
e
y
e
a
r
s
.
t
h
e
p
e
r
c
e
n
t
a
g
e
o
f
succe
ss
i
n
m
u
lti
ple-
c
h
o
i
ce
a
nd
c
o
m
p
l
e
x
mul
t
iple
-ch
o
i
ce
que
sti
o
n
s
has
incr
ease
d
.
whi
l
e
t
h
e
per
c
e
n
t
a
g
e
o
f
succe
ss
w
i
t
h
“
s
t
r
uc
ture
d
respons
e”
t
y
p
e
s
h
a
s
d
ec
rea
s
e
d
.
Loo
k
i
n
g
a
t
the
g
e
nera
l
a
v
era
g
e
.
t
he
r
esul
ts
o
f
t
h
e
P
I
S
A
2003
and
PIS
A
201
2
t
e
s
t
s
se
em
to be
si
m
ilar
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SSN: 2252-
8822
IJERE
V
ol
.
7,
N
o.
1,
Ma
r
ch 201
8
: 57
– 64
62
4.
DISC
USSION
Th
is
s
t
u
d
y
a
i
m
e
d
a
t
eva
l
ua
t
i
n
g
s
t
ude
n
t
s
uc
cess
ra
te
s
i
n
T
urke
y
a
c
c
o
r
d
i
n
g
to
d
i
f
fe
rent
t
ype
s
o
f
que
st
ions
i
n
P
I
S
A
200
3
an
d
PI
SA
201
2
ma
t
h
e
m
a
t
i
c
s
l
i
t
era
c
y
te
sts
.
F
o
r
th
is
p
urp
o
se
.
que
st
i
ons
i
n
t
h
e
ma
them
at
i
c
al
l
itera
c
y
t
e
s
t
w
e
re
f
irst
c
la
ss
ifi
e
d
ac
cor
d
i
n
g
t
o
d
i
ffe
ren
t
t
y
p
e
s
of
q
ues
t
i
o
ns
t
ak
i
n
g
in
to
a
c
c
oun
t
t
h
e
c
o
d
e
boo
kl
et
a
nd
t
h
e
t
echn
i
c
a
l
re
po
rt
s.
I
n
th
e
seco
nd
s
t
a
g
e
.
t
h
e
a
n
sw
ers
of
s
tu
de
nt
s
w
ho
par
tic
i
p
a
t
ed
i
n
t
h
e
PISA
t
est
s
f
ro
m
Tu
rk
ey
w
ere
e
x
ami
n
ed
i
n
d
e
t
a
i
l
a
n
d
the
ra
t
e
s
of
c
o
m
ple
t
e
l
y
c
o
rr
ect.
pa
rti
a
ll
y
corr
ect
a
n
d
i
n
co
rrec
t
an
s
we
rs we
r
e d
e
t
e
r
m
i
n
ed
.
Wh
en
t
h
e
P
ISA
2
0
0
3
t
e
st
i
s
e
x
amin
ed
a
cc
o
r
din
g
t
o
t
h
e
di
ff
ere
n
t
t
ype
s
of
q
ue
st
io
ns
i
n
t
h
e
ma
them
at
i
c
al
l
it
e
r
ac
y
test.
it
i
s
se
e
n
t
hat
t
h
e
h
i
g
h
es
t
numb
e
r
of
c
omp
l
ete
l
y
c
o
rrec
t
a
ns
w
e
r
s
(
44.9%)
and
t
h
e
low
e
s
t
numbe
r
of
i
nc
or
rec
t
a
nsw
e
rs
(
51.9%
)
are
w
i
t
h
m
u
l
t
i
ple
c
h
o
i
c
e
q
ue
sti
o
ns.
2
8
.6
%
of
s
t
u
de
nts
ga
ve
corr
ect
r
e
s
po
n
s
es
t
o
c
o
mp
l
i
c
a
te
d
m
u
lti
p
l
e-c
h
o
i
c
e
que
st
io
ns
t
ha
t
p
ro
vide
s
t
u
d
e
nt
s
wit
h
o
n
e
o
r
mo
re
c
o
r
re
c
t
answ
e
r
o
p
t
io
n
s
i
n
some
c
as
es.
The
q
u
e
s
ti
on
t
ype
f
or
w
h
i
c
h
s
t
ude
n
t
s
ga
ve
t
he
l
e
a
st
c
omple
t
e
l
y
c
o
rr
ect
answ
e
r
s
(31.7
%
)
and
the
m
o
st
i
nc
orr
ect
(
65.
8
%
)
a
n
sw
e
r
s
ar
e
also
c
omp
l
e
x
m
ul
tip
l
e
c
h
o
i
c
e
qu
est
i
on
s.
S
t
ude
n
t
s
scor
e
d
t
he
l
ow
e
s
t
p
o
i
n
ts
o
n
str
u
ct
ured
r
esp
onse
que
sti
ons
(
%2
4.
5).
In
t
his
qu
est
i
o
n
t
y
p
e.
t
h
e
re
a
r
e
part
ial
l
y
c
or
rec
t
a
nsw
e
r
s
a
s
w
e
ll
a
s
c
o
mpl
e
te
ly
c
orre
ct
a
nsw
e
rs
a
nd
i
n
c
o
rrec
t
a
n
s
we
rs.
Two
poi
nt
f
ou
r%
o
f
stude
n
t
a
n
s
w
e
rs
t
o
s
t
ruc
t
ure
d
r
espo
nse
q
u
e
s
t
i
o
n
s
w
e
re
p
a
r
tia
ll
y
c
orre
ct.
T
h
e
h
i
g
h
e
s
t
num
ber
of
q
u
e
st
i
o
ns
(3.3%
)
l
eft
b
l
ank
w
a
s
from
the
ques
tio
n
ty
pe
“
struc
t
ur
ed
r
e
s
p
o
n
se
q
uestions”.
W
hen
s
u
ccess
rates
by
que
st
ion
ty
pe
s
w
e
re
e
xa
min
e
d
for
t
h
e
st
u
dy
prob
lem
.
t
he
h
ighe
s
t
s
u
c
ce
ss
r
ate
w
a
s
ac
hie
v
e
d
o
n
m
u
lti
p
l
e
cho
i
ce
q
ues
t
i
ons
(
4
4
.6
3%)
a
nd
t
h
e
l
o
w
e
st
s
ucce
ss
rate
w
as
a
c
h
i
e
ved
on
c
o
m
p
l
e
x
m
u
lt
ip
le
c
ho
i
c
e
que
st
ions
(
3
1
.4
9%)
.
Wh
en
t
h
e
P
ISA
2
0
1
2
t
e
st
i
s
e
x
amin
ed
a
cc
o
r
din
g
t
o
t
h
e
di
ff
ere
n
t
t
y
pe
s
of
q
ue
st
ions
i
n
t
h
e
ma
them
at
i
c
al
l
it
e
r
ac
y
test.
it
i
s
se
e
n
t
hat
t
h
e
h
i
g
h
es
t
numb
e
r
of
c
omp
l
ete
l
y
c
o
rrec
t
a
ns
w
e
r
s
(
51.3%)
and
t
h
e
low
e
s
t
n
um
ber
of
i
nc
o
r
rec
t
a
nsw
e
rs
(
49
.
3
%
)
w
e
r
e
give
n
t
o
m
ulti
p
le
c
h
o
i
ce
q
u
es
tio
ns.
A
hi
g
h
e
r
n
um
ber
of
c
o
rre
c
t
r
e
s
ponse
s
w
e
r
e
gi
v
e
n
t
o
c
o
m
p
l
e
x
m
u
l
tip
l
e
c
h
oi
c
e
q
u
e
st
i
o
ns
t
h
a
n
ot
her
que
stio
n
typ
e
s
(2
9.97
%)
.
wher
eas
t
he
r
ate
of
c
orr
ect
a
nswe
rs
t
o
s
t
r
u
c
t
ured
r
esp
ons
e
que
s
t
i
ons
i
s
2
7
.4
4%
.
The
l
e
as
t
n
u
m
be
r
o
f
com
p
le
te
l
y
c
or
rec
t
a
nsw
e
rs
(
32
.
7
9%).
t
he
h
ig
he
st
n
umbe
r
of
i
nc
o
rr
ect
a
nsw
e
rs
(
65.1
%
)
a
nd
the
h
i
ghe
st
numbe
r
of
b
l
a
nk
r
e
s
p
onse
s
(
1.02
%)
w
e
r
e
give
n
to
s
truc
tur
e
d
re
sp
o
n
se
q
u
e
st
io
ns.
Whe
n
e
xa
mine
d
i
n
t
e
r
ms
o
f
succe
ss
p
e
r
ce
nta
g
e
s
.
m
u
l
t
i
p
le
-c
hoic
e
q
ues
t
ions
c
om
e
first
.
f
ol
l
ow
ed
by
c
o
mple
x
m
u
l
t
i
p
le-c
ho
i
c
e
que
st
ion
s
and
struc
t
ur
ed
r
espo
nse q
u
e
s
ti
ons.
Whe
n
s
ucce
ss
r
a
t
es
a
re
c
ompa
red
ac
cordi
n
g
t
o
q
ues
t
i
o
n
t
y
pes
i
n
the
r
e
levan
t
t
es
ts.
an
i
ncre
ase
fr
o
m
44.6
3
% to
5
0.0
5
%
i
s
s
e
e
n
for mult
ipl
e
c
h
o
ic
e
que
st
io
ns
a
n
d
a
n i
ncre
ase
from
31.49
%
to
42.7
1
%
for
c
o
m
p
lex
mult
i
p
le
c
h
o
i
c
e
que
st
i
o
ns.
w
h
ile
a
d
e
c
re
ase
fr
om
38.7
0
%
to
3
2.3
5
%
i
s
s
e
e
n
i
n
s
t
ruc
t
ured
r
e
s
p
onse
q
u
e
stio
ns.
W
h
il
e
a
hi
g
h
e
r
su
cc
e
s
s
ra
t
e
w
as
e
xp
ect
e
d
w
it
h
st
ru
ct
u
r
e
d
r
es
p
o
n
se
q
ues
t
io
ns
i
n
a
n
sw
erin
g
w
h
i
c
h
s
t
ud
en
ts
a
r
e
requ
ire
d
t
o
bu
i
l
d
t
h
e
i
r
ow
n
u
n
iq
ue
a
nsw
e
rs
a
c
c
ord
i
n
g
t
o
t
h
e
c
o
n
struc
t
i
v
ist
te
ac
hi
n
g
a
p
p
roac
h
in
t
r
od
u
c
ed
i
n
the
a
c
a
d
em
i
c
y
ea
r
200
5-
2
006.
t
he
o
p
p
o
sit
e
o
cc
urred.
W
hile
t
he
r
esu
l
ts
o
b
t
a
i
ne
d
a
r
e
com
p
are
d
w
i
t
h
t
he
si
m
i
lar
st
u
d
i
es
i
n
t
h
e
l
i
t
e
ra
tu
re
.
i
t
i
s
obse
r
v
e
d
tha
t
t
he
r
esu
l
t
s
are
o
v
er
al
l
cons
iste
nt
.
Th
e
stud
y
b
y
[
2
7
]
a
nd
[3
2] is an exa
mple of t
h
e
s
e
stu
d
i
es. [3
6] exa
mined how
d
i
f
fere
n
t
m
easure
m
e
n
t-eva
l
ua
ti
on
a
pproac
h
es suc
h as
mult
i
p
le
c
hoic
e
t
e
s
ts.
w
r
it
te
n
e
x
am
i
n
a
tio
ns
a
nd
p
e
rform
an
c
e
t
a
s
k
s
a
ff
e
c
t
e
d
achi
e
v
e
me
nt
i
n
sci
e
n
ce
c
l
ass
e
s.
F
o
r
th
is
p
urp
o
se.
he
u
sed
data
c
o
l
l
e
ct
i
o
n
t
o
ols
m
easu
r
in
g
t
h
e
s
am
e
ac
hi
e
v
em
en
ts
.
d
e
vel
o
ped
b
y
t
he
in
ves
tiga
t
or.
He
f
o
u
nd
tha
t
5
9
s
t
ude
nts
per
f
orm
e
d
hi
ghe
r
on
m
ul
t
i
p
l
e-
c
h
o
i
ce
tes
t
s
tha
n
w
r
i
t
t
e
n
e
xa
ms
a
nd
perform
ance
t
ask.
C
ons
i
d
eri
ng
t
h
at
t
he
q
ue
sti
o
ns
e
xa
m
i
ne
d
are
d
eri
v
ed
f
rom
tea
c
h
er
-ma
d
e
te
sts.
i
t
i
s
no
tew
o
r
t
h
y
t
ha
t the
y
s
h
o
w
a
similar
d
i
strib
u
t
ion
t
o
ac
h
ie
ve
m
e
nt
r
ates on
the
que
st
i
on t
y
p
e
s in
t
he
P
IS
A tes
t
.
[2
7]
i
n
v
e
s
ti
ga
t
e
d
st
ude
n
t
ach
i
e
ve
me
nt
r
a
t
es
acc
ord
i
n
g
t
o
d
i
ffe
r
en
t
type
s
o
f
que
sti
ons
on
the
ba
si
s
of
all
su
b-f
i
el
ds
i
n
t
h
e
P
I
S
A
2003
a
n
d
P
I
S
A
200
6
tes
t
s.
H
e
fo
u
n
d
t
ha
t
stu
d
e
nt
s
ach
ie
ve
d
t
h
e
h
i
ghe
st
s
c
o
r
e
s
w
i
t
h
mult
i
p
le
c
h
o
i
c
e
que
sti
o
n
s
i
n
t
h
e
su
b-fiel
ds
o
f
re
a
d
i
n
g
ski
l
l
s
a
nd
s
cie
n
c
e
.
w
h
i
c
h
i
s
c
on
sis
t
en
t
w
i
th
t
he
r
esul
t
s
ob
ta
ine
d
w
i
t
h
i
n
t
h
e
sc
o
p
e
of
t
he
s
t
u
d
y
.
H
o
w
e
ve
r.
t
he
t
y
p
e
s
of
q
ues
t
i
o
ns
w
h
e
re
s
tude
nt
s
have
b
ee
n
th
e
m
o
st
succe
ssfu
l
i
n
t
h
e
su
b-fie
l
d
o
f
m
a
t
he
ma
t
i
c
a
l
l
i
t
era
c
y
ove
r
the
y
ea
rs
a
re
s
emi
-
st
ru
ct
u
r
ed
a
n
d
mu
lt
i
p
l
e
-ch
o
i
c
e
que
st
ions.
re
sp
ecti
v
e
l
y. This diffe
r
e
n
ce
m
ay
be d
u
e t
o
the
fa
c
t
t
ha
t
[2
7] de
t
ai
le
d t
h
e
q
u
es
t
i
o
n
ty
p
es differe
ntl
y
f
r
o
m
t
he
t
yp
e
s
o
f
qu
est
i
on
s
d
i
sc
u
s
se
d
in
t
hi
s
st
udy
.
a
n
d
t
h
at
t
he
P
IS
A
20
0
6
t
e
s
t
w
a
s
prima
r
il
y
foc
u
se
d
on
me
asurin
g
sc
ie
nce
l
i
tera
cy.
A
l
l
o
p
e
n
-e
n
d
ed.
sem
i
-struct
u
re
d
.
a
n
d
s
hor
t-an
swer
que
s
t
i
o
ns
w
ere
categ
or
iz
ed
a
s
st
r
u
c
t
ure
d
r
esp
ons
e
q
u
es
ti
ons
u
n
d
er
a
s
i
n
gl
e
hea
d
ing
i
n
t
h
i
s
st
ud
y.
T
h
i
s
ma
y
be
a
ttri
b
u
t
ed
t
o
t
h
e
P
I
SA
2
0
0
3
Cog
n
i
t
i
ve
T
es
t
C
ode
B
o
okle
t
a
n
d
t
he
P
IS
A
20
1
2
T
ec
h
n
ic
al
R
e
por
t
.
pub
l
i
sh
ed
b
y
t
h
e
O
E
C
D
.
Th
e
ov
e
r
al
l
ou
tc
ome
of
t
he
s
tu
dy
is
t
hat
s
t
u
d
e
n
t
a
c
h
ie
ve
me
nt
r
ates
a
re
h
i
g
h
er
i
n
mul
t
i
ple
c
h
o
i
ce
que
stio
ns
t
ha
n
i
n
o
t
h
er
ty
pes
o
f
q
ue
sti
ons.
Th
i
s
f
i
n
d
i
ng
is
s
u
p
p
o
rte
d
b
y
ma
ny
o
t
h
er
s
tu
die
s
.
[8],
[10],[1
5
],
a
nd
[18]
c
om
par
e
d
suc
cess
rate
s
o
n
t
he
b
a
s
is
o
f
que
st
i
on
ty
pes use
d
i
n mid-te
rm
o
r
fi
nal
e
x
am
s
f
o
r
di
ffere
n
t
d
i
sci
p
l
i
nes of hea
lt
h
st
ud
ies.
A
c
c
o
rdi
n
g
t
o
t
he
r
e
s
ults
o
f
t
h
e
st
udy.
s
tu
d
e
nt
s
ha
d
m
o
re
s
ucc
e
s
s
w
i
t
h
m
ulti
pl
e
ch
o
i
c
e
ques
t
i
ons
t
h
a
n
w
ith
ope
n-e
n
de
d
q
u
es
tio
ns.
[1
2]
,
[2
0]
a
n
d
[
3
3
]
a
i
m
ed
t
o
de
term
ine
th
e
succes
s
rate
o
f
s
t
udents
in
d
iff
e
rent
di
sc
ip
li
nes
s
u
c
h
a
s
nurs
i
n
g
.
chem
is
t
r
y
a
nd
p
hys
ics
a
c
c
o
r
d
ing
t
o
d
if
fe
rent
t
yp
es
o
f
qu
esti
on
s
and
inv
e
st
ig
at
e
the
effec
t
o
f
d
i
f
f
er
en
t
t
ypes
of
q
ues
tio
ns
o
n
a
cade
m
ic
a
c
h
ie
vem
en
t.
F
or
t
his
p
u
r
pose
.
s
t
u
de
n
t
s
w
e
r
e
a
sked
Evaluation Warning : The document was created with Spire.PDF for Python.
IJERE
I
S
S
N
:
2252-
88
22
T
h
e
Stu
d
e
n
t
s
Achiev
em
e
n
t i
n
T
u
rke
y
,
Acc
o
rdin
g t
o
Q
u
es
t
i
o
n T
y
pes
U
s
e
d
in
PISA ..
.
(Ye
ş
i
m
Özer
Özka
n
)
63
di
ffe
re
nt
t
y
p
es
o
f
q
u
es
ti
ons
s
uc
h
a
s
m
ul
t
i
p
l
e
cho
i
ce
q
ues
t
i
ons.
t
w
o
-s
tag
e
m
ul
t
i
p
l
e
c
h
oi
c
e
que
st
io
ns.
e
ssays.
and
re
stric
t
e
d
e
ssa
y
s.
a
nd
t
he
d
ata
we
re
c
oll
e
c
t
e
d
.
The
re
sults
r
e
v
eal
e
d
t
h
a
t
t
h
e
st
ud
en
ts
h
ad
t
h
e
h
i
g
h
e
st
r
a
t
es
of
s
uc
ce
ss
w
i
t
h
m
ultip
le
c
h
o
ic
e
ques
t
i
o
ns.
I
n
a
s
t
u
d
y
c
o
n
duc
t
e
d
b
y
[
34]
o
p
e
n-en
de
d
a
nd
6
0
m
ul
tip
le
-
c
ho
ice
tes
t
s
w
e
re
c
om
pare
d
i
n
e
va
lua
t
ing
the
rea
d
i
ng
c
o
m
p
rehe
nsio
n
ski
l
l
s
o
f
8
th
g
ra
de
s
t
u
den
t
s
in prima
ry
e
duc
a
tio
n.
A
c
c
ordi
n
g
to
t
he
f
ind
i
ngs
o
f
t
h
e
stud
y.
t
he
s
tu
d
e
nt
s
w
e
re
m
ore
suc
cess
f
u
l
w
it
h
m
u
lti
p
l
e
ch
oice
q
ues
t
i
ons
t
ha
n
op
e
n
-en
d
e
d
q
u
es
tio
ns
w
he
n
it
ca
me
t
o
me
asu
r
in
g
t
h
eir
re
ad
i
ng
s
k
il
ls.
w
h
ic
h
is
c
onsi
s
te
n
t
w
it
h
t
h
e
resu
l
t
s
o
b
t
a
i
ne
d
w
i
t
h
i
n
t
he
s
c
o
pe
o
f
the
stud
y.
[
9]
i
n
v
e
s
t
i
ga
te
d
t
h
e
te
net
s
o
f
t
h
e
c
o
n
s
truc
tiv
is
t
t
e
ac
hi
n
g
a
p
p
roa
c
h
a
do
p
t
ed
a
s
tod
a
y's
e
duc
a
t
i
o
n
m
odel
in his s
tu
dy an
d prese
n
te
d
th
e
m
w
ith a
cr
i
tic
a
l
p
oi
n
t
o
f
vie
w
.
The i
n
ve
st
i
g
at
or has f
o
c
us
ed o
n the im
pli
c
atio
ns
of
t
hi
s
a
pproa
c
h
i
n
t
h
e
fie
l
d
of
a
p
p
l
i
ca
t
i
on.
I
n
t
ro
duc
t
i
o
n
o
f
t
he
c
o
n
s
truc
ti
v
i
st
t
eac
hin
g
a
pproa
ch
t
o
o
u
r
c
o
unt
ry
’s
e
du
c
a
t
i
o
n
sy
st
e
m
b
y
lo
oki
ng
a
t
su
cc
e
s
sf
ul
out
co
mes
a
c
hie
v
e
d
b
y
o
t
her
cou
n
tr
i
e
s
w
i
t
h
ou
t
rea
l
l
y
un
dersta
n
d
i
n
g
w
h
at
it
is.
w
h
a
t
it
e
n
ta
i
l
s
a
nd
w
h
a
t
it
s
e
e
k
s
t
o
ac
h
i
e
v
e.
h
a
s
o
n
l
y
serve
d
t
o
c
h
a
nge
t
he
curr
i
c
u
l
um
c
o
n
t
e
n
t
a
nd
n
o
t
hin
g
m
ore
.
M
or
eove
r.
t
he
f
a
c
t
t
h
at
t
h
e
con
s
t
r
u
c
tiv
i
s
t
t
ea
chin
g
a
p
p
r
o
ach
h
as
b
e
e
n
a
d
opt
e
d
w
it
hou
t
t
r
ai
ni
ng
t
e
a
c
h
e
r
s
on
a
p
p
l
i
c
a
t
i
o
n
me
th
ods
a
nd
t
e
ch
ni
que
s
and
of
f
e
ri
n
g
t
h
em
t
o
ch
a
n
c
e
t
o
p
r
a
c
ti
ce
h
a
s
l
ed
t
o
p
r
ac
ti
cal
c
h
a
ll
en
ge
s
a
nd
i
n
e
f
fi
ci
en
ci
e
s
.
wh
ic
h
in
t
ur
n
de
cre
a
ses
the
effi
c
i
e
n
cy
l
ev
els
in
educ
a
t
i
o
n.
T
he
s
t
u
d
y
r
e
s
ults
a
r
e
c
onsis
te
nt
w
it
h
the
re
su
l
t
s
o
b
t
a
in
e
d
w
ith
in
t
h
e
s
co
p
e
o
f
th
is
s
t
udy
.
Th
e
grea
tes
t
i
ndic
a
t
or
o
f
suc
h
i
n
e
ff
i
c
ie
nc
ies
a
r
e
su
c
c
e
s
s
rate
s
fal
lin
g
from
38.7
0
%
to
32.
35
%
o
v
e
r
t
h
e
y
e
a
rs
betw
ee
n P
I
S
A
20
0
3
a
nd 2
0
1
2
w
hen
in
f
ac
t
q
u
i
t
e
t
h
e
co
ntrar
y
w
as
e
x
p
e
ct
ed.
S
o
me
r
esearche
r
[
11],[24]
a
nd
[2
6]
c
o
n
d
u
c
te
d
em
p
i
rica
l
stu
d
i
es
t
o
i
n
v
e
st
ig
at
e
th
e
ef
fe
ct
o
f
the
con
s
truc
t
i
v
i
s
t
t
e
a
c
h
i
n
g
ap
pro
a
ch
on
s
t
u
d
e
n
t
succ
ess
in
d
iffer
e
n
t
d
i
sc
ip
l
i
ne
s
s
u
ch
a
s
ge
ogra
p
h
y
a
nd
s
c
ie
nc
e
teac
h
i
ng.
A
nal
y
ses
sh
o
w
t
ha
t
c
onstruc
t
i
v
i
s
t
t
ea
ch
i
ng
gener
a
tes
h
i
ghe
r
ac
adem
ic
a
c
h
iev
e
m
e
nt
t
ha
n
t
r
ad
iti
on
a
l
met
h
od
s
of
t
e
a
c
h
ing
.
T
hi
s
sug
g
e
st
s
th
at
c
o
n
st
ru
c
t
ivi
s
t
t
e
a
c
h
i
ng
p
r
o
mo
tes
t
h
e
c
o
gn
i
tive
leve
ls
o
f
s
t
ude
nts
i
n
com
p
aris
on
t
o
tra
d
iti
o
n
a
l
m
eth
o
d
s.
T
he
s
t
udy
r
esu
lts
a
r
e
i
nc
o
n
s
i
s
te
n
t
w
it
h
t
h
e
res
u
l
t
s
ob
ta
ine
d
w
i
t
h
i
n
t
h
e
sco
p
e
o
f
t
hi
s
stud
y.
W
hen
on
e
lo
o
k
s
at
t
he
s
t
r
uc
ture
d
re
spons
e
que
s
t
i
o
ns
i
n
P
I
S
A
2003
a
nd
P
I
S
A
201
2
t
e
s
t
s
and
t
h
e
gen
e
ral
s
u
cc
ess
rat
e
,
it
i
s
see
n
t
ha
t
t
h
er
e
is
a
d
ec
re
a
se
i
n
s
u
cce
ss
r
ates
w
he
n
i
n
f
a
c
t
hig
h
e
r
a
c
a
dem
i
c
ac
hi
e
v
em
en
ts
w
er
e
bein
g
ex
pe
cte
d
f
o
l
l
o
w
i
ng
t
h
e
i
n
t
r
od
uc
tio
n
o
f
t
h
e
c
o
ns
truct
i
vi
st
t
e
a
c
hi
n
g
a
p
p
roac
h.
T
his
i
n
d
i
ca
te
s
th
a
t
c
o
n
st
ru
cti
v
i
s
t
te
ac
h
i
n
g
do
e
s
n
o
t
h
ave
an
i
mp
ac
t
o
n
a
w
ide
-
rangi
ng
tes
t
s
uch
as
t
he
P
I
S
A
te
st
do
ne o
n
an
i
n
t
e
r
nat
i
ona
l
sca
l
e
.
5.
CONCL
U
S
ION
It
c
a
n
b
e
c
o
nc
l
u
de
d
tha
t
s
t
u
den
t
a
c
h
i
e
vem
e
nt
s
dif
f
er
a
cc
ord
i
ng
t
o
t
he
y
e
a
rs
w
hen
th
e
t
e
sts
w
e
re
take
n
a
nd
the
t
y
pe
s
of
q
uestio
ns.
The
q
u
es
tio
n
t
y
pe
w
i
t
h
the
h
i
ghe
s
t
p
e
r
ce
nt
a
g
e
o
f
s
uc
cess
ar
e
m
u
l
t
i
p
le-
cho
i
ce
q
uest
io
ns
i
n
b
o
t
h
P
IS
A
tests.
T
he
t
y
p
e
o
f
que
st
ion
t
h
a
t
has
t
h
e
l
o
we
st
p
e
r
ce
n
t
a
g
e
o
f
s
ucc
e
ss
are
th
e
com
p
le
x
m
u
l
t
i
ple-
c
h
o
i
ce
q
u
e
s
t
i
ons
i
n
t
h
e
P
I
S
A
2
0
03
te
st
a
nd
s
t
r
uc
t
u
r
e
d
r
e
sp
onse q
u
e
s
ti
ons
i
n t
h
e
P
I
S
A
201
2
tes
t
.
A
ccor
d
in
g
to
t
h
e
c
ons
truc
t
i
v
i
st
e
d
u
c
a
t
i
o
n
a
pproac
h
e
f
f
ec
t
ua
te
d
in
2
00
5-2
0
06a
c
a
de
mic
year
,
i
t
i
s
expe
c
t
e
d
t
o
o
b
s
erve
a
r
i
s
e
i
n
c
on
struc
t
e
d
r
espo
nse
ques
t
i
o
n
typ
e
;
h
owever
,
fi
n
d
i
n
gs
o
f
t
h
e
stu
d
y
r
e
v
eal
t
h
a
t
the s
u
c
cess of
c
on
struc
t
e
d
re
s
po
nse
q
u
es
ti
on
s
i
s
d
ecr
ease
d
ac
c
o
rd
in
g t
o
th
e
appl
ica
tio
n y
e
ars.
REFE
RENCES
[1]
M.
F
.
Tu
rgu
t
an
d
Y. Bay
ku
l
,
“
E
ğ
iti
md
e
ö
l
ç
m
e v
e
değ
erlend
i
r
me,”
(6
. Bask
ı
),
An
k
a
ra,
Peg
e
m Ak
ad
emi
, 2
01
4
.
[2]
M
.
Ç
akan
,
“
T
h
e
r
o
l
e
an
d
imp
o
rtance
o
f
l
arge-s
c
a
l
e
achi
e
vem
e
nt
,”
Ed
ucat
io
n
a
nd Scien
ce,
v
ol/is
sue:
2
8(128),
pp.
1
9
-2
6, 2
00
3.
[3]
MEB
, “
T
I
MMS
2
01
5 u
l
u
s
la
ra
ra
sı
m
a
t
em
a
t
i
k
v
e fen
eği
l
iml
e
ri araş
t
ı
r
ması. u
l
u
s
a
l
ön rapor,”
20
16
.
[4]
M
E
B, “P
I
RL
S 2
0
01
Ul
u
slararası
ok
um
a b
ecerileri
nd
e gelişim p
roje
si. u
lu
sa
l r
a
p
o
r
,
”
20
03
.
[5]
M
E
B,
“
P
I
SA
2012
U
l
u
slararası
öğ
renci
değ
e
rlend
i
rm
e
p
r
og
ra
mı.
u
l
u
s
a
l
n
ih
a
i
r
a
p
o
r
,”
2
0
1
5
.
[6]
TÜ
S
İ
AD,
S
.
R.
Ş
i
r
in
a
n
d
S
.
Vatanartı
r
an,
“
P
ISA
20
12
d
eğ
erl
e
nd
ir
mes
i:
Türki
y
e
içi
n
v
eri
y
e
dayalı
eğiti
m
ref
o
rm
u
ön
erileri,
”
2
01
4.
[7]
S
.
G
ült
e
k
i
n
an
d
N
.
Ç
.
D
e
mi
rtaş
lı,
“
C
o
m
p
a
ring
t
he
T
es
t
i
n
f
o
rm
ati
o
n
o
bt
ain
e
d
t
h
ro
u
g
h
m
u
lti
p
l
ech
oi
ce
op
en-end
ed
and
mi
xed
ite
m
t
e
sts
bas
e
d
on
ite
m
res
pon
se
t
he
ory
,
El
ement
a
r
y
Edu
c
atio
n Online,
v
o
l/is
s
u
e
:
1
1
(1),
p
p.
2
51
-263,
20
12
.
[8]
P.
K
u
m
a
r
,
et a
l
.,
“
Co
m
p
arasion
of
p
erfo
rm
ance
in
p
h
y
s
i
olog
y
s
u
b
j
ect
a
s
s
es
sed
b
y
sh
ort
e
s
sa
y
ty
pe
q
ue
stio
ns
a
nd
m
u
lti
p
le
c
ho
i
ce qu
est
i
o
n
s
o
f
f
i
r
st year m
ed
ical st
u
den
t
s
,
”
Asia
n
jour
nal o
f
m
e
dical
scien
c
s,
v
o
l
. 8
,
pp
.
8
2
-
84
, 2
01
7
.
[9]
C.
E
.
Bay
r
ak
taro
ğl
u,
“
Co
nstru
c
tivist appro
ach in edu
c
at
io
n
an
d
a critical vi
e
w,”
Mas
t
er
T
hes
i
s
,
M
alt
e
pe
U
ni
versity
,
İsta
n
b
u
l
,
T
urk
e
y, 2
01
1.
[10]
S
.
A
al
ae
i,
et al.
,
“
A
C
om
paris
o
n
of
m
ultip
l
e-ch
oi
ce
a
n
d
e
ss
ay
q
u
e
st
ions
i
n
th
e
e
v
a
lu
ati
on
of
d
ent
a
l
st
ud
ent
s
,
”
Int
e
rna
t
i
o
n
a
l
Jo
u
r
n
a
l of Ad
vanced
Bi
o
t
echn
o
lo
g
y
and
Res
e
ar
ch
,
vol
/i
ss
ue:
7(5
)
,
p
p
.
1
67
4-16
80
,
201
6.
[11]
O.
Çeti
n
a
n
d
Y
.
G
ü
n
a
y,
“
T
h
e
Effect of
co
nst
r
uctiv
i
st
t
heo
r
y
o
n
s
t
u
d
en
ts’ achi
e
ve
m
e
nt
a
nd
t
h
e
ir way of
co
ns
truct
i
ng
kn
ow
ledg
e in s
c
i
ence ed
u
cat
ion
,
”
E
d
u
c
ati
on an
d Scien
ce,
v
ol/issue:
3
2(1
46),
p
p
.
24-38
,
2
00
7.
[12]
M
.
D
elaram
a
nd
A
.
S
h
a
rif
i
,
“A
c
o
m
paris
o
n
of
s
tudn
ts
s
co
res
in
m
ultip
l
e
-choice
and
essay
quest
i
ons
i
n
mother
a
nd
chi
l
d
healt ex
am
i
n
atio
n,”
Fut
u
re
of
medical educ
ation
journa
l
,
vo
l/issue
:
4(2
),
p
p
.
1
5-1
8
, 2
01
4.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N: 2
2
5
2
-
88
22
I
J
ERE
V
ol.
7,
N
o.
1
,
Mar
c
h 20
1
8
:
57
–
64
64
[13]
J
.
D
.
Ed
uw
em
a
n
d
I
.
E.
U
moin
y
a
n
g
,
“
I
tem
ty
pes
an
d
u
pper
bas
i
c
e
du
cati
on
s
t
u
d
ent
s
’
perf
o
r
m
a
nce
in
m
ath
e
m
a
ti
cs
in
t
h
e
s
o
u
t
h
e
r
n
se
n
a
to
r
i
a
l
d
i
s
t
r
i
c
t
o
f
c
r
o
ss
r
iv
e
r
s
t
a
te
N
ig
e
r
i
a,”
J
o
urn
a
l
o
f
M
ode
r
n
s
Ed
uca
t
i
on
Revi
ew,
vol/i
ssue:
4(1
)
, pp
. 5
7-7
3
,
2
01
4.
[14]
R.
U
.
E
n
ej
a
and
E.
F
.
Ikeh
,
“A
n
a
l
y
sis
o
f
s
choo
l
typ
e
a
nd
t
es
t
i
te
m
f
o
rm
at
o
n
studen
t
s
’
achi
e
vem
e
nt
i
n
f
i
n
a
nci
a
l
acco
un
tin
g,”
In
te
rn
atio
na
l J
o
urn
a
l
o
f
Edu
c
a
tion
an
d Re
se
a
r
c
h
,
vol
/i
ss
ue:
4(9
)
,
pp.
1
05
-114
,
2
0
1
6
.
[15]
A.
G
a
y
e
f
,
et al.
,
“
I
s as
k
i
ng
sam
e
q
ues
tio
n in
d
iff
e
re
n
t
w
ays
h
a
s any im
pact
on
st
udent
achi
e
vem
e
nt
?”
Sci
e
nc
e
d
i
r
ect
p
r
o
cedia
so
cial and
beh
a
vi
or
al sci
e
nces,
v
o
l
.
1
5
2
, pp
.
3
39
-3
42
, 2
01
4.
[16]
R.
L
uk
he
le
,
et
al
.
,
“
O
n
th
e
rel
a
ti
v
e
v
alu
e
o
f
m
u
ltipl
e-cho
i
ce.
c
ons
tru
c
ted
resp
on
se.
and
ex
am
inee-s
e
lect
ed
items
o
n
t
w
o
achi
e
vem
e
nt tests,”
P
r
og
ram S
t
atisti
cs
R
es
ea
rch
.
T
echni
cal
R
e
p
ort
No.
9
3-28,
1
99
3.
[17]
B.
L
uz,
“
C
om
p
a
ri
ng
s
t
u
d
e
nt
p
e
r
f
o
rm
an
ce
on
d
i
fferen
t
item
f
o
rm
at
s
re
l
a
ti
ve
t
o
ach
iev
e
men
t
l
evel
s
cu
tp
oints,”
Ed
uc
a
t
io
na
l Re
so
urc
e
s
In
for
m
a
t
i
o
n
Ce
nte
r
, 1
99
8.
[18]
D.
J
.
Pep
p
le,
et a
l
.
,
“
A
c
o
m
p
a
ris
o
n
of
s
tu
dent
p
erf
o
rm
an
ce
i
n
m
ulti
pl
e-c
h
oi
ce
an
d
l
on
g
e
s
sa
y
qu
e
s
tio
n
s
in
t
h
e
MBBS
s
ta
ge
I
p
hy
sio
l
og
y
e
x
a
m
ina
t
io
n
a
t
t
h
e
U
nive
r
s
i
t
y
o
f
t
h
e
W
e
s
t
I
n
d
i
e
s
(
M
o
n
a
C
a
m
p
u
s
)
,
”
Amer
ican
p
h
ys
iol
ogica
l s
o
ci
ety,
v
o
l
.
3
4,
pp.
8
6
-
89
,
2
010
.
[19]
K
.
S
cou
l
l
e
r,
“
The
i
n
f
l
u
e
nce
o
f
a
ss
esm
e
nt
m
etho
d
o
n
s
tu
den
t
s’
l
e
arni
ng
a
pp
roach
es:
M
u
ltip
le
c
h
o
i
ce
qu
est
i
o
n
ex
am
inat
io
n
v
e
rsu
s
assi
g
m
ent ess
a
y,
”
Hi
gher
E
d
uca
t
i
o
n
,
v
o
l.
3
5
, p
p.
453
-47
2
,
1998.
[20]
S.
T
e
m
e
l
,
et
a
l
.
,
“
T
he
e
f
f
ect
o
f
d
i
ff
erent
t
y
p
e
s
of
t
est
o
n
p
res
e
rv
ice
ch
em
istr
y
t
eachers’
achi
e
v
e
m
e
nt
r
el
ated
t
o
“
c
h
e
mi
cal
b
on
d
i
ng
,
”
Pr
ob
lem
s
of ed
uca
t
i
on in th
e
21
st
cent
u
r
y
,
vol.
4
1
,
p
p.
1
23
-129,
201
2.
[21]
OECD
,
“Codebook
for
cogniti
ve
ite
m data
fi
le,”
200
3.
[22]
D
.
Ö
zdemir,
“
C
on
st
rativ
e
a
n
a
ly
sis
of
p
s
y
ch
om
e
t
ric
p
r
op
erties
o
f
sh
ort
an
swe
r.
m
ulti
pl
e
-
ch
oi
c
e
and
es
say
ty
pe
t
e
sts,” M
a
s
t
er Thes
i
s
,
Hacet
tep
e
Un
i
versit
y
, An
k
ara, T
urk
e
y, 1
9
97.
[23]
S
.
T
osyalı,
“
Th
e
Com
p
ariso
n
o
f
i
t
em
a
n
d
t
est
feat
ures
o
f
m
u
ltip
le
c
h
o
i
ce.
s
hort
an
swered
a
nd
t
rue-f
a
ls
e
tes
t
s
in
aptitude t
e
s
ts,”
M
a
s
t
er T
hesis, Hacet
tepe
U
ni
ve
rs
it
y,
A
nk
ara, Turkey
, 2
00
2.
[24]
H
.
W
a
i
n
e
r
and
D
.
T
h
i
ss
en,
“
C
o
m
bi
n
i
ng
m
u
l
tiple-cho
ice
an
d
co
n
s
tr
u
c
ted
res
ponse
tes
t
s
cores
:
t
o
w
ard
a
M
a
rx
is
t
T
h
eor
y
o
f
test
c
onstruction,”
P
r
ogram
St
a
ti
st
ics Research.
Tech
n
i
cal
Repo
r
t N
o
. 92
-
93,
19
9
2
.
[25]
Z
.
K
arasu
and
M
.
Ü
n
l
ü,
“
Th
e
ef
f
ect
o
f
ach
ievemen
t
academ
ic
o
f
c
o
n
s
t
ruc
t
iv
e
m
e
th
od
i
n
g
e
o
g
raph
y
ed
uc
at
io
n
,
”
M
a
rmar
a
Geog
ra
ph
y
Jou
r
na
l,
vol.
12
,
pp
.
1
0
5
-
128,
2
00
6.
[26]
T.
Y
a
n
p
a
r
,
et
al
.
,
“
A
p
p
licat
ion
o
f
t
he
c
o
n
s
t
ru
ctivist
t
eachin
g
ap
pro
a
ch
i
n
so
lu
b
i
l
ity
s
ubject
t
h
r
oug
h
g
r
ou
p
act
iv
iti
es,
”
İ
nö
n
ü
Un
iversit
y
Jou
r
n
a
l of Ed
ucationa
l S
c
ien
ces,
vol/issue: 7(11),
pp.
11
3
-1
22
,
2
006
.
[27]
E.
D
e
m
i
r
,
“Th
e
s
tu
den
t
s
ach
iev
e
m
e
n
t
i
n
T
u
rk
ey.
accord
in
g
to
t
he
q
u
e
sti
o
n
t
y
pes
us
ed
i
n
Program
f
o
r
Internati
o
na
l
S
t
ud
e
n
t
Ass
e
ssment
(P
ISA)
c
o
gniti
ve
d
o
m
ai
n
t
e
st
s,
”
M
a
ster T
hesis, Hacet
tep
e
Unive
rs
it
y
, An
k
ara, Turk
e
y, 2
01
0.
[28]
E
A
RG
E
D
,
“PI
S
A 20
0
3
pro
jesi.
u
l
u
s
al
niha
i
rapo
r
,” 20
0
5
.
[29]
O
E
CD,
“
C
od
ebo
ok
fo
r
co
gniti
ve
ite
m
d
a
ta
fi
le,
”
201
2.
[30]
O
E
CD,
“
P
IS
A
200
3
t
echn
i
cal
r
eport
,
”
2
005.
[31]
O
E
CD,
“
P
IS
A
201
2
t
echn
i
cal
r
e
port
,
”
2
014.
[32]
M
.
O.
Ç
e
t
i
n
,
“
A
s
t
u
d
y
o
n
st
uden
t
s
?
achi
v
em
en
ts th
a
t are m
eas
ured th
r
oug
h mul
t
i
p
ly
cho
i
ce t
e
st
s.
w
ri
tt
en
ex
a
ms an
d
p
e
rf
orm
a
nce
tas
k
s
o
n
s
ien
ce
an
d
tech
nol
og
y
cou
r
se
a
nd
st
ud
ent
?
s
c
onc
ep
ti
on
s
towards
p
e
rf
orm
a
nce
tas
k
s
,
”
M
a
s
t
er
T
h
es
is, Ab
a
nt İzzet Bay
sal
Unive
r
s
i
t
y
,
Bo
lu
, T
urk
e
y,
200
9.
[33]
U
.
E
ffiong,
et al.
,
“Test
t
y
p
e
s.
s
tud
e
n
t
s
’
ach
ieve
m
e
n
t
i
n
sen
i
or
s
eco
nd
ary
s
c
ho
ol
p
hy
sic
s
a
n
d
e
ra
dic
a
t
io
n
o
f
p
ov
e
r
ty
an
d h
u
n
g
er i
n N
i
g
e
ria,”
IO
SR jo
u
r
nal o
f
r
e
sea
r
ch &
me
t
hod
i
n
ed
u
c
a
tio
n,
vol.
4,
p
p
.
0
1
-
05,
2
0
1
4
.
[34]
M
.
T
emizk
a
n
a
nd
M
.
E
.
S
a
llab
a
ş
,
“
O
k
u
d
u
ğun
u
anl
a
ma
b
e
ceris
in
in
d
eğerlen
d
i
r
ilmes
i
n
d
e
ço
kt
an
s
eçm
eli
testl
e
rle
açı
k
u
ç
lu
y
az
ı
l
ı
y
o
k
l
am
aları
n
k
arşılaş
t
ı
r
ılm
a
sı,”
Du
m
l
up
ı
nar Üni
v
ersi
t
e
si Sosy
al
B
ilimler
Dergisi,
vo
l.
3
0
,
pp.
2
07
-
22
0, 1
99
8.
BIOGRAPHI
E
S
OF
AUT
HORS
Yeşim
Özer
Ö
z
k
an
i
s
an
E
d
u
ca
ti
on
a
l
M
eas
urem
ent
and
Ev
aluati
on
s
p
eciali
s
t
.
R
es
earc
h
i
nteres
ts
incl
ud
e
it
em
r
espo
ns
e
th
eory,
differen
t
i
a
l
item
f
u
n
c
ti
oning
.
La
rge
S
cale
A
s
s
e
ssm
en
t
(P
IS
A
,
PIRL
S,TIM
M
S
)
e
d
u
cat
i
o
n
a
l
s
t
a
t
is
t
i
cs,
mul
t
ivar
iate
s
t
a
tist
i
cal a
na
l
y
sis
.
Nesri
n
Ö
ZA
SLA
N
i
s
a
t
eacher
a
t
M
i
nistry
o
f
Nat
i
o
n
al
E
d
u
catio
n
i
n
Tu
rkey.
Research
i
nteres
ts
mat
h
em
atics
educat
io
n. Large
S
c
a
l
e
As
s
essm
ent (P
IS
A
,
P
IRLS
,T
IM
M
S
)
e
ducat
io
n
a
l
st
ati
s
t
i
cs.
Evaluation Warning : The document was created with Spire.PDF for Python.