Int
ern
at
i
onal
Journ
al of E
valua
tio
n
an
d
Rese
arch
in
Educati
on (I
JE
RE)
Vo
l.
8
, No
.
3
,
Septem
ber
201
9
, p
p.
376
~
383
IS
S
N: 22
52
-
8822
,
DOI: 10
.11
591/
ije
re
.
v8
i
3
.
19642
376
Journ
al h
om
e
page
:
http:
//
ia
es
core
.c
om/
journa
ls
/i
ndex.
ph
p/IJE
R
E
A comp
uterized
adapti
ve test for
measuri
ng the p
hysics cr
itical
thin
kin
g
skills
Aang Z
ainul A
bidi
n
1
, E
di I
stiyon
o
2
, Nun
ung Fadi
lah
3
,
Wipsar
Su
n
u
Br
ams Dwan
d
aru
4
1,2
,
3
Depa
rtment
o
f
Ph
y
si
cs
Educat
ion,
Gradu
ate
Sc
hool,
Yog
y
ak
arta
Sta
te
Univer
sit
y
,
Indone
si
a
4
Depa
rtment of
Ph
y
sics
Edu
cati
on
,
Yog
y
ak
art
a
Stat
e
Univer
sit
y
,
Indone
sia
Art
ic
le
In
f
o
ABSTR
A
CT
Art
ic
le
history:
Re
cei
ved
A
pr
21
, 201
9
Re
vised
Ju
l
3
0
,
201
9
Accepte
d
Aug 2
2
, 201
9
Cla
ss
ic
al
assess
m
ent
s
tha
t
are
not
comprehe
ns
ive
and
do
not
disti
nguish
student
s'
in
it
i
al
abi
li
t
ie
s
m
ake
m
ea
surem
ent
result
s
far
from
the
ac
tua
l
abi
litie
s
.
Thi
s
st
ud
y
was
condu
c
te
d
to
produc
e
a
computer
ized
a
dap
ti
v
e
te
st
for
ph
y
s
ic
s
c
rit
i
ca
l
thi
nk
ing
skil
ls
(CAT
-
Ph
y
sCr
iT
S)
th
at
m
et
th
e
fe
asibil
i
t
y
cri
t
eri
a
.
Th
e
t
est
was
pre
sent
ed
for
the
ph
y
si
cs
subjec
t
o
f
11
th
g
rad
e
h
igh
school
student
s
with
two
-
ti
er
m
ult
ipl
e
-
choi
c
e
form
at
.
Thi
s
deve
lopmen
t
rese
arc
h
was
ba
sed
on
th
e
4
-
D
m
odel
combined
with
th
e
te
st
d
eve
lopmen
t
m
odel
b
y
Orion
do
&
Antonio.
E
le
ven
exp
ert
s
an
d
577
11
th
gra
de
high
school
student
s
in
Kulonprogo,
Indone
sia,
hav
e
par
ticip
at
ed
.
The
m
edi
a
fea
sibil
i
t
y
and
the
cont
en
t
v
al
idit
y
of
th
e
item
s
was
assess
e
d
b
y
expe
rts,
wh
il
e
item
and
abi
litie
s
par
ame
te
rs
were
esti
m
at
ed
b
y
item
response
the
or
y
.
The
result
s
obta
in
ed:
1)
CAT
m
edi
a
was
dec
la
r
ed
ver
y
f
ea
s
ibl
e
and
con
te
nt
val
idi
t
y
of
136
it
ems
was
dec
l
are
d
va
li
d;
2)
al
l
i
te
m
s
fit
te
d
to
par
tial
c
red
i
t
m
odel
,
the
it
em
rel
i
ability
was
cl
assifie
d
as
good,
and
the
d
iffi
cu
lty
ind
ex
of
it
ems
was
good;
3)
the
re
sults
of
the
CAT
-
Ph
y
sCriTS
were
equi
v
al
en
t
to
student
s
ac
ad
emic
ac
h
ie
v
ement.
Based
o
n
the
result
s,
C
AT
-
Ph
y
sCriTS
has
fulfi
lle
d
the
req
u
ire
m
ents
as
a
m
ea
surin
g
instrument
wi
th
m
ea
surem
ent
ti
m
es
were
faste
r and
m
ore
comprehe
nsive
f
or
la
rg
e
-
sca
le a
s
sess
m
ent
s.
Ke
yw
or
d
s
:
Ad
a
ptive test
Assessm
ent of
ph
ysi
cs lea
rn
i
ng
Crit
ic
al
think
in
g
s
kill
s
Item
r
esponse
t
heory
Tw
o
-
ti
er m
ultip
le
-
c
ho
ic
e
Copyright
©
201
9
Instit
ut
e
o
f Ad
vanc
ed
Engi
n
ee
r
ing
and
S
cienc
e
.
Al
l
rights re
serv
ed
.
Corres
pond
in
g
Aut
h
or
:
Aang Z
ai
nul A
bid
in
,
Dep
a
rtm
ent o
f
Ph
ysi
cs E
ducat
ion
,
G
ra
du
at
e
School,
Yogyaka
rta St
at
e
Un
i
ver
sit
y,
Kar
a
ng Mal
an
g,
Ca
t
ur
Tu
ngga
l, De
pok, Sle
m
an,
D
.I.
Yog
ya
kar
ta
, 5
5281
, In
donesia.
Em
a
il
:
aang
.za
inu
l.a
bid
in
@
gm
ai
l.co
m
1.
INTROD
U
CTION
Crit
ic
al
think
in
g
sk
il
ls
are
the
m
ai
n
and
first
thing
s
that
m
u
st
be
con
side
re
d
in
le
arn
in
g
in
the
era
of
industrial
re
voluti
on
4.0,
as
i
n
the
le
ar
ning
fr
am
ewo
r
k
de
velo
ped
by
Pa
rtnershi
p
f
or
21
st
Ce
ntury
Le
arn
i
ng
(P
21
)
[
1]
.
Althou
gh
t
he
crit
ic
al
think
in
g
is
of
te
n
ref
e
rr
e
d
to
as
the
m
os
t
i
m
po
rta
nt
el
e
m
ent
in
physi
cs
le
arn
in
g,
i
n
re
al
i
ty
,
crit
ic
al
t
hinki
ng
asses
s
m
ent
is
an
are
a
that
was
oft
en
ov
e
rlo
ok
e
d
[2
,
3]
.
Wh
er
e
as
the
assessm
ent
is
extrem
el
y
need
ed
t
o
fi
nd
ou
t
wh
et
her
st
udents
ha
ve
ha
d
su
f
fici
ent
ph
y
sic
s
crit
ic
al
think
i
ng
sk
il
ls
an
d
w
ha
t
treatm
ent
shou
l
d
be
gi
ve
n
[4]
.
O
ne
r
eas
on
causi
ng
the
assessm
ent
of
crit
ic
al
think
i
ng
in
ph
ysi
cs
le
ar
ning
is
oft
en
ov
e
r
looked
is
t
he
la
ck
of
av
ai
la
bili
ty
of
te
sts
that
sp
eci
fical
ly
m
easur
e
the
phys
ic
s
crit
ic
al
think
in
g
sk
il
ls.
S
uch
problem
s
hav
e
act
ually
of
te
n
be
en
com
pl
ai
ned
of
by
seve
ral
researc
her
s
[5
,
6]
.
It
m
eans
that
the
dev
el
opm
ent
of
sta
nda
rd
iz
e
d
crit
ic
al
thinki
ng
te
sts
m
us
t
be
adjuste
d
to
the
existi
ng
needs
(physic
s)
s
o t
ha
t suc
h prob
l
e
m
s w
il
l no
t al
ways be
r
e
peat
ed
.
Anothe
r
a
risin
g
prob
le
m
is
the
ina
ppr
opria
te
te
st
fo
rm
at
for
m
easur
in
g
crit
ic
al
think
in
g
s
kill
s
an
d
accom
m
od
at
ing
m
at
erial
coverage
.
T
he
for
m
s
of
te
sts
tha
t
are
of
te
n
us
e
d
a
re
essay
s
a
nd
m
ulti
ple
-
cho
ic
es.
The
essay
is
t
he
m
os
t
co
m
pr
e
hensi
ve
form
of
a
te
st
[
7]
.
Howev
e
r,
it
has
m
any
weaknes
ses
su
c
h
as
a
li
m
it
ed
m
easur
e
of
high
-
le
vel
thin
king,
m
or
e
was
te
f
ul
in
te
rm
s
of
tim
e
and
cost,
high
s
ubj
ect
ivi
ty
,
and
it
is
dif
fic
ult
to
dete
rm
ine
its
validit
y
a
nd
reli
abili
ty
[8
,
9]
.
Me
an
w
hile,
m
ulti
ple
-
cho
i
ce
ha
ve
m
any
ad
van
ta
ges
suc
h
a
s
Evaluation Warning : The document was created with Spire.PDF for Python.
In
t J
E
val & R
es Educ
.
IS
S
N:
22
52
-
8822
A com
pu
te
rize
d ada
ptive
test
for
m
eas
ur
i
ng
the physic
s c
rit
ic
al thinki
n
g sk
il
ls i
n
...
(
Aa
ng Za
i
nu
l
Abi
din
)
377
easy
to
im
ple
m
ent
in
la
rg
e
c
la
sses,
a
high
l
evel
of
obj
ect
i
vit
y,
broa
d
m
a
te
rial
cov
e
rag
e
,
an
d
ca
n
be
c
orrecte
d
easi
ly
[10]
.
The
disad
va
nta
ges
of
the
m
ulti
ple
-
cho
ic
e
a
re
le
ss
com
pr
ehensi
ve
an
d
the
stud
e
nt'
s
think
in
g
pro
cess
ca
nnot
be
seen
cl
earl
y
[11]
.
Ba
se
d
on
t
he
se
stre
ngths
a
nd
wea
knesses,
t
he
su
i
ta
ble
form
of
t
he
te
st
m
us
t
be
deter
m
ined
by
c
ons
iderin
g
validit
y,
reli
abili
ty
,
and
com
pr
e
hen
s
iveness
.
T
he
s
olu
ti
on
to
this
pro
blem
is
the
im
ple
m
e
ntati
on
of
tw
o
-
ti
er
m
ult
iple
-
cho
ic
e
te
sts
[
12]
.
This
form
at
con
sist
s
of
se
ve
ral
ans
we
r
opti
ons
and
se
ve
ral
op
t
ion
s
for
reas
on
s,
so
it
will
require
stu
de
nts
to
thin
k
ab
out
de
te
rm
ining
the
reasons
s
uita
bl
e
for
their
c
ho
ic
e
of
answers
[
13
]
.
This
is
i
n
acc
orda
nce
with
w
hat
wa
s
m
entioned
in
[
3]
that
“reaso
n”
ca
n
c
hange
or
im
pr
ov
e
one'
s
m
ind
to
act
m
or
e
pr
eci
sel
y.
In
ad
diti
on,
the
us
e
of
reas
on
s
wh
e
n
an
s
wer
i
ng
m
ulti
pl
e
-
ch
oice
te
sts i
s a sensi
ti
ve
an
d
e
ff
e
ct
ive
way to a
dd
v
al
ue
t
o
th
e c
om
pr
ehen
sive
ne
ss of
qu
e
sti
on it
e
m
s.
The
t
hird
pr
oble
m
is
the
use
of
cl
assic
al
te
st
theo
ry
w
hich
has
m
any
weaknesse
s
al
so
m
akes
m
easur
em
ent
resu
lt
s
far
f
ro
m
the
act
ual
abiliti
es.
In
[14]
,
it
was
sai
d
that
in
In
do
nesia,
there
are
sti
ll
m
any
cl
assic
al
te
st
theo
rie
s
us
e
d
i
n
a
naly
zi
ng
te
sts
an
d
est
im
a
ti
ng
a
bili
ti
es.
The
cl
assic
al
t
est
theo
ry
has
m
any
disad
va
ntages,
one
of
wh
ic
h
is
the
cha
rac
te
risti
cs
of
it
e
m
s
dep
end
i
n
g
on
the
te
st
pa
rtic
ipants
[15
]
.
Fo
r
exam
ple,
a
quest
ion
would
be
an
ea
sy
quest
ion
if
a
nsw
ered
by
a
cl
ev
er
pa
rtic
ipant,
oth
e
rw
ise
,
a
quest
io
n
would
be
dif
fi
cult
if
answ
e
r
ed
by
th
e
le
ss
intel
li
gen
t
par
ti
ci
pan
ts
[
16
]
.
Likewise,
the
scor
e
re
flect
ing
th
e
par
ti
ci
pa
nts'
abili
t
ie
s
dep
e
nds
on
the
te
st.
If
this
ha
ppe
ns
,
t
he
act
ual
le
vel
of
abili
ty
of
the
te
st
par
ti
ci
pa
nt
will
nev
e
r
be
know
n
[
17]
.
T
he
s
olu
ti
on
to
t
his
pr
ob
le
m
is
the
im
ple
m
entat
ion
of
it
e
m
response
theo
ry
(
IRT
).
T
his
is
beca
us
e
t
he
le
vel
of
the
pa
r
ti
ci
pan
t'
s
abili
ty
and
e
stim
at
i
on
of
t
he
it
em
par
am
et
ers
is
m
easur
ed
acc
urat
el
y
[18]
.
T
her
e
f
or
e
, in
de
velo
ping
the c
onstructs
of the
physi
cs
crit
ic
al
think
in
g
te
sts m
us
t be
don
e
w
it
h t
he IRT
.
The
fou
rth
prob
le
m
is
the
eff
ic
ie
ncy
a
nd
eff
ec
ti
ve
nes
s
of
the
te
st.
I
n
recent
ye
a
rs
,
the
us
e
of
com
pu
te
rs
in the assessm
ent
process
beg
a
n t
o
be
i
m
ple
m
e
nted
by the
governm
ent o
f
Indonesia in the
nationa
l
high
sch
ool
ex
a
m
kn
own
as
a
com
pu
te
r
-
ba
sed
te
st
(CBT)
.
The
natio
nal
exam
us
es
the
sam
e
te
st
design
f
or
stud
e
nts
with
t
he
sam
e
age
or
le
vel
of
ed
uc
at
ion
ass
um
ing
that
they
have
the
sam
e
abili
ty
[19]
.
I
n
re
al
it
y,
stud
e
nts
betwe
en
sch
oo
ls
have
sign
ific
ant
di
ff
e
ren
ces
in
a
bi
li
t
ie
s.
Test
s
that
ign
ore
par
ti
ci
pan
ts'
abili
ti
e
s
will
be
s
uitable
for
the
pur
po
se
of
m
easur
in
g
th
e
achiev
em
ent
of
gr
a
du
at
e
c
om
petency
sta
nd
a
r
ds
becaus
e
each
stud
e
nt
has
th
e
sa
m
e
co
m
pe
te
n
cy
sta
nd
ar
ds.
Howe
ver,
it
is
le
ss
eff
ic
ien
t
to
m
easur
e
par
ti
ci
pa
nts'
a
bili
ti
es,
especial
ly
par
t
ic
ipa
nts
with
low
a
nd
high
abili
ti
es
[20]
.
This
is
bec
ause
m
any
quest
ion
it
em
s
c
anno
t
disti
nguish
te
st
pa
rtic
ipants
[
21
]
.
O
ne
s
ol
ution
is
to
ap
ply
ada
ptive
te
sts.
The
im
ple
m
e
ntati
on
of
c
ompu
te
r
-
base
d
a
dap
ti
ve
te
sts
is
kn
own
as
com
pu
te
r
iz
ed
a
dap
ti
ve
te
sti
ng
(CA
T).
CAT
has
t
he
adv
a
ntage
s
of
bei
ng
m
or
e ef
fici
ent b
ecau
se it
e
m
s t
hat are
to
o
eas
y or
to
o diff
ic
ul
t can b
e a
vo
i
de
d
s
o
that t
he
l
eng
t
h
of the te
st t
i
m
e
can
be
re
du
ce
d
with
ou
t
re
duci
ng
the
m
eas
ur
em
ent
pr
eci
s
ion
le
vel,
te
st
secur
it
y
is
m
or
e
gu
ara
ntee
d,
and
th
e
par
ti
ci
pa
nt'
s
scor
e
is
im
m
edia
te
ly
kn
ow
n
[22
,
23]
.
With
t
he
am
az
ing
s
olut
ion
s
offer
e
d
by
CAT,
it
w
ould
be
ver
y
us
e
fu
l i
f
a
pp
li
ed
in
m
easur
i
ng the
physi
cs criti
cal
thin
king s
kill
s of st
ud
e
nts.
The
rele
va
nt
r
esearch
was
t
he
dev
el
opm
ent
of
C
AT
for
as
sessing
pro
ble
m
-
so
lv
ing
s
kill
s
co
nducted
by
[
24
]
.
T
he
r
esults
obta
ine
d
fou
nd
t
hat
te
sts
dev
el
op
e
d
usi
ng
C
AT
were
eff
ect
ive
t
o
app
ly
.
In
[13]
al
so
dev
el
op
e
d
c
riti
cal
think
i
ng
te
st
instr
um
ent
s
in
t
he
f
orm
of
tw
o
-
ti
er
m
ulti
ple
-
cho
ic
e.
The
res
ults
ob
ta
ine
d
sh
ow
t
hat
two
-
ti
er
m
ulti
ple
-
cho
ic
e
quest
io
ns
wer
e
i
n
ac
corda
nce
with
the
cha
racteri
sti
cs
of
ph
ysi
c
s
that
pr
i
or
it
iz
e
the
r
easo
ns
in
answ
erin
g
a
pro
blem
.
2.
RESEA
R
CH MET
HO
D
This
is
a
re
se
arch
a
nd
de
ve
lop
m
ent
stud
y
(R
&
D)
.
Th
e
resu
lt
e
d
pro
du
ct
was
C
A
T
-
P
hysCriTS
pro
gr
am
.
This
pro
gr
am
con
sist
s
of
a
we
bs
it
e
-
base
d
CAT
m
edia
and
an
i
tem
ban
k
co
nc
ern
i
ng
ph
ysi
cs
crit
ic
al
think
i
ng
sk
il
ls.
The
de
velo
pme
nt
m
od
el
us
e
d
w
as
the
4
-
D
(d
e
fine,
desi
gn,
de
velo
p,
&
di
sse
m
inate
)
by
[25]
,
wh
ic
h
wa
s
co
m
bin
ed
with
th
e
te
st
dev
el
opm
ent
m
od
el
by
[26]
.
T
he
c
ombinati
on
o
f
t
he
two
m
od
el
s
pr
oduce
d
a
de
velo
pm
ent p
r
oce
dure
s
ho
wn in Fi
gure
1.
A
total
of
577
stud
e
nts
par
ti
ci
pated
i
n
this
st
ud
y.
They
we
r
e
11
th
-
gra
de
st
ud
e
nts
from
si
x
se
nior
hi
gh
sch
oo
ls
in
Ku
l
onpro
go
Re
ge
ncy,
I
ndonesi
a
,
in
2018
.
T
he
su
bject
s
du
rin
g
the
lim
it
ed
t
rial
sta
ge
wer
e
344
stud
e
nts.
T
his
nu
m
ber
wa
s
in
accor
dan
ce
wi
th
the
IRT
re
quirem
ents
[27]
.
Me
anwhil
e,
th
e
su
bject
s
du
ring
th
e
fiel
d
te
st
sta
ge
we
re
233
stu
de
nts.
All
par
ti
ci
pan
ts
we
re
sel
ect
ed
based
on
the
cat
e
gory
of
sch
ool
aca
dem
ic
ach
ie
vem
ent
determ
ined
by
the
2017
Nati
onal
Exam
inatio
n
sc
or
e
s.
The
sa
m
ple
sel
ect
i
on
was
m
ade
so
tha
t
the
inf
or
m
at
ion
obta
ine
d
ca
n
represe
nt
stude
nts'
abili
ti
es
in
gen
e
ral.
In
t
his
dev
el
op
m
ent
proce
dure
, d
at
a
wer
e
colle
ct
ed
th
r
ough
:
1)
quest
io
nn
ai
re
giv
in
g
to
e
xperts
t
o
obta
in
m
edia
fe
asi
bili
ty
data
of
CAT
a
nd
c
on
te
nt
validit
y
of
the
it
e
m
s,
2)
pape
r
-
base
d
te
st
(PB
T)
in
lim
it
ed
tria
ls
to
find
ou
t
the
c
har
act
erist
ic
s
of
the
it
e
m
s,
and 3) C
AT
i
n t
he
fiel
d t
rial
stage to
f
i
nd out
the d
em
ograph
ic
s o
f
stu
de
nts' ab
il
it
ie
s.
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2252
-
8822
In
t.
J
.
Eval
.
&
Re
s
.
E
du
c
.
Vo
l
.
8
, No
.
3
,
Sept
e
m
ber
2019
:
376
-
383
378
Figure
1. The
dev
el
op
m
ent pro
ce
dure
Fo
r
data
analy
sis,
exp
e
rts'
j
udgm
ent
was
use
d
to
assess
the
co
ns
tr
uctio
n
of
the
CA
T
-
Ph
ysC
riTS
pro
gr
am
.
The
c
on
st
ru
ct
io
n wa
s r
el
at
ed
t
o
m
e
dia f
easi
bili
ty
an
d co
ntent
val
idit
y of
the
it
em
s.
The
feasibi
li
ty
o
f
the
m
edia
was
analy
zed
by
com
par
in
g
the
ave
rag
e
sc
or
e
s
(
̅
)
with
ide
al
aver
a
ge
(
=
1
2
⁄
(
max
s
core
+
min
s
core
)
)
an
d
i
deal
sta
nd
a
r
d
dev
ia
ti
ons
(
=
1
6
⁄
(
max
s
core
−
min
s
core
)
)
[28]
.
If
̅
≤
−
1
.
8
×
i
s
c
at
eg
or
i
zed
as
“
ver
y
wea
k”,
−
1
.
8
×
<
̅
≤
−
0
.
6
×
is
c
at
eg
or
i
ze
d
as
“wea
k”,
−
0
.
6
×
<
̅
≤
+
0
.
6
×
is
c
at
eg
ori
zed
as
“su
ff
ic
ie
nt”,
+
0
.
6
×
<
̅
≤
+
1
.
8
×
is
c
at
eg
ori
zed
a
s
“
good
”,
an
d
̅
>
+
1
.
8
×
is
c
at
e
gori
zed
as
“
ve
ry
go
od
”.
Me
an
w
hile,
the
c
on
te
nt
validit
y
of
it
e
m
s
w
as
ana
ly
zed
usi
ng
the A
i
kens’ f
orm
ula
[29]
.
T
he
it
e
m
is said to be
valid if
it
h
a
s a c
oeffici
ent
value
a
bove
th
e thr
es
hold.
The
it
em
char
act
erist
ic
s
were
rev
ie
wed
ba
s
ed
on
t
he
em
pirical
validit
y,
the
te
st
reli
abili
ty
,
and
t
he
it
e
m
diff
ic
ulty
ind
e
x.
The
em
pirical
validit
y of
the
it
em
s
was
determ
ined
base
d
on
the f
i
tt
ing
ness
bet
w
een
th
e
it
e
m
with
the
scor
i
ng
m
od
el
.
The
i
te
m
fit
was
based
on
the
res
ults
of
a
lim
i
te
d
tria
l
sta
ge
.
T
he
s
t
udent
s’
answers
we
re
analy
zed
by
a
par
ti
al
cred
it
m
od
el
(P
CM
)
in
the
form
of
po
ly
tom
ou
s
da
ta
,
n
am
e
ly
scor
e
1
if
wrong
a
ns
w
er
and
wro
ng
rea
so
n,
sc
or
e
2
if
correct
ans
we
r
an
d
w
ron
g
re
aso
n,
sc
or
e
3
if
w
ron
g
ans
we
r
an
d
correct
reas
on,
and
sc
or
e
4
if
the
ans
wer
a
nd
reason
are
bo
t
h
co
rr
ect
[30
,
31]
.
Sc
or
e
s
we
re
a
naly
zed
us
ing
th
e
Qu
est
program
.
The
re
su
lt
s
ob
ta
ine
d
the
e
stim
ation
of
the
I
NF
I
T
m
ea
n
of
square
va
lue
(INF
IT
MNS
Q).
Ov
e
rall
,
the
te
s
t
is
sai
d
to
be
f
it
if
t
he
MNS
Q
INFIT
val
ue
i
s
1
.
0
±
0
.
0
[
32
]
.
The
go
odness
of
fit
of
each
it
em
can
be
see
n
i
n from
the I
N
FIT M
NSQ
valu
e on eac
h
it
em
with the
crite
ri
a receive
d
a
re
0.77 to
1.3
0.
The
reli
abili
ty
of
the
sc
ores
of
te
st
res
ults
was
est
im
a
t
ed
acco
r
ding
to
the
IRT
ba
sed
on
th
e
reli
abili
ty
of
it
e
m
est
i
m
a
te
s
and
t
he
reli
abili
ty
of
cas
e
est
im
at
e
[33]
.
Bot
h
est
im
at
es
wer
e
ob
ta
ine
d
fro
m
the
analy
sis
of
the
Qu
est
pro
gram
.
I
f
t
he
reli
a
bili
ty
value
(
)
<
0
.
67
is
c
at
egori
zed
as
“
poor
”,
0
.
67
≤
<
0
.
80
is
c
at
eg
or
i
ze
d
as
“
s
uffici
ent
”,
0
.
80
≤
≤
0
.
90
is
c
at
eg
ori
zed
a
s
“
good
”,
0
.
90
<
≤
0
.
94
is
c
at
eg
or
i
ze
d
a
s
“
ver
y
go
od
”
,
a
nd
>
0
.
94
is
c
at
e
gori
zed
as
“
e
xcell
e
nt
”
[
34]
.
The
r
el
ia
bili
ty
was
t
hen
pro
ven
by
a
gra
ph
of
the
relat
ion
s
hi
p
betwee
n
in
form
ation
f
unct
i
on
an
d
SE
M
ob
t
ai
ne
d
f
ro
m
the
Parscale
pr
og
ram
[35]
.
Me
anwhil
e,
th
e
it
e
m
diff
ic
ulty
ind
ex
(
b
)
w
a
s
ob
ta
ine
d
thr
ough
analy
sis
of
the
Qu
e
st
pro
gr
am
.
The
it
e
m
was
sta
te
d
to
be
go
od
if
the
diff
ic
ulty
ind
ex
wa
s
m
or
e
than
-
2.0
or
le
ss
than
+2.0
[
36
]
.
I
f
it
was
cl
os
e
to
-
2,
the
it
e
m
w
as said t
o be easy,
whe
reas if
it
w
as cl
os
e t
o
+
2
the
n t
he
it
em
w
as said to be
dif
ficu
lt
.
The
la
st
data
was
relat
e
d
t
o
t
he
est
im
at
i
on
of
the
stu
den
ts
’
a
bili
ty
(
)
.
M
axim
um
l
i
kelihoo
d
est
i
m
ation
(M
L
E
)
was
us
e
d
as
a
m
et
ho
d
of
est
im
a
ti
ng
stud
e
nt
a
bili
ti
es
ap
plied
to
the
CA
T
-
P
hy
sCriTS
pro
gr
am
[37]
.
The
value
ca
n
be
known
dire
ct
ly
after
t
he
st
ud
e
nt
c
om
plete
s
the
te
st.
Stu
den
ts'
a
bili
ti
es
wer
e
then
a
naly
zed
by lo
ok
i
ng at t
he
a
ver
a
ges
i
n ea
ch
sc
hool cat
egory.
3.
RESU
LT
S
A
ND AN
ALYSIS
3.1.
Constr
ucti
on
of
t
he
CAT
-
P
hy
s
CriT
S
pr
og
ra
m
The
CAT
-
P
hy
sCriTS
co
ns
ist
s
of
tw
o
m
ai
n
com
po
ne
nts,
nam
el
y
web
sit
e
-
base
d
CA
T
m
edia
an
d
an
it
e
m
ban
k.
T
he
con
st
r
uctio
n
of
the
C
AT
m
edia
was
relat
ed
to
the
form
at
and
t
he
al
gori
thm
us
ed.
The
CAT
m
edia
is
a
we
bs
it
e
with
a
te
st
pr
ese
ntati
on
syst
e
m
as
sh
own
in
Fi
gure
2,
w
hile
the
exam
ple
of
stude
nt
te
st
resu
lt
s
a
re sh
own
in Fi
gure
3.
Evaluation Warning : The document was created with Spire.PDF for Python.
In
t J
E
val & R
es Educ
.
IS
S
N:
22
52
-
8822
A com
pu
te
rize
d ada
ptive
test
for
m
eas
ur
i
ng
the physic
s c
rit
ic
al thinki
n
g sk
il
ls i
n
...
(
Aa
ng Za
i
nu
l
Abi
din
)
379
Figure
2. Flo
w
char
t
of
CAT
-
Ph
ysC
riTS
Figure
3. Exa
m
ple o
f
stu
dent
test
r
esults
The
CA
T
al
go
rithm
was
adap
te
d
from
the
al
gorithm
dev
el
op
e
d
in
[38]
wh
ic
h
wa
s
ba
sed
on
IRT,
log
ic
,
a
nd
sim
ple
sta
ti
sti
cs.
In
Fig
ur
e
2
we
can
see
that
s
tud
e
nts
were
gi
ven
an
i
niti
al
it
e
m
with
m
od
erate
diff
ic
ulty
(
b
= 0
)
a
nd
the initi
al
ab
il
it
y
was
al
so
assu
m
ed
to b
e zero.
If
stu
den
ts a
ns
we
re
d corr
ect
ly
(
sco
r
e 3
o
r
4)
t
hen
stu
de
nt
s
wer
e
gi
ven
a
m
or
e
diff
ic
ul
t
qu
est
io
n,
i
f
wrong
(sc
or
e
1
or
2)
the
n
st
ud
e
nt
s
wer
e
gi
ven
an
easi
er
qu
est
io
n.
Wh
e
n
t
his
process
t
ook
pla
ce,
the
abili
ty
(
)
was
est
im
a
ted
by
the
m
axi
m
u
m
l
ikeli
hood
est
i
m
ation
(M
LE).
T
his
est
im
at
ion
was
do
ne
by
cal
culat
ing
t
he
abili
ty
(
)
,
the
pro
bab
il
it
y
of
co
rr
ect
a
nswe
r
(
(
)
)
,
the
pro
ba
bili
ty
of
w
r
ong
a
nswer
(
(
)
)
,
inf
or
m
at
i
on
functi
on
(
(
)
)
,
sta
nd
a
r
d
er
ror
(
(
)
)
,
and
diff
e
re
nce
of
SE
[37]
.
T
his
al
gorithm
was
al
so
il
lustrat
ed
in
Fig
ur
e
3
.
When
stu
de
nts
an
swe
r
c
orrectl
y
diff
ic
ult
it
e
m
,
stud
e
nts'
abili
ties
al
so
rise.
T
he
dec
rease
in
a
bili
ty
reco
r
de
d
in
it
em
s
5,
13,
an
d
16
was
be
cause
the
stu
den
t
a
nswere
d
inc
orrec
tl
y
(sco
re
1
or
2)
on
the
previ
ou
s
it
e
m
.
The
final
abili
ty
of
stud
e
nts
was
0.1
9.
The
te
st
will
s
top
if
the
it
em
s
r
un
out
or
t
he
le
vel
of
accu
racy
of
the
a
bi
li
ty
est
i
m
a
te
(
diff
er
en
ce
of
~
0
)
has bee
n
reac
he
d
a
nd or if
the
test
tim
e w
as
up.
The
CA
T
m
e
dia
has
been
assessed
for
it
s
feasibil
it
y
by
el
even
ex
p
e
rts
.
Th
ere
a
re
two
as
pects
assessed
,
nam
e
ly
the
app
eara
nce
an
d
functi
on
of
com
pone
nts.
The
av
era
ge
sco
re
giv
e
n
b
y
the
exp
erts
in
the
app
ea
ra
nce
as
pect
was
5.8
2,
wh
il
e
the
fun
ct
ion
aspect
ge
ts
the
aver
a
ge
scor
e
of
8.9
1.
T
he
overall
aver
a
ge
m
edia
feasibil
i
ty
scor
e
was
14.
73.
In
acc
ord
ance
with
the
cat
egory
in
[28
]
,
al
l
asp
ect
s
wer
e
cat
egorized
ver
y
well
.
T
hu
s
, th
e
CAT m
ed
ia
h
a
s fulfil
le
d
the
re
qu
i
rem
ents u
s
ed
as
a test
sys
tem
in
te
rm
s o
f
it
s constr
uctio
n.
The
sec
ond
co
m
po
nen
t
of
t
he
CAT
-
Ph
ysC
r
iTS
pro
gr
am
is
the
it
em
ban
k.
Sp
eci
ficat
io
n
ta
ble
of
the
it
e
m
b
ank
can
be
see
n
i
n
Ta
bl
e 1
.
Tab
le
1
.
Sp
eci
f
ic
at
ion
s
of
t
he
it
e
m
b
ank
Asp
ect
Su
b
-
A
sp
ect
Ind
icato
r
Ph
y
sics
Su
b
jects
No
.
o
f
Ite
m
A
B
C
D
E
Elementa
ry
cla
rifica
tio
n
F
o
cu
sin
g
on
th
e
q
u
estio
n
F
o
r
m
u
latin
g
a
q
u
e
stio
n
abo
u
t even
ts
in
every
d
a
y
lif
e
.
4
1*
4
4
4
17
An
aly
zin
g
an
argu
m
e
n
t
An
aly
zin
g
state
m
e
n
ts an
d
deter
m
in
in
g
th
e si
m
ila
rities
o
r
d
if
f
erences in
an even
t giv
en
.
4
4
4
4
1*
17
Bas
ic su
p
p
o
rt
Co
n
sid
ering
so
u
rce
credib
ility
Exa
m
in
in
g
parts th
at can be co
n
sid
ere
d
to
b
e tr
u
sted
b
ased
on
ar
g
u
m
en
t
s/
ex
p
eri
m
en
ts an
d
th
eir
in
terpretation
s
.
1*
4
1*
4
4
14
Co
n
sid
ering
the
resu
lts o
f
o
b
serv
atio
n
s
Exp
ressin
g
r
easo
n
s b
ased
on
ob
serv
a
tio
n
s o
n
an
ev
en
t
.
4
4
4
4
4
20
Infer
en
ce
Ded
u
ctin
g
and
consid
ering
th
e r
esu
lts o
f
ded
u
ctio
n
Interpretin
g
state
m
en
ts an
d
clar
if
y
in
g
data
.
4
4
4
1*
4
17
Ind
u
ctin
g
and
con
s
id
ering
th
e r
esu
lts o
f
in
d
u
c
tio
n
Makin
g
a
h
y
p
o
th
esis
bas
ed
on
sev
eral
state
m
en
ts/ev
en
ts
.
4
4
1*
4
1*
14
Adva
n
ced
cla
rifica
tio
n
Def
in
in
g
ter
m
s
an
d
co
n
sid
ering
a
d
ef
in
itio
n
Def
in
in
g
a
state
m
e
n
t and
con
sid
ering
d
ef
in
itio
n
s
b
ased
on
exa
m
p
les
in ev
ery
d
a
y
lif
e
.
4
4
4
4
4
20
S
tra
teg
y
&
ta
ctics
Deter
m
in
in
g
an act
io
n
Makin
g
decis
io
n
s b
ased
on
strategies
and
so
lu
tio
n
s,
an
d
f
o
r
m
u
latin
g
app
rop
riate
alternativ
es
.
4
4
4
1*
4
17
Nu
m
b
e
r
o
f
Ite
m
29
29
26
26
26
136
A
=
Equ
ilib
riu
m
a
n
d
rotatio
n
al
d
y
n
a
m
i
cs,
B
=
Ho
o
k
e's
elasticity
an
d
law,
C
=
Static
f
lu
id
,
D
=
Dy
n
a
m
ic
f
lu
id
,
E
=
Te
m
p
e
ratur
e,
h
eat,
an
d
heat transf
er,
a
n
d
*
=
An
ch
o
r
ite
m
.
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2252
-
8822
In
t.
J
.
Eval
.
&
Re
s
.
E
du
c
.
Vo
l
.
8
, No
.
3
,
Sept
e
m
ber
2019
:
376
-
383
380
The
c
onstr
uction
of
t
he
it
em
bank
is
prese
nt
ed
in
Table
1
.
The
as
pects,
s
ub
-
as
pects,
a
nd
in
dicat
ors
of
c
riti
cal
thin
king
s
kill
s
us
e
d
we
re
ref
e
rr
i
ng
to
as
pects
a
nd
s
ub
-
as
pects
dev
el
op
e
d
i
n
[
39
]
.
T
he
five
physi
cs
top
ic
s
te
ste
d
w
ere
physi
cs
m
a
te
rial
s
of
t
he
2013
c
urricul
um
(K
-
13)
rev
is
ed
in
2016
for
11
th
-
grade
stu
de
nts
in
the
odd
sem
es
te
r.
The
it
em
bank
c
onsist
ed
of
fou
r
te
st
pa
ckag
e
s
with
t
he
num
ber
of
each
pac
ka
ge
was
40
it
e
m
s
acco
m
pan
ie
d
by
8
anc
hor
it
em
s
(total
=
136
it
em
s).
The
qu
e
sti
on
f
or
m
at
us
ed
wa
s
two
-
ti
er
m
ult
iple
-
cho
ic
e.
T
he fir
st t
ie
r
co
ntains
five
a
nswer
op
ti
on
s a
nd the
s
econd tie
r
cont
ai
ns
f
i
ve reaso
n op
ti
ons
.
The
it
em
ban
k
has
b
ee
n
te
st
ed
for
it
s
c
on
t
ent
validit
y
th
r
ou
gh
the
ju
dgm
ent
of
ni
ne
e
xp
e
rts
(
two
le
ct
ur
ers
a
nd
s
even
physi
cs
te
achers
).
Ba
se
d
on
the
as
ses
sm
ent
giv
en
,
t
hen
t
he
Ai
ken’s
V
c
oeffici
ent
was
cal
culat
ed
for
each
it
em
.
The
res
ults
of
Ai
ke
n'
s
V
c
oe
ff
ic
i
ents
of
al
l
it
e
m
s
we
re
i
n
t
he
r
ang
e
of
0.8
9
t
o
1.0
0.
In
acco
r
d
ance
with
w
hat
wa
s
prese
nted
i
n
[
29
]
,
the
te
sts
wer
e
sai
d
to
be
valid
if
t
hey
we
re
m
or
e
t
ha
n
t
he
thres
ho
l
d
val
ue
s
in
the
Aike
n’s
V
ta
ble.
T
he
thres
ho
l
d
f
o
r
9
raters
a
nd
a
ppr
oval
cat
egor
y
of
4
was
0.7
4.
Sin
ce
Aike
n’
s
V
c
oe
ff
ic
ie
nts
of
al
l
it
e
m
s
wer
e
in
the
ra
ng
e
of
0.89
t
o
1.0
0,
t
hen
al
l
it
em
s
wer
e
sta
te
d
to
hav
e
cov
e
re
d
the
en
t
ire area
of the
con
te
nts fr
om
t
he
as
pect
of
ph
ysi
cs criti
cal
think
in
g skil
ls (
va
l
id).
3.2.
Items
c
ha
r
ac
t
eri
stics of
th
e
item b
an
k
The
it
em
s
chara
ct
erist
ic
s
were
re
viewe
d
bas
ed
on
the
em
pi
rical
validit
y,
t
he
reli
abili
ty
of
sc
o
res
,
a
nd
the
it
e
m
s
diff
ic
ulty
ind
ex
.
T
he
it
e
m
ban
k
was
te
ste
d
on
34
4
stud
e
nts
.
T
he
r
esults
of
the
e
m
pi
rical
validit
y
and
the
reli
abili
ty
val
ues of
t
he
te
s
t base
d on anal
ysi
s u
sin
g Que
st are
pr
ese
nte
d
in
Ta
ble 2.
Tab
l
e
2
.
Re
s
ults o
f
the
em
pirical
v
al
idit
y an
d
the
te
st reli
ab
il
ity
Para
m
eter
Ite
m
Esti
m
atio
n
Test
ee
Esti
m
a
tio
n
Av
erage valu
e and
stan
d
ard d
ev
iatio
n
of
I
NFIT
M
NSQ
1
.00
±
0
.05
1
.00
±
0
.13
Av
erage valu
e and
stan
d
ard d
ev
iatio
n
of
INFI
T
t
-
0
.01
±
0
.
49
-
0
.01
±
0
.
85
INFI
T
MNSQ
f
o
r
each ite
m
0
.88
-
1.
18
Reliab
ility
o
f
o
f
item
esti
m
ate
0
.87
R
eliab
ility
o
f
case
esti
m
ate
0
.72
T
he
good
ness
of
fit
pr
es
ente
d
in
Table
2
s
how
the
overal
l
aver
a
ge
val
ue
of
INFIT
M
NSQ
for
it
e
m
est
i
m
ation
w
as
1
.
00
±
0
.
05
an
d for the te
ste
e
was
1
.
00
±
0
.
13
.
I
n
a
ddit
ion
,
the
INF
IT
MN
SQ
valu
e for
eac
h
it
e
m
w
as in th
e range of
0.8
8
t
o
1.1
8.
In
acc
orda
nce w
it
h
t
he
ex
pla
nation
i
n
[
32
]
,
t
he o
ve
rall
it
e
m
is sai
d
to b
e
fitt
ed
with
t
he
m
od
el
if
it
has
an
I
NFIT
M
N
SQ
a
ver
a
ge
va
lue
of
ar
ound
1
.
00
±
0
.
00
,
wh
il
e
eac
h
it
em
is
fit
if
it
has
MNSQ
I
NF
I
T
value
of
0.77
to
1.3
0.
T
her
e
fore,
the
r
esult
of
the
go
odness
of
fit
as
a
wh
ole
or
for
each
it
e
m
can
be
sai
d
to
b
e
f
it
with
PCM
.
T
hu
s
, the
it
e
m
b
an
k
ca
n be
declare
d v
al
id as a
m
easu
rin
g
inst
ru
m
ent.
The
reli
abili
ty
of
the
sc
or
e
i
n
the
IRT
was
ind
ic
at
ed
by
t
he
it
e
m
separ
a
ti
on
in
dex
a
nd
the
per
s
on
separ
at
io
n
in
de
x
as
prese
nted
in
Table 2
.
T
he
value
of
t
he
it
e
m
reli
abili
t
y
was
cl
assifi
ed
as
good
(
0.8
7)
,
wh
il
e
the
pe
rson
reli
abili
ty
was
su
f
fici
ent
(
0.72
)
.
I
n
acc
orda
nc
e
with
e
xp
la
nation
in
[
33]
,
the
hi
gh
e
r
th
e
it
e
m
reli
abili
ty
,
the
m
or
e
con
vi
nci
ng
t
he
sam
ple
m
at
ches
the
te
ste
d
it
e
m
,
wh
il
e
the
higher
th
e
per
s
on
reli
ab
il
ity,
the
m
or
e
co
nv
i
ncin
g
the
m
eas
ur
em
ent
pr
ovi
des
co
ns
ist
ent
resu
lt
s.
Ba
se
d
on
the
se
res
ults
it
can
be
concl
ud
e
d
that t
he qua
li
ty of th
e
it
e
m
s w
as
go
od, while
the consist
en
cy
of stude
nts’
a
nswers
w
a
s s
uff
ic
ie
nt.
The
val
ue
of
r
el
ia
bili
ty
between
one
le
vel
of
a
bili
ty
with
oth
er
a
bili
ti
es
is
diff
ere
nt
.
T
o
fin
d
ou
t,
i
t
can
be
see
n
at
the
c
urve
of
th
e
relat
ionshi
p
betwee
n
i
nform
at
ion
f
un
ct
io
n
a
nd
sta
nda
rd
erro
r
of
m
eas
ur
em
ent
(S
EM)
fro
m
the P
arsca
le
pro
gram
p
resen
te
d i
n
Fig
ure
4
.
Figure
4
.
The
c
urve
of the
rela
ti
on
s
hip
beetw
een inf
or
m
at
ion
functi
on a
nd
SEM
Evaluation Warning : The document was created with Spire.PDF for Python.
In
t J
E
val & R
es Educ
.
IS
S
N:
22
52
-
8822
A com
pu
te
rize
d ada
ptive
test
for
m
eas
ur
i
ng
the physic
s c
rit
ic
al thinki
n
g sk
il
ls i
n
...
(
Aa
ng Za
i
nu
l
Abi
din
)
381
To
analy
ze
th
e
curve
in
Fi
gure
4
is
done
by
lookin
g
at
the
intersect
io
n
of
the
li
ne
betwee
n
the
inf
or
m
at
ion
functi
on
a
nd
the stand
a
rd
e
rro
r
[35]
.
Ba
sed
on th
e
cu
rv
es of
the
relat
io
ns
hi
p
betwee
n
in
for
m
at
ion
functi
on
and
S
EM
in
Figu
re
4
,
it
can
be
co
nclu
ded
that
th
e
it
e
m
ban
k
is
su
it
able
or
reli
able
for
stud
e
nt
s
wit
h
abili
ty
v
al
ues
(
)
of
-
1.7 to 3
.2.
The
la
st
c
ha
ra
ct
erist
ic
s
of
th
e
it
e
m
bank
wer
e
relat
ed
to
t
he
it
em
diff
ic
ulty
ind
e
x
(
b
).
T
he
it
em
diff
ic
ulty
ind
e
x
of
13
6
it
e
m
s
was
in
the
range
of
-
0.7
6
to
0.83.
The
a
ve
r
age
of
the
it
em
diff
ic
ulty
ind
e
x
as
a
whole
was
0.0
0
±
0.35,
wh
il
e
f
or each
asp
ect
and s
ub
-
as
pect is p
rese
nted
i
n Fi
gure
5
.
Figure
5
.
The
i
tem
d
ifficult
y i
nd
e
x f
or
eac
h
a
sp
ect
a
nd sub
-
aspect
The
it
e
m
is
sai
d
t
o
be
go
od
if
it
ha
d
a
value
of
−
2
≤
≤
2
[
36
]
.
Be
ca
us
e
t
he
it
e
m
diff
ic
ulty
in
de
x
of
th
e
it
e
m
bank
was
betwee
n
-
0.7
6
to 0
.
83, it
can
be
co
ncl
ud
e
d
that
t
he
it
e
m
diff
ic
ulty
ind
e
x
of
al
l
it
em
s
was
cl
assifi
ed
as
good
.
I
n
ad
diti
on,
Fi
gure
5
s
hows
that
the
seq
uen
ce
of
the
it
e
m
diff
ic
ulty
ind
e
x
of
ea
ch
a
sp
ect
is
in
acco
r
dan
ce
with
t
he
le
ve
ls
of
crit
ic
al
thin
king
pr
e
sented
in
[39
]
wh
ic
h
sta
rt
i
ng
f
ro
m
el
e
m
entary
cl
arifica
ti
on
,
basi
c suppo
rt, infere
nce,
a
dva
nc
e cla
rificat
ion, as
well
as stra
te
gy and tact
ic
s.
3.3.
Te
st
tri
als u
si
ng
th
e
C
AT
-
P
hy
s
CriT
S
pr
ogra
m
Af
te
r
the
C
AT
-
Ph
ysC
riTS
pr
ogram
was
te
st
ed
on
233
st
udents,
the
lo
wes
t
an
d
highest
a
bili
ti
es
(
)
ob
ta
ine
d
wer
e
in
the
ra
ng
e
of
0.03
t
o
0.6
0
with
a
n
a
ve
rag
e
of
0.1
5.
T
he
p
hysic
s
c
riti
cal
think
in
g
sk
il
ls
ob
ta
ine
d base
d o
n
the
sc
hool
academ
ic
achievem
ent cat
e
go
r
ie
s ar
e
pr
ese
nt
ed
in
Fig
ure
6
.
Figure
6. The
ph
ysi
cs c
riti
cal
thin
king s
kill
s in
eac
h
sc
hool
cate
gory
The
ave
ra
ge
a
bili
ty
value
show
n
in
Fi
gure
6
is
eq
uiv
al
ent
to
the
sch
oo
l
cat
egory.
A
good
m
easur
in
g
instru
m
ent
can
disti
ng
uis
h
ac
cordi
ng
to
the
abili
ti
es
of
stud
e
nts
[
40
]
.
Ba
sed
on
the
sui
ta
bili
ty
between
the
resu
lt
s
of
m
ea
su
rem
ent
us
in
g
CAT
-
P
hysCriTS
an
d
sc
hool
academ
ic
a
chievem
ent,
w
e
can
say
the
CAT
-
Ph
ysC
riTS
pr
ogram
can dist
inguis
h
the
phy
sic
s cr
it
i
cal
think
i
ng s
kill
s o
f
stu
de
nts
.
4.
CONCL
US
I
O
N
The
res
ults
of
t
he
st
ud
y
sho
w
that
the
C
AT
-
P
hysCriTS
pro
gram
has
m
et
the
re
qu
i
rem
ents
to
be
us
e
d
to
m
easur
e
ph
y
sic
s
crit
ic
al
thi
nk
i
ng
s
kill
s.
T
erm
s
that
hav
e
been
f
ulfill
ed
inclu
de:
1)
in
te
rm
s
of
const
ruct
ion,
the
CAT
m
edia
has
bee
n
de
cl
ared
valid
w
it
h
a
ve
ry
good
cat
eg
or
y
i
n
te
rm
s
of
ap
pe
aran
ce
a
nd
functi
on
aspects.
T
he
con
te
nt
validit
y
of
136
it
em
s
has
bee
n
decla
red
vali
d
;
2)
in
te
rm
s
of
the
char
act
erist
ic
s
of
the
Evaluation Warning : The document was created with Spire.PDF for Python.
IS
S
N
:
2252
-
8822
In
t.
J
.
Eval
.
&
Re
s
.
E
du
c
.
Vo
l
.
8
, No
.
3
,
Sept
e
m
ber
2019
:
376
-
383
382
it
e
m
s,
al
l
it
e
m
s
wer
e
declare
d
fit
to
the
PC
M.
T
he
qu
al
it
y
of
the
it
em
was
good,
wh
il
e
the
co
ns
ist
ency
of
t
he
answers
from
t
he
stu
de
nts
wa
s
suffici
ent
.
T
he
it
e
m
ban
k
w
as
su
it
able
for
stud
e
nts
wit
h
a
bili
ti
es
(
θ
)
f
rom
-
1.7
to
3.2.
T
he
di
ff
ic
ulty
inde
x
of
al
l
it
e
m
s
was
cl
assifi
ed
a
s
good;
an
d
3)
the
CA
T
-
P
hy
sCriTS
pro
gra
m
is
capab
le
of
dist
inguishi
ng
stu
den
ts'
crit
ic
al
thin
king
s
kill
s,
wh
ic
h
is
in
dic
at
ed
by
t
he
m
e
asur
em
ent
res
ul
ts
of
the CAT
-
P
hys
CriTS
program
equivale
nt
to
s
tud
e
nt aca
dem
i
c achiev
em
ent.
Su
ccess
f
ully
dev
el
oped
C
AT
-
Ph
ysC
riTS
wi
ll
hav
e
i
m
p
li
c
at
ion
s
f
or
e
ff
e
ct
iveness,
e
ff
i
ci
ency,
an
d
com
pr
ehe
ns
ive
ness
in
e
valuat
ing
physi
cs
cri
ti
cal
think
ing
sk
il
ls
of
stu
de
nts.
T
he
CAT
-
Ph
ysC
riTS
pro
gr
am
pr
ese
nts
a
fast
er
m
easur
em
e
nt
with
out
re
duci
n
g
it
s
acc
uracy
an
d
is
m
or
e
com
pr
eh
e
ns
ive
f
or
la
rge
-
scal
e
asses
sm
ent.
T
he
res
ults
of
this
stud
y
can
al
so
be
us
e
d
as
a
ref
erenc
e
f
or
ed
ucati
on
r
egu
la
to
rs
to
ut
il
iz
e
the
CAT
syst
em
on
la
r
ge
-
scal
e
assessm
ents
su
c
h
as
natio
nal
exam
inatio
ns.
F
or
furt
her
researc
h,
si
m
il
ar
dev
el
op
m
ents
can
be
m
ade
fo
r
ot
her
im
po
r
ta
nt
sk
il
ls.
In
add
it
io
n,
the
nu
m
ber
of
it
e
m
s
can
be
m
ulti
pli
ed
so
that t
he
it
em
d
ifficult
y i
nde
x
c
an be e
ven
ly
di
stribu
te
d
at
all
le
vels of st
ud
e
nt ab
il
it
y.
ACKN
OWLE
DGE
MENTS
We
are
ver
y
gr
at
ef
ul
to
Gradu
at
e
Sc
hool
,
Yogyaka
rta
S
ta
te
Un
ive
rsity
for
al
l
the
fa
ci
li
ti
es
and
su
pp
or
t
in
co
m
ple
ti
ng
this
researc
h.
We
al
so
ackno
wle
dg
e
gen
e
rous
su
pp
or
t
f
ro
m
the
governm
ent
of
Ind
on
esi
a th
r
ough the
Direct
or
at
e
of
Resea
r
ch
an
d
Com
m
un
it
y Serv
ic
e,
w
hich has s
uppo
rted
this
projec
t with
fun
ding th
rou
gh the
gra
du
at
e
sch
oo
l
gr
a
nt
re
sea
rch sc
hem
e.
REFERE
NCE
S
[1]
S.
E.
Sale
h,
"
C
rit
ical
th
inki
ng
as
a
21st
ce
ntu
r
y
skill
:
concep
ti
ons,
implemen
ta
ti
on
and
cha
l
l
enge
s
in
the
efl
cl
assroom
,"
Eur.
J. F
ore
ign
Lang
.
Teach
.
,
vol
.
4
(
1
),
pp
.
1
-
16
,
2019
.
[2]
Mundila
rto
and
H.
Ism
oy
o
,
"
Eff
ec
t
of
probl
em
-
base
d
le
a
rning
o
n
improvem
ent
ph
y
sics
a
chi
ev
e
m
ent
and
cri
t
ic
a
l
thi
nking
o
f
senio
r
high
schoo
l
stu
dent
,
"
J. B
a
lt
.
Sc
i.
Educ.
,
vol. 16
(
5
),
pp
.
761
-
779
,
2017.
[3]
R.
H.
Enn
is,
"
Cr
it
ical
thi
nk
ing ass
essm
ent
,
"
Theory
Pract.
,
vol
.
3
2
(
3
),
pp
.
179
-
18
6,
1993
.
[4]
R.
W
.
El
li
o
tt,
"
Understa
nding
fac
ul
t
y
enga
g
e
m
ent
in
assessment
through
fee
dba
ck
and
di
al
ogues :
A
m
ixed
m
et
hods a
pproa
c
h
,
"
Int
.
J. E
va
l.
Re
s. Educ
.
,
vol
.
7
(
3
)
,
pp
.
16
7
-
17
5,
2018
.
[5]
F.
S.
Putri
and
E.
Isti
y
ono
,
"
Th
e
developm
ent
o
f
per
form
anc
e
a
ss
ess
m
ent
of
STEM
-
base
d
cr
it
i
c
al
th
inki
ng
skil
l
in
the
h
igh
school
ph
y
sics
le
ss
on
s,"
Int. J. Env
iron
.
Sci
.
Edu
c.
,
vol
.
1
2
(
5
),
pp
.
1269
-
1
281,
2017
.
[6]
R.
S.
Dam
a
y
an
t
i,
A.
Su
y
a
tna,
W
arsono,
and
U.
Rosidin,
"
Deve
lopment
of
authentic
assess
m
e
nt
instruments
for
cri
tica
l
thi
nk
ing
skill
s
in
glob
al
warm
ing
with
a
scie
ntific
appr
o
a
ch,
"
Int.
J.
S
ci
.
Appl
.
S
ci
.
Con
f.
Ser.
,
vol
.
2
(
1
),
p
p
.
289
-
299,
2017
.
[7]
G.
N.
Khan,
N.
Ishrat
,
and
A.
Q.
Khan
,
"
Us
i
ng
it
em
ana
l
y
si
s
on
essay
t
y
p
e
s
questi
ons
give
n
in
sum
m
at
ive
exa
m
ina
t
ion
of
m
edi
ca
l
colleg
e
student
s
:
Fac
il
i
t
y
v
al
u
e,
d
iscri
m
i
nat
ion
inde
x
,"
In
t.
J.
Re
s.
Me
d
.
S
ci
.
,
vol
.
3
(
1
)
,
pp
.
178
-
182,
2015
.
[8]
K.
A.
Shaaba
n,
"
As
ses
sm
ent
of
cri
tica
l
thi
nking
skill
s
through
r
ea
ding
compreh
ension
,"
Int.
J
.
Lang.
Stud.
,
vol
.
8
(
2
),
pp
.
117
-
14
0,
2014
.
[9]
M.
Bai
g,
S
.
K.
Ali,
S.
Ali
,
and
N.
Huda,
"
Eval
uat
ion
of
m
ultip
le
cho
ic
e
and
sh
ort
essa
y
question
it
ems
in
basi
c
m
edi
ca
l
sci
enc
es
,"
Pa
ki
stan
J. M
ed.
S
ci.
,
vol
.
30
(
1
),
pp
.
3
-
6
,
2014
.
[10]
D.
D.
Kerkm
an
and
A.
T.
Johns
on,
"
Chal
l
engi
ng
m
ult
ipl
e
-
choi
c
e questi
ons t
o
eng
age
cr
it
i
ca
l think
i
ng
,"
InSight
A J
.
Sch.
Tea
ch.
,
vo
l. 9, pp. 92
-
97,
20
14.
[11]
E.
Isti
y
ono
,
D.
Marda
pi,
and
Suparno,
"
Deve
lop
m
ent
of
ph
y
sics
highe
r
orde
r
thinking
skill
s
te
st
(P
y
sTHO
TS)
for
high
school
stud
en
t
,"
J. Pene
li
t
.
dan
Ev
a
l. P
end
i
dik
.
,
vo
l. 18
(
1
),
pp.
1
-
12
,
2014
.
[12]
W
ina
rti
,
Ca
ri,
Suparm
i
,
W
.
S
unar
no,
and
E
.
Isti
y
ono
,
"
Deve
lopment
of
two
ti
er
te
st
to
a
ss
ess
conc
ept
ua
l
under
standi
ng
in
hea
t
and
te
m
per
at
ure
,"
in
IOP
C
onf.
Serie
s:
Journal
of
Phy
sics:
Conf.
Serie
s
795
(
2017
)
012052
,
,
pp.
1
-
5
,
2017
.
[13]
F.
S.
Putri
,
E
.
Ist
i
y
o
no
,
and
E
.
Nu
rca
h
y
an
to,
"
Dev
el
opm
ent
of test instrum
ent
s for
c
rit
ical thi
nking
skill
s in
the form
of
rea
son
abl
e
m
ult
iple
choi
c
e
(P
oli
tomus
)
in
DI
Y,
"
Unnes P
hys
.
Educ. J.
,
vo
l. 5
(
2
),
pp
.
76
-
84
,
20
16.
[14]
Fajri
ant
h
i,
W
.
H
endr
ia
n
i,
and
B
.
G.
Septa
rin
i,
"
D
eve
lopme
nt
of
cr
it
ical
th
inki
ng
t
e
st wi
th
appr
oa
ch
to
it
em re
sponse
the
or
y
,"
J
.
P
enel
it
.
dan
Ev
al
.
Pe
n
didi
k
.
,
vol. 20
(
1
)
,
pp
.
45
-
55
,
201
6.
[15]
E.
Isti
y
ono
,
W
.
B.
Dw
anda
ru,
a
nd
F.
Rahay
u
,
"
The
dev
el
oping
of
cre
a
ti
v
e
thi
nk
ing
skill
s
t
est
ba
sed
on
m
oder
n
te
st
the
or
y
in
p
h
y
si
c
s of
senior
high s
chool
s
,"
Cakra
wala
P
endi
di
k.
,
vol.
37
(
2
)
,
pp
.
1
90
-
200,
2018
.
[16]
A.
A.
Bic
hi
an
d
R.
Ta
li
b
,
"
Ite
m
response
th
eor
y
:
an
int
rod
uct
ion
to
latent
tra
it
m
odel
s
to
te
st
and
it
em
deve
lopment
,"
I
nt.
J. Eval
.
R
es.
Educ
.
,
vo
l. 7
(
2
),
pp.
142
-
151
,
20
18.
[17]
O.
A.
Aw
opej
u
and
E
.
R
.
I
.
Afol
abi
,
"
Com
par
at
i
ve
an
aly
sis
of
class
ic
al
te
st
th
eor
y
and
it
em
r
esponse
the
or
y
base
d
it
em
par
amet
er
esti
m
at
es
of
seni
or
school
ce
rti
f
i
ca
t
e
m
at
hematic
s
exa
m
ina
ti
on
,"
Eur.
Sci
.
J.
,
vol.
12
(
28
),
pp.
263
-
284,
2016
.
[18]
R.
K.
Ham
b
le
ton
and
H.
S
wam
ina
tha
n,
"
Fundam
ent
al
s
of
item
respon
the
or
y
,"
.
Newbur
y
Park,
Cal
if
:
Sage
Publ
ic
a
ti
ons
,
19
91.
[19]
H.
C.
Bagus,
"
The
nat
ion
al
ex
am
administra
tion
b
y
using
computer
iz
ed
ad
a
pti
ve
te
st
ing
(CAT)
m
ode
l,
"
J.
Pe
ndidik.
dan
K
ebud.
,
vol
.
18
(
1
)
,
pp
.
45
-
5
3
,
201
2.
Evaluation Warning : The document was created with Spire.PDF for Python.
In
t J
E
val & R
es Educ
.
IS
S
N:
22
52
-
8822
A com
pu
te
rize
d ada
ptive
test
for
m
eas
ur
i
ng
the physic
s c
rit
ic
al thinki
n
g sk
il
ls i
n
...
(
Aa
ng Za
i
nu
l
Abi
din
)
383
[20]
G.
Li
ng
,
Y.
Att
al
i
,
B.
Finn,
and
E.
A.
Ston
e,
"
I
s
a
compute
rize
d
ada
p
ti
ve
t
est
m
ore
m
oti
vat
ing
tha
n
a
fixe
d
-
it
e
m
tes
t
?
,
"
App
l. P
sy
chol
.
Me
as.
,
vo
l. 41
(
7
), pp.
495
-
5
11,
2017
.
[21]
E.
C.
A
y
b
ek
an
d
R.
N.
Dem
irtasli,
"
Com
pute
r
i
ze
d
ad
aptive
t
est
(CAT)
applic
a
ti
ons
and
i
te
m
r
esponse
the
o
r
y
m
odel
s for
pol
y
t
om
ous i
te
m
s
,"
Int.
J. Res. Educ. Sci.
,
vo
l. 3
(
2
), p
p.
475
-
487
,
201
7.
[22]
M.
Rezai
e
and
M.
Golshan,
"
C
om
pute
r
ada
p
ti
v
e
te
st
(CAT):
A
dvant
ag
es
and
l
i
m
it
at
ions
,"
In
t.
J.
Educ
.
Inv
estig.
,
vol.
2
(
5
)
,
pp
.
12
8
-
137
,
2015
.
[23]
H.
M.
W
u,
B.
C.
Kuo,
and
S.
C.
W
ang,
"
Com
pute
ri
ze
d
d
y
nam
ic
ad
apt
iv
e
t
ests
with
imm
edi
at
e
l
y
indi
vidu
al
i
zed
fee
dba
ck
for
pri
m
ar
y
schoo
l
m
athem
at
ic
s
learni
n
g,
"
Edu
c. Tec
hn
ol.
So
c.
,
vo
l. 20
(
1
),
pp
.
61
-
72
,
20
17.
[24]
E.
Isti
y
ono
,
W
.
S.
B.
Dw
anda
ru,
and
R.
Fa
i
za
h,
"
Mapping
of
phy
sics
probl
em
-
solving
skill
s
of
senior
high
schoo
l
student
s using
P
h
y
sP
roSS
-
CAT,
"
Re
s.
E
val
.
Edu
c.
,
vol
.
4
(
2
)
,
pp
.
144
-
154,
2018
.
[25]
S.
Thi
aga
r
aj
an
,
D.
S.
Semmel,
and
M.
I.
Semmel,
"
Instruc
ti
on
al
Deve
lopment
fo
r
Tra
ini
ng
T
ea
ch
er
of
Exc
epti
ona
l
chi
ldr
en
,"
B
loo
m
ingt
on
India
n
a
:
Indiana
Univ
ersity
,
1974.
[26]
L.
L.
Oriondo
and
E.
M.
D.
Antonio,
"
Eva
lu
at
ing
educat
ional
outc
om
es
(Te
st,
m
ea
surm
ent
and
eva
lua
t
ion
)
,"
Florent
ino
St:
R
ex
Print
ing
Company
,
In
c
,
1998.
[27]
E.
Muraki
and
R
.
D.
Bock
,
"
Parsca
l
e:
IRT
it
em
a
naly
s
is
and
t
est
scoring
for
ratin
g
sca
le
d
at
a
,"
C
hic
ago
:
Scientifi
c
Soft
were
Inte
rna
ti
onal, Inc
,
1998
.
[28]
S.
E. P.
W
idiy
ok
o,
"
T
ec
hniqu
es
f
or
pre
pa
ring
r
ese
arc
h
instruments
.
Yog
y
ak
art
a
:
Pu
staka
P
el
ajar
,
2
013.
[29]
L.
R.
Aiken
,
"
Conte
nt
va
li
di
t
y
a
nd
rel
ia
b
il
i
t
y
of
single
item
s
or
questi
onnaires
,"
Educ
.
Ps
yc
hol
.
Me
as.
,
vol
.
40
(
4
),
pp.
955
-
959
,
19
80.
[30]
S.
Krishnan
and
N.
Idris,
"
Us
ing
PC
M
to
i
m
prove
the
qual
ity
of
a
n
instrument
,"
Int.
J.
E
val.
Re
s.
Educ
.
,
vol.
7
(
4
)
,
p
p.
313
-
316
,
20
18.
[31]
E.
Isti
y
ono
,
"
T
he
an
aly
s
is
of
senior
high
sch
ool
student
s’
p
h
y
sics
HO
TS
i
n
Bant
ul
distric
t
m
ea
sured
usin
g
Ph
y
sReMChoT
HO
TS,
"
in
Ame
r
ic
an
Inst
it
ut
e
o
f Phy
sics
Con
fe
re
nce
Proceedi
ngs
,
vol
.
1868
,
pp
.
1
-
7,
2017
[32]
R.
Adam
and
S.
Khoo,
"
Quest:
t
he
int
e
racti
ve
tes
t
ana
l
y
sis
s
y
st
e
m
ver
sion
2.
1
,"
Vict
ori
a:
The
Au
stralian
Counci
l
for E
duca
ti
onal
Re
search
,
1996.
[33]
B.
Subal
i,
"
Te
st
deve
lopment
wit
h
empirical
inves
ti
gat
ion
and
re
liabil
i
t
y
.
Yog
y
akarta
:
UN
Y
Press
,
2016.
[34]
B.
Sum
int
ono
and
W
.
W
idhi
arso,
"
Rasch
mode
li
ng
applic
ation
on
educ
at
i
on
al
assess
m
ent
.
Cim
ahi
:
Tr
im
Komunikat
a
,
20
15.
[35]
K.
Hi
da
y
at
i
,
Budi
y
ono
,
and
Sugim
an,
"
Us
ing
al
ignment
inde
x
a
nd
pol
y
tomous
it
em
response
the
or
y
on
sta
ti
sti
c
s
essa
y
te
st
,"
Eura
sian J
.
Edu
c. Re
s.
,
vol
.
79
,
pp
.
1
15
-
132,
2019
.
[36]
R.
K.
Ham
ble
to
n
and
H
.
Sw
aminat
han
,
"
It
em
res
ponse
the
or
y
,"
B
oston,
MA
:
Klu
e
r Inc
,
1985.
[37]
K.
T.
Han,
"
Ma
ximum
li
kel
ihoo
d
score
esti
m
at
i
on
m
et
hod
with
fen
ce
s
for
short
-
le
ngth
te
sts
an
d
computer
ized
ada
pt
ive
te
sts
,"
Appl
.
Psy
chol
.
Me
as.
,
vol. 40
(
4
),
pp
.
289
-
301
,
2
016.
[38]
S.
Hadi
,
"
W
eb
-
b
ase
d
ad
apt
i
v
e te
s
t
-
base
d
compute
riz
ed
dev
el
opm
e
nt
,"
Yog
y
ak
art
a
:
Aswaja Presind
o
,
2013
.
[39]
R.
H.
Enn
is,
"
A
logi
c
al
basis
for
m
ea
suring
critical
th
inki
ng
skil
ls
,"
J. De
p.
Sup
erv
.
Curr
ic
.
D
ev
.
,
v
ol.
43
(
2
)
,
pp.
45
-
48,
1985
.
[40]
E.
Isti
y
ono
,
"
IT
-
base
d
HO
TS
ass
essm
ent
on
ph
y
sics
le
arn
ing
as
t
he
21st
ce
ntur
y
demand
at
senio
r
high
schoo
ls:
Expe
c
ta
t
ion
and rea
l
ity
,"
in
AIP
Confe
renc
e
Pro
c
ee
dings
,
vol
.
02
0014,
pp
.
1
-
6
,
2
018
.
Evaluation Warning : The document was created with Spire.PDF for Python.