I
A
E
S
I
n
t
e
r
n
at
io
n
al
Jou
r
n
al
of
A
r
t
if
ic
ia
l
I
n
t
e
ll
ig
e
n
c
e
(
I
J
-
AI
)
V
ol
.
10
, N
o.
1
,
M
a
r
c
h
202
1
, pp.
175
~
183
I
S
S
N
:
2252
-
8938
,
D
O
I
:
10.11591/
ij
a
i.
v
10
.i
1
.pp
175
-
183
175
Jou
r
n
al
h
om
e
page
:
ht
tp
:
//
ij
ai
.
ia
e
s
c
or
e
.c
om
Spike neuron optimization using d
eep reinforceme
nt
learning
T
an
S
z
i
H
u
i,
M
oh
am
ad
K
h
ai
r
i
I
s
h
ak
School o
f Elec
trica
l and E
lectr
onic E
nginee
ring, U
nivers
iti Sains
Malay
sia, 14
300, Ni
bong Te
bal, Pul
au Pina
ng,
Malaysia
A
r
t
ic
le
I
n
f
o
A
B
S
T
R
A
C
T
A
r
ti
c
le
h
is
to
r
y
:
R
e
c
e
iv
e
d
S
e
p 2
2, 20
20
R
e
vi
s
e
d
J
a
n
17
, 20
2
1
A
c
c
e
pt
e
d
F
e
b
9
, 20
21
Deep
reinforcement
learning
(DRL)
which
involved
reinforcement
l
earning
and artifi
cial neural
network
allows
agents
to tak
e the
best po
ssible
act
ions t
o
achieve
goals.
Spiki
ng
neural
network
(SNN)
faced
difficul
ty
in
traini
ng
due
to the non
-
differentiable
spike function of spike
neuron. In ord
er to ov
ercome
the
difficulty,
deep
Q
network
(DQN)
and
deep
Q
learning
with
nor
malized
advantage
function
(NAF)
are
proposed
to
interact
with
a
custom
environm
ent.
DQN
is
applied
for
discrete
action
space
whereas
NAF
is
implemented
for
continuous
action
space.
The
model
is
trained
and
te
sted
to
validate
its
performance
in
order
to
balance
the
firing
rate
of
excitat
ory
and
inhibitory
population
of
spike
neuron
by
using
both
algorithms.
T
raining
results
showed
both
agent
s
able
to
explore
in
the
custom
environme
nt
with
OpenAI
Gym
framework.
The
trained
model
for
both
algorithms
capable
to
balance
the
firing
rate
of
excitatory
and
inhibitory
of
the
spike
neuron.
NAF
achieved
0.80%
of
the
average
percentage
error
of
rate
of
difference
b
etween
target
and
actual
neuron
rate
where
as
DQN
obtained
0.96%.
NAF
a
ttained
the
goal
faster
than
DQN
with
only
3
steps
taken
for
actual
output
neuron
rate to meet
with or cl
ose to target
neuron firin
g rate.
K
e
y
w
o
r
d
s
:
D
e
e
p Q
ne
twor
k
D
e
e
p r
e
in
f
or
c
e
m
e
nt
l
e
a
r
ni
ng
S
pi
ke
ne
ur
on
This is an
open
acce
ss artic
le unde
r the
CC BY
-
SA
license.
C
or
r
e
s
pon
di
n
g A
u
th
or
:
M
oha
m
a
d K
ha
ir
i
I
s
ha
k
S
c
hool
of
E
le
c
tr
ic
a
l
a
nd E
le
c
tr
oni
c
E
ngi
ne
e
r
in
g
U
ni
ve
r
s
it
i
S
a
in
s
M
a
la
y
s
ia
14300, Nibong T
e
ba
l
P
ul
a
u P
in
a
ng, M
a
l
a
ys
ia
E
m
a
il
:
kha
ir
ii
s
ha
k@
us
m
.m
y
1.
I
N
T
R
O
D
U
C
T
I
O
N
D
e
e
p
r
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng
(
D
R
L
)
c
om
bi
ne
s
m
a
c
hi
ne
le
a
r
ni
ng
a
nd
a
r
ti
f
ic
ia
l
in
te
ll
ig
e
nc
e
te
c
hni
que
s
[
1]
.
R
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng
a
lg
or
it
hm
s
w
it
h
de
e
p
ne
ur
a
l
ne
twor
ks
a
r
e
im
pl
e
m
e
nt
e
d
in
D
R
L
to
s
e
le
c
t
th
e
be
s
t
pos
s
ib
le
a
c
ti
on
to
a
tt
a
in
go
a
ls
.
A
D
R
L
a
ge
nt
in
te
r
a
c
ts
w
it
h
a
vi
r
tu
a
l
e
nvi
r
onm
e
nt
a
s
s
how
n
in
F
ig
ur
e
1
a
nd
s
e
le
c
t
a
c
ti
on
s
to
s
ol
ve
c
om
pl
e
x
pr
obl
e
m
[
2]
.
D
e
e
p
ne
ur
a
l
ne
twor
k
is
u
s
e
d
by
a
ge
nt
s
to
a
ppr
oxi
m
a
te
a
va
lu
e
or
pol
ic
y
f
unc
ti
on
in
or
de
r
to
upda
te
a
nd
i
nde
x
th
e
da
ta
in
s
te
a
d
of
us
in
g
a
lo
okup
ta
bl
e
.
T
he
da
ta
c
on
s
is
ts
of
s
ta
te
s
, a
c
ti
ons
a
nd r
e
w
a
r
ds
. T
he
a
g
e
nt
t
a
ke
s
a
c
ti
ons
ba
s
e
d on the
c
ur
r
e
nt
s
ta
te
a
nd r
e
w
a
r
d
in
a
vi
r
tu
a
l
e
nvi
r
onm
e
nt
[
3]
.
T
he
a
ge
nt
r
e
c
e
iv
e
s
r
e
w
a
r
ds
or
p
e
na
lt
ie
s
ba
s
e
d
on
th
e
a
c
ti
ons
pe
r
f
or
m
e
d.
T
he
a
ge
nt
r
e
c
e
iv
e
s
pos
it
iv
e
r
e
w
a
r
ds
w
he
n t
he
out
c
om
e
i
s
c
lo
s
e
r
t
o t
he
t
a
r
ge
t
w
he
r
e
a
s
w
he
n t
he
r
e
i
s
a
f
a
ul
ty
a
c
ti
on
ta
ke
n
,
th
e
a
g
e
nt
obt
a
in
s
n
e
ga
ti
ve
r
e
w
a
r
ds
.
T
h
e
a
ge
nt
le
a
r
ns
f
r
o
m
e
xpe
r
ie
nc
e
to
d
e
c
id
e
th
e
be
s
t
s
ui
ta
bl
e
a
c
ti
on
to
a
tt
a
in
a
goa
l
[
4]
.
S
pi
ke
ne
ur
on
is
e
le
m
e
nt
a
r
y
uni
t
in
s
pi
ki
ng
ne
ur
a
l
ne
tw
or
k
(
S
N
N
)
.
S
pi
ke
ne
u
r
on
ha
s
th
e
c
ha
r
a
c
te
r
is
ti
c
of
s
pi
ki
ng be
ha
vi
our
. W
he
n
s
pi
ke
ne
ur
on i
s
f
ir
e
d,
a
s
pi
ke
i
s
ge
ne
r
a
te
d by us
in
g s
pi
ke
ge
ne
r
a
ti
on
f
unc
ti
on.
S
pi
ke
s
a
r
e
s
e
qu
e
nc
e
s
of
a
c
ti
on
pot
e
nt
ia
l
th
a
t
is
us
e
d
in
s
ig
na
l
tr
a
ns
m
is
s
io
n
in
s
pi
ke
n
e
ur
ons
[
5]
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2252
-
8938
I
nt
J
A
r
ti
f
I
nt
e
ll
,
V
ol
.
10
, N
o.
1
,
M
a
r
c
h
20
2
1
:
175
–
183
176
S
yna
ps
e
s
of
s
pi
ke
n
e
ur
on
w
hi
c
h
c
ons
i
s
ts
of
e
xc
it
a
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
o
n
a
r
e
r
e
qui
r
e
d
to
be
opt
im
iz
e
d
be
f
or
e
im
pl
e
m
e
nt
in
to
a
ne
twor
k
to
f
or
m
a
S
N
N
.
A
n
opt
im
i
z
a
ti
on
m
e
th
od
is
pr
opos
e
d
to
opt
im
iz
e
s
pi
ke
ne
ur
on by us
in
g m
a
xi
m
um
l
ik
e
li
hood
[
6]
. T
he
m
a
xi
m
um
li
ke
li
hood opti
m
iz
a
ti
on me
th
od i
s
us
e
d t
o
c
onf
ig
ur
e
th
e
s
in
gl
e
M
-
N
ne
ur
on
to
pr
e
di
c
t
th
e
f
ir
in
g
a
c
ti
vi
ty
of
th
e
ne
ur
on.
T
he
a
ve
r
a
ge
e
r
r
or
be
twe
e
n
th
e
a
c
tu
a
l
a
nd
pr
e
di
c
te
d
of
s
pi
ke
a
c
ti
vi
ty
is
3%
.
A
s
upe
r
vi
s
e
d
m
ul
ti
-
s
pi
ke
le
a
r
ni
ng
a
lg
or
it
hm
is
pr
opos
e
d
to
tr
a
in
ne
ur
ons
in
S
N
N
[
7]
. A
s
in
gl
e
ne
ur
on i
s
t
r
a
in
e
d t
o
l
e
a
r
n s
pi
ke
pa
tt
e
r
ns
i
n or
de
r
t
o ge
ne
r
a
te
s
pi
ke
t
r
a
in
s
. T
he
e
xpr
e
s
s
io
n of
m
e
m
br
a
ne
pot
e
nt
ia
l
is
s
im
pl
if
ie
d
by
th
e
a
lg
or
it
hm
a
nd
e
na
bl
e
s
th
e
opt
im
iz
a
ti
on
of
s
yna
pt
ic
w
e
ig
ht
s
th
r
ough
th
e
a
ppl
ic
a
ti
on
of
gr
a
di
e
nt
de
s
c
e
nt
.
T
h
e
r
e
s
ul
ts
s
how
e
d
th
a
t
th
e
a
lg
or
it
hm
a
bl
e
to
a
c
hi
e
ve
c
l
a
s
s
if
ic
a
ti
on
a
c
c
ur
a
c
y.
B
a
s
e
d
on
[
6]
a
nd
[
7]
,
th
e
ga
ps
c
a
n
be
s
e
e
n
f
r
om
th
e
ne
e
d
of
tr
a
in
in
g
da
ta
to
tr
a
in
th
e
m
ode
l.
M
a
xi
m
um
li
ke
li
hood
opt
im
iz
a
ti
on
m
e
th
od
a
nd
s
upe
r
vi
s
e
d
m
ul
ti
-
s
pi
ke
le
a
r
ni
ng
a
lg
or
it
hm
r
e
qui
r
e
d
tr
a
in
in
g
da
ta
s
e
t
to
tr
a
in
th
e
m
ode
l.
F
ur
th
e
r
m
o
r
e
,
a
n
uns
upe
r
vi
s
e
d
tr
a
in
in
g
a
lg
or
it
hm
is
im
pl
e
m
e
nt
e
d
to
t
r
a
in
S
N
N
[
8
]
.
S
pi
ke
ne
ur
on
m
ode
l
is
tr
a
in
e
d
us
in
g
s
yna
pt
ic
w
e
ig
ht
a
s
s
oc
ia
ti
on
tr
a
in
in
g
(
S
W
A
T
)
.
T
he
tr
a
in
in
g
a
nd
te
s
ti
ng
r
e
s
ul
ts
s
how
e
d
th
a
t
th
e
a
lg
or
it
hm
e
xhi
bi
ts
th
e
c
a
p
a
bi
li
ty
f
or
c
la
s
s
if
ic
a
ti
on
a
nd
c
onv
e
r
ge
nc
e
a
c
c
ur
a
c
y.
A
li
m
it
e
d pr
e
c
is
io
n (
S
N
N
/L
P
)
s
upe
r
vi
s
e
d l
e
a
r
ni
ng a
lg
or
it
hm
of
s
p
ik
in
g ne
ur
a
l
ne
twor
ks
i
s
i
m
pl
e
m
e
nt
e
d i
n S
N
N
tr
a
in
in
g
[
9]
.
S
yna
pt
ic
w
e
ig
ht
s
a
nd
s
yna
pt
ic
de
la
y
s
a
r
e
a
ppl
ie
d
w
it
h
li
m
it
e
d
pr
e
c
is
io
n
f
or
s
up
e
r
vi
s
e
d
le
a
r
ni
ng.
T
he
a
lg
or
it
hm
a
c
hi
e
ve
d
lo
w
m
e
a
n
s
qua
r
e
d
e
r
r
or
in
non
-
li
ne
a
r
X
O
R
c
la
s
s
if
ic
a
ti
on
pr
obl
e
m
a
nd
c
a
pa
bl
e
to
a
c
hi
e
ve
up t
o 97%
of
c
la
s
s
if
ic
a
ti
on a
c
c
ur
a
c
y.
I
n
s
pi
ki
ng
ne
ur
a
l
ne
twor
k
(
S
N
N
)
,
in
f
or
m
a
ti
on
is
e
m
it
te
d
a
nd
pr
oc
e
s
s
e
d
by
s
pi
ke
ne
ur
on
th
r
ough
a
s
e
que
nc
e
of
a
c
ti
on
pot
e
nt
ia
ls
w
hi
c
h
is
a
ls
o
known
a
s
s
pi
ke
s
[
10]
.
I
nf
or
m
a
ti
on
is
e
nc
ode
d
in
f
ir
in
g
r
a
te
of
s
pi
ke
ne
ur
on [
11]
.
S
pi
ke
ne
ur
on c
ons
is
ts
of
a
s
pi
ke
ge
ne
r
a
ti
on f
unc
ti
on f
or
f
ir
in
g
pur
pos
e
. T
he
s
pi
ke
f
unc
ti
on
is
non
-
di
f
f
e
r
e
nt
ia
bl
e
w
hi
c
h
c
r
e
a
te
a
di
s
c
ont
in
ui
ty
a
t
th
e
in
s
ta
n
c
e
of
f
ir
in
g
ti
m
e
.
N
on
-
di
f
f
e
r
e
nt
ia
bi
l
it
y
of
th
e
f
unc
ti
on
le
a
ds
to
di
f
f
ic
ul
ty
to
de
ve
lo
p
gr
a
di
e
nt
de
s
c
e
nt
to
pe
r
f
or
m
ba
c
kpr
opa
ga
ti
on
in
or
de
r
to
upda
te
th
e
w
e
ig
ht
of
s
pi
ke
ne
ur
on
f
or
m
in
im
iz
in
g
lo
s
s
[
12]
.
T
hi
s
ha
s
c
a
us
e
d
tr
a
in
in
g
of
S
N
N
us
in
g
ba
c
kpr
opa
ga
ti
on
be
c
om
e
di
f
f
ic
ul
t
a
s
c
om
pa
r
e
d t
o ot
he
r
a
r
ti
f
ic
ia
l
ne
ur
a
l
ne
twor
ks
(
A
N
N
)
[
13]
.
S
N
N
m
im
ic
s
bi
ol
ogi
c
a
l
ne
r
vous
s
ys
te
m
m
or
e
c
lo
s
e
ly
c
om
pa
r
e
d
to
c
onve
nt
io
na
l
a
r
ti
f
ic
ia
l
ne
ur
a
l
ne
twor
ks
[
14]
.
A
l
th
ough
S
N
N
is
b
io
lo
gi
c
a
ll
y
m
or
e
r
e
a
li
s
ti
c
th
a
n
a
r
ti
f
ic
ia
l
ne
ur
a
l
ne
twor
k
(
A
N
N
)
but
r
e
c
e
iv
e
s
le
s
s
a
tt
e
nt
io
n
th
a
n
A
N
N
due
to
th
e
di
f
f
ic
ul
ty
to
tr
a
in
S
N
N
[
15]
.
I
n
or
de
r
to
ove
r
c
om
e
th
e
non
-
di
f
f
e
r
e
nt
ia
bi
li
ty
of
s
pi
ke
f
unc
ti
on
th
a
t
le
a
ds
to
di
f
f
ic
ul
ty
in
S
N
N
tr
a
in
in
g,
de
e
p
r
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng
is
a
ppl
ie
d
to
ba
l
a
n
c
e
th
e
f
i
r
in
g
r
a
te
of
e
xc
it
a
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
on
of
s
pi
ke
ne
ur
on.
S
pi
ke
ne
ur
on
ha
s
di
f
f
e
r
e
nt
f
ir
in
g
r
a
te
of
s
pi
ke
s
w
he
n
di
f
f
e
r
e
nt
c
onf
ig
ur
a
ti
on
on
th
e
f
ir
in
g
r
a
te
of
e
xc
it
a
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
on
of
th
e
ne
ur
on
is
a
ppl
ie
d
[
16]
.
T
he
f
ir
in
g
r
a
te
of
in
hi
bi
to
r
y
popula
ti
on
o
f
th
e
s
pi
ke
ne
ur
on
is
in
it
ia
li
z
e
d
a
s
in
put
a
nd
a
dj
us
te
d
dur
in
g
tr
a
in
in
g
to
a
c
hi
e
ve
th
e
f
ir
in
g
r
a
te
of
e
xc
it
a
to
r
y
popu
la
ti
on
of
th
e
ne
ur
on
ha
s
th
e
s
a
m
e
r
a
te
w
it
h
th
e
ta
r
ge
t
ne
ur
on
f
ir
in
g
r
a
te
.
I
n
th
is
r
e
s
e
a
r
c
h,
two
a
lg
or
it
hm
s
of
r
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng
a
r
e
pr
opos
e
d
to
a
c
t
a
s
a
g
e
nt
s
w
hi
c
h
a
r
e
d
e
e
p
Q
n
e
twor
k
(
D
Q
N
)
a
nd
d
e
e
p
Q
-
le
a
r
ni
ng
w
it
h
nor
m
a
li
z
e
d
a
dva
nt
a
ge
f
unc
ti
ons
(
N
A
F
)
to
in
te
r
a
c
t
w
it
h
a
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
to
opt
im
iz
e
s
pi
ke
ne
ur
on
in
to
ba
la
nc
e
s
ta
te
.
O
th
e
r
th
a
n
pr
e
vi
ou
s
r
e
s
e
a
r
c
h
w
or
ks
th
a
t
us
in
g
de
e
p
le
a
r
ni
ng
or
r
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng,
th
is
r
e
s
e
a
r
c
h
w
or
k
a
ppl
ie
d
de
e
p
r
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng
to
s
ol
ve
th
e
di
f
f
ic
ul
ty
in
S
N
N
tr
a
in
in
g
by
us
in
g
ba
c
kpr
opa
ga
ti
on
a
lg
or
it
hm
.
T
h
e
a
lg
or
it
hm
c
ons
is
ts
of
r
e
in
f
or
c
e
m
e
nt
le
a
r
ni
ng
a
lg
or
it
hm
w
it
h
de
e
p
ne
ur
a
l
ne
twor
k
f
or
a
ppr
oxi
m
a
ti
on
of
Q
f
unc
ti
on.
T
he
m
ot
iv
a
ti
on
of
th
is
p
a
pe
r
is
to
tr
a
in
s
in
gl
e
s
pi
ke
n
e
ur
on
us
in
g
de
e
p
r
e
in
f
or
c
e
m
e
nt
l
e
a
r
ni
ng
w
it
h
th
e
a
bs
e
nc
e
of
tr
a
in
in
g
da
ta
s
e
t
in
or
de
r
to
a
tt
a
in
goa
ls
.
T
he
a
lg
or
it
hm
s
le
a
r
n
f
r
om
e
xpe
r
ie
nc
e
to
pe
r
f
or
m
a
n
a
c
ti
on t
o m
a
xi
m
iz
e
r
e
w
a
r
ds
.
F
ig
ur
e
1. A
r
c
hi
te
c
tu
r
e
of
D
e
e
p R
e
in
f
or
c
e
m
e
nt
L
e
a
r
ni
ng
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
A
r
ti
f
I
nt
e
ll
I
S
S
N
:
2252
-
8938
Spi
k
e
ne
ur
on opti
m
iz
at
io
n us
in
g d
e
e
p r
e
in
fo
r
c
e
m
e
nt
l
e
ar
ni
ng
(
T
an Sz
i
H
ui
)
177
2.
R
E
S
E
A
R
C
H
M
E
T
H
O
D
A
s
pi
ke
ne
ur
on
is
c
r
e
a
te
d
by
us
in
g
ne
ur
a
l
s
im
ul
a
ti
on
to
ol
(
N
E
S
T
)
s
im
ul
a
to
r
.
S
in
gl
e
s
pi
ke
ne
ur
on
is
us
e
d
in
tr
a
in
in
g
f
or
opt
im
iz
a
ti
on.
A
s
pi
ke
n
e
ur
on
is
m
ode
le
d
by
us
in
g
P
yN
E
S
T
c
om
m
a
nd
in
P
yt
hon
pr
ogr
a
m
m
in
g
la
ngua
ge
a
f
te
r
a
c
us
to
m
e
nvi
r
onm
e
nt
is
bui
lt
.
S
im
ul
a
ti
on
pa
r
a
m
e
te
r
s
a
r
e
r
e
qui
r
e
d
to
be
in
it
ia
li
z
e
d f
or
N
E
S
T
s
im
ul
a
to
r
t
o m
ode
l
a
s
pi
ke
ne
ur
on a
s
s
how
n i
n T
a
bl
e
1 [
17]
.
T
a
bl
e
1. S
im
ul
a
ti
on pa
r
a
m
e
te
r
s
of
s
pi
ke
ne
ur
on
S
i
m
ul
a
t
i
on P
a
r
a
m
e
t
e
r
s
V
a
l
ue
S
i
m
ul
a
t
i
on T
i
m
e
, t
_s
i
m
25000
S
i
z
e
of
E
xc
i
t
a
t
or
y P
opul
a
t
i
on, n_e
x
16000
S
i
z
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on, n_i
n
4000
M
e
a
n R
a
t
e
of
E
xc
i
t
a
t
or
y P
opul
a
t
i
on
5.0H
z
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on w
i
t
h a
r
a
ndom
num
be
r
r
a
nge
, r
_i
n
17.5
-
19.5H
z
P
e
a
k A
m
pl
i
t
ude
of
E
xc
i
t
a
t
or
y P
opul
a
t
i
on, e
ps
c
50
P
e
a
k A
m
pl
i
t
ude
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on, i
ps
c
-
50
S
yna
pt
i
c
D
e
l
a
y,d
0.01
L
ow
e
r
bound of
s
e
a
r
c
h i
nt
e
r
va
l
, s
e
l
f
.l
ow
e
r
0
Up
pe
r
bound of
s
e
a
r
c
h i
nt
e
r
va
l
, s
e
l
f
.uppe
r
50
A
c
us
to
m
e
nvi
r
onm
e
nt
i
s
c
r
e
a
t
e
d
us
in
g
O
pe
nA
I
gym
to
ol
ki
t.
T
he
s
pi
ke
n
e
ur
on
is
c
onv
e
r
te
d
in
to
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
a
f
te
r
th
e
c
us
to
m
e
nvi
r
onm
e
nt
is
bui
l
t.
T
he
e
nvi
r
onm
e
nt
s
e
t
th
e
in
it
ia
l
s
ta
te
f
or
th
e
pr
obl
e
m
s
to
be
s
ol
ve
d.
A
c
ti
on
s
pa
c
e
a
nd
obs
e
r
va
ti
on
s
pa
c
e
a
r
e
c
onf
ig
ur
e
d
f
o
r
bot
h
D
R
L
a
lg
or
it
hm
s
.
A
c
ti
on
s
pa
c
e
r
e
pr
e
s
e
nt
s
how
m
a
ny
po
s
s
ib
le
a
c
ti
ons
f
or
th
e
D
R
L
a
ge
nt
s
to
in
te
r
a
c
t
w
it
h
th
e
e
nvi
r
onm
e
nt
a
nd
obs
e
r
va
ti
on
s
p
a
c
e
r
e
pr
e
s
e
nt
s
a
ll
th
e
da
ta
th
a
t
ge
n
e
r
a
te
d
by
th
e
e
nvi
r
onm
e
nt
a
nd
to
b
e
ob
s
e
r
ve
d
by
th
e
a
g
e
nt
s
a
s
s
how
n i
n F
ig
ur
e
2.
F
ig
ur
e
2. I
nt
e
r
a
c
ti
on of
D
R
L
a
ge
nt
s
a
nd a
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
I
n
D
R
L
a
lg
or
it
hm
,
no
tr
a
in
in
g
da
ta
s
e
t
is
r
e
qui
r
e
d
a
s
in
put
to
pr
ovi
de
r
a
w
da
ta
f
or
tr
a
in
in
g.
T
he
D
R
L
a
ge
nt
s
s
e
le
c
t
th
e
a
c
ti
on
to
be
ta
ke
n
w
it
hout
tr
a
in
in
g
da
ta
. T
he
a
ge
nt
s
ge
ne
r
a
te
th
e
ir
ow
n
da
ta
a
c
c
or
di
ng
to
th
e
gi
ve
n
s
ta
te
,
a
c
ti
ons
ta
ke
n
a
nd
r
e
w
a
r
d
by
in
te
r
a
c
ti
ng
w
it
h
t
he
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
n
A
I
G
ym
f
r
a
m
e
w
or
k.
T
he
tr
a
in
in
g
da
ta
w
hi
c
h
a
ls
o
kno
w
n
a
s
e
xpe
r
ie
nc
e
is
s
to
r
e
d
in
m
e
m
or
y.
T
he
a
ge
nt
s
le
a
r
n
f
r
om
e
xpe
r
ie
nc
e
to
m
a
ke
de
c
is
io
ns
on
th
e
a
c
ti
on
to
be
ta
ke
n
to
obt
a
in
th
e
m
a
xi
m
um
r
e
w
a
r
ds
in
or
de
r
to
a
c
hi
e
ve
goa
l
[
18]
.
A
de
e
p
ne
ur
a
l
ne
twor
k
i
s
c
ons
tr
uc
te
d
in
D
Q
N
a
s
D
Q
N
is
va
lu
e
-
ba
s
e
d
m
e
th
od.
A
c
ti
on
s
pa
c
e
f
or
D
Q
N
i
n t
he
c
us
to
m
e
nvi
r
onm
e
nt
i
s
di
s
c
r
e
te
t
ype
w
it
h 4 pos
s
ib
le
a
c
ti
ons
. T
he
a
g
e
nt
t
a
ke
s
a
c
ti
on
s
ba
s
e
d on the
4
pos
s
ib
le
a
c
ti
on
s
de
f
in
e
d
a
s
s
how
n
in
T
a
bl
e
2. T
he
a
g
e
nt
r
e
c
e
i
ve
s
r
e
w
a
r
ds
a
c
c
or
di
ng
to
th
e
a
c
ti
on
ta
ke
n a
nd
s
ta
te
.
4000
t
r
a
in
in
g
s
te
ps
a
r
e
ta
ke
n
to
ba
la
nc
e
th
e
f
ir
in
g
r
a
te
of
e
xc
it
a
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
on
of
th
e
s
pi
ke
ne
ur
on.
T
he
tr
a
in
e
d
m
ode
l
i
s
us
e
d
f
or
te
s
ti
ng
to
va
li
da
te
t
he
pe
r
f
or
m
a
nc
e
f
or
5
e
pi
s
ode
s
.
T
h
e
f
lo
w
c
ha
r
t
of
t
hi
s
a
lg
or
it
hm
i
s
s
how
e
d i
n F
ig
ur
e
3(
a
)
.
T
a
bl
e
2. A
c
ti
on l
is
t
of
D
Q
N
A
c
t
i
on
D
e
t
a
i
l
s
of
a
c
t
i
on
0
C
ur
r
e
nt
i
nhi
bi
t
or
y r
a
t
e
+ 0.01
1
C
ur
r
e
nt
i
nhi
bi
t
or
y r
a
t
e
-
0.01
2
C
ur
r
e
nt
i
nhi
bi
t
or
y r
a
t
e
+ r
a
ndom
num
be
r
f
r
om
r
a
nge
of
0.02 t
o 0.05
3
C
ur
r
e
nt
i
nhi
bi
t
or
y r
a
t
e
-
r
a
ndom
num
be
r
f
r
om
r
a
nge
of
0.02 t
o
0.05
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2252
-
8938
I
nt
J
A
r
ti
f
I
nt
e
ll
,
V
ol
.
10
, N
o.
1
,
M
a
r
c
h
20
2
1
:
175
–
183
178
N
A
F
a
ge
nt
is
c
ont
r
uc
te
d
w
it
h
a
s
ta
te
-
va
lu
e
f
unc
ti
on
a
nd
a
n
a
dva
nt
a
ge
f
unc
ti
on
[
19]
.
I
n
N
A
F
,
th
r
e
e
ne
twor
ks
a
r
e
im
pl
e
m
e
nt
e
d
in
tr
a
in
in
g
to
a
ppr
oxi
m
a
te
Q
f
u
nc
ti
on
w
hi
c
h
a
r
e
m
u_m
ode
l,
V
_m
ode
l
a
nd
L
_m
ode
l.
T
he
V
_m
ode
l
is
th
e
ne
twor
k
us
e
d
to
le
a
r
n
s
ta
te
v
a
lu
e
f
unc
ti
on
a
nd
m
u_m
ode
l
is
th
e
ne
twor
k
a
ppl
ie
d
to
s
e
le
c
t
a
c
ti
on
to
be
ta
ke
n
th
a
t
c
a
n
m
a
xi
m
iz
e
s
Q
f
u
nc
ti
on.
A
n
a
dva
nt
a
ge
f
unc
ti
on
is
c
on
s
tr
uc
t
in
L
_m
ode
l.
A
c
ti
on
s
pa
c
e
of
N
A
F
a
lg
or
it
hm
is
c
ont
in
uous
dom
a
i
n
[
20
-
21]
.
T
he
a
c
ti
on
v
a
lu
e
is
r
a
ndom
s
e
le
c
te
d
by
th
e
N
A
F
a
ge
nt
f
r
om
th
e
r
a
nge
of
0
to
50
w
hi
c
h
is
th
e
s
e
a
r
c
h
in
te
r
va
l
r
a
nge
.
R
e
w
a
r
d
i
s
f
e
e
dba
c
k
to
th
e
a
ge
nt
f
r
om
th
e
e
nvi
r
onm
e
nt
.
T
he
tr
a
in
in
g
s
te
ps
is
s
e
t
to
26000
s
te
ps
to
ba
la
nc
e
th
e
f
ir
in
g
r
a
te
of
e
xhi
bi
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
on
of
s
pi
ke
ne
ur
on.
A
f
te
r
t
r
a
in
e
d
th
e
m
ode
l,
th
e
m
ode
l
is
im
pl
e
m
e
nt
e
d
in
te
s
ti
ng
in
or
de
r
t
o va
li
da
te
t
he
pe
r
f
or
m
a
nc
e
f
or
5 e
pi
s
ode
s
. T
he
f
lo
w
c
h
a
r
t
of
t
hi
s
a
lg
or
it
hm
i
s
s
how
e
d i
n F
ig
ur
e
3(
b)
.
(
a
)
(
b)
F
ig
ur
e
3. F
lo
w
c
ha
r
t
of
t
he
a
lg
or
it
hm
,
(
a
)
D
Q
N
, (
b)
N
A
F
3.
R
E
S
U
L
T
S
A
N
D
D
I
S
C
U
S
S
I
O
N
T
he
s
pi
ke
n
e
ur
on
m
ode
l
is
tr
a
in
e
d
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
bui
lt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k.
T
he
m
ode
l
is
tr
a
in
e
d
unt
il
it
a
bl
e
to
m
e
e
t
th
e
ta
r
ge
t
ne
ur
on
r
a
te
a
nd
a
c
hi
e
ve
c
onve
r
ge
n
c
y.
W
he
n
th
e
m
ode
l
is
not
gi
ve
n e
nough tr
a
in
in
g, t
he
m
ode
l
is
not
a
bl
e
t
o m
e
e
t
th
e
c
on
ve
r
ge
nc
e
a
nd t
he
goa
l
is
un
a
bl
e
t
o a
tt
a
in
.
3.1. DQN algor
it
h
m
S
pi
ke
ne
ur
on
is
opt
im
iz
e
d
us
in
g
D
Q
N
w
it
h
4000
s
te
ps
.
T
h
e
D
Q
N
a
ge
nt
in
te
r
a
c
te
d
w
it
h
th
e
c
us
to
m
e
nvi
r
onm
e
nt
a
nd
s
e
le
c
te
d
th
e
a
c
ti
ons
to
be
ta
ke
n.
A
pl
ot
of
e
pi
s
o
de
r
e
w
a
r
d
ve
r
s
us
e
pi
s
ode
s
is
ge
ne
r
a
te
d
a
s
s
how
n
in
F
ig
ur
e
4.
T
he
le
a
r
ni
ng
c
ur
ve
in
di
c
a
te
d
th
a
t
th
e
D
Q
N
a
ge
nt
c
a
pa
bl
e
to
e
xpl
or
e
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k.
T
he
tr
a
in
e
d
m
ode
l
a
bl
e
to
r
e
a
c
t
to
w
a
r
d
s
th
e
c
us
to
m
e
nvi
r
onm
e
nt
to
a
tt
a
in
th
e
goa
l
to
tr
a
in
s
pi
ke
ne
ur
on
in
to
ba
la
nc
e
s
ta
t
e
.
D
ur
in
g
th
e
in
it
ia
l
s
ta
te
,
th
e
a
g
e
nt
s
obt
a
in
e
d
ne
g
a
ti
ve
r
e
w
a
r
d
a
s
th
e
a
ge
nt
is
una
bl
e
to
e
xpl
or
e
in
th
e
c
us
to
m
e
n
vi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
to
opt
im
iz
e
s
pi
ke
ne
ur
on.
D
ur
in
g
tr
a
in
in
g,
th
e
a
ge
nt
le
a
r
ns
to
e
xpl
or
e
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
a
nd
r
e
c
e
iv
e
s
m
or
e
r
e
w
a
r
ds
.
T
he
a
ge
nt
r
e
c
e
iv
e
s
pos
it
iv
e
a
nd
ne
g
a
ti
ve
r
e
w
a
r
ds
ba
s
e
d
on
th
e
gi
ve
n
s
ta
te
a
nd
a
c
ti
on
ta
ke
n
th
r
oughout
th
e
tr
a
in
in
g.
E
a
c
h
a
c
ti
on
is
s
e
le
c
te
d
r
a
ndoml
y
f
r
om
4
di
s
c
r
e
te
a
c
ti
ons
th
a
t
de
f
in
e
d
in
a
c
ti
on
s
pa
c
e
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
A
r
ti
f
I
nt
e
ll
I
S
S
N
:
2252
-
8938
Spi
k
e
ne
ur
on opti
m
iz
at
io
n us
in
g d
e
e
p r
e
in
fo
r
c
e
m
e
nt
l
e
ar
ni
ng
(
T
an Sz
i
H
ui
)
179
of
th
e
a
lg
or
it
hm
.
W
it
h
th
is
c
a
p
a
bi
li
ty
to
e
xpl
or
e
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
,
th
e
m
ode
l
be
c
a
m
e
us
a
bl
e
f
or
te
s
ti
ng. Af
te
r
t
r
a
in
in
g,
th
e
m
ode
l
is
t
e
s
te
d f
or
5 e
pi
s
ode
s
t
o va
li
da
te
t
he
pe
r
f
or
m
a
nc
e
of
t
he
m
ode
l.
T
he
m
ode
l
r
e
c
e
iv
e
d r
e
w
a
r
ds
f
or
e
a
c
h e
p
i
s
ode
dur
in
g t
e
s
ti
ng a
s
s
how
n i
n F
i
gur
e
5.
F
ig
ur
e
4. L
e
a
r
ni
ng c
ur
ve
of
t
he
s
pi
ke
ne
ur
on mode
l
us
in
g D
Q
N
a
lg
or
it
hm
F
ig
ur
e
5. T
e
s
ti
ng c
ur
ve
of
t
he
t
r
a
in
e
d m
ode
l
us
in
g D
Q
N
a
lg
or
it
hm
T
he
i
nhi
bi
to
r
y popula
ti
on r
a
te
i
s
f
in
e
-
tu
ne
d by the
a
ge
nt
in
or
de
r
t
o a
tt
a
in
t
he
goa
l.
T
he
t
e
s
ti
ng r
e
s
ul
t
is
ta
bul
a
te
d
in
T
a
bl
e
3.
T
he
a
c
tu
a
l
out
put
ne
ur
on
r
a
te
o
f
two
e
pi
s
ode
s
a
r
e
c
lo
s
e
r
to
th
e
ta
r
ge
t
ne
ur
on
f
ir
in
g
r
a
te
w
it
h
th
e
di
f
f
e
r
e
nc
e
of
0.04H
z
in
th
ir
d
a
nd
f
if
th
e
pi
s
od
e
s
w
he
r
e
a
s
th
e
a
c
tu
a
l
out
put
n
e
ur
on
r
a
te
is
a
tt
a
in
e
d
th
e
goa
l
in
th
e
f
our
th
e
pi
s
ode
.
I
n
f
ir
s
t
a
nd
s
e
c
ond
e
pi
s
ode
s
of
te
s
ti
ng
obt
a
in
e
d
th
e
hi
ghe
s
t
va
lu
e
of
di
f
f
e
r
e
nc
e
be
twe
e
n
a
c
tu
a
l
a
nd
ta
r
ge
t
n
e
ur
on
f
ir
in
g
r
a
te
w
hi
c
h
i
s
0.08H
z
.
T
he
pe
r
c
e
nt
a
g
e
of
e
r
r
or
be
twe
e
n
th
e
r
a
te
of
di
f
f
e
r
e
nc
e
of
a
c
tu
a
l
a
nd t
a
r
ge
t
out
put
ne
ur
on
r
a
te
a
nd goa
l
is
c
a
lc
ul
a
te
d i
n T
a
bl
e
4. T
he
a
ve
r
a
g
e
pe
r
c
e
nt
a
ge
of
e
r
r
or
a
c
hi
e
ve
d
0.96%
.
T
he
lo
w
e
s
t
s
t
e
ps
ta
ke
n
f
or
a
c
tu
a
l
out
put
ne
ur
on
r
a
te
to
m
e
e
t
w
it
h
ta
r
ge
t
ne
ur
on
f
ir
in
g
r
a
te
is
84
s
t
e
ps
.
T
he
r
e
s
ul
t
s
how
e
d
th
a
t
th
e
c
a
pa
bi
li
ty
of
th
e
tr
a
in
e
d
m
ode
l
to
in
te
r
a
c
t
w
it
h
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
to
opt
im
iz
e
th
e
f
ir
i
ng
r
a
te
of
e
xc
it
a
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
on
of
t
he
s
pi
ke
ne
ur
on i
nt
o ba
la
nc
e
s
t
a
te
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2252
-
8938
I
nt
J
A
r
ti
f
I
nt
e
ll
,
V
ol
.
10
, N
o.
1
,
M
a
r
c
h
20
2
1
:
175
–
183
180
T
a
bl
e
3.
T
e
s
ti
ng r
e
s
ul
t
of
t
he
t
r
a
in
e
d m
ode
l
us
in
g D
Q
N
a
lg
o
r
it
hm
E
pi
s
ode
S
i
m
ul
a
t
i
on P
a
r
a
m
e
t
e
r
O
pt
i
m
a
l
V
a
l
ue
gi
ve
n
by D
Q
N
A
ge
nt
(
H
z
)
T
ot
a
l
R
e
w
a
r
d
S
t
e
ps
t
a
ke
n t
o
a
t
t
a
i
n goa
l
1
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.76
-
5540000.00
89
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
18.66
O
ut
put
N
e
ur
on r
a
t
e
5.08
2
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.78
-
5964000.00
92
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
18.66
O
ut
put
N
e
ur
on r
a
t
e
4.92
3
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.81
-
4522000.00
91
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
18.66
O
ut
put
N
e
ur
on r
a
t
e
5.04
4
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.73
-
5462000.00
84
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
18.66
O
ut
put
N
e
ur
on r
a
t
e
5.00
5
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y
P
opul
a
t
i
on
20.83
-
1645000.00
86
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
18.66
O
ut
put
N
e
ur
on r
a
t
e
4.96
T
a
bl
e
4.
T
e
s
ti
ng r
e
s
ul
t
f
or
out
put
a
nd t
a
r
ge
t
ne
ur
on f
ir
in
g r
a
te
i
n D
Q
N
a
lg
or
it
hm
E
pi
s
ode
T
a
r
ge
t
ne
ur
on
f
i
r
i
ng r
a
t
e
(
H
z
)
A
c
t
ua
l
out
put
ne
ur
on r
a
t
e
(
H
z
)
D
i
f
f
e
r
e
nc
e
of
out
put
a
nd t
a
r
ge
t
ne
ur
on
f
i
r
i
ng r
a
t
e
(
H
z
)
P
e
r
c
e
nt
a
ge
of
E
r
r
o
r
(%)
A
ve
r
a
ge
P
e
r
c
e
nt
a
ge
of
E
r
r
or
(
%
)
1
5.00
5.08
0.08
1.60
0.96%
2
4.92
0.08
1.60
3
5.04
0.04
0.80
4
5.00
0.00
0.00
5
4.96
0.04
0.80
3.2. NA
F
al
gor
it
h
m
S
pi
ke
ne
ur
on
is
opt
im
iz
e
d
us
in
g
N
A
F
w
it
h
26000
tr
a
in
in
g
s
te
p
s
.
T
he
N
A
F
a
ge
nt
is
le
a
r
nt
to
in
te
r
a
c
t
w
it
h
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
a
n
d
to
s
e
le
c
t
a
c
ti
on
to
be
ta
ke
n
to
ge
t
m
a
xi
m
um
r
e
w
a
r
ds
.
A
pl
ot
of
e
pi
s
ode
r
e
w
a
r
d
ve
r
s
us
e
pi
s
ode
s
i
s
ge
ne
r
a
t
e
d
a
s
s
how
n
in
F
ig
ur
e
6.
T
he
l
e
a
r
ni
ng
c
ur
ve
s
how
e
d
th
e
e
xpl
or
a
ti
on
of
N
A
F
a
ge
nt
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
.
T
he
r
e
s
ul
t
pr
ove
d
th
a
t
th
e
N
A
F
a
ge
nt
ha
s
th
e
c
a
pa
bi
li
ty
to
e
xpl
or
e
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k.
T
he
tr
a
in
e
d
m
o
de
l
c
a
pa
bl
e
t
o r
e
a
c
t
to
w
a
r
ds
t
he
e
nvi
r
onm
e
nt
t
o
opt
im
iz
e
t
he
f
ir
in
g
r
a
te
of
e
xc
it
a
to
r
y
a
nd i
nhi
bi
to
r
y
popula
ti
on o
f
th
e
s
pi
ke
ne
ur
on
in
to
b
a
la
nc
e
s
ta
te
.
T
he
a
ge
nt
obt
a
in
e
d
pos
it
iv
e
a
nd
ne
g
a
ti
ve
r
e
w
a
r
ds
f
lu
c
tu
a
t
e
ly
due
to
th
e
c
ont
in
uous
a
c
ti
on
s
pa
c
e
.
T
h
e
r
a
nge
of
th
e
a
c
ti
on
va
lu
e
is
be
tw
e
e
n
0
to
50.
D
if
f
e
r
e
nt
a
c
ti
on
va
lu
e
s
a
r
e
be
in
g
s
e
le
c
te
d
r
a
ndoml
y
f
or
a
c
ti
on
s
ta
ke
n
in
tr
a
in
in
g.
T
he
m
ode
l
a
bl
e
to
pe
r
f
or
m
e
xpl
or
a
ti
on
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
us
in
g
N
A
F
a
lg
or
it
hm
a
nd
th
e
m
ode
l
c
a
n
be
a
ppl
ie
d
f
or
te
s
ti
ng.
5
e
pi
s
ode
s
of
te
s
ti
ng
is
a
ppl
ie
d
to
te
s
t
th
e
pe
r
f
or
m
a
nc
e
of
th
e
tr
a
in
e
d
m
ode
l
f
or
va
li
da
ti
on
pur
pos
e
a
s
s
how
n
in
F
ig
ur
e
7. T
he
te
s
ti
ng
r
e
s
ul
t
is
r
e
c
or
de
d i
n T
a
bl
e
5.
F
ig
ur
e
6. L
e
a
r
ni
ng c
ur
ve
of
t
he
s
pi
ke
ne
ur
on mode
l
us
in
g N
A
F
a
lg
or
it
hm
F
ig
ur
e
7. T
e
s
ti
ng c
ur
ve
of
t
he
t
r
a
in
e
d m
ode
l
us
in
g
N
A
F
a
lg
or
it
hm
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
A
r
ti
f
I
nt
e
ll
I
S
S
N
:
2252
-
8938
Spi
k
e
ne
ur
on opti
m
iz
at
io
n us
in
g d
e
e
p r
e
in
fo
r
c
e
m
e
nt
l
e
ar
ni
ng
(
T
an Sz
i
H
ui
)
181
T
he
f
ir
in
g
r
a
te
of
in
hi
bi
to
r
y
popula
ti
on
is
f
in
e
-
tu
ne
d
by
th
e
a
ge
nt
to
a
tt
a
in
th
e
goa
l.
T
h
e
pe
r
c
e
nt
a
g
e
of
e
r
r
or
be
twe
e
n
th
e
r
a
te
of
di
f
f
e
r
e
nc
e
of
a
c
tu
a
l
out
put
ne
ur
on
r
a
te
a
nd
goa
l
i
s
r
e
c
or
de
d
in
T
a
bl
e
6.
T
h
e
a
ve
r
a
ge
pe
r
c
e
nt
a
ge
of
e
r
r
or
be
twe
e
n r
a
te
of
di
f
f
e
r
e
nc
e
of
out
put
a
nd t
a
r
ge
t
ne
ur
on f
ir
in
g r
a
te
a
c
hi
e
ve
d 0.80%
.
T
he
lo
w
e
s
t
s
te
p
s
ta
k
e
n
f
or
a
c
tu
a
l
out
put
ne
ur
on
r
a
te
to
m
e
e
t
w
it
h
or
c
lo
s
e
to
th
e
ta
r
ge
t
e
xc
it
a
to
r
y
popula
ti
on
r
a
te
is
3
s
te
ps
.
T
he
r
e
s
ul
t
s
ho
w
e
d
th
e
tr
a
in
e
d
m
ode
l
a
bl
e
to
in
te
r
a
c
t
w
it
h
th
e
c
u
s
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k t
o a
c
hi
e
ve
t
he
ba
la
nc
e
s
ta
te
of
s
pi
ke
n
e
ur
on.
T
a
bl
e
5.
T
e
s
ti
ng r
e
s
ul
t
of
t
he
t
r
a
in
e
d m
ode
l
us
in
g N
A
F
a
lg
or
it
h
m
E
pi
s
ode
S
i
m
ul
a
t
i
on P
a
r
a
m
e
t
e
r
O
pt
i
m
a
l
V
a
l
ue
gi
ve
n
by D
Q
N
A
ge
nt
(
H
z
)
T
ot
a
l
R
e
w
a
r
d
S
t
e
ps
t
a
ke
n t
o
a
t
t
a
i
n goa
l
1
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.80
-
480.00
11
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
19.23
O
ut
put
N
e
ur
on r
a
t
e
5.08
2
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y
P
opul
a
t
i
on
20.80
40.00
3
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
19.23
O
ut
put
N
e
ur
on r
a
t
e
5.04
3
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.80
-
710.00
13
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
19.23
O
ut
put
N
e
ur
on r
a
t
e
4.96
4
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.80
-
2030.00
21
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
19.23
O
ut
put
N
e
ur
on r
a
t
e
5.00
5
M
e
a
n R
a
t
e
of
I
nhi
bi
t
or
y P
opul
a
t
i
on
20.80
-
130.00
8
I
ni
t
i
a
l
R
a
t
e
of
I
nhi
bi
t
or
y P
opu
l
a
t
i
on
19.23
O
ut
put
N
e
ur
on r
a
t
e
4.96
T
a
bl
e
6.
T
e
s
ti
ng r
e
s
ul
t
f
or
out
put
a
nd t
a
r
ge
t
ne
ur
on f
ir
in
g r
a
te
i
n N
A
F
a
lg
or
it
hm
E
pi
s
ode
T
a
r
ge
t
ne
ur
on
f
i
r
i
ng r
a
t
e
(
H
z
)
A
c
t
ua
l
out
put
ne
ur
on r
a
t
e
(
H
z
)
D
i
f
f
e
r
e
nc
e
of
out
put
a
nd
t
a
r
ge
t
ne
ur
on f
i
r
i
ng
r
a
t
e
(
H
z
)
P
e
r
c
e
nt
a
ge
of
E
r
r
or
(
%
)
A
ve
r
a
ge
P
e
r
c
e
nt
a
ge
of
E
r
r
or
(
%
)
1
5.00
5.08
0.08
1.60
0.80%
2
5.04
0.04
0.80
3
4.96
0.04
0.80
4
5.00
0.00
0.00
5
4.96
0.04
0.80
3.3. E
val
u
at
io
n
of
D
Q
N
an
d
N
A
F
al
gor
it
h
m
S
pi
ke
ne
ur
on
is
opt
im
iz
e
d
to
ba
la
nc
e
th
e
f
ir
in
g
r
a
te
of
e
xc
it
a
to
r
y
a
nd
in
hi
bi
to
r
y
popula
ti
on
by
us
in
g
D
Q
N
a
nd
N
A
F
a
lg
or
it
hm
s
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k.
T
he
e
va
lu
a
ti
on
of
th
e
pe
r
f
or
m
a
nc
e
of
t
he
D
Q
N
a
nd N
A
F
t
r
a
in
e
d m
ode
l
is
t
a
bul
a
te
d i
n T
a
bl
e
7.
D
Q
N
a
lg
or
it
hm
is
a
ppl
ie
d
to
t
r
a
in
th
e
s
pi
ke
ne
ur
on
in
di
s
c
r
e
te
a
c
ti
on
s
pa
c
e
w
he
r
e
a
s
N
A
F
a
lg
or
it
hm
is
im
pl
e
m
e
nt
e
d
to
tr
a
in
th
e
s
pi
ke
ne
ur
on
in
c
ont
in
uou
s
dom
a
in
.
T
he
ty
pe
s
of
a
c
ti
on
s
pa
c
e
to
us
e
de
pe
nd
s
on
th
e
a
ppl
ic
a
ti
ons
.
T
he
tr
a
in
in
g
s
te
p
s
of
D
Q
N
is
lo
w
e
r
th
a
n
N
A
F
a
s
4000
tr
a
in
in
g
s
te
ps
a
r
e
e
xe
c
u
te
d
on
th
e
m
ode
l
a
nd
a
bl
e
to
m
e
e
t
th
e
goa
l.
T
he
tr
a
in
in
g
ti
m
e
is
lo
nge
r
f
o
r
N
A
F
a
s
th
e
m
ode
l
is
tr
a
in
e
d
f
or
26000
s
te
ps
to
a
tt
a
in
th
e
goa
l.
T
he
a
ve
r
a
ge
pe
r
c
e
nt
a
ge
e
r
r
or
of
r
a
te
of
di
f
f
e
r
e
nc
e
be
twe
e
n
ta
r
ge
t
a
nd
a
c
tu
a
l
out
put
ne
ur
on
f
ir
in
g
r
a
te
in
N
A
F
is
lo
w
e
r
th
a
n
D
Q
N
.
N
A
F
a
bl
e
to
a
c
hi
e
ve
0.80%
of
pe
r
c
e
nt
a
ge
e
r
r
or
in
th
e
te
s
ti
ng
tr
a
in
e
d
m
ode
l.
F
ur
th
e
r
m
or
e
,
s
te
ps
ta
ke
n
f
or
a
c
tu
a
l
out
put
ne
ur
on
r
a
te
t
o
m
e
e
t
w
it
h
or
c
lo
s
e
to
th
e
ta
r
ge
t
ne
ur
on
f
ir
in
g
r
a
te
in
N
A
F
is
lo
w
e
r
c
om
pa
r
e
d
to
D
Q
N
w
hi
c
h
onl
y
3
s
te
p
s
ta
ke
n
to
a
tt
a
in
th
e
goa
l.
T
hi
s
in
di
c
a
te
s
th
a
t
N
A
F
a
lg
or
it
hm
a
bl
e
t
o opti
m
iz
e
s
pi
ke
ne
ur
on i
nt
o ba
la
nc
e
s
ta
t
e
f
a
s
te
r
t
ha
n D
Q
N
.
T
he
pe
r
f
or
m
a
nc
e
of
D
Q
N
a
nd
N
A
F
is
c
om
pa
r
e
d
w
it
h
a
p
r
e
vi
ous
r
e
s
e
a
r
c
h
w
or
k
w
hi
c
h
us
in
g
m
a
xi
m
um
l
ik
e
li
hood opti
m
iz
a
ti
on me
th
od t
o opti
m
iz
e
s
pi
ke
ne
ur
on a
s
s
how
n i
n T
a
bl
e
8.
T
a
bl
e
7. E
va
lu
a
ti
on r
e
s
ul
t
of
D
Q
N
a
nd N
A
F
A
lg
or
it
hm
DQN
NAF
A
c
t
i
on S
pa
c
e
D
i
s
c
r
e
t
e
C
ont
i
nuous
T
r
a
i
ni
ng s
t
e
ps
4000
26000
A
ve
r
a
ge
pe
r
c
e
nt
a
ge
of
e
r
r
or
of
r
a
t
e
of
di
f
f
e
r
e
nc
e
be
t
w
e
e
n out
put
a
nd t
a
r
ge
t
ne
ur
on f
i
r
i
ng
r
a
t
e
0.96%
0.80%
T
he
l
ow
e
s
t
s
t
e
ps
t
a
k
e
n f
or
a
c
t
ua
l
out
put
ne
ur
on r
a
t
e
t
o m
e
e
t
w
i
t
h or
c
l
os
e
t
o t
he
t
a
r
ge
t
e
xc
i
t
a
t
or
y
popul
a
t
i
on r
a
t
e
84
3
T
he
hi
ghe
s
t
s
t
e
ps
t
a
ke
n f
or
a
c
t
ua
l
out
put
ne
ur
on r
a
t
e
t
o m
e
e
t
w
i
t
h or
c
l
os
e
t
o t
h
e
t
a
r
ge
t
e
xc
i
t
a
t
or
y
popul
a
t
i
on r
a
t
e
92
21
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2252
-
8938
I
nt
J
A
r
ti
f
I
nt
e
ll
,
V
ol
.
10
, N
o.
1
,
M
a
r
c
h
20
2
1
:
175
–
183
182
T
a
bl
e
8. C
om
pa
r
is
on of
S
pi
ke
N
e
ur
on O
pt
im
iz
a
ti
on M
e
th
od
M
e
t
hod
A
ve
r
a
ge
e
r
r
or
be
t
w
e
e
n a
c
t
ua
l
a
nd t
a
r
ge
t
out
put
M
a
xi
m
um
l
i
ke
hood opt
i
m
i
z
a
t
i
on m
e
t
hod
3.04%
D
e
e
p Q
N
e
t
w
or
k (
D
Q
N
)
0.96%
D
e
e
p Q
-
L
e
a
r
ni
ng w
i
t
h N
or
m
a
l
i
z
e
d
A
dva
nt
a
ge
F
unc
t
i
on (
N
A
F
)
0.80%
B
ot
h
pr
opos
e
d
a
lg
or
it
hm
s
a
c
hi
e
ve
d
lo
w
e
r
a
ve
r
a
ge
e
r
r
or
be
twe
e
n
a
c
tu
a
l
a
nd
ta
r
ge
t
out
put
c
om
pa
r
e
d
to
m
a
xi
m
um
li
ke
hood
opt
im
iz
a
ti
on
m
e
th
od.
T
he
e
nvi
r
onm
e
nt
us
e
d
in
D
Q
N
a
nd
N
A
F
a
r
e
di
f
f
e
r
e
nt
w
it
h
th
e
m
a
xi
m
um
li
ke
hood
opt
im
iz
a
t
io
n
m
e
th
od
a
s
th
e
c
us
to
m
e
nvi
r
o
nm
e
nt
of
D
Q
N
a
nd
N
A
F
i
s
c
ons
tr
uc
te
d
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k.
T
he
e
nvi
r
onm
e
nt
f
or
D
Q
N
a
nd
N
A
F
is
c
us
to
m
iz
e
d
in
or
de
r
to
e
ns
ur
e
bot
h
a
ge
nt
s
is
c
a
pa
bl
e
t
o e
xpl
or
e
i
n t
he
e
nvi
r
onm
e
nt
i
n or
de
r
t
o opti
m
iz
e
s
p
ik
e
ne
ur
on.
4.
C
O
N
C
L
U
S
I
O
N
D
e
e
p
r
e
in
f
or
m
e
nt
le
a
r
ni
ng
is
p
r
opos
e
d
a
s
a
m
e
th
od
to
ove
r
c
o
m
e
th
e
di
f
f
ic
ul
ty
in
S
N
N
tr
a
in
in
g
due
to
non
-
di
f
f
e
r
e
nt
ia
bl
e
of
s
pi
ke
f
unc
ti
on
of
s
pi
ke
ne
ur
on.
D
e
e
p
Q
n
e
twor
k
a
nd
D
e
e
p
Q
-
le
a
r
ni
ng
w
it
h
nor
m
a
li
z
e
d a
dva
nt
a
ge
f
unc
ti
ons
a
lg
or
it
hm
s
a
r
e
pr
opos
e
d t
o
ba
l
a
nc
e
t
he
f
ir
in
g r
a
te
of
e
xc
it
a
to
r
y a
nd i
nhi
bi
to
r
y
popula
ti
on
of
a
s
pi
ke
ne
ur
on.
A
s
pi
ke
ne
ur
on
is
tr
a
in
e
d
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
y
m
f
r
a
m
e
w
or
k.
B
ot
h
a
lg
or
it
hm
s
a
bl
e
to
in
te
r
a
c
t
w
it
h
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
F
r
a
m
e
w
or
k
t
o
a
tt
a
in
th
e
goa
l.
T
he
a
v
e
r
a
ge
pe
r
c
e
nt
a
g
e
e
r
r
or
of
r
a
te
of
di
f
f
e
r
e
nc
e
be
twe
e
n
t
a
r
ge
t
a
nd
a
c
tu
a
l
out
put
ne
ur
on
f
ir
in
g
r
a
te
f
or
N
A
F
a
nd
D
Q
N
a
lg
or
it
hm
s
obt
a
in
e
d
0.80
%
a
nd
0
.96%
r
e
s
pe
c
ti
ve
ly
.
I
n
te
r
m
s
of
s
te
ps
ta
ke
n
f
or
a
c
tu
a
l
out
put
ne
ur
on
r
a
te
to
m
e
e
t
w
it
h
th
e
ta
r
ge
t
ne
u
r
on
f
i
r
in
g
r
a
te
,
N
A
F
a
c
hi
e
ve
d
f
a
s
te
r
th
a
n
D
Q
N
to
m
e
e
t
th
e
t
a
r
ge
t
ne
ur
on f
i
r
in
g r
a
te
. T
he
r
e
s
ul
ts
pr
ove
d t
ha
t
th
e
a
lg
or
i
t
hm
s
a
bl
e
t
o e
xpl
or
e
i
n t
he
c
us
to
m
e
nvi
r
onm
e
nt
to
opt
im
iz
e
th
e
s
pi
ke
ne
ur
on.
I
n
f
ut
ur
e
w
or
k,
D
Q
N
a
nd
N
A
F
a
lg
or
it
hm
c
a
n
be
us
e
d
f
or
f
ur
th
e
r
de
ve
lo
pm
e
nt
to
tr
a
in
a
s
pi
ki
ng
ne
ur
a
l
ne
twor
k
(
S
N
N
)
s
in
c
e
bot
h
a
lg
or
it
hm
s
a
r
e
c
a
pa
bl
e
to
e
xpl
or
e
in
th
e
c
us
to
m
e
nvi
r
onm
e
nt
w
it
h
O
pe
nA
I
G
ym
f
r
a
m
e
w
or
k
by
us
in
g
D
R
L
to
opt
im
iz
e
a
s
pi
ke
n
e
ur
on.
T
he
d
e
ve
lo
pe
d
S
N
N
c
a
n
be
de
m
on
s
tr
a
te
d
in
va
r
io
us
ty
pe
s
of
a
ppl
i
c
a
ti
ons
s
uc
h
a
s
p
la
yi
ng
ga
m
e
,
c
la
s
s
if
ic
a
ti
on,
im
a
ge
r
e
c
ogni
ti
on
a
nd s
o on.
A
C
K
N
O
WL
E
D
G
E
M
E
N
T
S
T
he
a
ut
hor
s
w
oul
d
li
ke
to
th
a
nk
th
e
U
ni
v
e
r
s
it
i
S
a
in
s
M
a
la
y
s
ia
f
or
f
in
a
nc
ia
l
s
uppor
t
th
r
ough
th
e
R
e
s
e
a
r
c
h
U
ni
ve
r
s
it
y G
r
a
nt
S
c
he
m
e
P
E
L
E
C
T
/
1001/8014049.
R
E
F
E
R
E
N
C
E
S
[1]
V.
Francois
-
Lavet,
et
al.
,
“An
Introduction
to
Deep
Reinforceme
nt
Learning,”
Foundation
and
Trends®
in
Machine Lear
ning
, pp. 219
-
354, 2018.
[2]
K.
Arulkumaran,
et
al.
,
“Deep
Reinforcement
Learning:
A
Brief
Sur
vey,”
IEEE
Signal
Processing
Magazine
,
pp.
26
-
38, 2017. DOI: 10.1109/MSP.2017.2743240.
[3]
R.
Sutton,
and
A.
Barto,
“Reinforcement
Learning:
An
Introduction,
”
IEEE
Transactions
on
Neural
Networks
,
p.
1054, 1998.
[4]
M.
P
ieters
and
M.
A.
Wiering,
"
Q
-
learning
with
experie
nce
repla
y
in
a
dynamic
environmen
t,"
2016
IEE
E
Symposium
Series
on
Computational
Intelligence
(SSCI
)
,
Athens,
pp.
1
-
8,
2016.
DOI:
10.1109/SSCI.2016.7849368.
[5]
H.
Jang,
et
al
,
“An
Introduction
to
Probabilisti
c
Spiking
Neural
Net
works:
Probabilistic
Models,
Learning
Rules
,
and
Applications,”
in
IEEE
Signal
Processing
Magazine
,
vol.
36,
no.
6,
pp.
64
-
77,
Nov.
2019.
DOI
:
10.1109/MSP.2019.2935234.
[6]
A.
Russell
et
al.
,
“Optimization
methods
for
spiking
neurons
and
net
w
orks,”
IEEE
transactions
on
neural
networks
,
vol. 21, no. 12, pp. 1950
-
1962, 2010. DOI: 10.1109/TNN.2010.2083685.
[7]
Y.
Miao,
et
al.
,
“A
supervised
Multi
-
Spike
Learning
Algorithm
for
S
piking
Neural
Networks
,”
2018
International
Joint C
onference o
n Neura
l Net
works (I
JCNN)
, pp. 1
-
7, 2018. DOI: 10.1109/IJCNN.2018.8489175.
[8]
J.
Wade,
et
al.
,
“SWAT:
An
Unsupervised
SNN
Training
Algorith
m
for
Classification
Problems,”
2008
IEEE
Internati
onal
Joint
Conference
on
Neural
Networks
(IEEE
World
Congress
on
Computat
ional
In
telligence
)
,
pp.
2648
-
2655, 2008. DOI: 10.1109/IJCNN.2008.4634169.
[9]
E.
Stromatias
and
J.
Marsland,
“Supervised
learning
in
Spiking
Neural
Networks
with
Limited
Precision:
SNN/LP,”
2015
International
Joint
Conference
on
Neural
N
etworks
(IJCNN)
,
pp.
1
-
7,
2015
.
DOI:
10.1109/IJCNN.2015.7280732.
[10]
S.
Park,
et
al.
,
“Fast
and
Efficient
Information
Transmission
with
Burst
Spikes
in
Deep
Spiking
Neural
Networks,”
2019 56th ACM/IEEE Design Automation Conference (DAC)
, pp. 1
-
6, 2019.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
A
r
ti
f
I
nt
e
ll
I
S
S
N
:
2252
-
8938
Spi
k
e
ne
ur
on opti
m
iz
at
io
n us
in
g d
e
e
p r
e
in
fo
r
c
e
m
e
nt
l
e
ar
ni
ng
(
T
an Sz
i
H
ui
)
183
[11]
S.
Shrestha
and
G.
Orchard,
“SLAYER:
Spike
Layer
Error
Reassign
ment
in
Time,”
32nd
Conference
on
Neural
Informati
on Processi
ng Systems (N
eurIPS 2018)
, 2018.
[12]
Y. Oniz,
et al.
, “Spiking Neural
Networks for the Control
of a Servo S
ystem,”
2013 IEEE International Conferenc
e
on Mechatronics (ICM)
, pp. 94
-
98, 2013. DOI: 10.1109/ICMECH.2013.6518517.
[13]
Y.
Wu,
et
al.
,
“Spatio
-
Temporal
Backpropagation
for
T
raining
Hi
gh
Performance
Spiking
Neural
Networks,”
Frontiers in Neuroscience
, 2018. https://doi.org/10.3389/fnins.2018.00331.
[14]
A.
Abusnaina,
et
al.
,
“S
upervised
Training
of
Spiking
Neural
Net
-
w
ork
by
Adapting
the
E
-
MWO
Algorithm
for
Patter
n Classi
ficat
ion,”
Neural Pro
cess Le
tters
, pp. 661
-
682, 2019. https://doi.org/10.1007/s11063
-
018
-
9846
-
0.
[15]
R.
Abiyev,
et
al.
,
“Spiking
Neural
Networks
for
Identification
and
C
ontrol
of
Dynamic
Plants,”
2012
IEEE/ASME
Internati
onal
Conference
on
Advanced
Intelligen
t
Mechatronics
(AIM)
,
pp.
1030
-
1035,
2012.
DOI:
10.1109/AIM.2012.6265983.
[16]
B. Chandr
a and K
. V. N.
Babu, “A
new spik
ing neur
on model,”
The 201
3 Inte
rnationa
l Joint
Conferenc
e on Neu
ral
Networks (
IJCNN)
, pp. 1
-
5, 2013. DOI: 10.1109/IJCNN.2013.6706930.
[17]
J.
Eppler,
et
al.
,
“PyNEST:
a
convenient
interface
to
the
NEST
sim
ulator,”
Frontiers
in
neuroinformatics
,
2008.
DOI: 10.3389/neuro.
11.012.2008.
[18]
Y.
Wang
and
Z.
Zhang,
“E
xperience
Selection
in
Multi
-
agent
Deep
Reinforcement
Learning,”
2019
IEEE
31s
t
Internati
onal
Conference
on
Tools
with
Artifici
al
Intelligen
ce
(ICTAI)
,
pp.
864
-
870,
2019.
DOI:
10.1109/ICTAI.2019.00123.
[19]
C.
Qi,
et
al
,
“Deep
Reinforcement
Learning
With
Discre
te
Normalized
Advantage
Functions
for
Resource
Management in Ne
twork Slicing,” in
IEEE Communi
cations L
etters
, vol. 23, no. 8, pp. 1337
-
1341, Aug. 2019.
[20]
P.
Tuyen,
et
al.
,
“Deep
Reinforcement
Learning
Algorithms
for
S
teering
a
Underactuated
Ship,”
2017
IEE
E
Internati
onal
Conference
on
Multisens
or
Fusion
and
Integrati
on
for
Intelligen
t
Systems
(MFI
2017)
,
2017.
https://doi.org/10.1109/MFI.2017.8170388
.
[21]
O Ali, and MK Ishak.
“Bringing in
telligence to IoT Ed
ge: Machine Learning based S
mart City Image C
lassification
using
Microsoft
Azure
IoT
and
Custom
Vision,”
Journal
of
Physics:
Confer
ence
Series,
2020.
DOI:
10.1088/1742
-
6596/1529/4/042076.
B
I
O
G
R
A
P
H
I
E
S
O
F
A
U
T
H
O
R
S
Tan
Szi
Hui
received
B.
Sc.
degree
in
electronic
and
computer
en
gineering
from
Universiti
Teknikal Malaysia Melaka in 2019,
and the
M. Sc. degree
in embedde
d system engineering from
Universiti S
ains Malaysia in
2020.
Mohamad
Khairi
Ishak
received
the
B.Eng
degree
in
Electrical
an
d
Electronics
Engineering
from
the
International
Islamic
University
Malaysia
(IIUM),
Malaysi
a,
the
MSc.
in
Embedded
System,
from
the
Univer
sity
of
Essex,
United
Kingdom
and
PhD
fro
m
the
Univer
sity
of
Bristol
,
United
Kingdom.
He
is
a
member
of
IEEE
and
a
registered
graduate
engineer
with
the
Board
of
Engineers
Malaysia
(BEM).
Currently,
he
is
a
Senior
Lecturer
in
M
echatronics
Engineering
at
School
of
Electrical
and
Electronic
Engineering,
Universiti
Sains
Mal
a
ysia
(USM).
His
research
interests
are
Embedded
System,
Real
-
Time
Control
Communicatio
ns
and
Internet
of
Things
(IoT).
Emphasis
is
given
towards
the
development
of
theoretical
an
d
practical
methods
which
can
be
practically
validated.
Recently,
significant
research
effort
h
as
been
directed
towards
important industr
ial issues of
embedded
networke
d control sys
tems and Io
T.
Evaluation Warning : The document was created with Spire.PDF for Python.