I
AE
S In
t
er
na
t
io
na
l J
o
urna
l o
f
Art
if
icia
l In
t
ellig
ence
(
I
J
-
AI
)
Vo
l.
14
,
No
.
1
,
Feb
r
u
ar
y
2
0
2
5
,
p
p
.
500
~
515
I
SS
N:
2
2
5
2
-
8
9
3
8
,
DOI
: 1
0
.
1
1
5
9
1
/ijai.v
14
.i
1
.
p
p
5
0
0
-
5
1
5
500
J
o
ur
na
l ho
m
ep
a
g
e
:
h
ttp
:
//ij
a
i
.
ia
esco
r
e.
co
m
Enha
ncing
traf
fic
flow
t
hro
ug
h
mu
lti
-
a
g
ent
reinforc
ement
lea
rning
for
a
da
p
tive
traf
fic
lig
h
t
d
ura
tion
co
ntrol
Na
da
F
a
qir
1
,
J
a
o
ua
d
B
o
um
h
idi
1
,
Cha
k
ir
L
o
qm
a
n
1
,
Yo
un
ess
O
ub
e
na
lla
2
1
D
e
p
a
r
t
m
e
n
t
of
C
o
m
p
u
t
e
r
S
c
i
e
n
c
e
,
LI
S
A
C
L
a
b
o
r
a
t
o
r
y
,
F
a
c
u
l
t
y
of
of
S
c
i
e
n
c
e
s
D
h
a
r
El
M
e
h
r
a
z
,
S
i
d
i
M
o
h
a
mm
e
d
Ben
A
b
d
e
l
l
a
h
U
n
i
v
e
r
s
i
t
y
,
F
e
s,
M
o
r
o
c
c
o
2
D
e
p
a
r
t
me
n
t
of
C
o
m
p
u
t
e
r
S
c
i
e
n
c
e
,
F
a
c
u
l
t
y
of
S
c
i
e
n
c
e
s
a
n
d
T
e
c
h
n
o
l
o
g
i
e
s,
S
i
d
i
M
o
h
a
mm
e
d
B
e
n
A
b
d
e
l
l
a
h
U
n
i
v
e
r
s
i
t
y
,
F
e
s,
M
o
r
o
c
c
o
Art
icle
I
nfo
AB
S
T
RAC
T
A
r
ticle
his
to
r
y:
R
ec
eiv
ed
Mar
15
,
2
0
2
4
R
ev
is
ed
Oct
11
,
2
0
2
4
Acc
ep
ted
Oct
18
,
2
0
2
4
Th
is
st
u
d
y
a
d
d
re
ss
e
s
u
r
b
a
n
traffic
c
o
n
g
e
sti
o
n
th
r
o
u
g
h
d
e
e
p
lea
rn
in
g
fo
r
traffi
c
sig
n
a
l
c
o
n
t
ro
l
(TS
C)
.
In
c
o
n
tras
t
to
p
re
v
i
o
u
s
re
se
a
rc
h
on
sin
g
le
traffic
li
g
h
t
c
o
n
tro
ll
e
rs,
o
u
r
a
p
p
r
o
a
c
h
is
tail
o
r
e
d
to
t
h
e
TS
C
c
h
a
ll
e
n
g
e
with
in
a
n
e
two
rk
of
two
i
n
ters
e
c
ti
o
n
s.
Emp
l
o
y
in
g
c
o
n
v
o
l
u
ti
o
n
a
l
n
e
u
ra
l
n
e
two
r
k
s
(CNN
)
in
a
d
e
e
p
Q
-
n
e
two
rk
(DQ
N)
m
o
d
e
l,
our
m
e
th
o
d
a
d
o
p
ts
c
e
n
tralize
d
tra
in
in
g
a
n
d
d
istri
b
u
ted
e
x
e
c
u
ti
o
n
(C
TDE)
w
it
h
in
a
m
u
lt
i
-
a
g
e
n
t
re
in
f
o
rc
e
m
e
n
t
lea
rn
in
g
(M
ARL)
fra
m
e
wo
rk
.
Th
e
p
rima
ry
a
im
is
to
o
p
ti
m
ize
traffic
flo
w
in
a
two
-
in
ters
e
c
ti
o
n
se
tt
in
g
,
c
o
m
p
a
rin
g
o
u
tco
m
e
s
wit
h
b
a
se
li
n
e
stra
teg
ies
.
Ov
e
rc
o
m
in
g
sc
a
lab
il
it
y
a
n
d
p
a
rt
ial
o
b
se
rv
a
b
i
li
ty
c
h
a
ll
e
n
g
e
s,
o
u
r
a
p
p
ro
a
c
h
d
e
m
o
n
stra
tes
th
e
e
ffica
c
y
of
th
e
CTDE
-
b
a
se
d
M
ARL
fra
m
e
wo
rk
.
Ex
p
e
rime
n
ts
u
sin
g
u
r
b
a
n
m
o
b
i
li
ty
sim
u
latio
n
(
S
UMO)
e
x
h
i
b
i
t
a
6
8
%
p
e
rfo
rm
a
n
c
e
e
n
h
a
n
c
e
m
e
n
t
o
v
e
r
b
a
sic
traffic
li
g
h
t
c
o
n
tro
l
sy
ste
m
s,
v
a
li
d
a
ti
n
g
our
so
l
u
ti
o
n
a
c
ro
ss
d
iv
e
rse
sc
e
n
a
rio
s.
Wh
il
e
t
h
e
st
u
d
y
f
o
c
u
se
s
on
tw
o
in
ters
e
c
ti
o
n
s,
it
h
i
n
ts
at
b
ro
a
d
e
r
a
p
p
li
c
a
ti
o
n
s
in
c
o
m
p
lex
se
tt
in
g
s,
p
re
se
n
ti
n
g
a
p
ro
m
isin
g
a
v
e
n
u
e
f
o
r
m
it
ig
a
ti
n
g
u
rb
a
n
traffic
c
o
n
g
e
sti
o
n
.
T
h
e
re
se
a
rc
h
u
n
d
e
rsc
o
re
s
t
h
e
imp
o
r
tan
c
e
of
c
o
ll
a
b
o
ra
ti
o
n
wi
th
i
n
M
ARL
f
ra
m
e
wo
rk
s,
c
o
n
tri
b
u
ti
n
g
si
g
n
ifi
c
a
n
t
ly
to
t
h
e
a
d
v
a
n
c
e
m
e
n
t
of
a
d
a
p
ti
v
e
traffic
sig
n
a
l
c
o
n
tro
l
(ATS
C)
in
u
rb
a
n
e
n
v
iro
n
m
e
n
ts
fo
r
su
sta
in
a
b
le
tra
n
sp
o
r
tatio
n
so
l
u
ti
o
n
s.
K
ey
w
o
r
d
s
:
D
eep
Q
-
n
etwo
r
k
Deep
r
ein
f
o
r
ce
m
en
t
lear
n
in
g
Mu
lti
-
ag
en
t
Q
-
lear
n
in
g
T
r
af
f
ic
s
ig
n
al
co
n
tr
o
l
T
h
is
is
an
o
p
e
n
a
c
c
e
ss
a
rticle
u
n
d
e
r
th
e
CC
BY
-
SA
li
c
e
n
se
.
C
o
r
r
e
s
p
o
nd
ing
A
uth
o
r
:
Nad
a
Faq
ir
Dep
ar
tm
en
t
of
C
o
m
p
u
ter
Scie
n
ce
,
Facu
lty
of
Scien
ce
s
Dh
ar
El
Me
h
r
az
Sid
i
Mo
h
am
m
ed
B
en
Ab
d
ellah
Un
iv
er
s
ity
of
Fes
Fes
30000,
M
o
r
o
cc
o
E
m
ail:
n
ad
a.
f
aq
i
r
@
u
s
m
b
a.
ac
.
m
a
1.
I
NT
RO
D
UCT
I
O
N
T
r
af
f
ic
co
n
g
esti
o
n
is
a
p
er
v
asi
v
e
is
s
u
e
in
u
r
b
an
ar
ea
s
wo
r
ld
wid
e,
ca
u
s
in
g
s
ig
n
if
ican
t
ec
o
n
o
m
ic
lo
s
s
es
due
to
th
e
to
tal
tim
e
lo
s
t
by
co
m
m
u
ter
s
.
T
h
ese
ec
o
n
o
m
ic
lo
s
s
es
ar
e
e
s
tim
ated
to
r
ea
ch
b
illi
o
n
s
of
d
o
llar
s
an
n
u
ally
in
co
u
n
tr
ies
s
u
f
f
er
i
n
g
f
r
o
m
tr
a
f
f
ic
co
n
g
esti
o
n
.
As
a
r
esu
lt,
ad
d
r
ess
in
g
th
e
ec
o
n
o
m
ic
im
p
ac
t
of
allev
iatin
g
tr
af
f
ic
co
n
g
esti
o
n
h
as
b
ec
o
m
e
a
cr
u
cial
p
r
io
r
ity
.
Ma
n
ag
in
g
tr
a
f
f
ic
co
n
g
esti
o
n
h
as
em
er
g
ed
as
a
p
r
ess
in
g
an
d
u
n
s
o
lv
ed
p
r
o
b
le
m
in
n
u
m
er
o
u
s
u
r
b
an
ar
ea
s
ar
o
u
n
d
t
h
e
wo
r
ld
.
T
h
e
tr
ad
itio
n
al
s
tatic
tr
af
f
ic
li
g
h
t
co
n
tr
o
l
s
y
s
tem
(
T
L
C
S)
h
a
s
lo
n
g
b
ee
n
t
h
e
co
n
v
en
tio
n
al
s
o
lu
tio
n
f
o
r
m
an
ag
in
g
tr
a
f
f
ic
f
lo
w.
Ho
wev
er
,
with
th
e
in
cr
ea
s
in
g
co
m
p
l
ex
ity
an
d
s
ca
le
of
u
r
b
a
n
tr
an
s
p
o
r
tatio
n
n
etwo
r
k
s
,
tr
ad
itio
n
al
tr
af
f
ic
lig
h
t
c
o
n
tr
o
l
s
y
s
tem
is
no
lo
n
g
er
ad
eq
u
at
e
in
ef
f
ec
tiv
ely
m
itig
atin
g
tr
a
f
f
ic
co
n
g
esti
o
n
.
To
o
v
er
co
m
e
th
ese
lim
itatio
n
s
,
r
esear
ch
er
s
h
a
v
e
tu
r
n
ed
to
d
ee
p
r
ein
f
o
r
ce
m
e
n
t
lear
n
in
g
f
r
am
ewo
r
k
s
,
wh
ich
lev
er
ag
e
th
e
p
o
wer
of
r
ein
f
o
r
ce
m
en
t
lear
n
in
g
alg
o
r
ith
m
s
an
d
n
eu
r
al
n
etwo
r
k
s
s
u
ch
as
d
ee
p
n
eu
r
al
n
etwo
r
k
s
(
DNN)
an
d
c
o
n
v
o
lu
tio
n
al
n
eu
r
al
n
etwo
r
k
s
(
C
NN)
[
1
]
.
No
tab
l
e
ad
v
an
ce
m
en
ts
in
th
is
f
ield
i
n
clu
d
e
s
y
s
tem
s
lik
e
in
tellili
g
h
t
,
wh
ich
u
tili
ze
d
ee
p
r
ein
f
o
r
ce
m
e
n
t
lear
n
i
n
g
f
o
r
a
d
a
p
tiv
e
tr
af
f
ic
s
ig
n
al
co
n
t
r
o
l
(
AT
SC
)
,
d
em
o
n
s
tr
atin
g
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J Ar
tif
I
n
tell
I
SS
N:
2252
-
8
9
3
8
E
n
h
a
n
ci
n
g
tr
a
ffic flo
w
th
r
o
u
g
h
mu
lti
-
a
g
en
t rein
fo
r
ce
men
t le
a
r
n
in
g
fo
r
a
d
a
p
tive
tr
a
ffic lig
h
t
…
(
N
a
d
a
F
a
q
ir
)
501
s
ig
n
if
ican
t
im
p
r
o
v
em
e
n
ts
in
t
r
af
f
ic
ef
f
icien
cy
[
2
]
.
In
a
d
d
r
e
s
s
in
g
u
r
b
an
tr
af
f
ic
co
n
g
esti
o
n
,
o
u
r
f
o
cu
s
lies
in
o
p
tim
izin
g
tr
af
f
ic
f
lo
w
ef
f
icie
n
cy
with
in
a
two
-
in
ter
s
ec
tio
n
s
ce
n
ar
io
.
We
em
p
h
asize
th
e
u
s
e
of
C
NN
in
our
d
ee
p
Q
-
n
etwo
r
k
(
DQN)
m
o
d
e
l
an
d
em
p
lo
y
a
ce
n
tr
alize
d
tr
ai
n
in
g
an
d
d
is
tr
ib
u
ted
ex
ec
u
tio
n
(
C
T
DE
)
ap
p
r
o
ac
h
with
in
a
m
u
lti
-
ag
en
t
r
ein
f
o
r
ce
m
en
t le
ar
n
in
g
(
MA
R
L
)
f
r
am
e
wo
r
k
[
3
]
.
T
h
is
ar
ticle
p
r
o
p
o
s
es
a
n
o
v
el
m
u
lti
-
ag
en
t
p
er
s
p
ec
tiv
e
to
e
n
h
an
ce
t
h
e
p
e
r
f
o
r
m
an
ce
of
tr
af
f
ic
lig
h
t
co
n
tr
o
l
s
y
s
tem
s
,
r
ec
o
g
n
izin
g
t
h
e
im
p
o
r
ta
n
ce
of
a
g
en
t
co
llab
o
r
atio
n
in
s
u
p
p
o
r
tin
g
ef
f
icien
t
tr
af
f
ic
f
lo
ws
[
4
]
.
By
ad
o
p
tin
g
a
MA
R
L
ap
p
r
o
ac
h
,
o
u
r
g
o
al
is
to
d
e
v
elo
p
a
s
y
s
tem
wh
er
e
m
u
ltip
le
ag
e
n
ts
r
eg
u
lat
e
tr
af
f
ic
at
ad
jace
n
t
in
ter
s
ec
tio
n
s
,
co
llab
o
r
atin
g
to
o
p
tim
ize
tr
af
f
ic
f
lo
w.
We
ac
k
n
o
wled
g
e
th
e
p
o
ten
tial
f
o
r
b
r
o
ad
er
ap
p
licatio
n
s
in
more
co
m
p
le
x
s
ettin
g
s
,
an
d
o
u
r
wo
r
k
aim
s
to
ad
d
r
ess
s
ca
la
b
ilit
y
an
d
p
ar
tial
o
b
s
er
v
a
b
ilit
y
ch
allen
g
es
in
lar
g
e
-
s
ca
le
m
u
lti
-
ag
en
t
d
ee
p
r
ein
f
o
r
ce
m
en
t
lear
n
in
g
s
y
s
tem
s
[
5
]
.
To
v
alid
ate
th
e
ef
f
ec
tiv
en
ess
of
o
u
r
ap
p
r
o
ac
h
,
we
co
n
d
u
cte
d
s
ev
er
al
ex
p
er
im
en
t
s
u
s
in
g
tr
af
f
ic
s
im
u
latio
n
s
in
v
o
lv
in
g
two
in
ter
s
ec
tin
g
r
o
ad
s
.
T
h
ese
ex
p
er
im
en
ts
aim
ed
to
o
b
s
er
v
e
h
o
w
ag
en
ts
ac
h
iev
e
co
llab
o
r
atio
n
a
n
d
i
m
p
r
o
v
e
tr
af
f
ic
f
lo
w
th
r
o
u
g
h
in
d
ep
en
d
en
t
d
ee
p
Q
n
etwo
r
k
(
I
DQN)
tech
n
iq
u
es.
By
ad
d
r
ess
in
g
th
e
ec
o
n
o
m
ic
im
p
ac
t
of
tr
af
f
ic
co
n
g
esti
o
n
an
d
in
tr
o
d
u
cin
g
a
n
o
v
el
m
u
lti
-
ag
en
t
p
er
s
p
ec
tiv
e,
o
u
r
p
r
o
p
o
s
ed
ap
p
r
o
ac
h
o
f
f
er
s
a
p
r
o
m
is
in
g
s
o
lu
t
io
n
f
o
r
o
p
tim
izin
g
tr
a
f
f
ic
f
lo
w
in
co
m
p
lex
u
r
b
an
en
v
ir
o
n
m
en
ts
.
T
h
r
o
u
g
h
th
e
i
n
teg
r
atio
n
of
r
ea
l
-
tim
e
d
ata,
s
ca
lab
ilit
y
,
an
d
co
n
s
id
er
atio
n
s
f
o
r
m
ix
ed
tr
af
f
ic
s
ce
n
ar
io
s
,
we
aim
to
im
p
r
o
v
e
o
v
er
all
tr
af
f
ic
ef
f
icien
cy
an
d
cr
e
ate
s
m
ar
ter
,
more
s
u
s
tain
ab
le
c
ities
.
T
h
is
r
e
s
ea
r
ch
p
r
o
v
id
es
v
alu
a
b
le
in
s
ig
h
ts
in
to
th
e
ch
allen
g
es
an
d
p
o
te
n
tial
s
o
lu
tio
n
s
f
o
r
m
an
ag
in
g
tr
af
f
i
c
co
n
g
esti
o
n
u
s
in
g
d
ee
p
r
ein
f
o
r
ce
m
e
n
t
lear
n
in
g
te
ch
n
iq
u
es.
Nu
m
er
o
u
s
s
tu
d
ies
h
av
e
d
em
o
n
s
tr
ated
th
e
ef
f
ec
tiv
en
ess
of
r
ein
f
o
r
ce
m
e
n
t
lear
n
i
n
g
in
a
d
ap
tiv
e
tr
af
f
ic
lig
h
t
co
n
tr
o
l,
with
a
p
ar
ticu
la
r
f
o
cu
s
on
s
in
g
le
tr
af
f
ic
lig
h
t
co
n
tr
o
ller
s
[
6
]
,
[
7
]
.
Ho
wev
er
,
wh
en
ad
d
r
ess
in
g
m
u
lti
-
ag
en
t
s
y
s
tem
s
,
th
e
ch
allen
g
e
ar
is
es
in
d
eter
m
in
in
g
wh
eth
er
tr
af
f
ic
co
n
tr
o
l
s
h
o
u
ld
be
ap
p
r
o
ac
h
e
d
as
a
co
llab
o
r
ativ
e
or
co
m
p
etitiv
e
p
r
o
b
lem
[
8
]
.
In
s
u
c
h
en
v
ir
o
n
m
e
n
ts
,
th
e
co
n
f
lict
b
etwe
en
lo
ca
l
an
d
g
lo
b
al
o
p
tim
a,
co
u
p
led
with
t
h
e
in
ter
d
e
p
en
d
en
ce
of
a
g
en
t
ac
tio
n
s
,
a
d
d
s
c
o
m
p
lex
ity
to
th
e
d
y
n
am
ics
of
tr
an
s
p
o
r
t
n
etwo
r
k
s
.
Ad
d
itio
n
ally
,
f
ac
to
r
s
s
u
ch
as
d
r
iv
er
b
eh
a
v
io
r
an
d
co
-
ev
o
lu
tio
n
f
u
r
th
e
r
co
m
p
licate
th
e
ch
alle
n
g
es
of
tr
af
f
ic
lig
h
t
co
n
tr
o
l
in
u
r
b
a
n
en
v
ir
o
n
m
e
n
ts
.
Ou
r
wo
r
k
co
n
tr
ib
u
tes
to
th
is
lan
d
s
ca
p
e
by
d
elv
in
g
in
t
o
th
e
co
m
p
lex
ities
of
tr
af
f
ic
s
ig
n
al
co
n
tr
o
l
(
T
SC
)
in
n
etwo
r
k
s
with
m
u
ltip
le
in
ter
s
ec
tio
n
s
.
I
n
tr
o
d
u
cin
g
a
n
o
v
el
MA
R
L
f
r
am
ewo
r
k
,
our
a
p
p
r
o
ac
h
em
p
h
asizes
co
llab
o
r
atio
n
am
o
n
g
ag
en
ts
to
en
h
an
ce
tr
af
f
ic
f
lo
w.
It
ca
p
italizes
on
t
h
e
ad
v
an
tag
es
of
in
d
ep
en
d
en
t
t
r
af
f
ic
lig
h
t
co
n
tr
o
l,
e
n
s
u
r
in
g
s
tab
le
o
p
e
r
atio
n
s
ev
en
in
th
e
e
v
en
t
of
f
ailu
r
es
in
o
th
er
co
m
p
o
n
e
n
ts
.
To
en
ab
le
ef
f
icien
t
d
ec
is
io
n
-
m
ak
in
g
,
we
em
p
lo
y
C
NNs
to
m
o
d
el
our
ag
en
ts
an
d
in
co
r
p
o
r
ate
a
co
m
p
r
eh
e
n
s
iv
e
r
ep
r
esen
tatio
n
of
th
e
s
tate
of
ad
jace
n
t
in
ter
s
ec
tio
n
s
.
W
h
ile
ex
is
tin
g
s
tu
d
ies
[
6
]
,
[
7
]
h
a
v
e
p
r
ed
o
m
in
an
tl
y
f
o
cu
s
ed
on
s
in
g
le
tr
af
f
ic
lig
h
t
co
n
tr
o
ller
s
,
o
u
r
wo
r
k
ex
t
en
d
s
th
e
s
co
p
e
to
m
u
lti
-
in
ter
s
ec
tio
n
s
ce
n
ar
io
s
,
p
r
esen
tin
g
a
p
r
o
m
is
in
g
ap
p
r
o
ac
h
f
o
r
a
d
d
r
ess
in
g
tr
af
f
ic
co
n
g
esti
o
n
in
u
r
b
an
en
v
ir
o
n
m
e
n
ts
.
T
h
e
o
u
tlin
ed
co
m
p
a
r
is
o
n
s
aim
to
u
n
d
e
r
s
co
r
e
th
e
in
n
o
v
ativ
e
asp
ec
ts
of
o
u
r
p
r
o
p
o
s
ed
s
y
s
tem
with
in
th
e
b
r
o
ad
e
r
lan
d
s
ca
p
e
of
tr
af
f
ic
lig
h
t
co
n
tr
o
l
m
eth
o
d
o
lo
g
ies.
In
th
e
d
o
m
ain
of
d
ee
p
r
ein
f
o
r
c
em
en
t
lear
n
in
g
,
a
DNN
ty
p
ically
r
ep
r
esen
ts
th
e
Q
f
u
n
ctio
n
.
Valu
e
-
b
ased
m
eth
o
d
s
,
s
u
c
h
as
DQN
[
9
]
,
u
tili
ze
DNN
to
ex
tr
ac
t
s
tate
r
e
p
r
esen
tatio
n
s
d
ir
ec
tly
f
r
o
m
th
e
en
v
ir
o
n
m
e
n
t.
T
h
e
DQN
o
u
tp
u
ts
Q
v
alu
es
f
o
r
all
p
o
s
s
ib
le
ac
tio
n
s
,
en
a
b
lin
g
t
h
e
ap
p
r
o
x
im
atio
n
of
o
p
tim
al
s
tr
a
teg
ies.
To
e
n
h
an
ce
th
e
p
er
f
o
r
m
an
ce
,
s
tab
ilit
y
,
an
d
ef
f
ec
tiv
en
ess
of
th
e
DQN
ag
e
n
t,
two
co
m
m
o
n
ly
em
p
lo
y
e
d
m
ec
h
an
is
m
s
,
n
am
ely
th
e
tar
g
et
n
etwo
r
k
an
d
r
ep
lay
m
em
o
r
y
,
ar
e
in
teg
r
ated
d
u
r
in
g
tr
ain
in
g
.
Ou
r
ex
p
l
o
r
atio
n
alig
n
s
with
th
is
ap
p
r
o
ac
h
,
a
n
d
we
f
u
r
th
er
co
n
t
r
ib
u
te
by
ex
a
m
in
in
g
v
ar
io
u
s
DQN
-
b
ased
m
o
d
els,
in
clu
d
in
g
d
o
u
b
le
d
ee
p
Q
-
n
etwo
r
k
(
DDQN
)
[
1
0
]
,
d
u
elin
g
DQN
[
1
1
]
,
DDQN
with
p
r
o
p
o
r
tio
n
al
p
r
i
o
r
itizatio
n
b
ased
on
DDQN
[
1
2
]
,
an
d
d
y
n
a
m
ic
f
r
am
e
s
k
ip
d
ee
p
Q
-
n
etw
o
r
k
(
DFDQN)
[
1
3
]
.
W
h
ile
ar
tific
ial
n
eu
r
al
n
etwo
r
k
s
(
ANN)
h
av
e
b
ee
n
wid
ely
u
s
ed
as
m
o
d
el
ar
ch
itectu
r
es
[
1
4
]
,
[
1
5
]
,
th
e
r
e
is
a
g
r
o
win
g
u
tili
za
tio
n
of
C
NN
in
th
is
co
n
tex
t
[
1
6
]
–
[
1
9
]
.
By
in
t
eg
r
atin
g
C
NN
-
b
ased
m
o
d
els
a
n
d
e
x
am
in
in
g
th
eir
im
p
ac
t
on
th
e
lea
r
n
in
g
q
u
ali
ty
of
ag
e
n
ts
,
our
p
ap
er
co
n
t
r
ib
u
tes
to
ad
v
a
n
cin
g
th
e
u
n
d
er
s
tan
d
in
g
of
d
ee
p
r
ein
f
o
r
ce
m
e
n
t
lear
n
in
g
in
tr
a
f
f
ic
lig
h
t
c
o
n
tr
o
l.
T
h
e
f
in
d
in
g
s
s
h
ed
lig
h
t
on
th
e
s
tr
en
g
th
s
an
d
lim
itatio
n
s
of
d
if
f
er
en
t
n
eu
r
al
n
etwo
r
k
a
r
ch
it
ec
tu
r
es,
p
r
o
v
id
in
g
i
n
s
ig
h
ts
in
to
th
eir
ef
f
ec
tiv
e
n
ess
in
o
p
tim
i
zin
g
tr
af
f
ic
f
lo
w.
2.
T
H
E
CO
M
P
RE
H
E
NS
I
VE
T
H
E
O
RE
T
I
CA
L
B
ASI
S
AN
D
T
H
E
P
RO
P
O
SE
D
M
E
T
H
O
D
2
.
1
.
Sin
g
le
-
a
g
ent
re
info
rc
ement
lea
rning
Fig
u
r
e
1
illu
s
tr
ates
how
a
s
in
g
le
ag
en
t
in
r
ein
f
o
r
ce
m
en
t
lear
n
in
g
lear
n
s
by
in
ter
ac
tin
g
with
a
d
y
n
am
ic
en
v
ir
o
n
m
en
t
u
s
in
g
a
tr
ial
-
an
d
-
er
r
o
r
ap
p
r
o
ac
h
.
T
h
e
g
o
al
is
to
d
er
iv
e
th
e
b
est
p
o
s
s
ib
le
p
o
licy
by
m
ax
im
izin
g
th
e
ex
p
ec
ted
t
o
tal
of
cu
m
u
lativ
e
r
ewa
r
d
s
.
I
n
a
r
ei
n
f
o
r
ce
m
en
t
le
ar
n
in
g
p
r
o
b
lem
,
t
h
e
ag
e
n
t
iter
ativ
ely
o
b
s
er
v
es
its
en
v
ir
o
n
m
en
t,
an
d
th
e
n
it
tak
es
an
ac
tio
n
.
C
o
n
s
eq
u
en
tly
,
h
e
r
ec
eiv
es
f
r
o
m
th
e
en
v
ir
o
n
m
en
t
a
r
ewa
r
d
f
o
llo
win
g
h
is
ch
o
s
en
ac
tio
n
;
h
is
o
b
jectiv
e
is
to
ac
cu
m
u
late
th
e
r
ewa
r
d
s
in
th
e
lo
n
g
te
r
m
.
T
h
en
ac
co
r
d
in
g
to
its
r
u
les
an
d
th
e
s
tate
tr
an
s
itio
n
p
r
o
b
ab
ilit
y
,
th
e
en
v
i
r
o
n
m
e
n
t
m
o
v
es
to
th
e
n
ex
t
s
tate.
T
h
e
f
o
u
n
d
ati
o
n
o
f
r
ein
f
o
r
ce
m
en
t
lear
n
in
g
is
th
e
Ma
r
k
o
v
d
ec
is
io
n
p
r
o
ce
s
s
,
a
s
to
ch
asti
c
m
o
d
el
d
escr
ib
ed
b
y
th
e
t
u
p
le
(
S,
A,
R
,
P).
W
h
er
e
S
is
th
e
s
tates
s
et
an
d
∈
th
e
s
tate
o
f
t
h
e
en
v
ir
o
n
m
en
t
at
th
e
tim
e
-
s
tep
t
,
A
th
e
ac
tio
n
s
s
et
∈
th
e
ch
o
s
en
ac
tio
n
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8
9
3
8
I
n
t J Ar
tif
I
n
tell
,
Vo
l.
14
,
No
.
1
,
Feb
r
u
ar
y
2
0
2
5
:
500
-
5
1
5
502
at
th
e
tim
e
-
s
tep
t
b
y
th
e
a
g
en
t,
∈
th
e
im
m
ed
iate
r
ewa
r
d
v
alu
e
r
ec
eiv
e
d
b
y
th
e
a
g
en
t
in
th
e
s
tate
p
er
f
o
r
m
in
g
ac
tio
n
an
d
f
in
ally
P th
e
s
tate
tr
an
s
itio
n
p
r
o
b
a
b
ilit
y
.
Fig
u
r
e
1.
Sin
g
le
a
g
en
t
r
ei
n
f
o
r
c
em
en
t
lear
n
in
g
f
o
r
T
SC
In
tr
af
f
ic
s
ig
n
al
p
r
o
b
lem
s
,
th
e
ag
en
t
in
ter
ac
ts
with
th
e
i
n
ter
s
ec
tio
n
at
d
is
cr
ete
tim
e
s
tep
s
t
=
0,
1,
2,
f
ir
s
t
o
b
s
er
v
in
g
th
e
s
tate
of
th
e
in
ter
s
ec
tio
n
St
at
th
e
s
tar
t
of
th
e
tim
e
s
tep
t,
th
e
n
s
elec
ts
an
a
ctio
n
At
d
ep
e
n
d
in
g
on
a
p
o
licy
π.
Af
ter
th
e
v
eh
icl
e
m
o
v
es
u
n
d
er
th
e
ac
tiv
e
tr
af
f
i
c
lig
h
t,
th
e
s
tate
of
th
e
i
n
ter
s
ec
tio
n
ch
an
g
es
to
th
e
n
ew
s
tate
of
St+1
.
At
th
e
en
d
of
th
e
tim
e
s
tep
t,
th
e
ag
en
t
r
ec
eiv
es
a
r
ewa
r
d
Rt
f
o
llo
win
g
its
d
ec
is
io
n
to
ch
o
o
s
e
a
s
ig
n
al.
Ass
u
m
in
g
th
at
th
e
r
ewa
r
d
at
each
tim
e
s
tep
t
m
u
s
t
be
m
u
ltip
lied
by
t
h
e
d
is
co
u
n
t
f
ac
to
r
γ
ϵ
[
0
,
1]
to
d
eter
m
in
e
th
e
im
p
o
r
tan
ce
g
i
v
en
to
im
m
e
d
iate
an
d
f
u
t
u
r
e
r
e
war
d
s
,
th
e
cu
m
u
lativ
e
r
ewa
r
d
f
r
o
m
tim
e
s
tep
t
to
th
e
en
d
of
tim
e
T
is
ex
p
r
ess
ed
as
f
o
llo
ws:
=
∑
+
∞
=
0
.
T
h
e
v
alu
e
f
u
n
ctio
n
Q
π
(
s
,
a
)
r
e
p
r
esen
ts
th
e
ac
tio
n
A
t
tak
en
in
th
e
c
u
r
r
en
t
s
tate
S
t
,
f
o
llo
wed
by
th
e
ap
p
licatio
n
of
p
o
licy
π
u
n
til
th
e
ep
is
o
d
e
c
o
n
clu
d
es.
Du
r
in
g
t
h
is
p
r
o
ce
s
s
,
th
e
cu
m
u
lativ
e
r
ewa
r
d
s
ea
r
n
ed
by
th
e
a
g
en
ts
can
be
ex
p
r
ess
ed
as
(
1
)
.
(
,
)
=
[
|
=
,
=
]
(
1
)
T
he
ag
e
n
t'
s
o
b
jectiv
e
is
to
id
en
tify
th
e
o
p
tim
al
p
o
licy
f
o
r
s
elec
tin
g
ac
tio
n
s
,
r
e
p
r
esen
ted
as
π
∗
in
(
2
)
,
wh
ich
m
ax
im
izes
th
e
cu
m
u
lativ
e
f
u
t
u
r
e
r
ewa
r
d
or
Q
-
v
alu
es
an
d
e
n
ab
les
th
e
ag
en
t
to
ac
c
o
m
p
lis
h
its
o
b
jectiv
e.
∗
=
(
,
)
∈
,
∈
(
2
)
T
h
e
Q
-
v
alu
es
th
at
co
r
r
esp
o
n
d
to
th
e
o
p
tim
al
p
o
licy
∗
ar
e
d
e
n
o
ted
as
∗
(
,
)
=
∗
(
,
)
an
d
can
be
ap
p
r
o
x
im
ated
u
s
in
g
lin
ea
r
f
u
n
ctio
n
ap
p
r
o
x
im
ato
r
s
.
T
h
e
p
r
im
ar
y
o
b
jectiv
e
is
to
id
e
n
tify
th
e
o
p
tim
al
p
o
licy
∗
an
d
s
u
b
s
eq
u
e
n
tly
,
d
eter
m
i
n
e
th
e
o
p
tim
al
Q
-
v
alu
es
∗
(
,
)
.
T
h
e
r
ec
u
r
s
iv
e
r
elatio
n
f
o
r
t
h
e
o
p
tim
al
Q
-
v
alu
es
∗
(
,
)
is
p
r
o
v
id
ed
by
B
ellm
an
'
s
o
p
tim
ality
eq
u
atio
n
.
∗
(
,
)
=
[
+
∗
(
+
1
,
+
1
)
|
=
,
=
]
(
3
)
L
in
ea
r
f
u
n
ctio
n
a
p
p
r
o
x
im
ato
r
s
,
as
well
as
n
o
n
lin
ea
r
f
u
n
ctio
n
ap
p
r
o
x
im
ato
r
s
s
u
ch
as
DNN
,
c
an
be
em
p
lo
y
ed
to
esti
m
ate
ac
tio
n
-
v
alu
e
f
u
n
ctio
n
s
or
p
o
licy
.
2
.
2
.
M
ulti
-
a
g
ent
re
info
rc
e
ment
lea
rning
MA
R
L
is
b
u
ilt
ar
o
u
n
d
a
s
to
c
h
asti
c
g
am
e
th
at
r
elies
on
a
Ma
r
k
o
v
d
ec
is
io
n
p
r
o
ce
s
s
.
T
h
i
s
p
r
o
ce
s
s
is
d
ef
in
ed
by
th
e
tu
p
le
,
1
,
…
,
,
1
,
…
,
,
,
wh
er
e
n
r
ep
r
esen
ts
th
e
n
u
m
b
er
of
ag
en
ts
in
v
o
lv
ed
.
,
as
s
h
o
wn
in
Fig
u
r
e
2.
T
h
e
jo
in
t
ac
tio
n
s
p
ac
e
=
1
×
…
×
an
d
1
,
…
,
r
ep
r
esen
t
t
h
e
r
ewa
r
d
s
of
ea
c
h
ag
e
n
t
.
In
m
u
lti
-
ag
en
t
en
v
i
r
o
n
m
e
n
ts
,
s
tate
tr
an
s
itio
n
s
ar
is
e
f
r
o
m
th
e
co
llectiv
e
ac
tio
n
s
of
all
ag
en
ts
,
an
d
each
ag
e
n
t'
s
r
ewa
r
d
s
ar
e
in
f
lu
en
ce
d
by
th
e
jo
in
t
p
o
licy
d
e
n
o
ted
as
:
×
→
[
0
,
1
]
.
T
he
r
ewa
r
d
s
f
o
r
ea
c
h
ag
e
n
t
is
ca
lcu
lated
as
f
o
llo
ws:
=
[
+
1
|
=
,
,
=
,
]
.
C
o
n
s
eq
u
en
tly
,
t
h
e
B
ellm
an
eq
u
atio
n
can
be
ex
p
r
ess
ed
as
(
4
)
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J Ar
tif
I
n
tell
I
SS
N:
2252
-
8
9
3
8
E
n
h
a
n
ci
n
g
tr
a
ffic flo
w
th
r
o
u
g
h
mu
lti
-
a
g
en
t rein
fo
r
ce
men
t le
a
r
n
in
g
fo
r
a
d
a
p
tive
tr
a
ffic lig
h
t
…
(
N
a
d
a
F
a
q
ir
)
503
(
,
)
=
[
+
(
+
1
,
+
1
)
|
=
,
=
]
(
4
)
Fig
u
r
e
2.
Ag
e
n
ts
'
in
ter
ac
tio
n
with
th
e
en
v
ir
o
n
m
en
t
Sto
ch
asti
c
g
am
es
can
be
ca
teg
o
r
ized
i
n
to
th
r
ee
ty
p
es:
f
u
lly
c
o
o
p
er
ativ
e,
f
u
lly
c
o
m
p
etitiv
e,
an
d
m
i
x
ed
g
am
es.
In
f
u
lly
co
o
p
er
ativ
e
g
am
es,
all
ag
en
ts
s
h
ar
e
th
e
s
a
m
e
r
ewa
r
d
f
u
n
ctio
n
,
aim
in
g
to
m
ax
im
ize
th
e
t
o
tal
r
ewa
r
d
co
llectiv
ely
.
C
o
o
p
er
ati
o
n
am
o
n
g
ag
e
n
ts
is
ess
en
tial
to
r
ea
ch
th
is
o
b
jectiv
e.
Ap
p
r
o
a
ch
es
b
ased
on
th
e
Q
-
lear
n
in
g
alg
o
r
ith
m
h
av
e
b
ee
n
d
ev
elo
p
ed
to
tack
le
co
o
r
d
i
n
a
tio
n
ch
allen
g
es,
f
o
cu
s
in
g
on
a
s
in
g
le
o
p
tim
al
jo
in
t
ac
tio
n
[
2
0
]
.
L
au
e
r
'
s
d
is
tr
ib
u
ted
Q
-
lear
n
in
g
al
g
o
r
ith
m
[
2
1
]
a
d
d
r
ess
es
th
e
is
s
u
e
wh
il
e
m
ain
tain
in
g
lo
w
co
m
p
u
tatio
n
al
c
o
m
p
lex
ity
.
Ho
wev
er
,
th
is
alg
o
r
ith
m
o
n
ly
a
p
p
lies
to
d
eter
m
in
is
tic
s
ce
n
ar
io
s
with
non
-
n
eg
ativ
e
r
ewa
r
d
f
u
n
ctio
n
s
.
W
h
en
d
ea
lin
g
with
a
p
er
f
ec
tly
co
m
p
etitiv
e
g
am
e,
wh
er
e
th
er
e
ar
e
two
ag
e
n
ts
with
o
p
p
o
s
in
g
g
o
als
an
d
R1
=
-
R
2
,
m
o
s
t
of
th
e
liter
atu
r
e
on
r
ein
f
o
r
ce
m
en
t
lear
n
in
g
f
o
cu
s
es
on
th
e
two
-
ag
en
t
s
ce
n
ar
io
.
T
h
e
m
in
im
ax
p
r
in
cip
le
is
ap
p
lied
in
s
u
ch
a
s
itu
atio
n
,
wh
ich
aim
s
to
m
ax
im
ize
an
ag
en
t'
s
p
r
o
f
it
wh
ile
co
n
s
id
er
in
g
th
e
wo
r
s
t
-
ca
s
e
s
ce
n
ar
io
,
wh
e
r
e
th
e
o
p
p
o
n
e
n
t
will
alwa
y
s
tr
y
to
m
i
n
im
ize
it.
To
ac
h
iev
e
t
h
is
,
ad
v
er
s
ar
y
-
in
d
e
p
en
d
e
n
t
alg
o
r
ith
m
s
ar
e
u
s
ed
,
s
u
c
h
as
t
h
e
m
in
im
ax
-
Q
alg
o
r
ith
m
[
20
]
,
[
2
2
]
,
wh
ich
co
m
p
u
tes
s
tag
e
g
am
e
s
tr
ateg
ies
an
d
v
alu
es
u
s
in
g
th
e
m
in
im
ax
p
r
in
cip
le
an
d
p
r
o
p
a
g
ates
v
alu
es
th
r
o
u
g
h
s
tate
p
air
s
-
s
to
ck
u
s
in
g
a
tim
e
-
d
if
f
er
e
n
ce
r
u
le
s
im
ilar
to
Q
lear
n
in
g
.
Mix
ed
s
to
ch
asti
c
g
am
es
in
v
o
l
v
e
ag
en
ts
with
d
if
f
er
en
t
an
d
c
o
r
r
elate
d
r
ewa
r
d
s
,
wh
e
r
e
no
c
o
n
s
tr
ain
ts
ar
e
im
p
o
s
ed
on
th
e
a
g
en
ts
'
r
ewa
r
d
f
u
n
ctio
n
s
.
T
h
is
m
o
d
el
is
s
u
itab
le
f
o
r
s
elf
-
in
ter
ested
ag
en
ts
,
but
e
v
en
co
o
p
er
atin
g
ag
en
ts
m
a
y
en
co
u
n
ter
co
n
f
licts
of
in
ter
est,
s
u
c
h
as
wh
en
co
m
p
etin
g
f
o
r
r
eso
u
r
ce
s
.
Gam
e
th
eo
r
y
co
n
ce
p
ts
s
u
ch
as
e
q
u
ilib
r
iu
m
p
lay
a
s
ig
n
if
ica
n
t
r
o
le
in
alg
o
r
ith
m
s
f
o
r
m
ix
ed
s
to
ch
asti
c
g
am
es.
Sin
g
le
-
ag
en
t
r
ein
f
o
r
ce
m
e
n
t
lear
n
in
g
to
MA
R
L
m
eth
o
d
s
m
ak
e
no
ass
u
m
p
tio
n
s
ab
o
u
t
th
e
task
an
d
can
be
ap
p
lied
to
m
ix
ed
s
to
ch
asti
c
g
am
es,
alo
n
g
with
o
th
er
tech
n
i
q
u
es
s
u
ch
as
ag
e
n
t
-
in
d
ep
e
n
d
en
t,
a
g
en
t
-
tr
ac
k
in
g
,
an
d
a
g
en
t
-
awa
r
e
m
eth
o
d
s
.
Ho
wev
er
,
th
er
e
is
no
g
u
ar
an
tee
of
s
u
cc
ess
in
th
ese
s
ce
n
ar
io
s
.
Settin
g
lear
n
in
g
g
o
als
f
o
r
ag
en
ts
can
be
ch
allen
g
in
g
s
in
c
e
it
is
d
if
f
icu
lt
to
d
ef
in
e
good
g
en
er
a
l
o
b
jectiv
es.
T
h
e
two
p
r
im
a
r
y
asp
ec
ts
of
lear
n
in
g
g
o
als
ar
e
s
tab
ilit
y
an
d
ad
ap
ta
b
ilit
y
.
Stab
ilit
y
r
ef
er
s
to
an
ag
e
n
t'
s
ab
ilit
y
to
co
n
v
er
g
e
to
a
s
tab
le
p
o
licy
,
wh
ile
ad
ap
tab
ilit
y
en
s
u
r
es
th
at
an
ag
en
t'
s
p
er
f
o
r
m
an
ce
d
o
es
not
d
ec
lin
e
wh
en
o
th
er
ag
en
ts
ch
a
n
g
e
th
ei
r
s
tr
ateg
ies
[
2
3
]
,
[
2
4
]
.
3.
M
E
T
H
O
D
In
th
is
s
tu
d
y
,
we
in
tr
o
d
u
ce
an
in
n
o
v
ativ
e
ap
p
r
o
ac
h
to
ad
d
r
ess
th
e
p
er
s
is
ten
t
ch
allen
g
e
of
u
r
b
an
tr
af
f
ic
co
n
g
esti
o
n
u
s
in
g
ad
v
a
n
ce
d
ar
t
if
icial
in
tellig
en
ce
tech
n
iq
u
es.
T
h
e
p
r
im
ar
y
o
b
jectiv
e
is
to
o
p
tim
ize
tr
af
f
ic
f
lo
w
at
co
m
p
lex
in
ter
s
ec
tio
n
s
by
i
n
teg
r
atin
g
C
NN
in
to
a
DQN
m
o
d
el
with
in
a
MA
R
L
f
r
am
ewo
r
k
.
Ou
r
a
p
p
r
o
ac
h
lev
er
ag
es
a
C
T
DE
a
r
ch
itectu
r
e,
cr
u
cial
f
o
r
s
ca
lab
ilit
y
an
d
ef
f
ec
tiv
e
co
o
r
d
in
atio
n
am
o
n
g
m
u
ltip
le
ag
e
n
ts
co
n
tr
o
llin
g
t
r
af
f
ic
s
ig
n
als.
To
f
o
r
m
u
late
th
e
p
r
o
b
lem
,
th
e
p
r
im
ar
y
aim
is
to
en
h
an
ce
tr
af
f
ic
f
lo
w
o
p
tim
izatio
n
in
a
s
ce
n
ar
io
with
each
in
ter
s
ec
tio
n
'
s
s
ta
te
is
d
is
c
r
etize
d
in
to
ir
r
eg
u
lar
ly
s
h
ap
ed
ce
lls
u
s
in
g
d
is
cr
ete
tr
af
f
ic
s
tate
en
co
d
in
g
(
DT
SE)
.
T
h
ese
ce
lls
r
ep
r
esen
t
d
if
f
er
en
t
zo
n
es
m
o
n
ito
r
ed
by
th
e
ag
en
ts
,
ca
p
tu
r
in
g
v
e
h
icle
p
r
esen
ce
an
d
tr
af
f
ic
d
y
n
am
ics.
T
h
e
ag
en
t'
s
s
tate
r
ep
r
esen
tatio
n
,
d
en
o
ted
as
two
in
ter
s
ec
tio
n
s
,
an
d
we
b
e
n
ch
m
ar
k
o
u
r
r
esu
lts
ag
ain
s
t
b
aselin
e
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8
9
3
8
I
n
t J Ar
tif
I
n
tell
,
Vo
l.
14
,
No
.
1
,
Feb
r
u
ar
y
2
0
2
5
:
500
-
5
1
5
504
ap
p
r
o
ac
h
es.
Ou
r
in
n
o
v
ativ
e
ap
p
r
o
ac
h
le
v
er
ag
es
d
ata
s
u
c
h
as
v
eh
icle
p
o
s
itio
n
s
,
q
u
e
u
e
len
g
t
h
s
,
an
d
c
u
r
r
en
t
tr
af
f
ic
lig
h
t
s
tates.
T
h
is
d
etailed
s
itu
atio
n
al
awa
r
en
ess
en
ab
les
ag
en
ts
to
m
ak
e
in
f
o
r
m
e
d
d
ec
is
io
n
s
to
o
p
tim
ize
t
r
af
f
ic
f
lo
w.
T
h
e
ac
tio
n
s
p
ac
e
c
o
n
s
is
ts
of
f
o
u
r
d
is
tin
ct
ac
tio
n
s
p
er
a
g
en
t:
ad
ju
s
tin
g
tr
af
f
ic
li
g
h
ts
to
p
r
io
r
itize
n
o
r
th
/s
o
u
th
or
ea
s
t/west
tr
af
f
ic,
en
co
m
p
ass
in
g
b
o
th
s
tr
aig
h
t
a
n
d
tu
r
n
in
g
m
o
v
em
en
ts
.
Actio
n
s
ar
e
s
elec
ted
b
ased
on
th
e
ag
en
t'
s
o
b
s
er
v
atio
n
of
th
e
cu
r
r
en
t
s
tate
aim
ed
to
m
in
im
i
ze
cu
m
u
lativ
e
waitin
g
tim
es.
T
h
e
r
ewa
r
d
f
u
n
ctio
n
ac
r
o
s
s
all
v
eh
icles
at
th
e
in
te
r
s
ec
tio
n
is
d
ef
in
ed
as
th
e
r
ed
u
ctio
n
in
cu
m
u
lativ
e
waitin
g
tim
e
f
r
o
m
t
-
1
to
t,
em
p
h
asizin
g
th
e
a
g
en
t'
s
r
o
le
in
im
p
r
o
v
in
g
t
r
af
f
ic
ef
f
icien
cy
o
v
er
tim
e.
T
h
e
C
T
DE
f
r
a
m
ewo
r
k
f
ac
ilit
ates
co
llab
o
r
ativ
e
ac
tio
n
s
elec
tio
n
am
o
n
g
ag
en
ts
by
s
h
a
r
in
g
th
eir
last
ch
o
s
en
ac
tio
n
s
,
en
h
an
cin
g
by
s
h
ar
in
g
th
eir
last
ch
o
s
e
n
ac
tio
n
s
,
en
h
a
n
cin
g
c
o
o
r
d
i
n
atio
n
an
d
tr
af
f
ic
s
y
n
ch
r
o
n
izatio
n
b
etwe
en
ad
j
ac
en
t
in
ter
s
ec
tio
n
s
.
T
h
is
s
h
ar
in
g
m
ec
h
a
n
is
m
o
p
tim
izes
tr
af
f
ic
f
lo
w
ac
r
o
s
s
in
ter
co
n
n
ec
ted
r
o
a
d
n
etwo
r
k
s
,
cr
u
cial
f
o
r
m
itig
atin
g
co
n
g
esti
o
n
h
o
ts
p
o
ts
an
d
im
p
r
o
v
in
g
o
v
e
r
all
u
r
b
a
n
m
o
b
ilit
y
.
To
ev
alu
ate
th
e
ef
f
ec
tiv
en
ess
of
our
ap
p
r
o
ac
h
,
we
s
im
u
la
te
r
ea
lis
tic
tr
af
f
ic
s
ce
n
ar
io
s
u
s
in
g
th
e
W
eib
u
ll
d
is
tr
ib
u
tio
n
:
h
ig
h
-
tr
a
f
f
ic
(4
,
0
0
0
ca
r
s
)
,
lo
w
-
tr
af
f
ic
(
6
0
0
ca
r
s
)
,
NS
-
tr
af
f
ic
(2
,
000
ca
r
s
,
90%
n
o
r
th
/s
o
u
th
)
,
an
d
EW
-
tr
af
f
ic
(2
,
000
ca
r
s
,
90%
e
ast/
west).
T
h
ese
s
ce
n
ar
io
s
s
im
u
late
d
iv
er
s
e
tr
af
f
ic
co
n
d
itio
n
s
to
r
o
b
u
s
tly
test
th
e
ad
ap
tab
ilit
y
an
d
p
er
f
o
r
m
a
n
ce
of
our
C
T
DE
-
b
ased
MA
R
L
f
r
am
ewo
r
k
u
n
d
er
v
ar
y
in
g
lo
ad
an
d
d
ir
ec
tio
n
al
tr
af
f
i
c
d
is
tr
ib
u
tio
n
s
.
T
h
r
o
u
g
h
r
ig
o
r
o
u
s
ex
p
er
im
en
tatio
n
a
n
d
s
im
u
lat
io
n
s
,
we
v
alid
ate
th
e
ef
f
icac
y
of
o
u
r
ap
p
r
o
ac
h
we
v
alid
ate
th
e
ef
f
ec
tiv
en
ess
an
d
ef
f
icien
cy
of
o
u
r
C
T
DE
-
b
ased
MA
R
L
f
r
am
ewo
r
k
in
en
h
an
cin
g
tr
af
f
ic
m
an
ag
em
en
t
s
tr
ateg
ies.
Per
f
o
r
m
an
ce
m
et
r
ics
s
u
ch
as
a
v
er
ag
e
waitin
g
tim
es,
in
ter
s
ec
tio
n
th
r
o
u
g
h
p
u
t,
an
d
o
v
er
all
tr
af
f
ic
f
lo
w
ef
f
icien
c
y
p
r
o
v
id
e
q
u
an
titativ
e
in
s
ig
h
ts
in
to
th
e
f
r
am
ewo
r
k
'
s
ca
p
ab
ilit
y
to
o
p
tim
ize
u
r
b
a
n
tr
af
f
ic
o
p
e
r
atio
n
s
ef
f
ec
tiv
ely
.
3
.
1
.
E
nv
iro
nm
ent
des
ig
n
T
h
e
test
en
v
ir
o
n
m
en
t
f
o
r
e
v
alu
atin
g
our
ap
p
r
o
ac
h
co
n
s
is
t
s
of
two
n
eig
h
b
o
r
in
g
in
t
er
s
ec
tio
n
s
,
as
illu
s
tr
ated
in
Fig
u
r
e
3.
We
i
n
ten
tio
n
ally
o
p
te
d
f
o
r
th
is
m
in
i
m
alis
t
s
im
u
latio
n
s
etu
p
to
en
s
u
r
e
a
s
tr
aig
h
tf
o
r
war
d
im
p
lem
en
tatio
n
p
r
o
ce
s
s
an
d
m
in
im
ize
u
n
n
ec
ess
ar
y
co
m
p
le
x
ities
.
T
h
e
s
im
u
latio
n
o
f
u
r
b
a
n
m
o
b
ilit
y
(
SUMO)
to
o
l
was
u
tili
ze
d
f
o
r
m
o
d
elin
g
an
d
s
im
u
latin
g
th
e
u
r
b
an
tr
af
f
ic
s
ce
n
ar
io
s
[
2
5
]
.
Fig
u
r
e
3.
E
n
v
ir
o
n
m
en
t
of
th
e
s
im
u
latio
n
Ou
r
en
v
ir
o
n
m
en
t
is
co
m
p
o
s
e
d
of
an
in
teg
r
ated
tr
af
f
ic
lig
h
t
s
y
s
tem
wh
er
e
each
4
-
ar
m
in
ter
s
ec
tio
n
(
s
h
o
wn
in
Fig
u
r
e
4)
co
n
s
is
ts
of
eig
h
t
lan
es
each
(
f
o
u
r
in
b
o
u
n
d
lan
es
an
d
f
o
u
r
o
u
tb
o
u
n
d
lan
e
s
)
.
E
ac
h
ar
m
is
7
5
0
m
eter
s
lo
n
g
f
r
o
m
th
e
v
eh
icle
ac
ce
s
s
p
o
in
t
to
th
e
in
ter
s
ec
tio
n
tr
af
f
ic
lig
h
ts
.
E
ac
h
v
eh
icle
c
h
o
o
s
es
its
in
co
m
i
n
g
lan
e
ac
co
r
d
in
g
to
its
d
esti
n
atio
n
.
T
h
e
p
o
s
s
ib
le
d
ir
ec
tio
n
s
f
o
r
each
v
eh
icle
ar
e
ex
p
lain
e
d
as f
o
llo
ws:
−
Go
s
tr
aig
h
t
u
s
in
g
th
e
two
ce
n
t
r
al
lan
es
an
d
th
e
one
on
th
e
f
a
r
r
ig
h
t.
−
T
u
r
n
r
i
g
h
t
o
n
ly
u
s
in
g
t
h
e
r
ig
h
tm
o
s
t
lan
e.
−
T
u
r
n
lef
t
o
n
ly
u
s
in
g
th
e
lef
tm
o
s
t
lan
e.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J Ar
tif
I
n
tell
I
SS
N:
2252
-
8
9
3
8
E
n
h
a
n
ci
n
g
tr
a
ffic flo
w
th
r
o
u
g
h
mu
lti
-
a
g
en
t rein
fo
r
ce
men
t le
a
r
n
in
g
fo
r
a
d
a
p
tive
tr
a
ffic lig
h
t
…
(
N
a
d
a
F
a
q
ir
)
505
Fig
u
r
e
4.
Deta
ils
of
th
e
p
o
s
s
ib
le
d
ir
ec
tio
n
s
of
ea
c
h
in
ter
s
ec
tio
n
T
h
e
tr
af
f
ic
lig
h
ts
r
eg
u
latin
g
each
in
ter
s
ec
tio
n
ar
is
e
f
r
o
m
th
e
f
o
llo
win
g
eq
u
atio
n
a
n
d
s
h
o
wn
in
Fig
u
r
e
5
.
As in
r
ea
l lif
e,
th
e
tr
af
f
ic
lig
h
ts
ar
e
eith
er
r
ed
,
y
ell
o
w,
o
r
g
r
ee
n
an
d
at
ea
ch
tim
e
s
tep
,
th
e
tr
af
f
ic
lig
h
t
is
eith
er
y
ello
w
o
r
g
r
ee
n
i
n
p
h
ase.
T
h
e
co
lo
r
p
h
ase
tr
an
s
itio
n
is
cy
clica
l
(
r
ed
-
g
r
ee
n
-
y
ello
w
-
r
ed
)
an
d
m
an
d
at
o
r
y
to
clea
r
t
h
e
ce
n
te
r
o
f
th
e
in
ter
s
ec
tio
n
an
d
to
av
o
i
d
p
r
o
b
a
b
le
a
cc
id
en
ts
.
T
h
e
d
u
r
atio
n
o
f
ea
ch
p
h
ase
is
f
i
x
ed
,
1
0
s
ec
o
n
d
s
f
o
r
th
e
g
r
ee
n
tim
e
an
d
4
s
ec
o
n
d
s
f
o
r
th
e
y
ello
w
tim
e.
T
h
e
r
ed
p
h
ase,
th
e
r
ef
o
r
e,
la
s
ts
th
e
tim
e
elap
s
ed
s
in
ce
th
e
p
r
ev
io
u
s
p
h
ase
ch
a
n
g
e.
=
{
,
,
,
,
,
,
,
}
(
5
)
Fig
u
r
e
5.
T
r
af
f
ic
lig
h
ts
f
o
r
ea
c
h
in
b
o
u
n
d
la
n
e
of
each
in
ter
s
e
ctio
n
in
th
e
s
im
u
lated
e
n
v
ir
o
n
m
en
t
3
.
2
.
Ag
ent
s
In
th
e
co
n
tex
t
of
MA
R
L
,
AT
S
C
in
v
o
lv
es
co
o
r
d
in
atin
g
th
e
ac
tio
n
s
of
m
u
ltip
le
in
ter
s
ec
tio
n
tr
af
f
ic
lig
h
t
co
n
tr
o
ller
s
to
ac
h
iev
e
n
etwo
r
k
-
wid
e
tr
af
f
ic
o
p
tim
izatio
n
.
E
ac
h
in
ter
s
ec
tio
n
tr
af
f
ic
lig
h
t
c
o
n
tr
o
ller
ac
ts
as
an
ag
en
t
with
in
th
e
MA
R
L
f
r
am
e
wo
r
k
,
p
er
ce
iv
in
g
tr
af
f
ic
co
n
d
itio
n
s
at
th
eir
r
esp
ec
tiv
e
in
ter
s
ec
tio
n
s
an
d
m
a
k
in
g
d
ec
is
io
n
s
on
p
h
ase
co
n
tr
o
l
s
ig
n
als
ac
co
r
d
in
g
ly
.
T
h
r
o
u
g
h
in
t
er
ac
tio
n
an
d
co
llab
o
r
atio
n
,
ag
en
t
co
n
tr
o
ller
s
ad
ap
t
th
eir
p
o
licies
to
ch
an
g
in
g
tr
af
f
ic
d
y
n
am
ics,
lev
er
ag
i
n
g
co
llec
tiv
e
n
etwo
r
k
in
tellig
en
ce
to
o
p
tim
ize
tr
af
f
ic
f
lo
w
ac
r
o
s
s
th
e
en
tire
s
y
s
tem
.
T
h
is
co
llab
o
r
ativ
e
ap
p
r
o
ac
h
allo
w
s
o
f
f
icer
s
to
lear
n
f
r
o
m
ea
c
h
o
th
er
'
s
ex
p
er
ien
ce
s
,
im
p
r
o
v
e
d
ec
is
io
n
-
m
ak
i
n
g
p
r
o
c
ess
es,
an
d
u
ltima
tely
im
p
r
o
v
e
t
h
e
ef
f
icien
cy
of
th
e
T
SC
s
y
s
tem
.
By
u
s
in
g
MA
R
L
in
AT
SC
,
th
e
n
etwo
r
k
-
wid
e
c
o
n
tr
o
l
lo
o
p
s
h
o
wn
in
Fig
u
r
e
6,
f
ac
ilit
ates
th
e
co
o
r
d
in
atio
n
a
n
d
s
y
n
ch
r
o
n
izatio
n
of
ag
en
t
ac
tio
n
s
,
lea
d
in
g
to
a
m
o
r
e
ef
f
icie
n
t
an
d
ad
a
p
tiv
e
tr
af
f
ic
m
an
ag
e
m
en
t
s
o
lu
tio
n
.
T
h
is
ap
p
r
o
ac
h
n
o
t
o
n
ly
co
n
s
id
er
s
lo
ca
l
o
p
tim
izatio
n
at
each
in
ter
s
ec
tio
n
,
but
also
co
n
s
id
er
s
th
e
o
v
er
all
im
p
ac
t
of
s
ig
n
al
co
n
tr
o
l
d
ec
is
io
n
s
,
r
esu
ltin
g
in
im
p
r
o
v
ed
tr
af
f
ic
f
lo
w,
r
ed
u
ce
d
c
o
n
g
e
s
tio
n
,
an
d
im
p
r
o
v
ed
tr
af
f
ic
o
v
er
all
p
er
f
o
r
m
an
ce
of
th
e
tr
an
s
p
o
r
t
n
etwo
r
k
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8
9
3
8
I
n
t J Ar
tif
I
n
tell
,
Vo
l.
14
,
No
.
1
,
Feb
r
u
ar
y
2
0
2
5
:
500
-
5
1
5
506
Fig
u
r
e
6.
Netwo
r
k
-
wid
e
MA
R
L
co
n
tr
o
l l
o
o
p
:
P
o
licy
o
p
tim
iz
atio
n
in
AT
SC
as a
n
in
ter
ac
tiv
e
MD
P
p
r
o
ce
s
s
with
2
ag
en
ts
Usi
n
g
d
ee
p
Q
-
L
ea
r
n
i
n
g
,
each
ag
en
t
in
o
u
r
p
r
o
p
o
s
ed
m
et
h
o
d
r
etr
iev
es
th
e
d
is
cr
etize
d
r
ep
r
esen
tatio
n
of
th
e
en
v
ir
o
n
m
en
t
s
tate,
d
en
o
ted
as
St
=
(
,
,
∆
,
)
(
d
etailed
in
th
e
n
e
x
t
s
ec
tio
n
)
.
As
our
ap
p
r
o
ac
h
is
b
ased
on
th
e
C
T
DE
d
esig
n
of
a
m
u
lti
-
ag
en
t
e
n
v
ir
o
n
m
en
t,
k
n
o
wled
g
e
s
h
ar
in
g
b
etwe
en
th
e
ag
en
ts
of
th
e
two
in
ter
s
ec
tio
n
s
is
cr
u
cial
in
a
p
a
r
tially
o
b
s
er
v
a
b
le
en
v
ir
o
n
m
en
t.
To
e
n
ab
le
t
h
is
k
n
o
wled
g
e
s
h
ar
in
g
,
at
each
tim
e
s
tep
t,
each
ag
en
t
s
h
ar
es
th
e
last
ex
ec
u
ted
ac
tio
n
at
tim
e
s
tep
t
-
1
with
th
e
o
th
er
a
g
en
t.
T
h
is
in
f
o
r
m
atio
n
is
p
r
o
v
id
e
d
as
in
p
u
ts
to
th
e
C
NN
n
etwo
r
k
,
wh
ich
esti
m
ates
th
e
Q
-
v
alu
es
(
,
,
)
f
o
r
all
p
o
s
s
ib
le
ac
tio
n
s
in
th
e
ac
tio
n
s
p
ac
e,
as
s
h
o
wn
in
Fig
u
r
e
7.
Fu
r
th
er
m
o
r
e
,
u
s
in
g
th
e
r
e
p
la
y
m
em
eo
r
y
m
ec
h
an
is
m
,
ea
c
h
ag
en
t
r
ec
o
r
d
s
th
e
o
b
s
er
v
ed
in
ter
ac
tio
n
ex
p
er
ien
ce
=
(
,
,
,
+
1
)
in
to
a
r
ep
lay
m
em
o
r
y
M
=
{
1
,
2
,
·
·
·
,
}
.
To
a
d
d
r
ess
th
e
is
s
u
e
of
o
s
cillatio
n
d
u
r
in
g
t
h
e
tr
ain
i
n
g
p
h
ase,
we
em
p
lo
y
th
e
tar
g
et
n
etwo
r
k
m
ec
h
a
n
is
m
,
wh
ic
h
u
tili
ze
s
th
e
s
am
e
ar
ch
itectu
r
e
as
th
e
C
NN
n
etw
o
r
k
d
ep
icte
d
in
Fig
u
r
e
7.
In
ad
d
itio
n
to
u
p
d
ati
n
g
th
e
C
NN
p
ar
am
eter
s
,
th
e
ag
en
t
also
u
p
d
ates
th
e
tar
g
et
n
etwo
r
k
p
ar
am
eter
s
u
s
in
g
a
m
eth
o
d
ca
lled
s
o
f
t
u
p
d
atin
g
,
as
ex
p
lai
n
ed
in
s
ec
tio
n
3
.
6
.
T
h
e
ag
en
t
s
y
s
tem
s
tr
u
ctu
r
e
of
our
p
r
o
p
o
s
ed
m
eth
o
d
an
d
h
is
tr
ain
in
g
p
r
o
ce
s
s
ar
e
d
ep
icted
in
Fig
u
r
e
7,
illu
s
tr
atin
g
th
e
in
teg
r
atio
n
of
th
ese
co
m
p
o
n
en
ts
an
d
th
e
o
v
er
all
f
l
o
w
of
i
n
f
o
r
m
atio
n
an
d
u
p
d
ates
with
in
th
e
s
y
s
tem
.
Fig
u
r
e
7.
Ag
e
n
t
s
y
s
tem
an
d
tr
ain
in
g
p
r
o
ce
s
s
3
.
3
.
St
a
t
e
re
presenta
t
io
n
To
ap
p
ly
r
ei
n
f
o
r
ce
m
en
t
lear
n
i
n
g
,
we
n
ee
d
to
f
o
r
m
u
late
th
e
p
r
o
b
lem
as
a
Ma
r
k
o
v
p
r
o
ce
s
s
,
so
we
f
ir
s
t
n
ee
d
to
d
ef
in
e
th
e
s
tate
r
ep
r
esen
tatio
n
u
s
ed
by
t
h
e
ag
e
n
t.
T
h
e
s
tate
is
th
e
ag
en
t'
s
p
er
ce
p
tio
n
of
th
e
en
v
ir
o
n
m
en
t
at
tim
e
s
tep
t
an
d
is
co
m
m
o
n
ly
d
en
o
ted
by
St.
C
h
o
o
s
in
g
th
e
r
ig
h
t
s
itu
atio
n
al
r
ep
r
esen
tatio
n
is
im
p
o
r
tan
t
f
o
r
th
e
1
[
,
,
+
1
,
+
1
,
]
=
+
(
,
)
=
,
,
,
(
,
)
2
=
(
,
,
∆
,
)
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J Ar
tif
I
n
tell
I
SS
N:
2252
-
8
9
3
8
E
n
h
a
n
ci
n
g
tr
a
ffic flo
w
th
r
o
u
g
h
mu
lti
-
a
g
en
t rein
fo
r
ce
men
t le
a
r
n
in
g
fo
r
a
d
a
p
tive
tr
a
ffic lig
h
t
…
(
N
a
d
a
F
a
q
ir
)
507
ag
en
t
to
lear
n
how
to
o
p
tim
ize
tr
af
f
ic
ef
f
ec
tiv
ely
.
T
h
is
ch
o
ic
e
is
al
s
o
m
o
tiv
ated
by
th
e
q
u
a
n
tity
an
d
q
u
ality
of
in
f
o
r
m
atio
n
p
r
o
v
id
ed
to
th
e
a
g
en
t.
Vid
ali
et
a
l.
[
1
9
]
u
s
ed
a
d
is
cr
etize
d
r
ep
r
esen
tatio
n
of
ea
c
h
a
r
m
of
a
j
u
n
ctio
n
wh
er
e
each
ce
ll
is
ir
r
eg
u
lar
b
ased
on
f
o
r
m
u
la
DT
SE
[
1
4
]
,
ex
ce
p
t
th
at
less
in
f
o
r
m
atio
n
is
en
co
d
ed
.
No
t
all
ce
lls
ar
e
th
e
s
am
e
s
ize,
as
s
h
o
w
n
in
Fig
u
r
e
8.
At
each
i
n
ter
s
ec
tio
n
,
th
er
e
a
r
e
20
ce
lls
p
er
a
r
m
,
with
10
ce
lls
allo
ca
ted
to
th
e
lef
t
lan
e
an
d
th
e
r
em
ain
in
g
10
ce
lls
co
v
er
in
g
th
e
o
th
er
th
r
ee
la
n
es,
r
esu
ltin
g
in
80
ce
lls
p
e
r
in
ter
s
ec
tio
n
.
T
h
ese
ce
lls
ar
e
not
of
th
e
s
am
e
s
ize
an
d
th
eir
len
g
t
h
is
d
eter
m
in
e
d
by
th
eir
d
is
tan
ce
f
r
o
m
th
e
s
to
p
lin
e.
It
is
i
m
p
o
r
tan
t
to
s
tr
ik
e
a
b
alan
ce
b
etwe
en
ce
ll
le
n
g
th
an
d
co
m
p
u
tatio
n
co
m
p
lex
ity
.
If
t
h
e
ce
ll
is
to
o
lo
n
g
,
ca
r
s
m
ay
n
o
t
be
d
etec
ted
,
wh
ile
if
it
is
to
o
s
h
o
r
t,
th
e
co
m
p
u
tatio
n
r
e
q
u
ir
ed
co
v
e
r
in
g
th
e
p
at
h
len
g
th
in
c
r
ea
s
es.
T
h
e
p
a
p
er
p
r
o
p
o
s
es
a
ce
ll
len
g
th
th
at
is
2
m
eter
s
lo
n
g
er
th
an
th
e
ca
r
'
s
len
g
th
f
o
r
th
e
s
h
o
r
test
ce
ll
clo
s
est
to
th
e
s
to
p
lin
e.
W
h
en
ev
er
th
e
ag
en
t
o
b
s
er
v
es
th
e
en
v
ir
o
n
m
e
n
t,
it
r
ec
eiv
es
a
s
et
of
ce
lls
th
at
in
d
ic
ate
th
e
p
r
esen
ce
of
v
e
h
icles
in
each
ap
p
r
o
ac
h
r
o
a
d
zo
n
e.
Fig
u
r
e
8.
State
r
ep
r
esen
tatio
n
d
esig
n
of
an
in
ter
s
ec
tio
n
ar
m
with
ce
ll
len
g
th
s
In
our
a
p
p
r
o
ac
h
an
d
f
o
r
each
in
ter
s
ec
tio
n
we
r
etr
iev
e
th
e
s
tate
f
r
o
m
s
u
m
o
,
in
t
h
e
f
o
r
m
of
f
o
u
r
m
etr
ics
as
f
o
llo
ws
:
−
T
h
e
p
h
ase
of
th
e
tr
af
f
ic
s
ig
n
al
at
tim
e
s
tep
t;
−
T
h
e
co
n
g
esti
o
n
lev
el
of
each
c
ell
c,
at
tim
e
s
tep
t;
−
T
h
e
v
ar
iatio
n
in
co
n
g
esti
o
n
le
v
el
∆
f
or
ea
c
h
ce
ll
c
d
u
r
in
g
a
s
in
g
le
s
tep
;
−
T
h
e
m
ea
n
s
p
ee
d
of
th
e
v
eh
icle
s
of
each
ce
ll
c,
at
tim
e
s
tep
t.
W
h
er
e
co
n
g
esti
o
n
lev
el
m
u
s
t
be
in
th
e
r
an
g
e
of
[
0
,
1
]
;
A
v
alu
e
of
1
is
as
s
ig
n
ed
wh
en
a
v
eh
icle
is
p
r
esen
t
,
up
to
th
e
m
ax
im
u
m
o
cc
u
p
an
c
y
of
th
e
ce
ll.
,
is
in
ce
ll
c
of
ea
c
h
in
co
m
in
g
lan
e
of
th
e
in
ter
s
ec
tio
n
an
d
0
wh
en
no
v
eh
icle
is
in
th
is
ce
ll.
We
d
ef
in
e
th
e
co
n
g
esti
o
n
lev
el
as
(
6
)
.
=
ℎ
(
ℎ
+
)
(
6
)
W
h
er
e
ℎ
th
e
n
u
m
b
e
r
of
v
eh
icle
s
in
ce
ll
c,
th
e
len
g
th
of
t
h
e
c
ell
c
,
ℎ
th
e
s
ize
of
th
e
v
eh
icle
v
eh
an
d
.
T
h
is
r
ef
e
r
s
to
t
h
e
m
in
im
u
m
d
is
tan
ce
m
ain
tain
e
d
b
etwe
en
v
eh
icles.
(
in
SUMO
ℎ
=
5
m
eter
s
an
d
=
2
.
5
).
To
id
en
tify
c
o
n
g
ested
lan
es,
th
e
co
n
g
esti
o
n
lev
el
is
em
p
lo
y
e
d
,
an
d
if
a
lan
e
h
as
h
ig
h
co
n
g
esti
o
n
lev
els,
it
s
h
o
u
ld
s
h
if
t
to
th
e
tr
a
f
f
ic
s
ig
n
al
p
h
ase
th
at
allo
ws
tr
af
f
ic
to
p
r
o
ce
e
d
an
d
r
ed
u
ce
s
co
n
g
e
s
tio
n
.
T
h
e
s
tatu
s
of
n
eig
h
b
o
r
in
g
tr
af
f
ic
s
ig
n
als
is
d
eter
m
in
ed
u
s
in
g
th
e
ch
a
n
g
e
in
co
n
g
esti
o
n
d
en
o
ted
as
∆
.
If
th
e
co
n
g
esti
o
n
lev
els
at
th
e
lan
es
of
ad
jace
n
t
in
ter
s
ec
tio
n
s
in
cr
ea
s
e,
th
e
s
ig
n
alin
g
p
h
ase
th
at
p
er
m
its
tr
af
f
ic
to
en
ter
th
e
in
ter
s
ec
tio
n
.
m
u
s
t
be
ad
o
p
ted
.
E
ac
h
tr
af
f
ic
s
ig
n
al
can
p
o
ten
ti
ally
ac
h
iev
e
c
o
o
p
e
r
ativ
e
co
n
tr
o
l
with
n
eig
h
b
o
r
i
n
g
s
ig
n
als
by
co
n
s
id
er
in
g
th
e
ch
a
n
g
e
in
co
n
g
esti
o
n
lev
el
as
a
s
tate.
∆
at
tim
e
s
tep
t
can
be
ca
lcu
lated
by
tak
in
g
th
e
d
if
f
er
e
n
ce
b
etwe
en
an
d
−
1
th
en
∆
can
be
in
t
h
e
r
an
g
e
[
-
1
,
1
]
.
We
n
o
r
m
alize
∆
as
(
7
)
.
∆
=
1
+
−
−
1
2
(
7
)
T
he
s
p
ee
d
i
s
d
eter
m
in
ed
by
n
o
r
m
alizin
g
th
e
s
p
ee
d
of
each
v
eh
icle
ag
ain
s
t
th
e
m
ax
im
u
m
s
p
ee
d
allo
wed
f
o
r
th
e
ed
g
e.
In
s
u
m
m
ar
y
,
th
e
tr
a
f
f
ic
s
ig
n
al
ag
en
t
in
th
is
en
v
ir
o
n
m
en
t
estab
lis
h
es
th
e
s
tate
at
a
s
p
ec
if
ic
d
is
cr
ete
tim
e
s
tep
t
as
St=(
,
,
∆
,
).
Ou
r
ap
p
r
o
ac
h
is
b
ased
on
th
e
C
T
DE
d
esig
n
of
a
m
u
lti
-
a
g
en
t
en
v
ir
o
n
m
en
t
wh
er
e
th
e
s
h
ar
in
g
of
k
n
o
wled
g
e
b
etwe
en
th
e
ag
en
ts
of
th
e
two
in
ter
s
ec
tio
n
s
in
a
p
ar
tially
o
b
s
er
v
ab
le
en
v
i
r
o
n
m
e
n
t
im
p
lies
th
at
each
ag
en
t
at
tim
e
s
tep
t
s
h
ar
es
with
th
e
o
th
e
r
th
e
last
ac
tio
n
ex
ec
u
ted
at
tim
e
s
tep
t
-
1.
T
h
e
d
etai
l
of
th
e
s
tate
of
th
e
en
v
ir
o
n
m
en
t
th
at
we
h
a
v
e
ad
o
p
ted
also
tak
es
th
e
p
ar
tially
o
b
s
er
v
ab
le
n
atu
r
e
of
th
is
en
v
ir
o
n
m
e
n
t
.
T
h
is
is
ju
s
tifie
d
by
th
e
c
h
an
g
e
c
o
n
g
es
tio
n
lev
el
m
etr
ic
∆
ad
d
ed
to
th
e
r
ep
r
esen
tatio
n
of
th
e
s
tate.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
2
5
2
-
8
9
3
8
I
n
t J Ar
tif
I
n
tell
,
Vo
l.
14
,
No
.
1
,
Feb
r
u
ar
y
2
0
2
5
:
500
-
5
1
5
508
3
.
4
.
Act
i
o
n
s
pa
ce
In
o
u
r
ca
s
e,
we
h
av
e
f
o
u
r
d
if
f
e
r
en
t
ac
tio
n
s
th
at
can
be
tak
en
by
each
a
g
en
t
as
s
h
o
wn
in
Fig
u
r
e
9
.
−
A1
:
A
ctiv
ate
g
r
ee
n
lig
h
ts
f
o
r
v
eh
icles
in
th
e
n
o
r
t
h
an
d
s
o
u
th
ar
m
s
g
o
in
g
s
tr
aig
h
t
or
t
u
r
n
in
g
r
ig
h
t.
−
A2
:
A
ctiv
ate
g
r
ee
n
lig
h
ts
ex
cl
u
s
iv
ely
f
o
r
v
eh
icles
co
m
in
g
f
r
o
m
th
e
n
o
r
th
an
d
s
o
u
th
wh
o
w
an
t
to
tu
r
n
lef
t.
−
A3
:
A
ctiv
ate
g
r
ee
n
lig
h
ts
f
o
r
v
eh
icles
th
at
ar
e
in
th
e
ea
s
t
an
d
west
ar
m
an
d
wis
h
to
co
n
tin
u
e
s
tr
aig
h
t
or
tu
r
n
r
ig
h
t.
−
A4
:
Activ
ate
g
r
ee
n
lig
h
ts
ex
cl
u
s
iv
ely
f
o
r
v
eh
icles
in
th
e
ea
s
t
an
d
west
ar
m
tu
r
n
in
g
le
f
t.
W
h
en
th
e
ac
tio
n
ch
o
s
en
at
tim
e
s
tep
t
d
if
f
e
r
s
f
r
o
m
th
e
one
s
elec
ted
at
tim
e
s
tep
t+1
,
a
tr
a
n
s
itio
n
p
h
ase
m
u
s
t
be
ex
ec
u
ted
.
T
h
is
tr
an
s
itio
n
co
n
s
is
ts
of
ac
tiv
atin
g
a
y
ello
w
lig
h
t
p
h
ase
last
in
g
4
s
ec
o
n
d
s
,
but
o
n
ly
if
th
e
ac
tio
n
at
tim
e
s
tep
t
is
not
th
e
s
am
e
as
th
e
p
r
e
v
io
u
s
o
n
e.
In
th
is
ca
s
e,
th
e
r
e
ar
e
10
s
im
u
latio
n
s
tep
s
s
ep
a
r
atin
g
two
id
en
tical
ac
tio
n
s
,
as
each
s
im
u
latio
n
s
te
p
in
SUMO
r
ep
r
esen
ts
1
s
ec
o
n
d
.
If
co
n
s
ec
u
tiv
e
ac
tio
n
s
ar
e
d
i
f
f
er
en
t,
th
e
y
ello
w
lig
h
t
p
h
ase
ad
d
s
4
more
s
im
u
latio
n
s
tep
s
,
r
esu
ltin
g
in
14
s
tep
s
b
etwe
en
th
e
two
ac
tio
n
s
.
Fig
u
r
e
9.
Po
s
s
ib
le
ac
tio
n
s
f
o
r
each
ag
en
t
3
.
5
.
Rew
a
rd
def
ini
t
io
n
At
th
e
co
r
e
of
r
ein
f
o
r
ce
m
e
n
t
lear
n
in
g
lies
th
e
f
u
n
d
am
en
tal
co
n
ce
p
t
wh
er
e
an
a
g
en
t
u
n
d
er
tak
es
an
ac
tio
n
an
d
s
u
b
s
eq
u
e
n
tly
g
a
r
n
e
r
s
f
ee
d
b
ac
k
f
r
o
m
th
e
en
v
i
r
o
n
m
en
t
in
th
e
f
o
r
m
of
a
r
ewa
r
d
,
d
en
o
ted
as
R
t.
In
our
ap
p
r
o
ac
h
,
we
d
ir
ec
t
o
u
r
atten
tio
n
s
p
ec
if
ically
to
n
eg
ativ
e
r
e
war
d
s
,
em
an
atin
g
f
r
o
m
s
u
b
o
p
t
im
al
ac
tio
n
s
,
with
a
p
r
im
ar
y
o
b
jectiv
e
of
m
itig
ati
n
g
th
eir
im
p
ac
t.
Ou
r
o
v
er
ar
c
h
in
g
aim
is
to
cu
r
tail
n
eg
ativ
e
r
ewa
r
d
s
,
th
er
eb
y
en
h
an
cin
g
th
e
f
lu
i
d
ity
of
tr
af
f
i
c
f
lo
w
with
in
th
e
in
ter
s
ec
tio
n
as
tim
e
p
r
o
g
r
ess
es.
W
h
ile
n
u
m
er
o
u
s
tr
af
f
ic
p
ar
a
m
eter
s
s
u
ch
as
th
r
o
u
g
h
p
u
t,
a
v
er
ag
e
d
elay
,
a
n
d
t
r
av
el
tim
e
o
f
f
er
d
iv
e
r
s
e
m
etr
ics,
our
d
elib
er
ate
ch
o
ice
ce
n
ter
s
on
th
e
m
etr
ic
of
to
tal
waitin
g
tim
e.
T
h
is
s
tr
ateg
ic
d
ec
is
io
n
is
r
o
o
ted
in
th
e
m
etr
ic'
s
ca
p
ac
ity
to
f
u
r
n
is
h
a
co
m
p
r
eh
e
n
s
iv
e
ev
alu
atio
n
of
T
SC
s
tr
ateg
ies
in
u
r
b
an
en
v
ir
o
n
m
e
n
ts
.
Un
lik
e
s
in
g
u
lar
p
ar
am
ete
r
s
,
to
tal
waitin
g
tim
e
en
ca
p
s
u
lates
th
e
cu
m
u
lativ
e
in
f
lu
en
ce
on
th
e
en
t
ir
e
tr
af
f
ic
s
y
s
tem
,
p
r
esen
tin
g
a
h
o
lis
tic
p
er
s
p
ec
ti
v
e
on
co
n
g
esti
o
n
lev
els.
C
o
m
p
u
ted
by
s
u
m
m
in
g
th
e
waitin
g
t
im
es
of
all
v
e
h
icles
in
th
e
en
v
ir
o
n
m
e
n
t
at
tim
e
s
te
p
t,
a
v
eh
icle's
wait
in
g
tim
e
is
i
n
cr
em
en
ted
wh
en
its
s
p
ee
d
f
alls
b
elo
w
0
.
1
m
/s
,
as
d
ef
in
ed
by
(
8
)
.
=
∑
,
(
8
)
W
h
er
e
,
p
r
esen
t
th
e
waitin
g
ti
m
e
of
th
e
v
eh
icle
i
at
tim
estep
t.
th
en
th
e
r
ewa
r
d
ca
n
be
d
ef
i
n
ed
as
(
9
)
.
=
−
1
(
9
)
T
he
waitin
g
tim
e
of
a
v
e
h
icle
is
th
e
tim
e
(
in
s
ec
o
n
d
s
)
af
t
er
th
e
ap
p
ea
r
an
ce
of
th
e
v
eh
i
cle
in
th
e
en
v
ir
o
n
m
en
t
wh
e
r
e
its
s
p
ee
d
is
less
th
an
0
.
1
m
/s
at
th
e
s
tep
of
th
e
ag
en
t
t.
As
a
m
etr
ic,
ea
ch
ag
en
t
co
n
s
id
er
s
th
e
to
tal
cu
m
u
lativ
e
waitin
g
tim
e
f
o
r
s
tep
t
of
ea
c
h
v
e
h
icle
(
_
).
If
th
e
v
eh
icle
m
an
a
g
es
to
m
o
v
e
f
o
r
war
d
b
u
t
f
ails
to
p
ass
th
e
i
n
ter
s
ec
tio
n
,
th
is
m
etr
ic
d
o
es
n
o
t
r
eset
th
e
v
alu
e
of
th
e
to
tal
cu
m
u
lativ
e
waitin
g
tim
e
(
1
0
)
.
T
h
is
s
av
es
us
f
r
o
m
m
is
lead
in
g
s
tatis
tic
s
d
u
r
in
g
ag
en
t
tr
ain
in
g
.
_
=
∑
_
,
(
1
0
)
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J Ar
tif
I
n
tell
I
SS
N:
2252
-
8
9
3
8
E
n
h
a
n
ci
n
g
tr
a
ffic flo
w
th
r
o
u
g
h
mu
lti
-
a
g
en
t rein
fo
r
ce
men
t le
a
r
n
in
g
fo
r
a
d
a
p
tive
tr
a
ffic lig
h
t
…
(
N
a
d
a
F
a
q
ir
)
509
W
h
er
e
_
,
is
th
e
ac
cu
m
u
lated
wa
itin
g
tim
e
of
th
e
v
e
h
icle
i
at
ti
m
estep
t.
T
h
en
our
r
ewa
r
d
f
u
n
ctio
n
is
th
e
ac
cu
m
u
lated
to
tal
waitin
g
tim
e
of
all
th
e
ca
r
s
in
t
h
e
in
te
r
s
ec
tio
n
ca
p
tu
r
e
d
r
esp
ec
tiv
ely
at
ag
e
n
t
s
tep
t
an
d
t
–
1
p
r
esen
ted
as
(
1
1
)
.
=
_
−
1
_
(
1
1
)
In
(
9
)
a
n
d
(
1
1
)
p
r
o
d
u
ce
n
eg
ati
v
e
v
alu
es.
As
m
o
r
e
v
eh
icles
a
cc
u
m
u
late
in
th
e
q
u
eu
es
at
t
h
e
ag
en
t'
s
s
tep
t,
th
e
r
ewa
r
d
b
ec
o
m
es
in
cr
ea
s
in
g
l
y
n
eg
ativ
e,
p
r
o
m
p
tin
g
t
h
e
ag
e
n
t
to
ass
ess
its
ac
tio
n
more
ca
r
ef
u
lly
.
C
o
n
s
eq
u
e
n
tly
,
in
our
r
esu
lts
,
we
f
o
cu
s
on
v
is
u
alizin
g
th
e
n
e
g
ativ
e
r
ewa
r
d
at
each
tim
e
s
tep
to
ev
alu
ate
th
e
b
eh
av
io
r
of
each
ag
en
t.
3
.
6
.
T
ra
ini
ng
a
lg
o
rit
h
m
Alg
o
r
ith
m
1
s
u
m
m
a
r
izes
th
e
co
m
p
lete
tr
ain
i
n
g
p
r
o
ce
s
s
o
f
ea
ch
ag
e
n
t.
Usi
n
g
d
ee
p
Q
-
l
ea
r
n
in
g
,
ea
ch
ag
en
t f
ir
s
t r
etr
iev
es th
e
en
v
ir
o
n
m
en
t state
St =
(
,
,
∆
,
)
an
d
p
r
o
v
i
d
es it a
s
in
p
u
ts
to
th
e
C
NN
n
et
wo
r
k
,
wh
ich
esti
m
ates
th
e
Q
-
v
alu
es
(
,
,
)
as
o
u
tp
u
ts
f
o
r
all
p
o
s
s
ib
le
ac
tio
n
s
in
th
e
ac
tio
n
s
p
ac
e.
As
illu
s
tr
ated
in
Fig
u
r
e
7
.
E
ac
h
ag
en
t
r
ec
o
r
d
s
o
b
s
er
v
ed
in
ter
ac
tio
n
ex
p
er
ie
n
ce
=
(
,
,
,
+
1
)
in
to
a
r
ep
lay
m
e
m
o
r
y
M
=
{
1
,
2
,
·
·
·
,
}
.
As p
ar
t
o
f
o
u
r
ad
o
p
ted
C
T
D
E
ap
p
r
o
ac
h
,
ag
en
ts
e
n
g
ag
e
in
r
ec
o
g
n
itio
n
-
s
h
a
r
in
g
,
wh
er
ein
ea
ch
ag
e
n
t
s
h
ar
es
its
last
tak
en
ac
tio
n
at
tim
e
t
-
1
with
o
th
er
s
,
an
d
v
ice
v
er
s
a.
T
h
is
co
llab
o
r
ativ
e
s
tr
ateg
y
allo
ws
u
s
to
h
a
r
n
ess
th
e
ad
v
an
tag
es
o
f
ce
n
tr
alize
d
tr
ain
in
g
wh
i
le
ad
d
r
ess
in
g
ch
allen
g
es
r
elate
d
to
p
ar
tial
o
b
s
er
v
a
b
ilit
y
an
d
m
in
im
izin
g
d
im
en
s
io
n
ality
co
n
ce
r
n
s
ass
o
ciate
d
with
ce
n
tr
alize
d
ex
ec
u
tio
n
.
Alg
o
r
ith
m
1
:
DQN
with
ex
p
er
ien
ce
r
ep
lay
a
n
d
tar
g
et
n
etwo
r
k
f
o
r
AT
SC
1:
Initialize CNN network with random weights
θ
0
2:
Initialize target network
with weights
3:
Initialize hyperparameters
ϵ
,
γ
,
α
, and the number of episodes
N
4: for
each episode episode = 1
,
2
,...,N do
5:
Initialize action
A
0
6:
Start a new time step
7:
for
each time step time = 1
,
2
,...,T do
8:
if
a new time step
t
begins
then
9:
The agent observes the current intersection state
S
t ;
10:
The agent selects an action
A
= argmax
a
Q
(
S
t
,a
;
θ
) with probability 1
−
ϵ
,
and randomly selects a random action
A
t
with probability
ϵ
,
11:
if A
==
A
t−1
then
12
No transition phase for traffic signals;
13:
else
14:
Apply a transition phase for traffic signals;
15:
end if
16:
end if
17:
Execute the selected action
A
t;
18:
Vehicles move according to the current traffic signals;
19:
Increment time time = time + 1;
20:
if
the time step
t
ends
then
21:
The agent observes the reward
R
and the current intersection state
St+1
;
22:
Store the observed experience (
S
t
,A
t
,R
t
,S
t+1
) in replay memory
M;
23:
Randomly sample a minibatch of 100 experiences (
S
i
,A
i
,R
i
,S
i+1
) from
M;
24:
Form the training data: input set
X
and targets
y;
25:
Update
θ
with the training data;
26:
Update the target network
θ
′
according to equation (12);
27:
end if
28:
end for
29: end for
T
h
e
r
ep
lay
m
em
o
r
y
M
h
as
a
f
in
ite
ca
p
ac
ity
of
50
,
0
0
0
s
a
m
p
les,
an
d
o
n
ce
f
u
ll,
th
e
o
ld
est
d
ata
is
d
is
ca
r
d
ed
.
To
lear
n
C
NN
p
ar
a
m
eter
s
th
at
b
est
ap
p
r
o
x
im
ate
∗
(
s
,
a
)
,
th
e
ag
e
n
t
r
e
q
u
ir
es
tr
ain
i
n
g
d
ata
:
in
p
u
t
d
ata
s
et
X
=
{
(
,
)
:
t
≥
1
}
an
d
th
e
co
r
r
esp
o
n
d
in
g
tar
g
ets
y
=
{
∗
(
,
)
∶
t
≥
1
}
can
be
r
etr
iev
ed
f
r
o
m
r
ep
lay
m
em
o
r
y
M
f
o
r
th
e
in
p
u
t
d
ata
s
et.
Ho
wev
er
,
th
e
tar
g
et
∗
(
,
)
is
u
n
k
n
o
wn
.
We
u
s
e
its
es
tim
at
ed
v
alu
e
+
γ
(
(
+
1
,
)
as
th
e
tar
g
et
in
s
tead
,
as
in
[
1
6
]
,
[
2
6
]
–
[
2
8
]
,
wh
e
r
e
(
+
1
,
)
is
th
e
o
u
tp
u
t
of
a
s
ep
ar
ate
tar
g
et
n
etwo
r
k
w
ith
p
ar
am
eter
s
.
T
h
ese
p
ar
am
et
er
s
ar
e
s
o
f
tly
u
p
d
ated
u
s
in
g
th
e
eq
u
atio
n
(
s
o
f
t
u
p
d
ate
f
o
r
weig
h
ts
)
,
an
d
th
e
tar
g
et
n
etwo
r
k
'
s
in
p
u
t
is
th
e
co
r
r
esp
o
n
d
in
g
+
1
f
r
o
m
i
n
ter
ac
tio
n
ex
p
er
ien
ce
=
(
,
,
,
+
1
)
.
T
h
e
tar
g
et
n
etwo
r
k
em
p
lo
y
s
th
e
s
am
e
ar
ch
itectu
r
e
as
th
e
C
NN
n
etwo
r
k
d
ep
icted
in
Fig
u
r
e
7.
T
h
u
s
,
tar
g
ets
y
=
{
+
γ
(
(
+
1
,
)
∶
t
≥
1
}
.
We
u
s
e
th
e
s
to
ch
asti
c
g
r
ad
ien
t
d
escen
t
alg
o
r
ith
m
r
o
o
t
m
e
an
s
q
u
ar
ed
p
r
o
p
ag
atio
n
(
R
MSPr
o
p
)
[
2
9
]
with
a
m
in
i
b
atch
s
ize
of
1
0
0
to
r
e
d
u
ce
co
m
p
u
tatio
n
al
c
o
s
ts
.
Du
r
in
g
th
e
tr
ai
n
in
g
p
r
o
ce
s
s
of
th
e
C
NN
n
etwo
r
k
,
th
e
ag
en
t
u
tili
ze
s
th
e
ex
p
er
ien
ce
r
ep
lay
tech
n
iq
u
e
by
r
a
n
d
o
m
ly
s
elec
tin
g
100
s
am
p
les
f
r
o
m
th
e
r
ep
lay
m
em
o
r
y
Evaluation Warning : The document was created with Spire.PDF for Python.