I
n
t
e
r
n
at
ion
al
Jou
r
n
al
of
E
lec
t
r
ical
an
d
Com
p
u
t
e
r
E
n
gin
e
e
r
in
g
(
I
JE
CE
)
Vol.
1
4
,
No.
4
,
Augus
t
20
2
4
,
pp.
4655
~
4663
I
S
S
N:
2088
-
8708
,
DO
I
:
10
.
11591/i
jec
e
.
v
1
4
i
4
.
pp
4
655
-
4663
4655
Jou
r
n
al
h
omepage
:
ht
tp:
//
ij
e
c
e
.
iaes
c
or
e
.
c
om
E
m
ai
l
su
b
j
e
c
t
s ge
n
e
r
at
io
n
w
it
h
l
ar
ge
l
a
n
gu
ag
e
m
od
e
ls
:
GP
T
-
3.5,
PaL
M
2, an
d
B
E
R
T
S
ou
m
aya
L
ou
k
il
i,
Abd
e
lh
ad
i
F
e
n
n
a
n
,
L
ot
f
i
E
laachak
D
a
ta
a
nd I
nt
e
ll
ig
e
nt
S
ys
te
ms
T
e
a
m F
S
T
T
, A
bde
lm
a
le
k E
s
s
a
a
di
U
ni
ve
r
s
it
y, T
e
to
ua
n,
M
or
oc
c
o
Ar
t
icle
I
n
f
o
AB
S
T
RA
CT
A
r
ti
c
le
h
is
tor
y
:
R
e
c
e
ived
F
e
b
29,
2024
R
e
vis
e
d
M
a
r
20,
2024
Ac
c
e
pted
Apr
22,
2024
In
o
rd
er
t
o
e
n
h
a
n
ce
mark
et
i
n
g
effo
r
t
s
an
d
i
m
p
ro
v
e
t
h
e
p
e
rfo
rma
n
ce
o
f
mark
et
i
n
g
camp
ai
g
n
s
,
t
h
e
effect
i
v
e
n
es
s
o
f
l
an
g
u
a
g
e
g
en
erat
i
o
n
mo
d
el
s
n
eed
s
t
o
b
e
ev
a
l
u
a
t
ed
.
T
h
i
s
s
t
u
d
y
ex
am
i
n
e
s
t
h
e
p
e
rfo
rma
n
c
e
o
f
l
ar
g
e
l
a
n
g
u
ag
e
mo
d
e
l
s
(
L
L
Ms
)
,
n
amel
y
G
PT
-
3
.
5
,
PaL
M
2
,
an
d
b
i
d
i
rect
i
o
n
al
en
c
o
d
er
rep
res
e
n
t
a
t
i
o
n
s
fro
m
t
ran
s
fo
rmer
s
(
BE
RT
)
,
i
n
g
e
n
erat
i
n
g
emai
l
s
u
b
j
ec
t
s
fo
r
ad
v
er
t
i
s
i
n
g
camp
a
i
g
n
s
.
By
co
m
p
ari
n
g
t
h
e
i
r
res
u
l
t
s
,
t
h
e
au
t
h
o
r
s
ev
al
u
at
e
t
h
e
effi
cacy
o
f
t
h
es
e
m
o
d
e
l
s
i
n
e
n
h
a
n
ci
n
g
mar
k
et
i
n
g
effo
r
t
s
.
T
h
e
o
b
j
ec
t
i
v
e
i
s
t
o
ex
p
l
o
re
h
o
w
L
L
Ms
co
n
t
ri
b
u
t
e
t
o
crea
t
i
n
g
c
o
mp
e
l
l
i
n
g
emai
l
s
u
b
j
ect
l
i
n
es
an
d
i
mp
r
o
v
i
n
g
o
p
en
i
n
g
rat
e
s
an
d
camp
a
i
g
n
p
erf
o
rman
ce,
w
h
i
ch
g
i
v
es
u
s
a
n
i
n
s
i
g
h
t
i
n
t
o
t
h
e
i
mp
ac
t
o
f
t
h
es
e
mo
d
el
s
i
n
d
i
g
i
t
al
mark
e
t
i
n
g
.
In
t
h
i
s
p
ap
er,
t
h
e
au
t
h
o
r
s
fi
r
s
t
g
o
o
v
er
t
h
e
d
i
ffere
n
t
t
y
p
e
s
o
f
l
a
n
g
u
a
g
e
mo
d
el
s
a
n
d
t
h
e
d
i
ffere
n
ces
b
e
t
w
een
t
h
em,
b
ef
o
re
g
i
v
i
n
g
a
n
o
v
erv
i
ew
o
f
t
h
e
m
o
s
t
p
o
p
u
l
ar
o
n
e
s
t
h
at
w
i
l
l
b
e
u
s
e
d
i
n
t
h
e
s
t
u
d
y
,
s
u
ch
as
G
PT
-
3
.
5
,
PaL
M
2
,
an
d
B
E
RT
.
T
h
i
s
s
t
u
d
y
a
s
s
e
s
s
e
s
t
h
e
re
l
ev
a
n
ce,
en
g
a
g
eme
n
t
,
an
d
u
n
i
q
u
e
n
es
s
o
f
G
PT
-
3
.
5
,
PaL
M
2
,
an
d
BE
RT
b
y
t
ra
i
n
i
n
g
an
d
fi
n
e
-
t
u
n
i
n
g
t
h
em
o
n
mark
e
t
i
n
g
t
e
x
t
s
.
T
h
e
f
i
n
d
i
n
g
s
p
ro
v
i
d
e
i
n
s
i
g
h
t
s
i
n
t
o
t
h
e
maj
o
r
p
o
s
i
t
i
v
e
i
mp
ac
t
o
f
ar
t
i
f
i
ci
a
l
i
n
t
el
l
i
g
en
ce
(
AI
)
o
n
d
i
g
i
t
al
mark
e
t
i
n
g
,
en
a
b
l
i
n
g
i
n
f
o
r
med
d
eci
s
i
o
n
-
ma
k
i
n
g
fo
r
A
I
-
d
r
i
v
e
n
emai
l
mark
et
i
n
g
s
t
rat
e
g
i
e
s
.
K
e
y
w
o
r
d
s
:
Ar
ti
f
icia
l
int
e
ll
igenc
e
De
e
p
lea
r
ning
Digit
a
l
mar
ke
ti
ng
GPT
L
a
r
ge
langua
ge
models
Th
i
s
i
s
a
n
o
p
en
a
c
ces
s
a
r
t
i
c
l
e
u
n
d
e
r
t
h
e
CC
B
Y
-
SA
l
i
ce
n
s
e.
C
or
r
e
s
pon
din
g
A
u
th
or
:
S
oumaya
L
oukil
i
Da
ta
a
nd
I
ntelli
ge
nt
S
ys
tems
T
e
a
m
F
S
T
T
,
Abde
lm
a
lek
E
s
s
a
a
di
Unive
r
s
it
y
T
e
touan,
M
or
oc
c
o
E
mail:
s
oumaya
.
louki
li
1@e
tu
.
ua
e
.
a
c
.
ma
1.
I
NT
RODU
C
T
I
ON
O
ve
r
t
he
las
t
f
e
w
ye
a
r
s
,
d
ig
it
a
l
m
a
r
ke
t
in
g
h
a
s
be
e
n
s
ig
ni
f
ica
nt
l
y
im
pa
c
te
d
by
a
r
t
i
f
i
c
i
a
l
i
nt
e
l
li
ge
nc
e
(
AI
)
[
1
]
,
c
o
mp
le
te
ly
a
l
te
r
i
ng
in
te
r
a
c
t
io
ns
be
t
we
e
n
c
omp
a
n
ies
a
n
d
t
he
ir
c
us
to
me
r
s
.
I
n
te
g
r
a
ti
on
o
f
la
r
ge
la
ng
ua
ge
m
o
de
ls
(
L
L
M
s
)
in
t
o
c
o
n
ten
t
c
r
e
a
t
i
on
tec
hn
iq
ue
s
ha
s
be
e
n
o
ne
of
t
he
mos
t
r
e
c
e
n
t
a
nd
s
ig
ni
f
ica
n
t
de
ve
lo
p
men
ts
i
n
t
h
is
f
ie
ld
[
2
]
.
W
it
h
i
ts
a
s
t
ou
nd
i
ng
1
75
bi
ll
io
n
c
h
a
r
a
c
t
e
r
is
ti
c
s
[
3
]
,
L
L
M
s
li
ke
ge
ne
r
a
t
ive
p
r
e
-
t
r
a
i
ne
d
t
r
a
n
s
f
o
r
m
e
r
3
.
5
(
G
P
T
-
3
.
5
)
h
a
ve
us
he
r
e
d
i
n
a
ne
w
e
r
a
o
f
c
on
te
nt
c
r
e
a
t
io
n
.
T
h
e
s
e
mo
de
ls
h
a
ve
de
m
ons
t
r
a
t
e
d
th
e
i
r
a
b
i
li
ty
to
p
r
od
uc
e
c
o
nt
e
n
t
o
f
u
np
r
e
c
e
d
e
n
te
d
q
ua
l
i
ty
a
nd
c
on
te
xt
ua
l
r
e
l
e
v
a
nc
e
o
n
a
mas
s
i
ve
s
c
a
le
.
F
o
r
ins
ta
nc
e
,
G
P
T
-
3
.
5
c
a
n
g
e
n
e
r
a
t
e
te
x
t
th
a
t
r
e
s
e
m
bl
e
s
h
um
a
n
w
r
it
in
g
a
c
r
os
s
a
wi
de
r
a
ng
e
o
f
t
op
ics
a
nd
ge
n
r
e
s
,
i
nc
l
ud
i
ng
p
os
ts
f
o
r
s
oc
i
a
l
me
di
a
,
p
r
o
du
c
t
de
s
c
r
ip
t
io
ns
,
a
n
d
m
o
r
e
.
W
it
h
t
he
us
e
o
f
t
his
c
a
p
a
c
it
y
,
m
a
r
ke
te
r
s
c
a
n
e
f
f
ic
ien
t
ly
c
r
e
a
te
m
a
s
s
ive
a
m
ou
nts
o
f
c
on
te
nt
wh
il
e
s
a
v
i
ng
ti
me
a
n
d
r
e
s
ou
r
c
e
s
a
nd
ma
in
ta
in
in
g
a
c
ons
is
te
nt
b
r
a
n
d
v
oi
c
e
.
L
L
M
s
c
a
n
b
e
c
o
me
e
s
s
e
n
ti
a
l
t
o
t
he
o
pt
im
iz
a
t
io
n
o
f
e
m
a
i
l
ma
r
k
e
t
in
g
c
a
mpa
ig
ns
,
wh
ic
h
c
a
n
le
a
d
t
o
a
s
ig
ni
f
ica
nt
in
c
r
e
a
s
e
i
n
o
pe
n
a
n
d
c
li
c
k
-
t
h
r
o
ug
h
r
a
t
e
s
.
T
h
is
c
a
n
be
d
on
e
b
y
u
ti
l
izi
ng
th
e
s
t
r
e
n
gt
h
o
f
t
he
s
e
m
o
de
ls
to
c
r
e
a
t
e
c
a
p
ti
va
t
in
g
e
ma
il
s
u
b
jec
t
li
ne
s
a
nd
i
nt
e
r
e
s
t
i
ng
c
o
nt
e
n
t
.
I
f
A/
B
tes
ti
ng
s
t
ud
ies
d
e
m
ons
t
r
a
te
t
ha
t
e
ma
i
ls
w
i
th
s
ub
jec
t
l
in
e
s
c
r
e
a
ted
by
L
L
M
s
c
a
n
pe
r
f
o
r
m
be
tt
e
r
tha
n
e
m
a
i
ls
w
it
h
s
ub
je
c
t
l
i
ne
s
w
r
i
t
ten
by
h
um
a
ns
,
g
e
ne
r
a
t
in
g
a
r
i
s
e
in
o
pe
n
r
a
tes
,
it
w
il
l
h
ig
hl
y
im
p
r
o
ve
e
ma
il
m
a
r
k
e
t
i
ng
's
e
f
f
ic
ien
c
y
w
hi
le
a
ls
o
f
r
e
e
in
g
up
m
a
r
ke
t
in
g
t
e
a
ms
t
o
c
on
c
e
n
t
r
a
te
o
n
ot
he
r
t
a
c
t
ic
a
l
f
a
c
e
ts
o
f
t
he
i
r
c
a
m
pa
i
g
ns
.
T
he
o
bj
e
c
t
i
ve
o
f
th
is
s
t
ud
y
is
t
wo
f
ol
d
.
As
s
e
s
s
in
g
th
e
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2088
-
8708
I
nt
J
E
lec
&
C
omp
E
ng
,
Vol
.
1
4
,
No.
4
,
Augus
t
20
2
4
:
4655
-
4663
4656
p
e
r
f
o
r
m
a
n
c
e
o
f
d
i
f
f
e
r
e
nt
l
a
n
gua
ge
m
ode
l
-
b
a
s
e
d
a
pp
r
oa
c
he
s
i
n
ma
r
k
e
t
in
g
,
s
pe
c
i
f
i
c
a
ll
y
e
ma
i
l
m
a
r
ke
t
ing
,
is
t
he
f
i
r
s
t
p
a
r
t
.
I
t
w
i
ll
a
l
low
f
or
a
c
o
mpa
r
is
o
n
b
e
t
we
e
n
t
he
L
L
M
-
g
e
ne
r
a
te
d
s
u
bj
e
c
ts
a
n
d
t
hos
e
c
r
e
a
ted
by
hu
man
e
f
f
or
ts
.
S
e
c
o
nd
,
th
e
in
te
nt
io
n
is
to
e
x
e
c
u
te
t
he
s
a
i
d
c
om
pa
r
is
o
n
ba
s
e
d
o
n
ke
y
m
e
t
r
ics
,
s
uc
h
a
s
o
pe
n
i
ng
r
a
te
a
n
d
o
t
he
r
l
a
n
gu
a
ge
ge
ne
r
a
ti
on
me
t
r
i
c
s
.
B
y
c
o
nd
uc
ti
ng
t
his
a
na
l
ys
is
,
th
e
a
ut
ho
r
s
a
i
m
t
o
g
a
i
n
ins
ig
hts
i
n
to
th
e
e
f
f
e
c
t
iv
e
us
e
o
f
LLM
i
n
im
pr
ov
i
ng
e
ma
i
l
ma
r
ke
ti
ng
c
a
m
pa
ig
ns
pe
r
f
o
r
ma
nc
e
.
T
h
is
pa
pe
r
is
s
t
r
uc
t
u
r
e
d
a
s
f
ol
lo
ws
:
s
e
c
ti
on
2
dis
c
us
s
e
s
t
he
k
e
y
c
o
nc
e
pts
o
f
lan
gu
a
g
e
m
od
e
l
in
g
,
e
lu
c
i
da
ti
ng
it
s
de
f
i
ni
t
io
n
,
va
r
io
us
t
ype
s
,
a
n
d
t
r
a
c
in
g
i
ts
e
vo
lu
t
io
n
ov
e
r
t
im
e
,
a
l
on
g
wi
th
in
t
r
o
du
c
i
ng
the
ke
y
f
oc
u
s
o
f
t
his
s
tu
dy
:
th
r
e
e
p
r
om
in
e
n
t
l
a
r
ge
la
ng
ua
ge
m
o
de
ls
,
P
a
L
M
2
,
GP
T
-
3
.
5
,
a
n
d
b
id
i
r
e
c
t
io
na
l
e
nc
od
e
r
r
e
p
r
e
s
e
nt
a
t
io
ns
f
r
om
t
r
a
ns
f
o
r
me
r
s
(
B
E
R
T
)
,
o
f
f
e
r
ing
a
n
in
-
de
pt
h
a
na
lys
is
o
f
t
he
i
r
f
e
a
tu
r
e
s
a
nd
r
e
lev
a
nc
e
to
t
his
r
e
s
e
a
r
c
h
.
S
e
c
t
io
n
3
o
ut
li
ne
s
th
is
s
tu
dy
’
s
me
th
od
ol
og
y
,
i
nc
lu
di
ng
d
a
t
a
c
o
ll
e
c
ti
on
a
nd
f
i
ne
tu
ni
ng
.
I
n
s
e
c
ti
o
n
4
,
t
he
a
ut
ho
r
s
p
r
e
s
e
n
t
th
e
r
e
s
u
lt
s
a
nd
e
n
ga
ge
i
n
a
c
om
p
r
e
he
ns
i
ve
d
is
c
u
s
s
i
on
,
u
nr
a
v
e
l
in
g
th
e
s
ig
ni
f
ic
a
n
c
e
a
nd
i
m
pl
ic
a
t
io
ns
o
f
t
he
i
r
f
in
di
ngs
.
F
in
a
l
ly
,
t
he
pa
p
e
r
c
o
nc
lu
de
s
s
uc
c
i
nc
tl
y
,
s
u
m
ma
r
i
z
i
ng
th
e
k
e
y
ta
ke
a
wa
ys
,
a
nd
s
ug
ge
s
ti
ng
po
te
nt
ia
l
a
v
e
n
ue
s
f
o
r
f
ut
u
r
e
r
e
s
e
a
r
c
h
.
2.
L
I
T
E
RA
T
UR
E
RE
VI
E
W
2.
1.
L
an
gu
age
m
od
e
li
n
g
L
a
ngua
ge
modeling
(
L
M
)
is
one
of
the
f
unda
menta
l
objec
ti
ve
s
of
na
tur
a
l
langua
ge
pr
oc
e
s
s
ing.
L
M
s
a
tt
e
mpt
to
pr
e
dict
the
(
+
1
)
ℎ
token
in
a
s
e
que
nc
e
given
the
n
pr
e
c
e
ding
tokens
[
4]
in
or
de
r
to
r
e
pr
e
s
e
nt
the
li
ke
li
hood
of
wo
r
d
s
e
que
nc
e
s
.
R
e
a
l
-
li
f
e
a
ppli
c
a
ti
ons
of
tr
a
ined
L
M
s
a
r
e
diver
s
e
,
a
nd
va
r
ious
,
a
nd
include
:
C
li
nica
lB
E
R
T
[
5]
,
a
langua
ge
model
t
r
a
ined
on
c
li
nica
l
notes
to
pr
e
dict
hos
pit
a
l
r
e
a
dmi
s
s
ion,
M
olE
[
6]
,
a
mol
e
c
ular
f
ounda
ti
on
langua
ge
model
f
or
dr
ug
dis
c
ove
r
y,
that
pr
e
dicts
p
r
ope
r
ti
e
s
ba
s
e
d
on
c
he
mi
c
a
l
s
tr
uc
tur
e
,
a
nd
s
e
nti
ment
a
na
lys
is
ba
s
e
d
on
L
M
f
or
ge
ne
r
a
l
US
e
lec
ti
ons
,
B
iden
vs
T
r
ump
[
7
]
.
F
our
main
s
tage
s
of
de
ve
lopm
e
nt/
types
c
a
n
be
identif
ied
f
or
l
a
ngua
ge
models
:
2.
1.
1.
S
t
at
is
t
ical
lan
gu
age
m
od
e
ls
T
he
y
f
oc
us
on
e
s
ti
mating
the
p
r
oba
bil
it
y
dis
tr
ib
uti
on
o
f
va
r
ious
li
ngu
is
ti
c
unit
s
,
s
uc
h
a
s
wor
ds
,
s
e
ntenc
e
s
,
a
nd
e
nti
r
e
doc
uments
ove
r
a
ll
pos
s
ibl
e
phr
a
s
e
s
,
in
a
n
e
f
f
o
r
t
to
c
a
ptur
e
the
pa
tt
e
r
ns
o
f
na
tur
a
l
langua
ge
[
8]
.
T
he
pur
pos
e
is
to
im
pr
ove
the
pe
r
f
or
manc
e
of
s
e
ve
r
a
l
na
tu
r
a
l
langua
ge
p
r
oc
e
s
s
ing
a
ppli
c
a
ti
ons
.
M
a
jor
S
tatis
ti
c
a
l
langua
ge
models
(
S
L
M
)
tec
hniq
ue
s
include
but
a
r
e
not
li
mi
ted
to:
n
-
gr
a
ms
,
de
c
is
ion
tr
e
e
models
,
li
nguis
ti
c
a
ll
y
mot
ivate
d
models
,
e
xpone
nti
a
l
models
,
a
nd
a
da
pti
ve
models
.
2.
1.
2.
Ne
u
r
al
la
n
gu
age
m
o
d
e
ls
Ne
ur
a
l
langua
ge
models
(
NL
M
s
)
a
nti
c
ipate
a
s
e
t
of
f
utur
e
wor
ds
given
the
his
tor
y
o
f
p
r
e
vious
wor
ds
,
s
im
il
a
r
to
other
langua
ge
models
.
How
e
ve
r
,
in
NL
M
s
,
wor
ds
a
r
e
pr
ojec
ted
on
to
a
lowe
r
dim
e
ns
ional
ve
c
tor
s
pa
c
e
via
a
hidden
laye
r
f
r
om
a
s
pa
r
s
e
,
1
-
of
-
V
e
nc
oding
(
whe
r
e
V
is
the
s
ize
o
f
the
voc
a
bu
lar
y)
[
9]
.
E
a
c
h
wor
d
in
the
voc
a
bular
y
is
de
picte
d
a
s
a
r
e
a
l
-
va
lued
f
e
a
tur
e
ve
c
tor
s
o
that
,
f
or
s
e
mantica
ll
y
r
e
late
d
wor
ds
,
the
c
os
ine
of
the
a
ngles
be
twe
e
n
thes
e
ve
c
tor
s
is
high.
2.
1.
3.
P
r
e
-
t
r
ain
e
d
la
n
gu
age
m
od
e
ls
Among
a
ll
de
e
p
lea
r
ning
methods
,
f
ine
-
tuni
ng
a
p
re
-
tr
a
ined
langua
ge
model
(
P
L
M
)
on
downs
tr
e
a
m
tas
ks
of
int
e
r
e
s
t
ha
s
be
c
ome
the
s
tanda
r
d
pipeli
ne
in
NL
P
tas
ks
.
T
he
f
unda
menta
l
c
onc
e
pt
is
to
us
e
the
r
e
s
ult
ing
r
e
p
r
e
s
e
ntations
on
tas
ks
a
f
ter
tr
a
ini
ng
a
l
a
r
ge
ge
ne
r
a
ti
ve
model
on
mas
s
ive
c
or
por
a
,
whe
r
e
labe
led
da
ta
is
s
c
a
r
c
e
[
10]
.
P
L
M
s
a
r
e
c
a
pa
ble
of
c
a
ptur
i
ng
the
mea
ning
of
wo
r
ds
dyna
mi
c
a
ll
y
in
c
ons
ider
a
ti
on
of
their
c
ontext.
T
he
ve
r
y
f
ir
s
t
ve
r
s
ions
of
ge
ne
r
a
ti
ve
pr
e
-
tr
a
ined
tr
a
ns
f
or
mer
(
GP
T
)
[
11]
by
Ope
n
AI
,
a
nd
Google
’
s
own
B
E
R
T
[
12]
a
r
e
wildl
y
us
e
d
P
L
M
s
that
ha
ve
be
c
ome
ve
r
y
popular
in
r
e
c
e
nt
ye
a
r
s
.
2.
1.
4.
L
ar
ge
lan
gu
age
m
od
e
ls
T
wo
major
de
ve
lopm
e
nts
mar
ke
d
the
a
dva
nc
e
ment
of
L
L
M
s
:
the
us
e
of
t
r
a
ns
f
or
mer
a
r
c
hit
e
c
tur
e
s
,
a
nd
the
int
r
oduc
ti
on
of
f
ine
-
tuni
ng
f
or
pr
e
tr
a
ined
models
.
A
LLM
is
f
ine
-
tuned
by
f
ur
the
r
t
r
a
ini
n
g
it
on
a
s
maller
,
tas
k
-
s
pe
c
if
ic
da
tas
e
t
to
a
da
pt
it
to
a
pa
r
ti
c
ular
tas
k
o
r
topi
c
[
13]
.
T
he
p
r
oc
e
s
s
invol
ve
s
a
djus
ti
ng
the
pa
r
a
mete
r
s
of
a
n
a
lr
e
a
dy
tr
a
ined
model
us
ing
a
s
maller
,
do
main
-
s
pe
c
if
ic
da
tas
e
t,
a
nd
twe
a
king
the
we
ight
s
a
nd
pa
r
a
mete
r
s
of
the
model
to
r
e
duc
e
the
los
s
f
un
c
ti
on
a
nd
e
nha
nc
e
it
s
tas
k
pe
r
f
o
r
manc
e
.
L
L
M
s
,
s
uc
h
a
s
B
a
r
d
by
Google
,
or
C
ha
tGP
T
by
Ope
nAI
,
ga
ined
e
xpone
nti
a
l
popular
it
y
ove
r
the
las
t
c
ouple
of
ye
a
r
s
,
mainly
be
c
a
us
e
the
tec
hnolog
y
matur
e
d
e
nough
to
be
c
ome
viable
f
or
c
ons
umer
us
e
.
Although
other
s
e
xis
ted
be
f
o
r
e
,
it
wa
s
C
ha
tGP
T
t
ha
t
made
the
br
e
a
kthr
ough
f
o
r
us
e
r
s
outs
ide
o
f
the
r
e
s
e
a
r
c
h
c
omm
unit
y
a
nd
AI
e
nthus
ias
ts
,
by
a
ll
owing
a
nyon
e
to
s
ign
up
f
or
f
r
e
e
,
ope
n
a
c
ha
t
dialogue
,
a
nd
s
tar
t
ge
tt
ing
legible
L
L
M
output
without
ne
e
ding
to
unde
r
s
tand
the
c
onc
e
pt
or
pr
ompt
e
nginee
r
ing.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
E
lec
&
C
omp
E
ng
I
S
S
N:
2088
-
8708
E
mail
s
ubjec
ts
ge
ne
r
ati
on
w
it
h
lar
ge
language
mode
ls
:
GP
T
-
3
.
5,
P
aL
M
2
,
and
B
E
R
T
(
Soumay
a
L
o
uk
il
i)
4657
I
n
a
ddit
ion
to
pr
e
s
e
nti
ng
va
r
ious
oppor
tuni
ti
e
s
,
th
e
uti
li
z
a
ti
on
of
L
L
M
s
is
a
c
c
ompanie
d
by
va
r
ious
e
thi
c
a
l
a
nd
s
oc
ial
c
ha
ll
e
nge
s
.
W
e
idi
nge
r
e
t
al
.
[
14
]
c
a
tegor
ize
d
the
r
is
ks
pr
e
s
e
nted
a
s
s
uc
h:
i
)
d
is
c
r
im
ination,
by
pe
r
pe
tuating
s
oc
ial
s
ter
e
otypes
a
nd
bias
e
s
,
ii
)
inf
or
mation
ha
z
a
r
ds
,
whic
h
invol
ve
s
dis
c
los
ing
s
e
ns
it
ive
inf
or
mation,
ii
i)
mi
s
inf
or
mation
ha
z
a
r
ds
,
by
dis
s
e
mi
na
ti
ng
mi
s
lea
ding
inf
or
mation
,
iv)
malicious
us
e
,
a
s
us
e
r
s
with
ba
d
int
e
nt
c
ould
take
a
dva
ntage
a
nd
g
e
ne
r
a
te
pe
r
s
ona
li
z
e
d
s
c
a
ms
a
nd
f
r
a
ud,
v)
human
-
c
omput
e
r
int
e
r
a
c
ti
on
ha
r
ms
,
mea
ning
us
e
r
s
may
us
e
the
h
umanlike
c
a
pa
bil
it
ies
of
L
L
M
s
in
a
n
uns
a
f
e
manne
r
,
a
nd
vi)
a
utom
a
ti
on
a
nd
e
nvir
onmenta
l
ha
r
ms
,
a
s
tr
a
ini
ng
a
nd
ope
r
a
ti
ng
L
L
M
s
r
e
quir
e
e
xtens
ive
c
omput
ing
powe
r
,
incur
r
ing
high
e
nvi
r
onmenta
l
c
os
ts
.
S
ome
of
the
main
li
mi
tations
a
nd
c
ha
ll
e
nge
s
thes
e
L
L
M
s
f
a
c
e
include
the
high
c
os
t
it
take
s
to
tr
a
in
them
[
15]
,
the
ha
ll
uc
inations
[
16]
incor
r
e
c
t
a
nd/o
r
inappr
opr
iate
a
ns
we
r
s
given
c
onf
idently,
the
b
ias
they
mi
ght
pr
e
s
e
nt
du
e
to
the
tr
a
ini
ng
da
ta
they
ha
ve
be
e
n
given
[
17]
a
nd
their
vulner
a
bil
it
ies
to
a
dve
r
s
a
r
ial
a
tt
a
c
ks
[
18
]
.
T
he
r
e
a
r
e
s
e
ve
r
a
l
methods
uti
li
z
e
d
to
e
va
luate
th
e
pe
r
f
or
manc
e
of
L
L
M
s
.
S
e
e
ing
that
L
L
M
s
c
a
n
c
a
r
r
y
out
di
f
f
e
r
e
nt
tas
ks
,
it
is
im
por
tant
to
pinp
oint
wha
t
to
e
va
luate
,
whe
r
e
,
a
nd
how.
T
he
s
e
e
va
luation
methods
[
19]
include
:
mea
s
ur
ing
f
e
a
tur
e
s
s
uc
h
a
s
langua
ge
f
luenc
y,
c
ohe
r
e
nc
e
,
c
ontext,
a
nd
mo
r
e
,
z
e
r
o
-
s
hot
e
va
luation,
whic
h
mea
s
ur
e
s
the
pe
r
f
o
r
manc
e
o
f
lar
ge
langua
ge
models
on
tas
ks
they
ha
ve
not
be
e
n
e
xpli
c
it
ly
tr
a
ined
on,
uti
li
z
ing
other
la
r
ge
langua
ge
models
f
or
e
va
luation,
whic
h
o
f
f
e
r
s
s
c
a
labili
ty
a
nd
he
lps
identif
y
a
r
e
a
s
f
or
im
p
r
ove
ment,
a
nd
c
onc
e
ptual
e
va
luation
f
r
a
mew
or
ks
.
2.
2.
L
ar
ge
lan
gu
age
m
od
e
ls
in
f
oc
u
s
2.
2.
1.
P
aL
M
2
b
y
Googl
e
W
he
n
it
c
omes
to
AI
br
e
a
kthr
oughs
ove
r
the
ye
a
r
s
,
Google
s
tands
out
a
s
a
hous
e
hold
na
me.
F
or
ins
tanc
e
,
they
a
r
e
the
one
s
be
hind
L
L
M
s
a
ll
a
r
ound,
a
s
they
a
r
e
ba
s
e
d
on
tr
a
ns
f
or
mer
,
a
ne
ur
a
l
ne
twor
k
a
r
c
hit
e
c
tur
e
Google
made
publi
c
ba
c
k
in
2017
[
20]
.
P
a
L
M
2
is
Google
’
s
ne
xt
-
ge
ne
r
a
ti
on
lar
ge
langua
ge
model
that
o
f
f
e
r
s
a
s
igni
f
ica
nt
im
pr
ove
ment
f
r
om
their
f
ir
s
t
one
,
pa
thwa
ys
langua
ge
model
(
P
a
L
M
)
,
a
nd
that
wa
s
a
nnounc
e
d
in
M
a
y
2023.
I
t
is
a
340
bi
ll
ion
pa
r
a
mete
r
model
t
r
a
ined
on
3
.
6
t
r
il
li
on
tokens
[
21]
,
a
lt
hough
much
s
maller
than
it
s
pr
e
de
c
e
s
s
or
,
they
f
ound
t
ha
t
the
tr
a
ini
ng
c
omput
e
s
ur
pa
s
s
e
s
s
ize
f
or
the
r
e
s
ult
ing
qua
li
ty.
P
a
L
M
2
is
a
v
a
il
a
ble
in
f
our
s
ize
s
:
Ge
c
ko,
Otter
,
B
is
on,
a
nd
Unic
or
n,
f
r
om
s
malles
t
to
lar
ge
s
t
[
22]
.
Ge
c
ko
is
s
o
s
mall
a
nd
li
ght
that
it
c
a
n
r
un
on
mobi
le
de
vice
s
.
P
a
L
M
2
c
a
n
be
a
da
pted
to
a
c
c
omm
oda
te
e
nti
r
e
pr
oduc
t
c
las
s
e
s
in
mor
e
wa
ys
a
nd
be
ne
f
it
mor
e
us
e
r
s
be
c
a
us
e
of
it
s
ve
r
s
a
ti
li
ty.
P
a
L
M
2
c
a
n
pe
r
f
or
m
a
wide
r
a
nge
of
tas
ks
[
23
]
,
a
l
l
langua
ge
-
r
e
late
d.
F
o
r
ins
tanc
e
,
it
c
a
n
unde
r
s
tand
ove
r
100
langua
ge
s
,
whic
h
a
ll
ows
i
t
to
be
a
ble
t
o
tr
a
ns
late
a
nd
ge
ne
r
a
te
text
(
poe
ms
a
nd
s
tor
ies
)
f
luently
.
P
a
L
M
2
pe
r
f
or
ms
e
xc
e
pti
ona
ll
y
we
ll
in
langua
ge
pr
of
icie
nc
y
e
xa
ms
a
nd
pa
s
s
e
s
the
a
dva
nc
e
d
leve
ls
.
I
t
a
ls
o
c
a
n
logi
c
a
ll
y
s
olve
c
ompl
e
x
pr
oblems
,
whe
ther
it
is
mathe
matica
l,
r
iddl
e
s
,
or
e
ls
e
.
P
a
L
M
2
is
the
f
ir
s
t
langua
ge
model
to
pe
r
f
or
m
a
t
a
n
e
xpe
r
t
leve
l
wi
th
mor
e
than
85%
a
c
c
ur
a
c
y
on
que
s
ti
ons
c
ompar
a
ble
to
thos
e
on
the
U.
S
.
M
e
dica
l
L
ice
ns
ing
E
xa
mi
na
ti
on
(
USM
L
E
)
.
P
a
L
M
2
goe
s
be
yond
na
tur
a
l
langua
ge
s
;
i
t
is
a
ls
o
c
a
pa
ble
of
c
oding
in
ove
r
20
langua
ge
s
,
ge
ne
r
a
ti
ng
c
ode
to
a
ns
we
r
s
pe
c
if
ic
pr
oblems
,
de
bugging
it
,
or
e
ve
n
tr
a
ns
f
or
mi
ng
c
ode
f
r
om
one
langua
ge
to
a
nother
.
Up
unti
l
now,
Google
int
e
gr
a
ted
P
a
L
M
2
in
ne
a
r
ly
25
of
it
s
p
r
oduc
ts
,
to
make
them
mor
e
int
e
ll
igent.
YouT
ube
,
Gma
il
,
Google
Doc
s
,
Google
S
he
e
ts
,
a
nd
e
ve
n
Google
T
r
a
ns
late
a
r
e
only
s
ome
of
them.
T
he
y
a
ls
o
r
e
c
e
ntl
y
r
e
ve
a
led
Google
B
a
r
d
AI
,
a
text
-
ba
s
e
d
a
r
ti
f
icia
l
int
e
ll
igenc
e
c
ha
tbot
that
is
buil
t
on
P
a
L
M
2.
B
a
r
d
ge
ne
r
a
tes
r
e
a
l
-
ti
me
a
ns
we
r
s
us
ing
NL
P
a
nd
mac
hi
ne
lea
r
ning.
I
t
take
s
Google
f
r
om
be
ing
a
s
e
a
r
c
h
e
n
gine
to
a
c
a
pa
ble
vir
tual
a
s
s
is
tant
that
pr
ovides
we
ll
-
r
ounde
d
a
ns
we
r
s
f
r
om
a
na
tur
a
l
langua
ge
que
r
y
[
24]
.
T
o
a
dva
nc
e
medic
a
l
L
L
M
e
f
f
or
ts
,
Google
's
M
e
d
-
P
a
L
M
2
us
e
s
a
c
ombi
na
ti
on
of
ba
s
e
L
L
M
a
dva
nc
e
ment
s
,
medic
a
l
domain
f
inetuning
,
a
nd
pr
ompt
ing
tac
ti
c
s
,
includi
ng
a
nove
l
e
ns
e
mbl
e
r
e
f
ineme
nt
methodology
[
25
]
.
T
his
is
pa
r
ti
c
ular
ly
im
po
r
tant
a
s
it
pr
ovides
high
-
qua
li
ty
a
ns
we
r
s
to
c
ompl
e
x
medic
a
l
que
s
ti
ons
.
2.
2.
2.
G
P
T
-
3.
5
b
y
O
p
e
n
AI
P
r
ior
to
the
de
ve
lopm
e
nt
o
f
models
li
ke
GPT
-
1,
n
a
tur
a
l
langua
ge
unde
r
s
tanding
(
NL
U)
tas
ks
r
e
li
e
d
he
a
vil
y
on
s
upe
r
vis
e
d
lea
r
ning
with
labe
led
da
tas
e
ts
,
li
mi
ti
ng
their
a
bil
i
ty
to
ge
ne
r
a
li
z
e
.
GPT
-
1,
a
ge
ne
r
a
ti
ve
pr
e
-
tr
a
ined
langua
ge
model,
int
r
oduc
e
d
a
tr
a
ns
f
or
mation
by
us
ing
uns
upe
r
vis
e
d
lea
r
ning
on
mas
s
iv
e
unlabe
led
da
ta,
s
im
pli
f
ying
f
ine
-
tuni
ng
f
or
downs
tr
e
a
m
tas
ks
[
26]
.
GPT
-
2,
buil
ding
on
GPT
-
1,
i
mpr
ove
d
model
s
tr
uc
tur
e
,
incr
e
a
s
e
d
tr
a
ini
ng
da
ta,
a
nd
s
c
a
led
up
pa
r
a
mete
r
s
.
T
his
e
nha
nc
e
ment
boos
ted
pe
r
f
or
manc
e
a
c
r
os
s
va
r
ious
tas
ks
,
r
e
duc
ing
the
ne
e
d
f
or
s
up
e
r
vis
e
d
tr
a
ini
ng
a
nd
high
li
ghti
ng
the
potential
of
lar
ge
r
models
.
GPT
-
3,
a
n
e
ve
n
la
r
ge
r
model
with
mor
e
d
a
ta,
br
oke
r
e
c
or
ds
in
langua
ge
models
.
I
t
e
xc
e
ll
e
d
in
z
e
r
o
-
s
hot
a
nd
f
e
w
-
s
hot
s
e
tt
ings
,
de
mons
tr
a
ti
ng
r
e
mar
k
a
ble
ve
r
s
a
ti
li
ty
a
c
r
os
s
tas
ks
li
ke
math
,
a
r
ti
c
le
ge
ne
r
a
ti
on,
a
nd
c
oding.
As
models
gr
ow
in
s
ize
,
their
c
a
pa
bil
i
ti
e
s
c
onti
nue
to
e
xpa
nd,
pr
omi
s
ing
e
ve
n
mor
e
powe
r
f
ul
AI
s
ys
tems
in
the
f
utur
e
.
GPT
-
3.
5
is
a
r
e
mar
ka
ble
mi
les
tone
in
the
r
e
a
lm
of
na
tur
a
l
langua
ge
pr
oc
e
s
s
ing.
I
t
r
e
pr
e
s
e
nts
the
e
volut
ion
of
Ope
nAI
's
gr
oundbr
e
a
king
t
r
a
ns
f
or
m
e
r
-
ba
s
e
d
model
s
,
taking
a
dva
ntage
of
the
les
s
ons
lea
r
ne
d
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2088
-
8708
I
nt
J
E
lec
&
C
omp
E
ng
,
Vol
.
1
4
,
No.
4
,
Augus
t
20
2
4
:
4655
-
4663
4658
f
r
om
it
s
pr
e
de
c
e
s
s
or
s
,
GPT
-
3
a
nd
GPT
-
2,
a
s
we
ll
a
s
the
f
ounda
ti
ona
l
tr
a
ns
f
or
mer
a
r
c
hit
e
c
tur
e
.
I
n
s
tea
d
of
making
GPT
-
3.
5
a
va
il
a
ble
in
it
s
f
ull
y
t
r
a
ined
s
tate
,
Ope
nAI
us
e
d
it
to
c
r
e
a
te
s
e
ve
r
a
l
s
ys
tems
that
a
r
e
e
a
c
h
s
pe
c
ially
opti
mi
z
e
d
f
or
a
di
f
f
e
r
e
nt
job
a
nd
a
r
e
a
l
l
a
c
c
e
s
s
ibl
e
thr
ough
the
Ope
nAI
API
.
One
of
them,
text
-
Da
vinci
-
003,
is
be
li
e
ve
d
to
be
a
ble
to
ha
ndle
mo
r
e
c
ompl
e
x
c
omm
a
nds
a
nd
ge
ne
r
a
te
be
tt
e
r
-
qua
li
ty,
longer
-
f
or
m
w
r
it
ing
than
models
buil
t
on
GP
T
-
3.
I
n
r
e
a
l
-
li
f
e
a
ppli
c
a
ti
ons
,
GPT
-
3.
5
ha
s
lef
t
a
n
indeli
ble
mar
k.
I
ts
ve
r
s
a
ti
li
ty
e
xtends
to
a
wide
r
a
nge
of
f
ields
a
nd
indus
tr
ies
.
I
t
is
be
e
n
ha
r
ne
s
s
e
d
f
o
r
na
tur
a
l
langua
ge
unde
r
s
tanding
tas
ks
,
e
na
bli
ng
it
t
o
e
xtr
a
c
t
ins
ight
s
f
r
om
va
s
t
volu
mes
of
text
da
ta.
C
ontent
ge
ne
r
a
ti
on,
including
a
r
ti
c
le
wr
i
ti
ng
[
27]
a
nd
c
r
e
a
ti
ve
s
tor
ytelli
ng,
ha
s
s
e
e
n
s
igni
f
ica
nt
im
p
r
ove
ments
thanks
to
GP
T
-
3.
5's
a
bil
i
ty
to
pr
oduc
e
c
ohe
r
e
nt
a
nd
c
ontextua
ll
y
r
e
leva
nt
text.
I
n
the
r
e
a
lm
of
c
us
tom
e
r
s
e
r
vice
a
nd
s
uppor
t,
GPT
-
3.
5
ha
s
powe
r
e
d
c
ha
t
bots
a
nd
vir
tual
a
s
s
is
tants
[
28]
,
p
r
ovidi
ng
e
f
f
icie
nt
a
nd
r
e
s
pons
ive
int
e
r
a
c
ti
ons
.
Addit
ionally,
it
plays
a
pivot
a
l
r
ole
in
a
utom
a
ted
tr
a
ns
lation
[
29]
,
making
mul
ti
l
ingual
c
o
mm
unica
ti
on
mor
e
a
c
c
e
s
s
ibl
e
.
2.
2.
2.
B
E
RT
b
y
Googl
e
B
E
R
T
whic
h
s
tands
f
o
r
bidi
r
e
c
ti
ona
l
e
nc
ode
r
r
e
p
r
e
s
e
ntations
f
r
om
tr
a
ns
f
or
mer
s
wa
s
publi
s
he
d
by
r
e
s
e
a
r
c
he
r
s
a
t
Google
AI
L
a
ngua
ge
[
12
]
in
201
9.
I
t
is
a
n
innovative
langua
ge
r
e
pr
e
s
e
ntation
m
ode
l
a
nd
pr
e
s
e
nts
s
tate
-
of
-
th
e
-
a
r
t
r
e
s
ult
s
in
a
wide
va
r
iety
o
f
NL
P
tas
ks
.
T
he
main
pu
r
pos
e
be
hind
B
E
R
T
is
to
i
mpr
ove
langua
ge
unde
r
s
tanding
by
pr
e
-
tr
a
ini
ng
on
lar
ge
a
mount
s
of
unlabe
led
da
ta,
s
o
it
c
a
n
de
tec
t
a
nd
lea
r
n
the
pa
tt
e
r
ns
be
twe
e
n
the
e
nc
ounter
e
d
wor
ds
,
ter
ms
,
a
n
d
phr
a
s
e
s
.
T
he
main
tec
hnica
l
innovation
o
f
B
E
R
T
is
the
a
ppli
c
a
ti
on
of
t
r
a
ns
f
or
mer
's
bid
ir
e
c
ti
ona
l
t
r
a
ini
ng,
a
n
e
xtr
e
mely
popular
a
tt
e
nti
on
model,
to
langua
ge
m
ode
li
ng.
I
n
c
ont
r
a
s
t,
pa
s
t
s
tudi
e
s
c
ons
ider
e
d
text
s
e
que
nc
e
s
f
r
om
e
it
he
r
a
lef
t
-
to
-
r
ight
or
a
c
ombi
ne
d
le
f
t
-
to
-
r
ight
a
nd
r
igh
t
-
to
-
lef
t
tr
a
ini
ng
pe
r
s
pe
c
ti
ve
[
30]
.
T
he
r
e
s
ult
s
of
the
s
tudy
s
how
that
bidi
r
e
c
ti
ona
ll
y
t
r
a
ined
langua
ge
models
ha
ve
a
be
tt
e
r
a
bil
it
y
to
c
ompr
e
he
nd
c
ontext
a
nd
langua
ge
f
low
mor
e
de
e
ply
than
s
ingl
e
-
dir
e
c
ti
on
langua
ge
models
.
Although
it
wa
s
pr
e
vious
ly
not
pr
a
c
ti
c
a
ble,
bidi
r
e
c
ti
ona
l
tr
a
ini
ng
be
c
a
me
pos
s
ibl
e
be
c
a
us
e
of
mas
ke
d
L
M
(
M
L
M
)
on
whic
h
B
E
R
T
r
e
li
e
s
.
B
E
R
T
is
ve
r
s
a
ti
le,
a
nd
the
tas
ks
it
c
a
n
pe
r
f
or
m
a
r
e
numer
ous
.
Among
i
ts
a
ppli
c
a
ti
ons
,
no
table
e
xa
mpl
e
s
include
:
text
c
las
s
if
ica
ti
on
[
31]
,
que
s
ti
on
a
ns
we
r
ing
[
32]
,
na
tu
r
a
l
langua
ge
inf
e
r
e
nc
e
[
33
]
,
p
r
e
-
tr
a
ini
ng
langua
ge
r
e
pr
e
s
e
ntations
[
12]
,
s
e
nti
ment
a
na
lys
is
[
34]
,
f
a
ke
ne
ws
de
tec
ti
on
[
35
]
,
a
nd
mor
e
.
F
or
many
us
e
c
a
s
e
s
,
B
E
R
T
p
r
ove
d
to
pe
r
f
o
r
m
be
tt
e
r
than
many
models
.
How
e
ve
r
,
i
t
a
ls
o
pr
e
s
e
nts
s
e
v
e
r
a
l
li
mi
tations
,
s
uc
h
a
s
the
li
mi
t
on
the
s
ize
of
input
text
that
it
c
a
n
ha
ndle,
wh
ich
is
a
c
ha
ll
e
nge
whe
n
de
a
li
ng
with
long
doc
uments
or
s
umm
a
r
ies
of
he
f
ty
a
r
ti
c
les
[
36]
.
I
ts
voc
a
bular
y
i
s
f
ixed,
whic
h
pr
e
ve
nts
it
f
r
om
ha
ndli
ng
r
a
r
e
or
out
-
of
-
voc
a
bular
y
ter
ms
.
Although
it
c
a
n
be
f
ine
-
tuned
f
or
s
pe
c
if
ic
tas
ks
,
it
s
pe
r
f
or
manc
e
mi
ght
be
be
low
s
tanda
r
d
if
t
he
y
a
r
e
s
igni
f
ica
ntl
y
dif
f
e
r
e
nt
f
r
om
the
one
s
it
wa
s
tr
a
ined
on
[
37]
.
Anothe
r
li
mi
tation
of
B
E
R
T
is
it
s
high
c
omput
a
ti
ona
l
e
xpe
ns
e
,
e
s
pe
c
ially
whe
n
it
c
omes
t
o
lar
ge
r
models
.
B
E
R
T
c
a
nnot
be
dis
c
us
s
e
d
without
a
ddr
e
s
s
ing
the
t
r
a
ns
f
or
mer
a
r
c
hit
e
c
tur
e
,
whic
h
is
a
pivot
a
l
a
dva
nc
e
ment
in
na
tur
a
l
langua
ge
pr
oc
e
s
s
ing
(
NL
P
)
that
r
e
volut
ioni
z
e
d
the
wa
y
langua
ge
m
ode
ls
a
r
e
de
s
igned
a
nd
tr
a
ined.
I
t
wa
s
int
r
oduc
e
d
in
the
pa
pe
r
ti
tl
e
d
"
Attention
is
All
You
Ne
e
d"
by
Va
s
wa
ni
e
t
al
.
[
20]
a
nd
quickly
be
c
a
me
the
f
ounda
ti
on
f
or
many
s
tate
-
of
-
the
-
a
r
t
langua
ge
models
.
I
ts
ke
y
c
omponents
include
:
i
)
a
s
e
lf
-
a
tt
e
nti
on
mec
ha
nis
m,
whic
h
e
na
bles
the
model
to
unde
r
s
tand
de
pe
nde
nc
ies
be
twe
e
n
the
dif
f
e
r
e
nt
input
text
pa
r
ts
,
by
c
a
lcula
ti
ng
a
tt
e
nti
on
we
ight
s
f
or
e
a
c
h
token
;
ii
)
mul
ti
-
he
a
d
a
tt
e
nti
on,
a
nd
e
ve
r
y
s
ingl
e
he
a
d
lea
r
ns
dif
f
e
r
e
nt
r
e
lations
hips
be
twe
e
n
the
wor
ds
,
c
a
ptur
ing
d
if
f
e
r
e
nt
a
s
pe
c
ts
of
the
c
ontext,
p
os
it
ional
c
oding
;
iii
)
pos
it
ional
e
nc
odings
,
that
a
r
e
a
dde
d
t
o
the
wor
d
e
mbeddings
to
p
r
ovide
in
f
or
mation
a
bout
the
wor
d's
pos
it
ion
in
the
s
e
que
nc
e
;
iv
)
f
e
e
df
or
wa
r
d
ne
ur
a
l
ne
twor
ks
,
e
na
bli
ng
the
model
to
lea
r
n
c
ompl
e
x
int
e
r
a
c
ti
ons
a
nd
t
r
a
ns
f
or
mations
;
a
nd
v
)
laye
r
no
r
maliza
ti
on
a
nd
r
e
s
idual
c
onne
c
ti
ons
,
whic
h
a
r
e
te
c
hniques
us
e
d
to
s
tabili
z
e
tr
a
ini
ng
a
nd
e
na
ble
the
ne
twor
k
to
lea
r
n
e
f
f
e
c
ti
ve
ly
in
ve
r
y
de
e
p
a
r
c
hit
e
c
tur
e
s
.
S
ome
of
the
be
ne
f
it
s
o
f
the
tr
a
ns
f
or
mer
a
r
c
hit
e
c
tur
e
f
o
r
L
L
M
s
a
r
e
:
pa
r
a
ll
e
li
s
m,
mea
ning
the
s
e
lf
-
a
tt
e
nti
on
mec
ha
nis
m
a
ll
ows
the
model
to
pr
oc
e
s
s
dif
f
e
r
e
nt
pa
r
ts
of
the
input
s
e
que
nc
e
in
pa
r
a
ll
e
l,
whic
h
s
igni
f
ica
ntl
y
s
pe
e
ds
up
tr
a
ini
ng
a
nd
inf
e
r
e
nc
e
c
o
mpar
e
d
to
R
NN
s
or
C
NN
s
,
it
s
a
bil
it
y
to
c
a
ptur
e
l
ong
-
r
a
nge
de
pe
nde
nc
ies
,
making
it
idea
l
f
o
r
unde
r
s
tanding
a
wor
d
withi
n
a
n
e
nti
r
e
doc
ument
,
ge
ne
r
a
ti
ng
mor
e
c
ohe
r
e
nt
a
nd
c
ontextua
ll
y
a
ppr
opr
iate
text
be
c
a
us
e
it
c
a
ptur
e
s
c
ontextua
l
inf
or
mation
f
o
r
both
lef
t
a
nd
r
ight
o
f
a
wor
d,
a
nd
it
s
s
c
a
labili
ty,
s
o
it
ha
ndles
lar
ge
r
da
tas
e
ts
a
nd
mor
e
c
ompl
e
x
tas
ks
.
3.
M
E
T
HO
D
I
n
thei
r
e
nde
a
vor
to
a
s
s
e
s
s
the
pe
r
f
or
manc
e
o
f
th
e
thr
e
e
pr
omi
ne
nt
L
L
M
s
:
GPT
-
3.
5,
P
a
L
M
2,
a
nd
B
E
R
T
,
in
digi
tal
mar
ke
ti
ng,
the
a
uthor
s
e
mbar
ke
d
on
a
p
r
oc
e
s
s
of
f
ine
-
tuni
ng
thes
e
models
.
T
he
ir
pr
im
a
r
y
objec
ti
ve
c
e
nter
e
d
a
r
ound
the
ge
ne
r
a
ti
on
of
c
omp
e
ll
ing
e
mail
s
ubjec
ts
tailor
e
d
f
or
p
r
omot
ional
c
a
mpaigns
.
Not
only
that
,
but
the
e
mail
s
ubjec
ts
ne
e
de
d
to
l
a
nd
in
the
pr
im
a
r
y
inbox,
a
voidi
ng
the
s
pa
m
f
o
l
de
r
a
t
a
ll
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
E
lec
&
C
omp
E
ng
I
S
S
N:
2088
-
8708
E
mail
s
ubjec
ts
ge
ne
r
ati
on
w
it
h
lar
ge
language
mode
ls
:
GP
T
-
3
.
5,
P
aL
M
2
,
and
B
E
R
T
(
Soumay
a
L
o
uk
il
i)
4659
c
os
ts
.
T
his
invol
ve
d
r
e
f
ini
ng
the
models
us
ing
a
da
tas
e
t
that
e
nc
ompas
s
e
d
a
wide
r
a
nge
of
s
ubjec
t
li
ne
s
,
that
we
r
e
pr
e
vious
ly
tes
ted
out,
a
ll
owing
the
models
to
be
tt
e
r
c
ompr
e
he
nd
the
langua
ge
s
pe
c
if
ic
to
t
he
e
mail
mar
ke
ti
ng
domain.
T
hr
ough
thi
s
e
xpe
r
im
e
nt
,
the
a
im
is
to
d
is
s
e
c
t
a
nd
c
ontr
a
s
t
the
e
f
f
e
c
ti
ve
ne
s
s
of
e
a
c
h
L
L
M
's
pr
opos
e
d
e
mail
s
ubjec
ts
by
e
va
luating
th
e
pe
r
c
e
ntage
o
f
r
e
c
ipi
e
nts
who
e
nga
ge
by
ope
ning
thes
e
e
mails
.
T
his
c
ompr
e
he
ns
ive
a
na
lys
is
pr
omi
s
e
s
ins
ight
s
int
o
the
diver
s
e
c
a
pa
bil
it
ies
of
thes
e
L
L
M
s
in
c
r
a
f
ti
ng
s
ubjec
t
li
ne
s
that
r
e
s
ona
te
with
r
e
c
ipi
e
nts
a
nd
d
r
ive
higher
e
mail
ope
n
r
a
tes
.
F
or
thi
s
pa
pe
r
,
the
a
utho
r
s
ha
ve
pa
r
tner
e
d
wi
th
a
n
undis
c
los
e
d
e
mail
mar
ke
ti
ng
c
ompany,
whic
h
ha
s
gr
a
nted
us
a
c
c
e
s
s
to
a
c
ur
r
e
nt
da
taba
s
e
c
ompr
is
ing
644,
600
e
mail
s
ubjec
ts
.
T
his
da
tas
e
t
is
s
tr
uc
tur
e
d
a
s
s
uc
h:
the
a
dve
r
ti
s
e
d
pr
oduc
t
na
me,
a
nd
the
e
mail
s
ubjec
t
that’
s
be
e
n
us
e
d
in
the
c
a
mpaign
,
a
c
c
ompanie
d
by
c
or
r
e
s
ponding
s
tatis
ti
c
s
,
c
ompr
is
ing
the
number
of
de
li
ve
r
ies
a
nd
ope
nings
a
s
s
oc
iate
d
with
e
a
c
h
e
mail
s
ubjec
t,
whic
h
gives
us
the
ope
ning
r
a
te.
T
he
pe
r
c
e
ntage
of
ope
nings
is
the
main
mea
s
ur
e
of
the
i
mp
a
c
t
of
a
n
e
mail
s
ubjec
t,
a
s
it
is
the
f
ir
s
t
thi
ng
the
c
us
tom
e
r
s
e
e
s
.
Due
to
the
c
ompany's
pr
e
f
e
r
e
nc
e
f
or
a
nonymi
ty,
their
identit
y
r
e
mains
undis
c
los
e
d.
B
e
ginni
ng
with
the
a
f
or
e
mentioned
C
S
V
da
tas
e
t
c
ontaining
im
por
tant
inf
or
mat
ion
including
pr
oduc
t
na
mes
,
c
or
r
e
s
ponding
e
mail
s
ubjec
ts
,
a
s
we
ll
a
s
de
li
ve
r
y
a
nd
ope
n
s
tatis
ti
c
s
,
the
a
uthor
s
ini
ti
a
ted
the
c
us
tom
a
r
y
pha
s
e
of
da
ta
c
lea
ning.
T
his
invol
v
e
d
a
c
ti
ons
s
uc
h
a
s
e
li
mi
na
ti
ng
blanks
,
a
nd
a
ddr
e
s
s
ing
incons
is
tenc
ie
s
to
e
ns
ur
e
the
da
tas
e
t's
int
e
gr
it
y.
S
ubs
e
que
ntl
y,
they
e
xe
c
uted
a
s
tr
a
tegic
t
r
a
ns
f
or
mation,
c
onve
r
ti
ng
e
a
c
h
da
tas
e
t
e
ntr
y
int
o
a
pa
ir
of
s
pe
c
ialize
d
pr
ompt
a
nd
output
s
tr
uc
tu
r
e
.
T
he
s
e
pr
ompt
s
we
r
e
meticulous
ly
de
s
igned
to
s
oli
c
it
e
mail
s
ubje
c
t
s
ug
ge
s
ti
ons
f
or
a
given
pr
oduc
t,
a
im
ing
to
a
c
hieve
a
s
pe
c
if
ied
tar
ge
t
ope
ning
r
a
te.
T
he
s
e
p
r
ompt
s
we
r
e
c
oupl
e
d
with
their
c
or
r
e
s
ponding
a
ns
we
r
s
,
the
or
igi
n
a
l
e
mail
s
ubjec
ts
.
L
a
ngC
ha
in
wa
s
us
e
d
f
or
thi
s
.
T
he
objec
ti
ve
be
hind
L
a
ngC
ha
in
is
to
pr
e
pa
r
e
da
ta
to
be
a
c
c
e
s
s
e
d
by
the
L
L
M
with
the
lea
s
t
a
mount
of
c
omput
a
ti
on
pos
s
ibl
e
.
I
t
doe
s
that
by
br
e
a
king
down
da
ta
int
o
s
maller
“
c
hunks
”
that
ge
t
e
mbedd
e
d
int
o
a
ve
c
tor
s
tor
e
[
38]
.
M
oving
f
or
wa
r
d
,
whe
n
it
is
ti
me
to
c
r
e
a
te
the
pr
ompt
-
c
ompl
e
ti
on
pa
ir
,
the
ve
c
tor
ize
d
r
e
pr
e
s
e
ntations
of
the
lar
ge
doc
ument
c
a
n
be
us
e
d
in
a
ddit
ion
to
the
L
L
M
.
L
a
ngC
ha
in
a
ls
o
f
a
c
il
it
a
tes
other
API
-
r
e
late
d
tas
ks
,
s
uc
h
a
s
s
ur
f
ing
the
we
b,
s
e
ndin
g
e
mails
a
nd
c
r
e
a
ti
ng
pr
ompt
s
.
F
inetuning
L
L
M
s
invol
ve
s
re
-
tr
a
ini
ng
pr
e
-
tr
a
ined
models
on
e
xtens
ive
da
tas
e
ts
,
whic
h
will
c
us
tom
ize
the
model
to
be
e
f
f
icie
nt
a
t
the
s
pe
c
if
ic
tas
k
a
t
ha
nd
[
39]
.
T
his
tec
hnique
ha
s
r
e
v
olut
ioni
z
e
d
na
tur
a
l
langua
ge
pr
oc
e
s
s
ing
(
NL
P
)
by
a
da
pti
ng
pr
e
-
tr
a
ined
models
to
s
pe
c
if
ic
a
ppli
c
a
ti
on
s
,
s
uc
h
a
s
que
s
ti
on
-
a
ns
w
e
r
ing,
langua
ge
tr
a
ns
lation,
na
med
-
e
nti
ty
r
e
c
ognit
ion,
doc
ument
s
umm
a
r
iza
ti
on,
s
e
nti
ment
a
na
lys
is
,
a
nd
mor
e
[
40]
.
F
or
the
f
ine
-
tuni
ng
pr
oc
e
s
s
of
P
a
L
M
2
,
the
a
uthor
s
c
hos
e
the
text
-
bis
on@001
model
a
s
it
s
objec
ti
ve
s
a
r
e
the
c
los
e
s
t
to
their
s
,
a
nd
they
s
tr
a
tegic
a
ll
y
r
a
n
3
e
poc
hs
,
meticulous
ly
c
hos
e
n
to
a
li
gn
with
the
da
tas
e
t's
c
ha
r
a
c
ter
is
ti
c
s
a
nd
the
ba
tch
s
ize
a
nd
s
teps
.
B
y
u
t
il
izing
the
Ada
m
opti
m
ize
r
with
a
de
f
a
ult
lea
r
nin
g
r
a
te
o
f
0.
001,
the
a
uthor
s
e
ns
ur
e
d
opti
mal
c
onve
r
ge
nc
e
dur
ing
t
r
a
ini
ng.
Additi
ona
ll
y
,
thei
r
i
mpl
e
m
e
ntation
int
e
gr
a
ted
T
e
ns
or
B
oa
r
d,
e
na
bli
ng
thor
ough
moni
tor
ing
a
nd
a
na
lys
is
of
tr
a
ini
ng
metr
ics
,
thus
f
a
c
il
it
a
ti
ng
c
ompr
e
he
ns
ive
pe
r
f
or
manc
e
e
va
luation
pos
t
-
tr
a
ini
ng.
T
o
e
ns
ur
e
a
n
a
c
c
ur
a
te
a
nd
f
a
ir
c
ompar
is
on,
the
a
u
thor
s
e
xtende
d
their
f
ine
-
tuni
ng
e
f
f
or
ts
to
include
the
GPT
-
3.
5
a
nd
B
e
r
t
models
,
e
mpl
oying
s
im
il
a
r
hype
r
pa
r
a
mete
r
s
a
s
thos
e
uti
li
z
e
d
f
or
P
a
L
M
2
,
t
he
s
a
me
opti
mi
z
e
r
,
lea
r
ning
r
a
te
a
nd
the
number
of
s
teps
.
T
his
a
ppr
oa
c
h
gua
r
a
ntee
s
that
the
c
ompar
is
on
be
twe
e
n
P
a
L
M
2,
GP
T
-
3.
5,
a
nd
B
e
r
t
is
c
onduc
ted
unde
r
c
ons
is
tent
c
ondit
ions
,
a
ll
owing
f
o
r
a
mea
ningf
ul
a
s
s
e
s
s
ment
of
their
r
e
s
pe
c
ti
ve
pe
r
f
or
manc
e
a
nd
e
f
f
e
c
ti
ve
ne
s
s
in
a
ddr
e
s
s
ing
the
a
uthor
s
’
objec
ti
ve
s
.
T
o
c
ompr
e
he
ns
ively
e
va
luate
the
pe
r
f
or
manc
e
of
the
f
inetune
d
L
L
M
s
,
B
L
E
U
a
nd
R
OU
GE
s
c
or
e
s
we
r
e
us
e
d
to
c
ompar
e
.
Although
the
a
uthor
s
ha
ve
the
model
a
c
c
ur
a
c
y
in
ha
nd,
us
ing
i
t
f
o
r
text
ge
ne
r
a
ti
on,
a
s
oppos
e
d
to
c
las
s
if
ica
ti
on
or
other
,
is
mea
ningl
e
s
s
.
T
ha
t
is
be
c
a
us
e
text
ge
ne
r
a
ti
on
a
ll
ows
f
or
c
r
e
a
ti
ve
f
r
e
e
dom,
un
li
ke
other
tas
ks
whe
r
e
the
a
ns
we
r
is
e
it
he
r
r
ight
or
wr
ong
.
B
L
E
U
a
nd
R
OU
GE
-
L
S
um
a
r
e
wide
ly
e
mpl
oye
d
e
va
luation
metr
ics
in
na
tu
r
a
l
langu
a
ge
pr
oc
e
s
s
ing,
notably
f
or
mac
hine
t
r
a
ns
lation
a
nd
s
umm
a
r
iza
ti
on.
B
L
E
U
a
s
s
e
s
s
e
s
the
e
xtent
of
ov
e
r
lap
be
twe
e
n
a
r
e
f
e
r
e
nc
e
s
e
ntenc
e
(
X)
a
nd
a
c
a
ndidate
s
e
ntenc
e
(
Y)
[
41]
,
while
R
OU
GE
-
L
S
um
e
va
luate
s
the
longes
t
c
omm
on
s
ubs
e
qu
e
nc
e
of
tokens
b
e
twe
e
n
a
c
a
ndidate
(
Y)
a
nd
r
e
f
e
r
e
nc
e
(
X
)
,
s
pe
c
if
ica
ll
y
de
s
igned
f
or
a
s
s
e
s
s
ing
s
umm
a
r
y
qua
li
ty.
Af
ter
wa
r
d,
to
s
e
e
the
pe
r
f
or
manc
e
of
the
f
inetun
e
d
models
in
r
e
a
l
li
f
e
,
the
a
uthor
s
opted
to
tes
t
3
pr
oduc
ts
that
ha
d
unde
r
gone
pr
e
vious
human
-
de
s
igned
c
a
mpaigns
a
nd
a
c
hieve
d
dif
f
e
r
e
nt
ope
ning
r
a
tes
,
a
nd
ge
ne
r
a
ted
e
mail
s
ubjec
ts
to
p
r
omot
e
e
a
c
h
us
ing
t
he
thr
e
e
f
inetune
d
L
L
M
s
:
GPT
-
3.
5,
P
a
L
M
2,
a
n
d
B
E
R
T
.
T
he
s
e
s
ubjec
ts
we
r
e
then
dis
tr
ibut
e
d
to
a
n
e
quivale
nt
number
of
e
mail
a
ddr
e
s
s
e
s
.
W
hil
e
the
a
ddr
e
s
s
e
s
we
r
e
not
identica
l,
they
did
s
ha
r
e
a
de
mogr
a
phic
s
i
mi
lar
it
y,
e
ns
ur
ing
a
c
ons
is
tent
c
ontext
f
or
c
om
pa
r
is
on.
Additi
ona
ll
y,
a
ll
the
s
e
lec
ted
a
ddr
e
s
s
e
s
we
r
e
of
e
xc
e
ll
e
nt
qua
li
ty
,
mar
ke
d
by
thei
r
a
c
ti
ve
e
nga
ge
ment
a
nd
r
e
c
e
nt
int
e
r
a
c
ti
ons
.
T
his
meticulous
a
ppr
oa
c
h
a
ll
o
we
d
the
a
uthor
s
to
ga
uge
the
L
L
M
s
'
e
f
f
ica
c
it
y
in
ge
ne
r
a
ti
ng
s
ubjec
t
li
ne
s
that
r
ival
both
human
e
f
f
or
ts
a
nd
e
a
c
h
other
.
T
o
s
um
it
up,
thes
e
models
we
r
e
e
va
l
ua
ted
by
s
e
ve
r
a
l
methods
:
us
ing
the
c
u
s
tom
a
r
y
B
L
E
U
a
nd
R
OU
GE
s
c
or
e
s
,
their
tr
a
ini
ng
a
c
c
ur
a
c
y,
a
nd
f
inall
y
tes
ti
ng
their
ge
ne
r
a
ted
s
ubjec
t
li
ne
s
in
r
e
a
l
li
f
e
,
c
a
lcula
ti
ng
their
ope
n
r
a
tes
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2088
-
8708
I
nt
J
E
lec
&
C
omp
E
ng
,
Vol
.
1
4
,
No.
4
,
Augus
t
20
2
4
:
4655
-
4663
4660
4.
RE
S
UL
T
S
AN
D
DI
S
CU
S
S
I
ON
R
e
s
ult
s
we
r
e
meticulous
ly
ga
th
e
r
e
d
ove
r
3
da
ys
f
oll
owing
the
c
a
mpaign
launc
he
s
,
e
ns
ur
ing
a
r
obus
t
da
tas
e
t
f
or
a
na
lys
is
.
T
he
c
hos
e
n
ti
mef
r
a
me
a
ll
owe
d
f
o
r
a
c
omp
r
e
he
ns
ive
a
s
s
e
s
s
ment
of
r
e
c
ipi
e
nt
e
ng
a
ge
ment
while
mi
nim
izing
the
im
pa
c
t
of
da
il
y
f
luctua
ti
o
ns
.
R
igor
ous
va
li
da
ti
on
pr
oc
e
dur
e
s
we
r
e
im
plem
e
nted
to
maintain
the
a
c
c
ur
a
c
y
of
the
f
inal
f
indi
ngs
.
T
he
s
e
include
d
va
li
da
ti
ng
the
e
mail
li
s
t
by
c
lea
ning
it
f
r
om
inac
ti
ve
or
invalid
a
ddr
e
s
s
e
s
,
im
pleme
nti
ng
pe
r
m
i
s
s
ion
-
b
a
s
e
d
opt
-
in,
ha
ndli
ng
uns
ubs
c
r
ibi
ng,
a
nd
t
r
a
c
ing
the
a
c
ti
ons
on
the
e
mail,
s
uc
h
a
s
ope
ning,
s
e
nding
to
s
pa
m
a
nd
c
li
c
king
on
the
li
nk
.
T
a
ble
1
pr
e
s
e
nts
thr
e
e
pe
r
f
or
manc
e
metr
ics
f
or
th
e
f
inetune
d
langua
ge
models
:
P
a
L
M
2,
GPT
-
3.
5,
a
nd
B
E
R
T
,
r
e
p
r
e
s
e
nted
a
s
c
olum
ns
,
while
e
a
c
h
r
ow
r
e
pr
e
s
e
nts
a
d
if
f
e
r
e
nt
e
va
luation
metr
ic
:
B
L
E
U
s
c
or
e
,
R
OU
GE
-
L
S
um
s
c
or
e
,
a
nd
tr
a
ini
ng
a
c
c
ur
a
c
y.
T
h
e
e
va
luation
r
e
s
ult
s
s
how
that
both
B
E
R
T
a
nd
P
a
L
M
2
outper
f
or
m
GP
T
-
3.
5.
A
higher
B
L
E
U
s
c
or
e
indi
c
a
tes
be
tt
e
r
p
r
e
c
is
ion,
while
a
highe
r
R
OU
GE
-
L
S
u
m
s
c
or
e
s
igni
f
ies
im
pr
ove
d
r
e
c
a
ll
.
T
he
r
e
f
or
e
,
the
f
inetune
d
GPT
-
3.
5
model
e
xhibi
ts
the
lowe
s
t
p
r
e
c
is
ion,
r
e
c
a
ll
,
a
nd
a
c
c
ur
a
c
y
a
mong
the
thr
e
e
models
whe
n
c
om
pa
r
e
d
a
ga
ins
t
human
-
c
r
e
a
ted
c
ontent.
T
he
pe
r
f
or
manc
e
va
r
iations
a
c
r
os
s
thes
e
metr
ics
a
r
e
a
tt
r
ibut
e
d
to
e
a
c
h
model's
unique
c
a
pa
bil
it
ies
.
B
E
R
T
's
bidi
r
e
c
ti
ona
l
c
ontext
c
ompr
e
he
ns
ion
e
na
bles
it
to
e
xc
e
l
in
ta
s
ks
li
ke
e
mail
s
ubjec
t
ge
ne
r
a
ti
on,
r
e
s
ult
ing
in
s
upe
r
ior
B
L
E
U
a
nd
R
OU
GE
-
L
S
um
s
c
or
e
s
.
P
a
L
M
2,
opti
mi
z
e
d
f
or
c
onve
r
s
a
ti
ona
l
int
e
r
a
c
ti
ons
,
a
c
hieve
s
s
li
ght
ly
lowe
r
pe
r
f
or
manc
e
leve
ls
.
On
the
other
ha
nd,
de
s
pit
e
it
s
e
xtens
ive
knowle
dge
ba
s
e
,
GPT
-
3.
5
may
lac
k
f
in
e
-
tuni
ng
f
or
thi
s
s
pe
c
if
ic
tas
k,
lea
ding
to
ove
r
a
ll
lowe
r
s
c
or
e
s
.
T
a
ble
1.
F
inetune
d
L
L
M
s
B
L
E
U
a
nd
R
OU
GE
-
L
S
um
s
c
or
e
s
a
nd
tr
a
ini
ng
a
c
c
ur
a
c
y
P
a
L
M
2
G
P
T
-
3.5
B
E
R
T
B
L
E
U
s
c
or
e
0.59
0.32
0.62
R
O
U
G
E
-
L
S
um s
c
or
e
0.72
0.58
0.76
T
r
a
in
in
g a
c
c
ur
a
c
y
0.66
0.51
0.68
T
he
e
mail
ope
n
r
a
te
,
a
ke
y
metr
ic
in
mar
ke
ti
ng
,
mea
s
ur
e
s
the
pe
r
c
e
ntage
of
r
e
c
ipi
e
nts
who
ope
n
c
omm
e
r
c
ial
e
mails
they
r
e
c
e
ive.
I
t
is
c
a
lcula
ted
b
y
divi
ding
the
numbe
r
o
f
ope
ne
d
e
mails
by
the
tot
a
l
e
mails
de
li
ve
r
e
d.
F
a
c
tor
s
inf
luenc
ing
ope
n
r
a
tes
include
indus
tr
y
type,
ti
mi
ng
of
de
li
ve
r
y
,
a
nd
e
mail
s
ubjec
t
li
ne
s
.
T
o
e
va
luate
the
im
pa
c
t
of
s
ubjec
t
li
ne
qua
li
ty
,
th
e
a
uthor
s
maintaine
d
c
ons
is
tent
pa
r
a
mete
r
s
a
nd
c
ompar
e
d
ope
n
r
a
tes
be
twe
e
n
e
mails
with
s
ubjec
t
li
ne
s
ge
ne
r
a
ted
by
the
f
inetune
d
models
a
ga
ins
t
thos
e
c
r
a
f
ted
by
humans
.
T
a
ble
2
pr
e
s
e
nts
the
ope
ning
r
a
tes
o
f
pr
omot
ional
e
mails
f
or
thr
e
e
dif
f
e
r
e
nt
pr
oduc
ts
,
of
whic
h
the
s
ubjec
ts
a
r
e
ge
ne
r
a
ted
by
the
f
inetune
d
L
L
M
s
.
T
he
r
e
s
ult
s
a
r
e
a
s
f
oll
ows
:
a
ll
thr
e
e
tes
ted
L
L
M
s
,
na
mely
GPT
-
3.
5,
B
E
R
T
,
a
nd
P
a
L
M
2,
de
mons
tr
a
ted
s
upe
r
ior
pe
r
f
o
r
manc
e
whe
n
c
ompar
e
d
to
human
-
ge
ne
r
a
ted
e
f
f
or
ts
.
How
e
ve
r
,
a
c
los
e
r
e
xa
mi
na
ti
on
of
the
o
pe
ning
r
a
tes
r
e
ve
a
ls
that
B
E
R
T
a
nd
GPT
-
3.
5
e
xhibi
ted
r
e
mar
ka
bly
s
im
il
a
r
r
e
s
ult
s
,
s
howc
a
s
ing
their
c
om
pa
r
a
ble
e
f
f
e
c
ti
ve
ne
s
s
in
c
r
a
f
ti
ng
e
nga
ging
s
ubjec
t
li
ne
s
.
I
n
c
ontr
a
s
t,
P
a
L
M
2,
while
s
ti
ll
outper
f
or
mi
ng
hu
man
-
ge
ne
r
a
ted
s
ubje
c
ts
,
dis
playe
d
s
li
ghtl
y
lowe
r
ope
ning
r
a
tes
,
indi
c
a
ti
ng
a
modes
t
ga
p
in
it
s
a
bil
i
ty
to
c
a
ptur
e
r
e
c
ipi
e
nt
a
tt
e
nti
on
c
ompar
e
d
to
it
s
A
I
c
ounter
pa
r
ts
.
T
he
s
e
f
indi
ngs
unde
r
s
c
or
e
the
a
dva
nc
e
ments
in
AI
-
dr
iven
text
ge
ne
r
a
ti
on
a
nd
their
potential
t
o
s
ur
pa
s
s
human
-
ge
ne
r
a
ted
c
ontent
in
opti
mi
z
ing
e
mail
mar
ke
ti
ng
s
tr
a
tegie
s
.
T
he
s
e
r
e
s
ult
s
go
ha
nd
in
ha
nd
with
the
B
L
UE
a
nd
R
OU
GE
s
c
or
e
s
e
a
c
h
f
inetune
d
model
ha
d,
a
s
the
f
inetune
d
B
E
R
T
model
e
mul
a
tes
hu
man
-
li
ke
r
e
s
pons
e
s
the
b
e
s
t,
r
e
pli
c
a
ti
ng
the
de
f
icie
nc
ies
a
s
we
ll
.
C
ons
ider
ing
that
the
human
-
ge
ne
r
a
ted
c
onte
nt
it
s
e
lf
ha
s
pe
r
f
or
med
the
wor
s
t
c
ompar
e
d
to
other
s
,
thi
s
s
uppor
ts
the
obtaine
d
r
e
s
ult
s
.
All
thr
e
e
of
thes
e
L
L
M
s
a
r
e
im
pr
e
s
s
ive
in
ter
ms
of
tas
ks
they
c
a
n
pe
r
f
or
m
,
a
nd
the
pe
r
f
o
r
manc
e
they
pr
e
s
e
nted.
How
e
ve
r
,
e
a
c
h
of
them
ha
s
it
s
own
s
tr
e
ngths
,
whic
h
led
to
the
dis
c
r
e
pa
nc
y
in
the
r
e
s
ult
s
.
Due
to
it
s
bidi
r
e
c
ti
ona
l
na
tur
e
,
B
E
R
T
is
known
to
be
mor
e
a
dva
ntage
ous
in
ter
ms
of
na
tur
a
l
langua
ge
unde
r
s
tanding
(
NL
U
)
a
nd
s
e
nti
ment
a
na
lys
is
,
a
s
oppos
e
d
to
GPT
a
nd
P
a
L
M
2
whic
h
pe
r
f
or
m
be
tt
e
r
a
t
c
omm
on
s
e
ns
e
tas
ks
,
pr
a
gmatic
inf
e
r
e
nc
e
,
a
nd
text
ge
ne
r
a
ti
on.
An
i
mpor
tant
pil
la
r
o
f
digi
tal
mar
ke
ti
ng
is
unde
r
s
ta
nding
the
c
us
tom
e
r
s
’
point
of
view
a
nd
s
howing
them
e
nga
ging
c
ontent
that
will
lea
d
to
c
onve
r
s
ions
e
ve
ntually,
whic
h
c
ould
e
xplain
why
B
E
R
T
wa
s
be
tt
e
r
.
Als
o,
the
va
r
iations
in
ope
ning
r
a
tes
c
ould
be
a
r
e
s
ult
of
dif
f
e
r
e
nt
opti
m
iza
ti
on
tec
hniques
e
mpl
oye
d
f
or
e
a
c
h
type
of
s
ubjec
t
li
ne
.
W
hil
e
A
I
models
li
k
e
P
a
L
M
2,
GPT
-
3.
5,
a
nd
B
E
R
T
c
a
n
unde
r
go
i
ter
a
ti
ve
A/B
tes
ti
ng
a
t
a
lar
ge
r
s
c
a
le
a
nd
with
mor
e
da
ta,
hum
a
ns
may
ha
ve
li
mi
tations
in
ter
ms
o
f
c
onduc
ti
ng
e
xtens
ive
e
xpe
r
im
e
nts
.
AI
models
c
a
n
f
ine
-
tune
s
ubjec
t
li
ne
s
ba
s
e
d
on
mas
s
ive
da
ta
they
a
na
lyze
a
nd
lea
r
n
the
pa
tt
e
r
ns
of
wha
t
wor
ks
a
nd
wha
t
doe
s
not.
Huma
n
-
ge
ne
r
a
ted
s
ubjec
t
l
ines
may
not
unde
r
go
the
s
a
me
leve
l
of
r
igor
ous
opti
mi
z
a
ti
on,
lea
ding
to
lowe
r
ope
ning
r
a
tes
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
E
lec
&
C
omp
E
ng
I
S
S
N:
2088
-
8708
E
mail
s
ubjec
ts
ge
ne
r
ati
on
w
it
h
lar
ge
language
mode
ls
:
GP
T
-
3
.
5,
P
aL
M
2
,
and
B
E
R
T
(
Soumay
a
L
o
uk
il
i)
4661
T
he
s
e
L
L
M
s
ha
ve
one
major
thi
ng
in
c
omm
on:
they
a
ll
ha
ve
the
tr
a
ns
f
o
r
mer
a
r
c
hit
e
c
tur
e
a
s
a
f
ounda
ti
on.
T
his
a
r
c
hit
e
c
tur
e
r
e
volut
ioni
z
e
d
the
N
L
P
f
ield
a
nd
a
ppe
a
r
s
to
be
going
nowhe
r
e
,
a
s
mo
s
t
r
e
c
e
nt
L
L
M
s
a
r
e
ba
s
e
d
on
it
a
nd
a
r
e
d
r
if
ti
ng
a
wa
y
f
r
om
s
e
que
nti
a
l
pr
oc
e
s
s
ing
models
,
s
uc
h
a
s
R
NN
s
a
nd
L
S
T
M
s
f
or
ins
tanc
e
.
T
ha
t
is
be
c
a
us
e
tr
a
ns
f
or
mer
’
s
ke
y
innovation,
a
tt
e
nti
on
mec
ha
nis
ms
,
im
pr
ove
s
the
model’
s
a
bil
it
y
to
ha
ndle
long
-
r
a
nge
de
pe
nde
nc
ies
withi
n
t
e
xt,
whic
h
is
idea
l
f
or
text
ge
ne
r
a
ti
on
,
a
mong
other
tas
ks
.
I
t
a
ppe
a
r
s
f
r
om
the
r
e
s
ult
s
of
the
c
ur
r
e
nt
s
tudy
that
incor
por
a
ti
ng
L
L
M
s
int
o
e
mail
mar
ke
ti
ng
c
a
mpaigns
ha
s
pr
oduc
e
d
pos
it
ive
outcome
s
.
Ho
we
ve
r
,
the
e
volvi
ng
na
tur
e
of
tec
hnology
a
nd
c
ons
umer
be
ha
vior
point
s
to
s
e
ve
r
a
l
pr
omi
s
ing
dir
e
c
ti
ons
f
o
r
f
u
r
ther
s
tudy
a
nd
r
e
a
l
-
wor
ld
a
ppli
c
a
ti
on.
F
ir
s
t,
c
onti
nue
d
im
pr
ove
ments
in
L
L
M
s
,
s
uc
h
a
s
the
de
ve
lopm
e
nt
of
mor
e
c
ompl
e
x
a
nd
domain
-
s
pe
c
if
ic
mod
e
ls
,
may
im
pr
ove
their
e
f
f
e
c
ti
ve
ne
s
s
in
c
r
e
a
ti
ng
int
r
igui
ng
e
mail
s
ubjec
t
li
ne
s
.
M
or
e
ove
r
,
a
s
pr
ivac
y
r
e
gulation
s
e
volve
a
nd
c
ons
umer
pr
e
f
e
r
e
nc
e
s
c
ha
nge
,
inves
ti
ga
ti
ng
the
e
thi
c
a
l
c
ons
ider
a
ti
ons
a
nd
pe
r
s
ona
li
z
a
ti
on
c
ha
ll
e
nge
s
a
s
s
oc
iate
d
with
AI
-
ge
ne
r
a
ted
c
ontent
r
e
mains
i
m
pe
r
a
ti
ve
.
F
u
r
ther
mo
r
e
,
e
xa
mi
ning
the
long
-
ter
m
e
f
f
e
c
ts
of
AI
-
ge
ne
r
a
ted
e
mail
s
ubjec
ts
on
c
us
tom
e
r
e
nga
ge
ment
a
nd
br
a
nd
loyalty
will
pr
ovide
a
c
ompr
e
he
ns
ive
unde
r
s
tanding
of
their
r
ole
in
s
ha
ping
f
utur
e
mar
k
e
ti
ng
s
tr
a
tegie
s
,
a
nd
may
be
a
ppli
e
d
to
o
ther
c
om
pone
nts
:
e
mail
body,
ti
tl
e
s
of
a
ds
a
nd
a
r
ti
c
le
t
it
les
.
T
a
ble
2.
E
mails
ope
n
r
a
tes
with
L
L
M
s
a
nd
human
ge
ne
r
a
ted
s
ubjec
ts
O
pe
ni
ng r
a
te
P
a
L
M
2
G
P
T
-
3.5
B
E
R
T
H
uma
n G
e
ne
r
a
te
d
P
r
oduc
t
1
39.5%
44.8%
47.7%
28.4%
P
r
oduc
t
2
14.6%
21.7%
20.3%
6.8%
P
r
oduc
t
3
28.7%
31.4%
33.8%
9.2%
5.
CONC
L
USI
ON
I
n
s
umm
a
r
y,
thi
s
pa
pe
r
e
xplo
r
e
d
the
e
f
f
ica
c
y
of
e
mail
s
ubjec
t
li
ne
s
ge
ne
r
a
ted
by
L
L
M
s
in
the
c
ontext
of
dig
it
a
l
mar
ke
ti
ng
,
mor
e
s
pe
c
if
ica
ll
y
e
mail
mar
ke
ti
ng
c
a
mpaigns
.
T
he
pr
im
a
r
y
objec
ti
v
e
s
of
thi
s
s
tudy
we
r
e
to
a
s
s
e
s
s
the
pe
r
f
or
manc
e
of
L
L
M
s
,
na
mely
GPT
-
3.
5,
B
E
R
T
,
a
nd
P
a
L
M
2,
in
c
ompar
is
on
to
human
-
ge
ne
r
a
ted
s
ubje
c
t
li
ne
s
,
a
nd
to
dis
c
e
r
n
the
f
a
c
tor
s
c
ontr
ibut
ing
to
va
r
iations
in
ope
ning
r
a
te
s
a
c
r
os
s
dif
f
e
r
e
nt
pr
oduc
ts
.
T
he
f
ind
ings
ha
ve
r
e
ve
a
led
that
a
ll
thr
e
e
L
L
M
s
s
ur
pa
s
s
e
d
human
-
ge
n
e
r
a
ted
c
ontent
in
ter
ms
of
c
a
ptur
ing
r
e
c
ipi
e
nt
a
tt
e
nti
on,
s
igni
f
yi
ng
the
potential
o
f
AI
-
dr
iven
text
ge
ne
r
a
ti
on
in
e
mail
mar
ke
ti
ng.
M
or
e
ove
r
,
whi
le
GP
T
-
3.
5
a
nd
B
E
R
T
de
mons
tr
a
ted
s
im
il
a
r
pe
r
f
o
r
manc
e
,
P
a
L
M
2
e
xh
ibi
ted
a
s
li
ghtl
y
lowe
r
ope
n
r
a
te,
s
ugge
s
ti
ng
r
oom
f
or
i
mpr
ove
ment.
T
he
dis
c
us
s
ion
e
nc
ompas
s
e
d
f
a
c
tor
s
s
uc
h
a
s
s
e
mantic
c
ohe
r
e
nc
e
,
ps
yc
hologi
c
a
l
tr
igger
s
,
a
nd
opti
mi
z
a
ti
on
s
tr
a
tegie
s
,
s
he
dding
li
ght
on
the
me
c
ha
nis
ms
be
hind
the
obs
e
r
ve
d
r
e
s
ult
s
.
L
ooking
a
he
a
d,
the
dyna
mi
c
lands
c
a
pe
of
AI
a
nd
e
mail
mar
ke
ti
ng
pr
e
s
e
nts
e
xc
it
ing
a
ve
nue
s
f
or
f
utur
e
r
e
s
e
a
r
c
h,
including
the
de
ve
lopm
e
nt
of
domain
-
s
pe
c
if
ic
L
L
M
s
,
the
e
xplor
a
ti
on
of
hybr
id
AI
-
human
a
ppr
oa
c
he
s
,
a
nd
the
e
xa
mi
na
ti
on
of
e
thi
c
a
l
c
ons
ider
a
ti
ons
.
As
thi
s
e
volvi
ng
f
ield
is
ope
ning
to
mor
e
a
nd
mor
e
oppor
tuni
ti
e
s
,
thi
s
s
tudy
c
ont
r
i
butes
va
luable
ins
ight
s
to
guide
mar
ke
ter
s
a
nd
r
e
s
e
a
r
c
he
r
s
towa
r
d
a
de
e
pe
r
unde
r
s
tanding
o
f
A
I
's
r
ole
in
s
ha
ping
the
f
utu
r
e
of
e
mail
ma
r
ke
ti
ng
s
tr
a
tegie
s
.
RE
F
E
RE
NC
E
S
[
1]
P
.
va
n
E
s
c
h
a
nd
J
.
S
te
w
a
r
t
B
la
c
k,
“
A
r
ti
f
ic
ia
l
in
te
ll
ig
e
nc
e
(
A
I
)
:
r
e
vol
ut
io
ni
z
in
g
di
gi
ta
l
ma
r
ke
ti
ng,”
A
us
tr
al
as
ia
n
M
ar
k
e
ti
ng
J
our
nal
, vol
. 29, no. 3, pp. 199
–
203, Aug. 2021, doi:
10.1177/183933492
11037684.
[
2]
A
.
K
.
K
us
hw
a
ha
a
nd
A
.
K
.
K
a
r
,
“
L
a
ngua
ge
mod
e
l
-
dr
iv
e
n
c
ha
t
bot
f
or
bus
in
e
s
s
to
a
ddr
e
s
s
ma
r
ke
ti
ng
a
nd s
e
le
c
ti
on
of
pr
oduc
t
s
,”
in
I
F
I
P
A
dv
anc
e
s
in
I
nf
or
m
at
io
n
and
C
om
m
uni
c
at
io
n
T
e
c
hnol
ogy
,
vol
.
617,
S
pr
in
ge
r
I
nt
e
r
na
ti
ona
l
P
ubl
is
hi
ng,
2
020,
pp. 16
–
28.
[
3]
M
.
F
ir
a
t,
“
H
ow
c
ha
t
G
P
T
c
a
n
tr
a
ns
f
or
m
a
ut
odi
da
c
ti
c
e
xpe
r
ie
nc
e
s
a
nd
ope
n
e
duc
a
ti
on?
f
ut
ur
e
of
e
duc
a
ti
on
vi
e
w
pr
oj
e
c
t
vi
s
ua
l
pe
r
c
e
pt
io
n i
n e
duc
a
ti
ona
l
de
s
ig
n vi
e
w
pr
oj
e
c
t,
”
J
a
n. 2023.
[
4]
S
.
M
e
r
it
y,
N
.
S
.
K
e
s
ka
r
,
a
nd
R
.
S
oc
he
r
,
“
A
n
a
na
ly
s
is
of
ne
ur
a
l
la
ngua
ge
mode
li
ng
a
t
mul
ti
pl
e
s
c
a
le
s
,”
ar
X
iv
pr
e
p
r
in
t
ar
X
iv
:
1803.08240
, M
a
r
. 2018.
[
5]
O
.
J
. B
.
D
. W
a
lk
I
v, T
.
S
un,
A
.
P
e
r
ot
te
,
a
nd
N
.
E
lh
a
da
d, “
C
li
ni
c
a
ll
y
r
e
le
va
nt
pr
e
tr
a
in
in
g
is
a
ll
you
ne
e
d,”
J
our
nal
of
th
e
A
m
e
r
i
c
an
M
e
di
c
al
I
nf
or
m
at
ic
s
A
s
s
oc
ia
ti
on
, vol
. 28, no. 9, pp. 1970
–
1976,
J
un. 2021, doi:
10.1093/j
a
mi
a
/o
c
a
b086.
[
6]
O
.
M
é
nde
z
-
L
uc
io
,
C
.
N
ic
ol
a
ou,
a
nd
B
.
E
a
r
ns
ha
w
,
“
M
ol
E
:
a
mol
e
c
ul
a
r
f
ounda
ti
on
mode
l
f
or
dr
ug
di
s
c
ove
r
y,”
ar
X
iv
pr
e
pr
in
t
ar
X
iv
:
2211.02657
, N
ov. 2022.
[
7]
R
.
C
ha
ndr
a
a
nd
R
.
S
a
in
i,
“
B
id
e
n
vs
t
r
ump:
mode
li
ng
U
S
ge
ne
r
a
l
e
le
c
ti
ons
us
in
g
B
E
R
T
la
ngua
ge
mode
l,
”
I
E
E
E
A
c
c
e
s
s
,
vol
.
9,
pp. 128494
–
128505, 2021, doi:
10.1109/AC
C
E
S
S
.2021.31110
35.
[
8]
R
.
R
os
e
nf
e
ld
,
“
T
w
o
de
c
d
e
s
of
s
ta
ti
s
ti
c
a
l
la
ngua
ge
mode
li
ng
w
he
r
e
do
w
e
go
f
or
m
he
r
e
?
W
he
r
e
do
w
e
go
f
r
om
he
r
e
?
,
”
P
r
oc
e
e
di
ngs
of
t
he
I
E
E
E
, vol
. 88, no. 8, pp. 1270
–
1275, Aug. 2
000, doi:
10.1109/5.880083.
[
9]
Y
.
K
im
,
Y
.
I
.
C
hi
u,
K
.
H
a
na
ki
,
D
.
H
e
gde
,
a
nd
S
.
P
e
tr
ov,
“
T
e
mpor
a
l
a
na
ly
s
i
s
of
la
ngua
ge
th
r
ough
ne
ur
a
l
la
ngua
ge
mode
ls
,”
P
r
oc
e
e
di
ngs
of
t
he
A
nnual M
e
e
ti
ng of
t
he
A
s
s
oc
ia
ti
on f
or
C
om
put
at
io
nal
L
in
gui
s
ti
c
s
, pp. 61
–
65, M
a
y 2014, doi:
10.3115/v1/
w
14
-
2517.
[
10]
S
.
E
dunov,
A
.
B
a
e
vs
ki
, a
nd
M
.
A
ul
i,
“
P
r
e
-
tr
a
in
e
d
la
ngua
ge
mo
de
l
r
e
pr
e
s
e
nt
a
ti
ons
f
or
la
ngua
g
e
ge
ne
r
a
ti
on,”
in
N
A
A
C
L
H
L
T
2019
-
2019
C
onf
e
r
e
nc
e
of
th
e
N
or
th
A
m
e
r
ic
an
C
hapt
e
r
of
th
e
A
s
s
oc
ia
ti
on
fo
r
C
om
put
at
io
nal
L
in
gui
s
ti
c
s
:
H
um
an
L
anguage
T
e
c
hnol
ogi
e
s
-
P
r
oc
e
e
di
ngs
of
t
he
C
onf
e
r
e
nc
e
,
M
a
r
. 2019, vol. 1, pp. 40
52
–
4059, doi:
10.18653/v1/
n19
-
1409.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
S
S
N
:
2088
-
8708
I
nt
J
E
lec
&
C
omp
E
ng
,
Vol
.
1
4
,
No.
4
,
Augus
t
20
2
4
:
4655
-
4663
4662
[
11]
G
. C
ha
lv
a
tz
a
ki
, A
. Y
oune
s
, D
. N
a
ndha
, A
. T
. L
e
, L
. F
.
R
. R
ib
e
ir
o, a
nd I
.
G
ur
e
vyc
h, “
L
e
a
r
ni
ng
t
o r
e
a
s
on ove
r
s
c
e
ne
gr
a
phs
:
a
c
a
s
e
s
tu
dy
of
f
in
e
tu
ni
ng
G
P
T
-
2
in
to
a
r
obot
la
ngua
ge
mode
l
f
or
gr
o
unde
d
ta
s
k
pl
a
nni
ng,”
F
r
ont
ie
r
s
in
R
obot
ic
s
and
A
I
,
vol
.
10,
A
ug.
2023, doi:
10.3389/f
r
obt
.2023.1221739.
[
12]
J
.
D
e
vl
in
,
M
.
W
.
C
ha
ng,
K
.
L
e
e
,
a
nd
K
.
T
out
a
nova
,
“
B
E
R
T
:
pr
e
-
tr
a
in
in
g
of
de
e
p
bi
di
r
e
c
ti
ona
l
tr
a
ns
f
or
me
r
s
f
or
la
ng
ua
ge
unde
r
s
ta
ndi
ng,”
N
A
A
C
L
H
L
T
2019
-
2019
C
onf
e
r
e
n
c
e
of
th
e
N
or
th
A
m
e
r
ic
an
C
hapt
e
r
of
th
e
A
s
s
o
c
ia
ti
on
fo
r
C
om
put
at
io
nal
L
in
gui
s
ti
c
s
:
H
um
an L
anguage
T
e
c
hnol
ogi
e
s
-
P
r
oc
e
e
di
ng
s
of
t
he
C
onf
e
r
e
n
c
e
, vol
. 1, pp. 4171
–
4186, Oc
t.
2019.
[
13]
R
.
B
e
hni
a
,
M
.
R
.
E
br
a
hi
mi
,
J
.
P
a
c
he
c
o,
a
nd
B
.
P
a
dma
na
bha
n,
“
E
W
-
T
une
:
a
f
r
a
me
w
or
k
f
or
pr
iv
a
te
ly
f
in
e
-
tu
ni
ng
la
r
ge
la
ngua
ge
mode
ls
w
it
h
di
f
f
e
r
e
nt
ia
l
pr
iv
a
c
y,”
in
I
E
E
E
I
nt
e
r
nat
io
nal
C
onf
e
r
e
nc
e
on
D
at
a
M
in
in
g
W
or
k
s
hop
s
,
I
C
D
M
W
,
N
ov.
2022,
vol
.
2
022
-
N
ove
m, pp. 560
–
566, doi:
10.1109/I
C
D
M
W
58026.2022.00078.
[
14]
L
.
W
e
id
in
ge
r
e
t
al
.
,
“
T
a
xonomy
of
r
is
ks
po
s
e
d
by
la
ngu
a
ge
m
ode
ls
,”
in
A
C
M
I
nt
e
r
nat
io
nal
C
onf
e
r
e
nc
e
P
r
oc
e
e
di
ng
Se
r
ie
s
,
J
un.
2022, pp. 214
–
229, doi:
10.1145/3531146.3
533088.
[
15]
O
.
S
ha
r
ir
,
B
.
P
e
le
g,
a
nd
Y
.
S
hoha
m,
“
T
he
c
o
s
t
of
tr
a
in
in
g
N
L
P
mode
ls
:
a
c
on
c
is
e
ov
e
r
vi
e
w
,”
ar
X
iv
p
r
e
pr
in
t
a
r
X
iv
:
2004.08900
,
A
pr
. 2020.
[
16]
N
. M
c
K
e
nna
, T
. L
i,
L
. C
he
ng, M
. J
. H
os
s
e
in
i,
M
. J
ohns
on, a
nd M
. S
te
e
dma
n, “
S
our
c
e
s
of
ha
ll
uc
in
a
ti
on
by l
a
r
ge
l
a
ngua
ge
mo
de
ls
on
in
f
e
r
e
nc
e
ta
s
ks
,
”
F
in
di
ngs
of
th
e
A
s
s
oc
ia
ti
on
fo
r
C
om
put
at
io
nal
L
in
gui
s
ti
c
s
:
E
M
N
L
P
2023
,
pp.
2758
–
2774,
M
a
y
2023,
doi
:
10.18653/v1/
2023.f
in
di
ngs
-
e
mnl
p.182.
[
17]
E
.
F
e
r
r
a
r
a
,
“
S
houl
d
C
ha
t
G
P
T
be
bi
a
s
e
d?
c
ha
ll
e
nge
s
a
nd
r
is
ks
of
bi
a
s
in
la
r
ge
la
ngua
ge
mode
ls
,”
F
ir
s
t
M
onday
,
vol
.
28,
no.
11,
A
pr
. 2023, doi:
10.5210/f
m.v28i
11.13346.
[
18]
R
.
P
e
dr
o,
D
.
C
a
s
tr
o,
P
.
C
a
r
r
e
ir
a
,
a
nd
N
.
S
a
nt
o
s
,
“
F
r
om
pr
omp
t
in
je
c
ti
ons
to
S
Q
L
in
je
c
ti
on
a
tt
a
c
ks
:
how
pr
ot
e
c
te
d
is
your
L
L
M
-
in
te
gr
a
te
d w
e
b a
ppl
ic
a
ti
on?
,”
a
r
X
iv
pr
e
pr
in
t
a
r
X
iv
:
2308.01990
, A
ug. 2023.
[
19]
Y
.
C
ha
ng
e
t
al
.
,
“
A
s
ur
ve
y
on
e
va
lu
a
ti
on
of
la
r
ge
la
ngu
a
ge
m
ode
ls
,”
A
C
M
T
r
ans
ac
ti
ons
on
I
nt
e
ll
ig
e
nt
S
y
s
te
m
s
and
T
e
c
hnol
ogy
,
vol
. 15, no. 3, pp. 1
–
45, M
a
r
. 2024, doi:
10.1145/3641289.
[
20]
A
.
V
a
s
w
a
ni
e
t
al
.
,
“
A
tt
e
nt
io
n
is
a
ll
you
ne
e
d,”
A
dv
anc
e
s
i
n
N
e
ur
al
I
nf
or
m
at
io
n
P
r
oc
e
s
s
in
g
Sy
s
te
m
s
,
vol
.
2017
-
D
e
c
e
m
be
r
,
pp. 5999
–
6009, J
un. 2017.
[
21]
J
.
E
li
a
s
,
“
G
oogl
e
’
s
P
a
L
M
2
us
e
s
ne
a
r
ly
f
iv
e
ti
me
s
mor
e
te
xt
da
ta
th
a
n
pr
e
de
c
e
s
s
or
,
”
c
nbc
.c
om
.
ht
tp
s
:/
/ww
w
.c
nbc
.c
om/
2023/05/
16/
google
s
-
P
a
L
M
2
-
us
e
s
-
ne
a
r
l
y
-
f
iv
e
-
ti
me
s
-
mor
e
-
te
xt
-
da
ta
-
th
a
n
-
pr
e
de
c
e
s
s
or
.ht
ml
(
a
c
c
e
s
s
e
d
M
a
y
16, 2023)
.
[
22]
G
oogl
e
, “
G
oog
le
A
I
P
a
L
M
2,”
G
oogl
e
A
I
,
2023. htt
ps
:/
/a
i.
goog
le
/d
is
c
ove
r
/p
a
lm
2/
(
a
c
c
e
s
s
e
d
M
a
y 16, 2023)
.
[
23]
R
. A
ni
l
e
t
al
.
, “
P
a
L
M
2 t
e
c
hni
c
a
l
r
e
por
t,
”
ar
X
iv
pr
e
pr
in
t
ar
X
iv
:
2305.10403
, M
a
y 2023.
[
24]
Ö
.
A
ydi
n,
“
G
oogl
e
ba
r
d
ge
ne
r
a
te
d
li
te
r
a
tu
r
e
r
e
vi
e
w
:
me
ta
v
e
r
s
e
,”
J
our
nal
of
A
I
,
vol
.
7,
no.
1,
pp.
1
–
14,
D
e
c
.
2023,
doi
:
10.61969/j
a
i.
1311271.
[
25]
K
.
S
in
gha
l
e
t
al
.
,
“
T
ow
a
r
ds
e
xpe
r
t
-
le
ve
l
me
di
c
a
l
que
s
ti
on
a
ns
w
e
r
in
g
w
it
h
l
a
r
ge
la
ngua
ge
mode
ls
,
”
ar
X
iv
p
r
e
p
r
in
t
ar
X
iv
:
2305.09617
, M
a
y 2023.
[
26]
M
.
Z
ha
ng
a
nd
J
.
L
i,
“
A
c
omm
e
nt
a
r
y
of
G
P
T
-
3
in
M
I
T
te
c
hnol
ogy
r
e
vi
e
w
2021,”
F
undame
nt
al
R
e
s
e
ar
c
h
,
vol
.
1,
no.
6,
pp. 831
–
833, Nov. 2021, doi:
10.1016/j
.f
mr
e
.2021.11.011.
[
27]
O
.
B
ur
uk,
“
A
c
a
de
mi
c
w
r
it
in
g
w
it
h
G
P
T
-
3.5
(
C
ha
t
G
P
T
)
:
r
e
f
le
c
ti
ons
on
pr
a
c
ti
c
e
s
,
e
f
f
ic
a
c
y
a
nd
tr
a
ns
pa
r
e
nc
y,”
A
C
M
I
nt
e
r
nat
io
nal
C
onf
e
r
e
nc
e
P
r
oc
e
e
di
ng Se
r
ie
s
, pp. 144
–
153, F
e
b. 2023, doi:
10
.1145/3616961.3616
992.
[
28]
A
.
S
ha
f
e
e
g,
I
.
S
ha
z
ha
e
v,
D
.
M
ih
a
yl
ov,
A
.
T
ul
a
r
ov,
a
nd
I
.
S
ha
z
ha
e
v,
“
V
oi
c
e
a
s
s
is
ta
nt
in
te
gr
a
te
d
w
it
h
c
ha
t
G
P
T
,”
I
ndone
s
ia
n
J
our
nal
of
C
om
put
e
r
S
c
ie
nc
e
, vol
. 12, no. 1, F
e
b. 2023, doi:
10.
33022/i
jc
s
.v12i1.3146.
[
29]
W
.
J
ia
o,
W
.
W
a
ng,
J
.
H
ua
ng,
X
.
W
a
ng,
S
.
S
hi
,
a
nd
Z
.
T
u,
“
I
s
C
ha
tG
P
T
a
good
tr
a
ns
la
to
r
?
ye
s
w
it
h
G
P
T
-
4
a
s
th
e
e
ngi
ne
,”
a
r
X
iv
pr
e
pr
in
t
ar
X
iv
:
2301.08745
, J
a
n. 2023.
[
30]
L
.
Z
hou,
J
.
Z
ha
ng,
a
nd
C
.
Z
ong,
“
S
ync
hr
onous
bi
di
r
e
c
ti
ona
l
ne
ur
a
l
ma
c
hi
ne
tr
a
n
s
la
ti
on,”
T
r
an
s
ac
ti
ons
of
th
e
A
s
s
o
c
ia
ti
on
fo
r
C
om
put
at
io
nal
L
in
gui
s
ti
c
s
, vol
. 7, pp. 91
–
105, Apr
. 2019, doi:
10.1162/t
a
c
l_
a
_00256.
[
31]
Q
.
Y
u,
Z
.
W
a
ng,
a
nd
K
.
J
ia
ng,
“
R
e
s
e
a
r
c
h
on
te
xt
c
la
s
s
if
ic
a
ti
on
ba
s
e
d
on
B
E
R
T
-
B
iG
R
U
mode
l,
”
J
ou
r
nal
of
P
hy
s
ic
s
:
C
onf
e
r
e
nc
e
Se
r
ie
s
, vol
. 1746, no. 1,
J
a
n. 2021, doi:
10.1088/1742
-
6596/1746/
1/
012019.
[
32]
Z
.
W
a
ng,
P
.
N
g,
X
.
M
a
,
R
.
N
a
ll
a
pa
ti
,
a
nd
B
.
X
ia
ng,
“
M
ul
ti
-
pa
s
s
a
ge
B
E
R
T
:
a
gl
oba
ll
y
nor
ma
li
z
e
d
B
E
R
T
mode
l
f
or
ope
n
-
do
ma
in
que
s
ti
on
a
ns
w
e
r
in
g,”
E
M
N
L
P
-
I
J
C
N
L
P
2019
-
2019
C
onf
e
r
e
nc
e
on
E
m
pi
r
ic
al
M
e
th
ods
in
N
at
ur
al
L
anguage
P
r
oc
e
s
s
in
g
and
9t
h
I
nt
e
r
nat
io
nal
J
oi
nt
C
onf
e
r
e
nc
e
on N
at
ur
al
L
anguage
P
r
oc
e
s
s
in
g, P
r
oc
e
e
di
ngs
of
t
he
C
onf
e
r
e
nc
e
, pp. 5878
–
5882, Aug. 2019,
doi
:
10.18653/v1/
d19
-
1599.
[
33]
N
.
J
ia
ng
a
nd
M
.
C
.
de
M
a
r
ne
f
f
e
,
“
E
va
lu
a
ti
ng
B
E
R
T
f
or
na
tu
r
a
l
la
ngua
ge
in
f
e
r
e
nc
e
:
a
c
a
s
e
s
tu
dy
on
th
e
C
omm
it
me
nt
B
a
nk,
”
in
E
M
N
L
P
-
I
J
C
N
L
P
2019
-
2019
C
onf
e
r
e
nc
e
on
E
m
pi
r
ic
al
M
e
th
ods
in
N
at
ur
al
L
anguage
P
r
oc
e
s
s
in
g
and
9t
h
I
nt
e
r
nat
io
nal
J
oi
nt
C
onf
e
r
e
nc
e
on N
at
ur
al
L
anguage
P
r
oc
e
s
s
in
g, P
r
oc
e
e
di
ngs
of
t
he
C
onf
e
r
e
n
c
e
, 2019, pp. 6086
–
6091, doi:
10.18653/v1/
d19
-
1630.
[
34]
X
.
L
i,
L
.
B
in
g,
W
.
Z
ha
ng,
a
nd
W
.
L
a
m,
“
E
xpl
oi
ti
ng
be
r
t
f
or
e
nd
-
to
-
e
nd
a
s
pe
c
t
-
ba
s
e
d
s
e
nt
im
e
nt
a
na
ly
s
is
,
”
I
n
P
r
oc
e
e
di
ngs
of
th
e
5t
h W
or
k
s
hop on N
oi
s
y
U
s
e
r
-
ge
ne
r
at
e
d T
e
x
t
(
W
-
N
U
T
2019)
, pp. 34
–
41, Oc
t.
2019, doi:
10.18653/v1/
d19
-
5505.
[
35]
S
. K
ul
a
a
nd R
. C
.
M
ic
h
a
ła
nd K
oz
ik
, “
A
ppl
ic
a
ti
on of
t
he
be
r
t
-
ba
s
e
d a
r
c
hi
te
c
tu
r
e
i
n f
a
ke
ne
w
s
de
te
c
ti
on,”
i
n
A
dv
anc
e
s
i
n I
nt
e
ll
ig
e
nt
Sy
s
te
m
s
and
C
om
put
in
g
, vol
. 1267 AI
S
C
, S
pr
in
ge
r
I
nt
e
r
na
ti
ona
l
P
ubl
is
hi
ng, 2021, pp. 239
–
249.
[
36]
M
.
K
ha
dhr
a
oui
,
H
.
B
e
ll
a
a
j,
M
.
B
e
n
A
mm
a
r
,
H
.
H
a
ma
m,
a
nd
M
.
J
ma
ie
l,
“
S
ur
ve
y
of
B
E
R
T
-
ba
s
e
mode
ls
f
or
s
c
ie
nt
if
ic
te
xt
c
la
s
s
if
ic
a
ti
on:
C
O
V
I
D
-
19 c
a
s
e
s
tu
dy,
”
A
ppl
ie
d Sc
ie
nc
e
s
(
Sw
it
z
e
r
la
nd)
, vol
. 12, no. 6, M
a
r
. 2022, doi:
10.3390/app120628
91.
[
37]
Y
. L
u, J
. P
a
n, a
nd Y
. X
u, “
P
ubl
ic
s
e
nt
im
e
nt
on C
hi
ne
s
e
s
oc
i
a
l
me
di
a
dur
in
g t
he
e
me
r
ge
nc
e
of
C
O
V
I
D
19,”
J
our
nal
of
Q
uant
it
at
iv
e
D
e
s
c
r
ip
ti
on:
D
ig
it
al
M
e
di
a
, vol
. 1, Apr
. 2021, doi:
10.51685/j
qd.2021.013.
[
38]
L
a
ngc
ha
in
,
“
I
nt
r
oduc
ti
on
la
ngc
ha
in
,”
P
y
th
on.L
angc
hai
n.C
o
m
,
2023.
ht
tp
s
:/
/p
yt
hon.la
ngc
ha
in
.c
om/
do
c
s
/c
omm
uni
ty
(
a
c
c
e
s
s
e
d
M
a
y 16, 2023)
.
[
39]
S
.
R
ude
r
,
M
.
P
e
te
r
s
,
S
.
S
w
a
ya
mdi
pt
a
,
a
nd
T
.
W
ol
f
,
“
T
r
a
ns
f
e
r
le
a
r
ni
ng
in
na
tu
r
a
l
la
ngua
ge
pr
oc
e
s
s
in
g
tu
to
r
ia
l,
”
in
N
A
A
C
L
H
L
T
2019
-
2019
C
onf
e
r
e
n
c
e
of
th
e
N
or
th
A
m
e
r
ic
an
C
hapt
e
r
of
t
he
A
s
s
oc
ia
ti
on
fo
r
C
om
put
at
io
nal
L
in
gui
s
ti
c
s
:
H
um
an
L
angu
age
T
e
c
hnol
ogi
e
s
-
T
ut
or
ia
l
A
bs
tr
ac
ts
, 2019, pp. 15
–
18, doi:
10.186
53/
v1/
n19
-
5004.
[
40]
A
.
M
a
da
a
n,
S
.
Z
hou,
U
.
A
lo
n,
Y
.
Y
a
ng,
a
nd
G
.
N
e
ubi
g,
“
L
a
ngua
ge
mode
ls
of
c
ode
a
r
e
f
e
w
-
s
hot
c
omm
ons
e
ns
e
le
a
r
ne
r
s
,”
in
P
r
oc
e
e
di
ngs
of
th
e
2022
C
onf
e
r
e
nc
e
on
E
m
pi
r
ic
al
M
e
th
ods
in
N
at
ur
al
L
anguage
P
r
oc
e
s
s
in
g,
E
M
N
L
P
2022
,
2022,
pp. 1384
–
1403, doi:
10.18653/v1/
2022.e
mnl
p
-
ma
in
.90.
[
41]
S
.
L
ouki
li
,
A
.
F
e
nna
n,
a
nd
L
.
E
la
a
c
ha
k,
“
A
ppl
ic
a
ti
ons
of
te
xt
ge
ne
r
a
ti
on
in
di
gi
ta
l
ma
r
ke
ti
ng:
a
r
e
vi
e
w
,”
A
C
M
I
nt
e
r
nat
io
nal
C
onf
e
r
e
nc
e
P
r
oc
e
e
di
ng Se
r
ie
s
,
M
a
y 2023, doi:
10.1145/360772
0.3608451.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
nt
J
E
lec
&
C
omp
E
ng
I
S
S
N:
2088
-
8708
E
mail
s
ubjec
ts
ge
ne
r
ati
on
w
it
h
lar
ge
language
mode
ls
:
GP
T
-
3
.
5,
P
aL
M
2
,
and
B
E
R
T
(
Soumay
a
L
o
uk
il
i)
4663
B
I
OG
RA
P
HI
E
S
OF
AU
T
HO
RS
So
um
a
y
a
L
o
uki
l
i
i
s
a
d
ed
i
ca
t
ed
Ph
.
D
.
s
t
u
d
en
t
,
p
as
s
i
o
n
at
e
ab
o
u
t
t
h
e
s
y
n
erg
y
b
et
w
een
mark
e
t
i
n
g
,
art
i
fi
c
i
al
n
e
u
ral
n
et
w
o
r
k
s
,
an
d
ar
t
i
f
i
ci
al
i
n
t
e
l
l
i
g
e
n
ce
.
H
er
res
earch
e
x
p
l
o
re
s
i
n
n
o
v
at
i
v
e
a
p
p
r
o
ach
e
s
i
n
t
h
es
e
f
i
el
d
s
,
s
h
o
w
ca
s
i
n
g
a
c
o
mmi
t
me
n
t
t
o
s
t
a
y
i
n
g
a
t
t
h
e
fo
refr
o
n
t
o
f
cu
t
t
i
n
g
-
e
d
g
e
re
s
earch
.
S
o
u
ma
y
a's
w
o
r
k
refl
ec
t
s
a
fre
s
h
p
ers
p
ect
i
v
e
o
n
mar
k
et
i
n
g
,
l
e
v
erag
i
n
g
t
h
e
p
o
w
er
o
f
art
i
f
i
ci
a
l
n
eu
ra
l
n
et
w
o
r
k
s
an
d
A
I
fo
r
i
n
s
i
g
h
t
fu
l
s
t
rat
e
g
i
es
.
Sh
e
al
s
o
h
as
a
k
een
i
n
t
eres
t
i
n
s
e
n
t
i
men
t
an
a
l
y
s
i
s
an
d
d
i
d
mu
l
t
i
p
l
e
s
t
u
d
i
es
i
n
t
h
i
s
fi
e
l
d
,
i
n
t
er
s
ect
i
n
g
w
i
t
h
art
i
f
i
ci
a
l
i
n
t
el
l
i
g
en
ce
as
w
el
l
.
Sh
e
can
b
e
co
n
t
act
e
d
v
i
a
ema
i
l
at
:
s
o
u
ma
y
a.
l
o
u
k
i
l
i
1
@
et
u
.
u
ae.
ac.
ma.
A
bdel
ha
d
i
F
enna
n
i
s
a
p
ro
fe
s
s
o
r
i
n
t
h
e
D
e
p
art
m
en
t
o
f
C
o
mp
u
t
er
Sc
i
en
ce
a
t
A
b
d
el
ma
l
ek
E
s
s
aa
d
i
U
n
i
v
ers
i
t
y
,
T
an
g
i
er,
Mo
r
o
cco
.
H
o
l
d
i
n
g
a
v
a
l
i
d
at
e
d
emai
l
ad
d
re
s
s
at
u
ae.
ac.
ma,
h
e
i
s
a
d
i
s
t
i
n
g
u
i
s
h
e
d
acad
em
i
c
w
i
t
h
a
f
o
c
u
s
o
n
art
i
fi
c
i
al
i
n
t
el
l
i
g
en
ce
,
s
o
ft
w
are
en
g
i
n
eer
i
n
g
,
an
d
i
n
f
o
rmat
i
o
n
s
y
s
t
ems
(b
u
s
i
n
e
s
s
i
n
f
o
rmat
i
cs
).
H
i
s
ex
p
er
t
i
s
e
ex
t
en
d
s
t
o
d
i
s
ci
p
l
i
n
e
s
s
u
ch
as
h
i
g
h
er
ed
u
cat
i
o
n
,
ed
u
cat
i
o
n
al
t
ech
n
o
l
o
g
y
,
a
n
d
c
u
rri
c
u
l
u
m
t
h
e
o
ry
.
A
b
d
el
h
ad
i
's
s
k
i
l
l
s
en
c
o
mp
a
s
s
e
-
l
ear
n
i
n
g
,
o
n
l
i
n
e
l
earn
i
n
g
,
s
eman
t
i
c
w
eb
,
s
e
n
t
i
men
t
an
al
y
s
i
s
,
an
d
t
ech
n
o
l
o
g
y
-
e
n
h
a
n
ced
l
earn
i
n
g
.
W
i
t
h
a
s
t
r
o
n
g
co
m
mi
t
me
n
t
t
o
re
s
earch
a
n
d
acad
em
i
a,
h
e
p
l
a
y
s
a
v
i
t
a
l
ro
l
e
i
n
ad
v
a
n
ci
n
g
k
n
o
w
l
e
d
g
e
an
d
i
n
n
o
v
at
i
o
n
i
n
t
h
e
s
e
d
o
mai
n
s
.
H
e
can
b
e
co
n
t
act
e
d
v
i
a
ema
i
l
at
:
afen
n
an
@
g
ma
i
l
.
c
o
m.
Lo
tfi
El
a
a
ch
a
k
i
s
an
as
s
o
c
i
at
e
p
r
o
fes
s
o
r
at
Facu
l
t
é
d
e
s
Sci
en
ce
s
et
T
ech
n
i
q
u
e
s
d
e
T
an
g
er.
W
i
t
h
a
s
t
ro
n
g
f
o
cu
s
o
n
ar
t
i
f
i
ci
a
l
i
n
t
e
l
l
i
g
e
n
ce
,
h
e
h
as
a
p
r
o
fo
u
n
d
u
n
d
er
s
t
a
n
d
i
n
g
o
f
al
g
o
ri
t
h
m
s
an
d
ar
t
i
f
i
ci
a
l
n
e
u
ral
n
et
w
o
r
k
s
.
H
i
s
e
x
p
er
t
i
s
e
ex
t
e
n
d
s
t
o
h
i
g
h
er
e
d
u
ca
t
i
o
n
,
w
h
ere
h
e
act
i
v
el
y
en
g
a
g
es
i
n
i
n
n
o
v
a
t
i
v
e
t
each
i
n
g
met
h
o
d
s
.
Pro
fi
ci
e
n
t
i
n
mach
i
n
e
l
ear
n
i
n
g
,
L
o
t
f
i
h
as
mad
e
s
i
g
n
i
fi
ca
n
t
co
n
t
r
i
b
u
t
i
o
n
s
t
o
t
h
e
fi
e
l
d
,
p
ar
t
i
c
u
l
ar
l
y
i
n
t
h
e
real
m
s
o
f
n
eu
ra
l
n
et
w
o
r
k
s
an
d
art
i
f
i
ci
a
l
i
n
t
el
l
i
g
en
ce.
Bey
o
n
d
acad
em
i
a,
h
e
i
s
i
n
v
o
l
v
e
d
i
n
t
h
e
rea
l
m
o
f
s
er
i
o
u
s
g
ames
,
v
i
d
e
o
g
ames
,
g
ame
d
es
i
g
n
,
an
d
g
ame
d
e
v
el
o
p
me
n
t
,
s
h
o
w
cas
i
n
g
a
d
i
v
ers
e
s
k
i
l
l
s
e
t
.
L
o
t
fi
i
s
p
as
s
i
o
n
a
t
e
ab
o
u
t
i
n
c
o
rp
o
rat
i
n
g
t
h
e
s
e
s
k
i
l
l
s
i
n
t
o
t
each
i
n
g
met
h
o
d
s
,
emp
h
a
s
i
z
i
n
g
t
h
e
i
n
t
e
g
rat
i
o
n
o
f
d
i
g
i
t
al
g
ame
s
,
g
ame
-
b
as
e
d
l
earn
i
n
g
,
an
d
n
ew
t
ech
n
o
l
o
g
i
es
l
i
k
e
v
i
r
t
u
a
l
real
i
t
y
a
n
d
au
g
me
n
t
e
d
real
i
t
y
.
H
i
s
p
ro
f
i
ci
e
n
cy
a
l
s
o
ex
t
en
d
s
t
o
A
ra
b
i
c
n
a
t
u
ra
l
l
a
n
g
u
ag
e
p
ro
ce
s
s
i
n
g
(N
L
P),
refl
ect
i
n
g
h
i
s
co
mm
i
t
me
n
t
t
o
ad
v
an
ci
n
g
res
earc
h
an
d
ed
u
cat
i
o
n
i
n
t
h
e
s
e
d
y
n
am
i
c
d
o
ma
i
n
s
.
H
e
can
b
e
co
n
t
ac
t
ed
v
i
a
emai
l
at
l
e
l
aach
a
k
@
u
ae.
ac.
ma.
Evaluation Warning : The document was created with Spire.PDF for Python.