IAES
Inter
national
J
our
nal
of
Articial
Intelligence
(IJ-AI)
V
ol.
14,
No.
2,
April
2025,
pp.
1605
∼
1615
ISSN:
2252-8938,
DOI:
10.11591/ijai.v14.i2.pp1605-1615
❒
1605
Multi-task
deep
lear
ning
f
or
V
ietnamese
capitalization
and
punctuation
r
ecognition
Phuong-Nhung
Nguy
en
1
,
Thu-Hien
Nguy
en
2
,
Nguy
en
T
ruong
Thang
1
,
Nguy
en
Thi
Thu
Nga
1
,
Nguy
en
Thi
Anh
Phuong
1
,
T
uan-Linh
Nguy
en
3
1
Institute
of
Information
T
echnology
,
V
ietnam
Academy
of
Science
and
T
echnology
,
Hanoi,
V
ietnam
2
F
aculty
of
Mathematics,
Thai
Nguyen
Uni
v
ersity
of
Education,
Thai
Nguyen,
V
ietnam
3
F
aculty
of
Electronics
Engineering,
Thai
Nguyen
Uni
v
ersity
of
T
echnology
,
Thai
Nguyen,
V
ietnam
Article
Inf
o
Article
history:
Recei
v
ed
Feb
29,
2024
Re
vised
Oct
21,
2024
Accepted
No
v
14,
2024
K
eyw
ords:
Capitalization
Deep
learning
Multitask
learning
Named
entity
recognition
Punctuation
ABSTRA
CT
Speech
recognition
is
the
process
of
con
v
erting
the
speech
signal
of
a
particular
language
into
a
sequence
of
corresponding
content
w
ords
in
te
xt
format.
The
output
te
xt
of
automatic
speech
recognition
(ASR)
systems
often
lacks
struc-
ture,
such
as
punctuation,
capitalization
of
the
rst
letter
of
a
sentence,
proper
nouns,
and
names
of
l
ocations.
This
absence
of
structure
complicates
compre-
hension
and
restricts
the
utility
of
ASR-generated
te
xt
in
v
a
rious
applications,
such
as
creating
mo
vie
subtitles,
generating
transcripts
for
online
meetings
,
and
e
xtracting
customer
information.
Therefore,
de
v
eloping
standardization
solu-
tions
for
the
output
te
xt
from
ASR
is
necessary
to
impro
v
e
the
o
v
erall
quality
of
ASR
systems.
In
this
article,
we
use
the
idea
of
multitask
deep
learning
for
the
task
of
capitalization
and
punctuation
recognition
(CPR)
for
the
output
te
xt
of
V
ietnamese
ASR,
with
the
aim
of
the
named
entity
recognition
(NER)
task
as
a
supplement
to
help
the
CPR
model
perform
better
,
and
proposed
to
use
te
xt-to-
speech
(TTS)
to
create
a
dataset
for
CPR-NER
multitask
model
training.
The
e
xperiment
results
sho
w
that
the
multi-task
deep
learning
model
has
impro
v
ed
CPR
results
by
6.2%
of
F1
score
with
ASR
output
and
7.1%
on
ra
w
te
xt.
This
is
an
open
access
article
under
the
CC
BY
-SA
license
.
Corresponding
A
uthor:
T
uan-Linh
Nguyen
F
aculty
of
Electronics
Engineering,
Thai
Nguyen
Uni
v
ersity
of
T
echnology
Thai
Nguyen,
V
ietnam
Email:
ntlinh@tnut.edu.vn
1.
INTR
ODUCTION
Automatic
speech
recognition
(ASR)
refers
to
the
v
arious
processes,
technologies,
and
met
h
odol
ogies
that
f
acilitate
impro
v
ed
interaction
between
humans
and
computers
by
con
v
erting
spok
en
language
into
written
te
xt
[1],
[2].
The
output
te
xt
of
the
ASR
system,
besides
containing
some
i
nsertion,
deletion,
and
w
ord
substitution
errors,
often
has
no
punctuation
and
no
capitalization.
The
issue
of
standardizing
te
xt
by
rest
o
r
ing
punctuation
and
capitalization
is
necessary
.
It
will
mak
e
the
te
xt
more
structured
and
understand-
able.
Furthermore,
it
will
f
acilitate
NLP
tasks
including
named
entity
recognition
(NER),
syntactic
parsing,
part-of-speech
tagging,
and
discourse
se
gmentation
[3].
Additionally
,
it
benets
practical
applications
such
as
generating
mo
vie
subtitles,
broadcasting
ne
ws
transcripts,
creating
documents
for
online
meetings,
and
e
xtract-
ing
customer
information
more
ef
fecti
v
ely
[4].
Research
in
this
eld
has
traditionally
focused
on
indi
vidual
tasks,
such
as
restoring
punct
uation
[5]–[7]
or
capitalization
[8]–[10].
Ho
we
v
er
,
these
separate
processing
techniques
are
challenging
to
impro
v
e
J
ournal
homepage:
http://ijai.iaescor
e
.com
Evaluation Warning : The document was created with Spire.PDF for Python.
1606
❒
ISSN:
2252-8938
the
o
v
erall
ef
fecti
v
eness
of
the
ASR
systems.
Therefore,
recently
,
authors
ha
v
e
proposed
models
to
simultane-
ously
handle
tw
o
tasks
[4],
[11]–[13].
Initially
,
research
in
this
eld
utilized
methods
such
as
maximum
entrop
y
[14],
hidden
mark
o
v
model
(HMM)
[15],
long
short-term
memory
(LSTM)
[16],
and
cond
i
tional
random
eld
(CRF)
[17],
[18].
Currently
,
the
most
adv
anced
models
for
both
punctuation
restoration
and
capitalization
recognition
use
neural
netw
ork
models.
St
udies
ha
v
e
e
xplored
the
character
-le
v
el
recurrent
neural
netw
ork
(RNN)
model
[19],
bidirectional
RNN
[20],
[21].
P
articularly
,
with
the
emer
gence
of
the
transformer
model
and
its
v
ariations
lik
e
BER
T
,
RoBER
T
a,
ALBER
T
,
DistilBER
T
,
mBER
T
,
and
XLM-RoBER
T
a,
studies
on
restoring
punctuation
and
capitalization
on
the
transformer
model
[12],
[22],
BER
T
[10],
[23],
RoBER
T
a
[24],
ha
v
e
sho
wn
positi
v
e
results.
Moreo
v
er
,
research
on
the
V
ietnamese
language
has
also
follo
wed
this
inno
v
ati
v
e
and
promising
trend,
using
CRFs
combined
with
CNNs
and
LSTM
[25],
transformer
[26]–[28].
Notably
,
the
V
iBER
T
model,
based
on
RoBER
T
a
for
V
ietnamese,
is
the
latest
model
that
achie
v
es
better
performance
in
both
tasks
[29].
There
are
also
ha
v
e
some
limitations
and
challenges
that
need
t
o
be
solv
ed
for
the
problem
of
re
storing
punctuation
and
capitalization
in
general
and
V
ietnamese
in
particular
.
Re
g
arding
data-related
issues,
to
deal
with
lo
w-resource
language
data,
some
studies
ha
v
e
e
xperimented
with
us
ing
standardized
te
xt
without
punc-
tuation
and
capitalization
[3],
[6],
[26].
Because
commas
and
periods
appear
more
frequently
than
other
marks,
most
research
focuses
only
o
n
these
marks
[12],
[30],
[31].
ASR
output
te
xt
does
not
ha
v
e
punctuation,
so
it
is
often
long
and
endless.
T
o
process
such
te
xt
in
models,
input
sequences
are
usually
randomly
cut
into
ranges
of
20-30
w
ords
[32]
or
20-50
w
ords
[24],
maximum
length
100
w
ords
[33]
and
150
w
ords
[25].
Determining
the
appropriate
length
for
cutting
is
a
matter
that
needs
to
be
considered.
The
proposal
to
use
o
v
erlap-chunking
in
cutting
and
concatenating
sequences
has
helped
impro
v
e
the
model
[9].
The
issue
of
mismatched
sequence
lengths
due
to
errors
lik
e
insertions,
deletions,
and
substitutions
in
the
ASR
output
te
xt
is
one
of
the
major
challenges
that
need
to
be
addressed.
Determining
the
order
of
punctuation
restoration
and
capitalization
is
still
an
important
issue
as
it
can
signicantly
impact
the
nal
results
[29].
Multi-task
deep
learning
(MTDL)
has
g
ai
ned
signicant
attention
in
the
eld
of
machine
learning
[34]
and
natural
language
processing
(NLP)
[35].
It
has
been
used
to
address
challenges
such
as
o
v
ertting
and
data
scarcity
in
NLP
tasks.
Dif
ferent
architectures,
including
parallel,
hierarchical,
modular
,
and
generati
v
e
adv
ersarial
architectures,
ha
v
e
been
emplo
yed
for
multi-task
learning
(MTL)
in
NLP
.
Optimization
techniques
such
as
loss
construction,
data
sampling,
and
task
scheduling
ha
v
e
been
utilized
to
train
multi-task
models
ef
fecti
v
ely
.
MTL
has
sho
wn
promising
results
in
v
arious
NLP
tasks,
including
machine
translation,
dialogue-
based
systems,
sentiment
analysis,
and
machine
reading
comprehension.
In
this
study
,
our
goal
is
to
propose
a
MTDL
for
simultaneous
capitalization
and
punctuation
recogni-
tion
(CPR).
W
e
combine
MTL
with
t
he
h
ypothesis
that
inte
grating
a
NER
model
will
impro
v
e
the
performance
of
punctuation
restoration
and
capitalization.
Additionally
,
we
also
emplo
y
the
o
v
erlap-chunking
technique
and
the
V
iBER
T
language
model,
which
ha
v
e
sho
wn
ef
fecti
v
eness
in
NER
[13].
P
articularly
,
to
cope
with
lo
w-resource
language
data,
we
consider
using
te
xt-to-speech
(TTS)
to
augment
the
training
data.
This
paper
is
structured
as
follo
ws:
the
initial
session
addresses
the
problem
and
re
vie
ws
related
research.
The
second
session
disc
u
s
ses
our
proposed
approach
and
the
architecture
of
the
MTDL
CPR
system.
In
the
third
session,
the
focus
is
on
the
dataset’
s
implementat
ion
and
preparation
for
both
training
and
testing.
Session
four
details
the
conguration
steps
for
the
testing
process
and
presents
the
e
v
aluation
outcomes.
Lastly
,
the
fth
session
of
fers
a
conclusion
and
recommendations
for
future
research
a
v
enues.
2.
METHOD
2.1.
Multi-task
deep
lear
ning
f
or
capitalization
and
punctuation
r
ecognition
Using
the
idea
from
MTL
for
the
task
of
punctuation
and
capitalization
restoration,
a
MTDL
approach
is
proposed
with
the
aim
of
impro
ving
the
performance
of
the
CPR
task.
Figure
1
illustrates
the
proposed
MTDL
model,
which
consists
of
the
main
CPR
task
combined
with
an
auxiliary
NER
stream
that
pro
vides
additional
information
about
the
named
entity
(NE)
for
the
punctuation
and
capitalization
restoration
task.
The
input
data
to
the
model
is
the
output
te
xt
of
V
ietnamese
ASR
without
punctuati
on
and
capitalization,
with
a
length
of
N
.
During
recognition,
some
sentences
may
contain
errors
such
as
substitutions,
inserti
ons,
and
deletions,
making
the
punctuation
and
capitalization
restoration
task
more
challenging.
The
sentence
from
input
is
transferred
through
the
V
iBER
T
,
a
V
ietnamese
language
representation
model.
In
this
study
,
a
transfer
learning
approach
is
applied
with
V
iBER
T
,
a
pre-trained
model
that
is
k
ept
x
ed
in
the
proposed
MTDL
model.
Int
J
Artif
Intell,
V
ol.
14,
No.
2,
April
2025:
1605–1615
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
1607
The
output
of
V
iBER
T
is
a
matrix
of
size
(
N
x
768)
.
This
representation
matrix
is
simultaneously
fed
into
three
blocks:
(i)
the
NER
information
e
xtraction
block,
(ii)
the
main
CPR
restoration
block,
and
(iii)
the
NER
auxiliary
learning
block
using
a
MTL
mechanism.
Figure
1.
Proposed
MTDL
CPR
model
Figure
1
sho
ws
the
proposed
multitask
deep
learning
model,
which
includes
the
main
CPR
stream
and
NER
stream
to
support
the
NE
information
for
CPR.
Both
CPR
and
NER
tagger
are
based
on
the
transformer
decoder
structure
in
[29].
The
input
data
consisted
of
unpunctuated
and
uncapitalized
V
ietnamese
sentences,
which
were
e
xtracted
from
the
output
of
an
ASR
system,
with
a
set
size
of
N.
The
input
sentence
underwent
processing
using
the
representation
of
V
ietnamese
language
m
odel
kno
wn
as
V
iBER
T
.
This
model
produced
an
output
in
the
form
of
a
matrix,
characterized
by
dimensions
of
(Nx768),
which
encode
the
repres
entation
of
the
input
sentence.
This
matrix
representation
w
as
simultaneously
directed
to
w
ard
three
distinct
components:
(i)
a
block
that
e
xtracts
NE
information,
referred
to
as
the
NE
encoder;
(ii)
the
main
CPR
block;
and
(iii)
an
auxiliary
block
for
NER.
The
supplemental
block
for
e
xtracting
NER
information
incorporated
a
NER
tagging
block
along
with
an
NER
embedding
block,
which
functioned
to
encode
recognizable
NER
tags.
These
encoded
tags
were
meant
to
supply
pertinent
information
to
the
CPR
block.
The
CPR
block
represented
the
core
task
of
the
MTDL
model
and
utilized
output
from
the
sentence’
s
representation
matrix
produced
by
V
iBER
T
.
This
input
w
as
mer
ged
(concatenated)
with
the
NER
tag
encoding
matrix,
emanating
as
the
output
from
block
(ii).
This
mer
ger
with
the
output
from
block
(ii)
enriched
t
he
NER
information,
thereby
enhancing
the
precision
of
the
CPR
labeling
process.
The
output
from
the
block
w
as
a
collection
of
probabilities
corresponding
to
NER
tags
for
the
s
p
e
cic
input
sentence,
and
the
loss
function
for
this
process
w
as
calculated
based
on
these
NER
labels.
The
encoder
block
w
as
link
ed
to
the
NER
embedding
block
through
a
mechanism
that
shared
parameters,
as
illustrated
in
(1).
W
ithin
this
conguration,
the
weight
matrix
of
the
NER
embedding
w
as
replicated
from
the
transposed
matrix
of
the
encoder
in
block
(iii).
W
emb
=
W
T
enc
(1)
The
netw
ork
w
as
trained
using
a
MTL
approach
that
in
v
olv
ed
tw
o
tasks:
CPR,
which
serv
ed
as
the
primary
task,
and
NER,
which
functioned
as
a
supporti
v
e,
auxiliary
task.
The
loss
v
alue
for
the
MTDL
model
Multi-task
deep
learning
for
V
ietnamese
capitalization
and
punctuation
...
(Phuong-Nhung
Nguyen)
Evaluation Warning : The document was created with Spire.PDF for Python.
1608
❒
ISSN:
2252-8938
w
as
determined
by
taking
a
weighted
sum
of
the
loss
v
alues
from
both
the
CPR
and
NER
tasks.
The
MTDL
model’
s
loss
v
alue
is
computed
as
follo
ws:
the
losses
from
the
CPR
and
NER
tasks
are
each
assigned
a
weight,
and
then
these
weighted
losses
are
summed
toget
her
to
yiel
d
the
nal
loss
v
alue
for
the
model,
as
sho
wn
in
(2).
L
mtl
=
α
L
CPR
+
β
L
NER
(2)
Where
α
is
the
weight
for
the
loss
v
alue
of
the
CPR
task,
and
β
is
the
weight
for
the
loss
v
alue
of
the
NER
task.
The
choice
of
α
and
β
depends
on
the
importance
of
each
task.
In
this
study
,
the
CPR
task
is
considered
the
primary
task
and
the
NER
task
as
the
auxiliary
task.
Therefore,
α
and
β
are
chosen
to
be
0.6
and
0.4,
correspondently
.
2.2.
Input
text
o
v
erlap
segmentation
The
input
for
the
CPR
model
recei
v
es
output
te
xt
from
ASR.
This
te
xt
typically
lacks
punctua-
tion,
resulting
in
a
long,
indenite
sequence
that
poses
considerable
challenges
for
processing
by
models.
Therefore,
before
being
fed
into
the
model,
the
input
sequence
is
often
se
gmented
into
x
ed-length
portions,
which
enhances
the
ability
to
process
independently
or
in
parallel
se
gments.
Related
research
particularly
focuses
on
the
se
gmentation
of
the
input
sentence
sequence
and
often
emplo
ys
a
strate
gy
of
random
cutting
within
a
range
of
20-30
w
ords
[32]
or
20-50
w
ords
[24].
Ho
we
v
er
,
this
approach
can
lead
to
a
lack
of
suf
cient
conte
xtual
information
around
the
boundaries
of
the
se
gments,
resulting
in
predictions
that
are
often
inaccurate.
T
o
o
v
ercome
this
limitation,
research
has
proposed
a
ne
w
technique
for
handling
the
cutting
and
concatenation
of
sequences
by
utilizing
o
v
erlapping
cuts.
The
main
idea
is
to
ensure
that
the
resulting
se
gments
ha
v
e
enough
conte
xtual
information
for
the
w
ords,
enabling
the
CPR
model
to
mak
e
the
best
predictions.
After
processing
the
o
v
erlapping
se
gments,
the
y
are
mer
ged
back
into
the
output
sequence
of
the
original
string.
Figure
2
details
this
architecture,
which
includes
three
components:
the
di
vision
of
o
v
erlapping
se
g-
ments,
the
CPR
model,
and
the
mer
ging
of
the
o
v
erlapping
se
gments.
It
can
be
seen
that
the
input
sentence
is
di
vided
into
three
se
gments
,
which
are
stack
ed
in
an
o
v
erlapping
f
ashion.
After
being
processed
by
the
CPR
model,
the
se
gments
are
recognized
in
the
middle
of
the
second
se
gment
ha
ving
more
surrounding
conte
xt,
thus
being
tagged
m
ore
accurately
than
the
w
ords
at
the
cutting
boundary
se
gments.
Finally
,
these
se
gment
s
will
be
mer
ged
in
an
o
v
erlapping
manner
to
form
the
restored
sentence.
Figure
2.
Ov
erlap
chunk-mer
ging
of
input
te
xt
The
chosen
length
for
the
se
gment
cuts
is
an
e
v
en
number
of
w
ords.
Let
l
represent
the
length
of
the
se
gment
cut,
and
k
is
the
length
of
the
o
v
erlapping
part,
then
we
ha
v
e
l
=
2
k
.
Each
input
string
S
,
containing
Int
J
Artif
Intell,
V
ol.
14,
No.
2,
April
2025:
1605–1615
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
1609
n
w
ords
denoted
as
w
1
,
w
2
,
...,
w
n
,
will
be
di
vided
into
n/l
+
(
n
−
k
)
/l
o
v
erlapping
se
gments.
Here,
the
i
th
se
gment
cut
is
a
substring
of
w
ords
w
i
−
1
k
+
1
,
.
.
.
,
w
i
+1
k
.
The
study
has
e
xamined
v
arious
v
alues
for
l
and
k
,
and
these
v
alues
ha
v
e
been
selected
empirically
to
be
appropriate.
As
the
input
sentence
is
di
vided
into
o
v
erlapping
se
gments,
the
challenge
in
mer
ging
these
se
gments
is
to
determine
which
w
ords
should
be
discarded
and
which
should
be
retained
in
the
mer
ged
portion
of
the
nal
sentence.
Let
c
be
the
length
of
the
se
gment
that
will
be
either
retained
or
discarded
in
the
o
v
erlappi
ng
parts.
F
or
simplicity
in
calculation,
we
tak
e
c
=
k
/
2
.
Observ
ations
indicate
that
the
ending
w
ords
of
the
rst
o
v
erlapping
se
gment
and
the
initial
w
ords
of
the
second
o
v
erlapping
se
gment
(the
w
ords
around
the
cutting
point)
lack
substantial
conte
xt.
Therefore,
the
algorithm
will
discard
the
c
se
gment
at
the
end
of
the
rst
o
v
erlapping
part
and
retain
the
c
se
gment
in
the
second
o
v
erlapping
part.
Consequently
,
the
remaining
w
ords
at
the
be
ginning
of
the
rst
o
v
erlapping
se
gment
are
k
ept,
and
the
remaining
w
ords
at
the
start
of
the
second
o
v
erlapping
se
gment
are
discarded.
This
ensures
that
t
h
e
w
ords
retai
ned
in
the
o
v
erlapping
sections,
al
w
ays
situated
in
the
m
iddle
of
se
gments,
will
ha
v
e
ample
conte
xt,
aiding
in
more
accurate
restoration.
The
process
of
discarding
and
retaining
portions
of
the
o
v
erlapping
se
gments
is
repeated
for
subsequent
o
v
erlapping
se
gments.
The
post-catenation
mer
ging
process
is
described
as
(3).
[[
w
1
,
.
.
.
w
(2
k
−
c
)
]
+
n
−
1
X
i
=2
[
w
((
i
−
1)
k
+
c
)
,
.
.
.
w
(
ik
+
c
)
]
+
[
w
(
n
−
2
k
+
c
)
,
.
.
.
w
n
]
(3)
2.3.
Dataset
pr
eparation
Numerous
datasets
for
V
ietnamese
speech
are
no
w
accessible
to
the
public,
e
xamples
of
which
in-
clude
MICA
VNSpeechCorpus
[36],
DEMEN567
[37],
and
V
O
V
[38].
Despite
these
resources
being
reported,
it
should
be
noted
that
the
y
may
not
encompass
audi
o
data
that
is
both
lar
ge-scale
and
of
high
quality
.
Besides,
the
e
xisting
speech
datasets
are
not
annotated
with
punctuation,
capitalization,
nor
NE
labels,
Therefore,
it
is
necessary
to
construct
a
suf
ciently
lar
ge
dataset
complete
with
labels
for
punctuation,
capitalization,
and
NE
in
order
to
train
the
CPR
model
using
a
MTL
approach.
The
dataset
for
NER
tailored
to
the
V
ietnamese
language
w
as
de
v
eloped
as
part
of
the
acti
vities
of
the
V
ietnamese
language
and
speech
processing
conference
(VLSP).
It
comprises
32,000
instances
in
the
training
dataset
with
an
additional
4,272
samples
designated
for
testing.
The
dat
aset
focuses
on
e
xtracting
specic
entities
which
include
indi
vidual
names
(PER),
or
g
anizational
names
(ORG),
and
geographic
locations
(LOC).
Detailed
information
about
this
dataset
can
be
found
in
[39].
The
ini-
tial
dataset
w
as
in
XML
format,
featuring
entities
that
were
hierarchically
nested.
The
dataset
w
as
transformed
into
the
CoNLL
format
for
NER,
which
simplies
the
process
by
only
identifying
the
primary-le
v
el
entities.
In
the
initial
stages
of
data
preparation,
we
transformed
all
te
xt
to
lo
wercase
and
e
xcised
an
y
speci
al
characters
and
punctuation,
wit
h
the
e
xception
of
commas,
periods,
and
question
marks.
Subsequently
,
we
structured
the
dataset
with
annotations
as
per
the
schema
presented
in
T
able
1,
which
is
arranged
into
three
columns.
The
rst
column
contains
the
lo
wercase
v
ersion
of
the
input
sentences.
The
second
column
holds
the
labels
for
CPR,
with
eight
labels.
The
third
column
contains
the
NER
labels
,
with
six
labels.
T
ables
2
and
3
sho
w
the
statistics
of
our
dataset.
T
able
1.
F
ormat
of
dataset
for
MTDL
training
T
e
xt
CPR
label
NER
label
minh
U$
B-PER
ph
´
uc
U,
I-PER
con
L$
O
trai
L$
O
ˆ
ong
L$
O
hai
U$
B-PER
long
U$
I-PER
l
`
am
L$
O
o
L$
O
h
`
a
U$
B-LOC
giang
U.
I-LOC
Multi-task
deep
learning
for
V
ietnamese
capitalization
and
punctuation
...
(Phuong-Nhung
Nguyen)
Evaluation Warning : The document was created with Spire.PDF for Python.
1610
❒
ISSN:
2252-8938
T
able
2.
Dataset
statistics
of
CPR
tag
T
ag
T
rain
De
v
T
est
L$
760,182
38,885
189,058
L,
35,647
1,884
9,327
L.
29,917
1,531
6,596
L?
860
35
151
U$
127,973
6,630
30,313
U,
10,644
572
2,607
U.
8,097
400
1,843
U?
75
3
21
T
able
3.
Dataset
statistics
of
NER
tag
NER
tag
T
rain
De
v
T
est
B-LOC
15,871
841
2,002
B-ORG
8,627
383
2,152
B-PER
16,299
883
3,466
I-LOC
15,810
830
1,850
I-ORG
19,132
1,026
3,899
I-PER
11,621
644
3,077
O
886,035
45,333
223,470
An
alternati
v
e
method
for
data
collection
utilizing
TTS
systems
has
been
outlined
in
Figure
3,
which
is
suggested
as
a
substitute
for
traditional
recording
techniques:
−
Phase
1:
T
ransform
written
te
xt
into
audible
speech
with
the
use
of
a
TTS
tool
TTS
technology
has
been
under
e
xploration
in
V
ietnam
for
some
time.
Initially
,
VnSpeech
w
as
recognized
as
the
pioneering
V
ietnamese
TTS
system.
The
V
ietSound
system,
which
is
a
product
of
Ho
Chi
Minh
City
Uni
v
ersity
of
T
echnology
,
follo
wed
suit.
Y
et,
these
early
systems
had
setbacks,
particularly
in
pro-
ducing
speech
that
sounded
disconnected
and
unnatural.
In
recent
times,
more
rob
ust
and
widely
implemented
TTS
systems
ha
v
e
emer
ged.
Notably
,
FPT
.AI
of
fers
a
TTS
service
reno
wned
for
its
ef
fecti
v
eness.
Additionally
,
the
V
iettel’
s
VTCC.AI
system
and
Vbee’
s
AI
TTS
system
are
among
the
po
werful
players
in
the
eld.
T
elecom
companies
lik
e
MobiPhone
ha
v
e
also
introduced
their
TTS
solutions.
Google
has
been
pro
viding
a
TTS
tool
that
is
celebrated
for
its
nat
ural-sounding
v
oice,
positioning
it
as
one
of
t
h
e
top
contenders
in
TTS
softw
are.
Le
v
eraging
DeepMind’
s
speech
synthesis
e
xpertise,
Google
has
crafted
a
v
oice
quality
that
listeners
may
nd
challenging
to
dif
ferentiate
from
human
speech.
In
our
study
,
we
utilized
the
dataset
of
VLSP
2018
along
with
Google’
s
TTS
system
to
con
v
ert
te
xt
into
speech
for
both
training
and
v
alidation
purposes.
Due
to
resource
constraints,
our
focus
w
as
restricted
to
recording
audio
for
the
test
set
within
the
dataset,
which
comprises
4,272
samples
totaling
approximately
242,000
w
ords.
The
speech
dataset
w
as
generated
by
four
indi
viduals
who
read
the
te
xt
in
v
arious
settings,
resulting
in
o
v
er
26
hours
of
recorded
audio.
−
Phase
2:
Utilizing
the
ASR
system
for
speech
recognition
F
ollo
wing
the
acquisition
of
the
audio
dataset
from
both
TTS
and
recordings,
the
data
underwent
processing
via
an
ASR
system.
F
or
this
phase,
the
V
ietnam
articial
intelligence
solutions
(V
AIS)
ASR
system
w
as
selected
due
to
its
superior
performance
in
recognizing
V
ietnamese
speech
when
compared
to
other
systems
such
as
those
by
FPT
,
V
iettel,
Zalo,
and
Google,
as
noted
in
[40].
Additionally
,
the
V
AIS
ASR
system
w
as
pre
viously
utilized
in
research
focused
on
identifying
mark
ers
in
V
ietnamese
speech
processing
using
a
Pi
peline
approach,
as
mentioned
in
[13].
T
o
maintain
consistenc
y
for
result
comparison,
we
opted
to
continue
with
the
V
AIS
system
for
our
e
xperimental
frame
w
ork.
−
Phase
3:
Ev
aluation
of
ASR
te
xt
from
synthesized
speech
T
able
4
presents
the
error
rates
for
the
TTS-ASR
process
and
the
recorded
te
xt
through
ASR
(REC-
ASR)
method,
assessed
across
241,899
w
ords
from
the
dataset.
The
ndings
sho
wed
only
a
minor
discrepanc
y
in
ASR
accurac
y
when
comparing
tw
o
data
collection
methods,
which
w
as
deemed
pro
visionally
acceptable.
In
order
to
amass
a
substantial
v
olume
of
training
data,
we
sourced
information
from
authoritati
v
e
V
ietnamese
ne
ws
websites,
including
vne
xpress.net,
dantri.com.vn,
and
vietnamnet.vn.
This
content
w
as
then
processed
using
a
speech
synthesis
system
to
generate
the
v
oice
data.
W
e
emplo
yed
the
recorded
speech
from
the
VLSP
2018
dataset,
featuring
four
distinct
speak
ers,
as
our
test
set.
Int
J
Artif
Intell,
V
ol.
14,
No.
2,
April
2025:
1605–1615
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
1611
The
dataset
of
VLSP2018
is
se
gmented
into
training,
de
v
elopment,
and
testing
subsets.
The
training
subset
contains
30,280
V
ietnamese
sentences;
the
de
v
elopment
subset
includes
1,593
sentences;
and
the
testing
subset
comprises
4,271
sentences,
all
formatted
with
uppercase
letters
and
punctuation.
The
VLSP
sentences
were
annotated
for
capitalization,
punctuation,
and
NE
recognition.
F
or
creating
the
test
subset’
s
audio,
four
indi
viduals
recorded
the
test
te
xt
of
the
subset.
The
V
AIS
ASR
system
(https://v
ais.vn/en/speech-to-te
xt-core/)
w
as
utilized
to
con
v
ert
the
spok
en
w
ords
into
te
xt
data
for
e
v
aluation
purposes.
F
or
the
generation
of
synthe-
sized
audio
data
from
the
te
xtual
VLSP
data,
Google’
s
TTS
system
w
as
utilized.
Additionally
,
V
AIS’
s
ASR
system
w
as
emplo
yed
to
transcribe
the
synthetic
audio,
producing
a
te
xt
dataset
for
training
the
MTDL
model.
Figure
3.
Data
preparation
process
T
able
4.
Analysis
of
ASR
error
rates
(%)
of
synthesized
and
recorded
speech
Number
(%)
Outlier
(%)
Other
(%)
T
otal
(%)
TTS-ASR
1.061
2.418
1.582
5.068
REC-ASR
1.427
2.371
2.228
6.032
2.4.
Model
setting
The
V
iBER
T
model
is
a
scaled-do
wn
v
ersion
of
the
R
oB
E
R
T
a
base
,
whi
ch
is
enforced
using
the
f
airseq
frame
w
ork.
This
streamlined
model
retains
4
encoder
layers
as
its
R
oB
E
R
T
a
base
counterpart.
The
number
of
attention
heads
has
been
reduced
from
12
in
the
original
to
4
in
V
iBER
T
.
Each
training
instance
can
accommodate
up
to
512
tok
ens.
The
V
iBER
T
model
is
trained
using
the
Adam
optimization
algorithm
with
a
batch
size
of
512.
It
has
a
peak
learning
rate
set
at
0.0003
and
a
w
arm-up
phase
consisting
of
3,000
updates.
The
o
v
erall
number
of
updates
during
training
reaches
800,000.
This
training
process
is
performed
on
tw
o
Nvidia
2080T
i
GPUs,
each
with
12
GB
of
memory
,
and
spans
a
duration
of
5
weeks.
The
suggested
MTL
model
features
both
a
CPR
tagger
and
an
NE
tagger
,
both
congured
i
d
e
ntically
.
Each
tagger
utilizes
a
transformer
decoder
to
e
xtract
te
xt
features.
At
the
output,
a
CRF
model
is
emplo
yed
to
produce
se
v
en
distinct
labels.
These
labels
include
the
be
ginning
(B-X)
and
inside
(I-X)
tags
for
entities
in
the
cate
gories
of
or
g
anizations
(ORG),
persons
(PER),
and
locations
(LOC),
as
well
as
t
he
O
tag,
which
denotes
tok
ens
that
do
not
correspond
to
an
y
NE.
F
or
the
output
of
CPR,
the
model
will
assign
labels
that
include
’U’
to
denote
uppercase,
’L
’
for
lo
wercase,
combine
with
l
abels
’$’
(indicating
the
absence
of
punctuation);
’.
’
(period);
’,
’
(comma);
’?’
(question
m
ark)
to
add
punctuation
to
the
input
w
ord.
Consequently
,
each
w
ord
will
be
associated
with
one
of
the
follo
wing
eight
labels:
’U$’;
’L$’;
’U.
’;
’L.
’;
’U,
’;
’L,
’;
’U?’;
’L?’.
3.
RESUL
T
AND
DISCUSSION
3.1.
Ev
aluation
metrics
Our
model’
s
performance
is
assessed
through
precision,
recall,
and
the
F1-score
metrics
for
both
tasks
as
sho
wn
in
(4)
to
(5).
pr
ecision
=
T
P
T
P
+
F
P
(4)
r
ecal
l
=
T
P
T
P
+
F
N
(5)
r
ecal
l
=
2
∗
pr
ecision
∗
r
ecal
l
pr
ecision
+
r
ecal
l
(6)
Multi-task
deep
learning
for
V
ietnamese
capitalization
and
punctuation
...
(Phuong-Nhung
Nguyen)
Evaluation Warning : The document was created with Spire.PDF for Python.
1612
❒
ISSN:
2252-8938
where
true
positi
v
e
(TP)
denotes
the
number
of
punctuation
and
capitalization
tags
that
were
correctly
identied
by
the
model.
F
alse
positi
v
e
(FP)
signies
the
number
of
punctuation
and
capitalization
tags
that
were
mistak-
enly
identied.
On
the
other
hand,
f
al
se
ne
g
ati
v
e
(FN)
refers
to
the
number
of
punctuation
and
capitalization
tags
that
the
model
f
ail
to
recognize.
3.2.
Experiment
r
esult
T
able
5
presents
the
performance
of
our
MTDL
model
in
comparison
to
the
baseline
models
(without
NER
auxilary
task)
transformer
decoder
CRF
.
F
or
the
single-task
training,
the
state-of-the-art
method
in
v
olv
es
rening
V
iBER
T
pre-trained
language
model
for
tok
en
classication
tasks
by
adding
a
CRF
layer
on
top
[29].
The
result
presents
the
F1
scores
associated
with
v
arious
types
of
inputs
and
highlights
the
shortcomings
of
the
ASR-generated
te
xt
when
contrasted
with
te
xt
that
is
not
case-sensiti
v
e
(the
reference
te
xt
without
capitalization
and
punctuation
information)
while
the
CPR
results
decreased
from
0.7818
to
0.6319.
T
able
5.
Capitalization
and
punctuation
recognition
results
Input
type
Single
task
Multi-task
Uncased
reference
te
xt
0.7818
0.8375
ASR
output
0.7319
0.7780
The
MTL
approach
outperforms
the
single-task
learning
by
a
signicant
mar
gin,
as
e
videnced
by
the
scores
of
0.8375
and
0.7818,
respecti
v
ely
.
This
suggests
that
MTL
is
more
ef
fecti
v
e
at
generalizi
ng
from
uncased
reference
te
xt,
potential
ly
due
to
its
ability
to
capture
and
utilize
shared
representations
from
NER
that
are
rele
v
ant
to
the
input
te
xt.
The
high
performance
of
0.8375
for
MTL
in
processing
uncased
refere
nce
te
xt
indicates
rob
ust
feature
e
xtraction
and
learning,
which
can
handle
the
absence
of
case
information
ef
fecti
v
ely
.
This
could
be
attrib
uted
to
the
multi-task
frame
w
ork’
s
potential
to
le
v
erage
auxiliary
information
from
NER
task,
impro
ving
the
model’
s
understanding
of
conte
xt.
The
ndings
further
conrmed
that
inte
grating
the
NER
component
enhances
the
performance
of
the
CPR
model
processing
ASR-generated
te
xt.
The
F1
score
for
the
CPR
model
sa
w
a
notable
increase
of
approximately
6.2%,
rising
from
0.7319
t
o
0.778,
after
adding
the
NE
recognition
model
to
the
te
xt
of
ASR
output.
Additionally
,
the
NER
model
pro
v
ed
to
be
highly
v
aluable,
as
it
substantially
raised
the
F1
score
of
the
CPR
model
by
7.1%
with
unprocessed
te
xt,
which
in
this
case,
lack
ed
capitalization
and
punctuation
in
the
reference
te
xt.
3.3.
Ablation
study
Our
model’
s
performance
assessment
re
v
eals
the
signicance
of
its
indi
vidual
components
in
comparison
to
our
optimal
model,
which
incorporates
V
iBER
T
as
the
encoding
layer
alongside
all
components.
T
able
6
illustrates
the
essential
role
each
com
ponent
plays
in
our
multitask
learning
model’
s
ef
fecti
v
eness.
System
performance
deteriorates
upon
alteration
or
omission
of
components.
The
complete
model
achie
v
ed
an
F1
score
of
0.8375,
b
ut
this
score
dropped
to
0.8034
when
we
e
xcluded
the
NER
feature
and
relied
solely
on
conte
xtual
embedding
for
the
capitalization
and
punctuation
prediction
tasks.
When
we
substituted
the
o
v
erlap-chunk-mer
ging
the
input
te
xt
with
t
he
non-o
v
erlap-chunk-mer
ging,
the
F1
scores
for
the
capitaliza-
tion
prediction
task
and
punctuation
prediction
task
fell
to
0.8182.
The
ndings
indicate
that
the
presence
of
o
v
erlapping
w
ords
in
the
model
supplies
additional
information
for
prediction,
and
our
method
of
mer
ging
chunks
is
capable
of
appropriately
selecting
se
gments
from
t
he
areas
of
o
v
erlap.
The
most
substantial
decline
occurred
when
we
replaced
conte
xtual
embeddings
with
w
ord
embeddings
(GloV
e),
resulting
in
a
0.16
decrease
in
F1
score.
T
able
6.
Performance
metrics
with
component
modications
Component
Modication
Precision
Recall
F1
Full
None
0.8398
0.8352
0.8375
NER
Remo
v
al
0.8114
0.7953
0.8034
(4.08
%
↓
)
Chunking
Non
o
v
erlap
0.8215
0.8149
0.8182
(2.29
%
↓
)
Embedding
GloV
e
0.6826
0.6720
0.6773
(19.12
%
↓
)
Int
J
Artif
Intell,
V
ol.
14,
No.
2,
April
2025:
1605–1615
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
1613
4.
CONCLUSIONS
In
our
research,
we
introduced
the
application
of
a
NER
model
to
enhance
capitalization
and
punctuation
reco
v
ery
in
V
ietnamese
speech
processing
with
the
MTL
scheme.
The
e
xperimental
results
pro-
vided
e
vidence
of
the
syner
gy
between
these
methodologies.
W
e
also
created
and
presented
the
inaugural
speech
dataset
that
establishes
a
foundation
for
future
e
xplorations
i
nto
entity
e
xtraction
from
V
ietnamese
speech.
Additionally
,
we
highlighted
the
signicant
benets
of
utilizing
a
pre-trained
language
model
tailored
for
the
V
ietnamese
language
processing
in
CPR
and
NER
multi-tasks,
which
set
a
ne
w
benchmark
on
the
VLSP
2018
dataset.
W
e
ha
v
e
made
this
pre-trained
model
a
v
ailable
to
the
research
community
to
encourage
further
e
xploration.
This
contrib
ution
is
particularly
v
aluable
for
underrepresented
languages
such
as
V
ietnamese.
A
CKNO
WLEDGEMENTS
This
w
ork
w
as
supported
by
the
Institute
of
Information
T
echnology
,
V
ietnam
Academy
of
Science
and
T
echnology
under
project
number
CS23.05.
REFERENCES
[1]
D.
Y
u
and
L.
Deng,
A
utomatic
speec
h
r
eco
gnition:
a
deep
learning
appr
oac
h
.
in
Signals
and
Communication
T
echnology
.
London:
Springer
,
2015,
doi:
10.1007/978-1-4471-5779-3.
[2]
A.
Jarin,
A.
Santos
a,
M.
T
.
Ul
iniansyah,
L.
R.
Aini,
E.
Nurf
adhilah,
and
Gunarso,
“
Automatic
speech
recognition
for
indonesian
medical
dictation
in
cloud
en
vironment,
”
IAES
International
J
ournal
of
Articial
Intellig
ence
,
v
ol.
13,
no.
2,
pp.
1762–1772,
2024,
doi:
10.11591/ijai.v13.i2.pp1762-1772.
[3]
R.
P
appag
ari,
P
.
Zelask
o,
A.
Mik
olajczyk,
P
.
Pezik,
and
N.
Dehak,
“Joint
prediction
of
truecasing
and
punctuation
for
con
v
ersational
speech
in
lo
w-resource
scenarios,
”
in
2021
IEEE
A
utomatic
Speec
h
Reco
gnition
and
Under
standing
W
orkshop
(ASR
U)
,
IEEE,
2021,
pp.
1185–1191,
doi:
10.1109/ASR
U51503.2021.9687976.
[4]
F
.
M.
M.
Batista
and
D.
N.
J.
N.
Mamede,
“Reco
v
ering
capitalization
and
punctuation
marks
on
speech
transcriptions,
”
Ph.D
Thesis
,
Fields
of
Science
and
T
echnology
,
Instituto
Superior
T
´
ecnico,
Lisboa,
Portug
al,
2011.
[5]
N.
Uef
ng,
M.
Bisani,
and
P
.
V
ozila,
“Impro
v
ed
models
for
automatic
punctuation
prediction
for
spok
en
and
written
te
xt,
”
in
Inter
speec
h
2013
,
ISCA,
2013,
pp.
3097–3101,
doi:
10.21437/Interspeech.2013-675.
[6]
Q.
H.
Pham,
B
.
T
.
Nguyen,
and
N.
V
.
Cuong,
“Punctuation
prediction
for
vietnamese
te
xts
using
condi
tional
random
elds,
”
in
Pr
oceedings
of
the
T
enth
International
Symposium
on
Information
and
Communication
T
ec
hnolo
gy
-
SoICT
2019
,
Ne
w
Y
ork,
USA:
A
CM
Press,
2019,
pp.
322–327,
doi:
10.1145/3368926.3369716.
[7]
P
.
˙
Zelask
o,
P
.
Szyma
´
nski,
J.
Mizg
ajski,
A.
Szymczak,
Y
.
Carmiel,
and
N.
Dehak,
“Punctuation
prediction
model
for
con
v
ersational
speech,
”
in
arXiv-Computer
Science
,
2018,
pp.
1–5.
[8]
R.
H.
Susanto,
H.
L.
Chieu,
and
W
.
Lu,
“Learning
to
capi
talize
with
character
-le
v
el
recurrent
neural
netw
orks:
an
empirical
study
,
”
in
Pr
oceedings
of
the
2016
Confer
ence
on
Empiric
al
Methods
in
Natur
al
Langua
g
e
Pr
ocessing
,
Stroudsb
ur
g,
USA:
Association
for
Computational
Linguistics,
2016,
pp.
2090–2095,
doi:
10.18653/v1/D16-1225.
[9]
H.
N.
T
.
Thu,
B.
N.
Thai,
H.
N.
V
.
Bao,
T
.
D.
Quoc,
M.
L.
Chi,
and
H.
N.
T
.
Minh,
“Reco
v
ering
capitalization
for
automatic
speech
recognition
of
vietnamese
using
transformer
and
chunk
mer
ging,
”
in
2019
11th
International
Confer
ence
on
Knowledg
e
and
Systems
Engineering
(KSE)
,
IEEE,
2019,
pp.
1–5,
doi:
10.1109/KSE.2019.8919342.
[10]
R.
Rei,
N.
M.
Guerreiro,
and
F
.
Batista,
“
Automatic
truecasing
of
video
subtitles
using
bert:
a
multilingual
adaptable
approach,
”
in
Information
Pr
ocessing
and
Mana
g
ement
of
Uncertai
nty
in
Knowledg
e-Based
Systems
,
2020,
pp.
708–721,
doi:
10.1007/978-3-
030-50146-4
52.
[11]
A.
Gra
v
ano,
M.
Jansche,
and
M.
Bacchiani,
“Restoring
punctuation
and
capitalization
in
transcribed
speech,
”
in
2009
IEEE
Interna-
tional
Confer
ence
on
Acoustics,
Speec
h
and
Signal
Pr
ocessing
,
IEEE,
2009,
pp.
4741–4744,
doi:
10.1109/ICASSP
.2009.4960690.
[12]
A.
V
¯
ara
vs
and
A.
Salimbaje
vs,
“Restoring
punctuation
and
capitalization
using
transformer
models,
”
in
Statistical
Langua
g
e
and
Speec
h
Pr
ocessing
,
Cham,
Switzerland:
Springer
,
2018,
pp.
91–102,
doi:
10.1007/978-3-030-00810-9
9.
[13]
T
.
B.
Nguyen,
Q.
M.
N
guyen,
T
.
T
.
H.
Nguyen,
Q.
T
.
Do,
and
C.
M.
Luong,
“Impro
ving
vietnamese
named
entity
recogni-
tion
from
speech
using
w
ord
capitalization
and
punctuation
reco
v
ery
models,
”
in
Inter
speec
h
2020
,
2020,
pp.
4263–4267,
doi:
10.21437/Interspeech.2020-1896.
[14]
F
.
Batista,
I.
T
rancoso,
and
N.
Mamede,
“
Automatic
reco
v
ery
of
punctuation
m
arks
and
capitalization
information
for
iberian
languages,
”
in
Pr
oceedings
of
the
I
Iberian
SL
T
ec
h
2009
,
2009,
pp.
99–102.
[15]
J.
P
.
Y
amron,
I.
Carp,
L.
Gillick,
S.
Lo
we,
and
P
.
V
.
Mulbre
gt,
“
A
hidden
mark
o
v
model
approach
to
te
xt
se
gmentation
and
e
v
ent
tracking,
”
in
Pr
oceedings
of
the
1998
IEEE
International
Confer
ence
on
Acoustics,
Speec
h
and
Signal
Pr
ocessing
,
IEEE,
1998,
pp.
333–336,
doi:
10.1109/ICASSP
.1998.674435.
[16]
O.
T
ilk
and
T
.
Alum
¨
ae,
“LSTM
for
punctuation
restoration
in
speech
transcripts,
”
in
Inter
speec
h
2015
,
ISCA,
2015,
pp.
683–687,
doi:
10.21437/Interspeech.2015-240.
[17]
X.
W
ang,
H.
T
.
Ng,
and
K.
C.
Sim,
“Dynamic
conditional
random
elds
for
joint
sentence
boundary
and
punctuation
prediction,
”
in
Inter
speec
h
2012
,
ISCA,
2012,
pp.
1384–1387,
doi:
10.21437/Interspeech.2012-398.
[18]
M.
Lui
and
L.
W
ang,
“Reco
v
ering
casing
and
punctuation
using
conditional
random
el
ds,
”
in
Pr
oceedings
of
A
ustr
alasian
Lan-
gua
g
e
T
ec
hnolo
gy
Association
W
orkshop
,
2013,
pp.
137–141.
[19]
G.
Ramena,
D.
Nag
araju,
S.
Moharana,
D.
P
.
Mohanty
,
and
N.
Purre,
“
An
ef
cient
architecture
for
predicti
ng
the
case
of
characters
using
sequence
models,
”
in
2020
IEEE
14th
International
Confer
ence
on
Semantic
Computing
(ICSC)
,
IEEE,
2020,
pp.
174–177,
doi:
10.1109/ICSC.2020.00035.
Multi-task
deep
learning
for
V
ietnamese
capitalization
and
punctuation
...
(Phuong-Nhung
Nguyen)
Evaluation Warning : The document was created with Spire.PDF for Python.
1614
❒
ISSN:
2252-8938
[20]
O.
T
ilk
and
T
.
Alum
¨
ae,
“Bidirectional
recurrent
neural
netw
ork
with
attention
mechanism
for
punctuation
restoration,
”
in
Inter
speec
h
2016
,
2016,
pp.
3047–3051,
doi:
10.21437/Interspeech.2016-1517.
[21]
A.
Zahra,
A.
F
.
Hidayatullah,
and
S.
Rani,
“Bidirectional
long-short
term
memory
and
conditional
random
eld
for
tourism
named
entity
recognition,
”
IAES
International
J
ournal
of
Articial
Intellig
ence
,
v
ol.
11,
no.
4,
pp.
1270–1277,
2022,
doi:
10.11591/ijai.v11.i4.pp1270-1277.
[22]
R
.
P
an,
J.
A.
Garc
´
ıa-D
´
ıaz,
and
R.
V
alencia-Garc
´
ıa,
“Ev
aluation
of
transformer
-based
models
for
punctuation
and
capitaliza-
tion
restoration
in
spanish
and
portuguese,
”
in
Natur
al
Langua
g
e
Pr
ocessing
and
Information
Systems
,
2023,
pp.
243–256,
doi:
10.1007/978-3-031-35320-8
17.
[23]
A.
Nagy
,
B.
B
ial,
and
J.
´
Acs,
“
Automatic
punctuation
restoration
with
bert
models,
”
arXiv-Computer
Science
,
pp.
1–11,
2021.
[24]
M
.
Courtland,
A.
F
aulkner
,
and
G.
McElv
ain,
“Ef
cient
automatic
punctuation
restoration
using
bidirectional
transformers
with
rob
ust
inference,
”
in
Pr
oceedings
of
the
17th
International
Confer
ence
on
Spok
en
Langua
g
e
T
r
anslation
,
2020,
pp.
272–279,
doi:
10.18653/v1/2020.iwslt-1.33.
[25]
H.
T
.
T
.
Uyen,
N.
A.
T
u,
and
T
.
D.
Huy
,
“V
ietnames
e
capitalization
and
punctuation
reco
v
ery
models,
”
in
Inter
speec
h
2022
,
ISCA,
2022,
pp.
3884–3888,
doi:
10.21437/Interspeech.2022-931.
[26]
B
.
Nguyen
et
al.
,
“F
ast
and
accurate
capitalization
and
punctuat
ion
for
automatic
speech
recognition
using
transformer
and
chunk
mer
ging,
”
in
2019
22nd
Confer
ence
of
the
Oriental
COCOSD
A
International
Committee
for
the
Co-or
dination
and
Standar
disation
of
Speec
h
Databases
and
Asses
sment
T
ec
hniques
(O-COCOSD
A)
,
IEEE,
2019,
pp.
1–5,
doi:
10.1109/O-
COCOSD
A46868.2019.9041202.
[27]
V
.
T
.
Bui
and
O.
T
.
T
ran,
“Punctuation
prediction
in
vietnamese
asrs
using
transformer
-based
models,
”
in
PRICAI
2021:
T
r
ends
in
Articial
Intellig
ence
,
2021,
pp.
191–204,
doi:
10.1007/978-3-030-89363-7
15.
[28]
H.
T
ran,
C.
V
.
Dinh,
Q.
Pham,
and
B.
T
.
Nguyen,
“
An
ef
cient
transformer
-based
model
for
vietnamese
punctuation
prediction,
”
in
International
Confer
ence
on
Indus
trial,
Engineering
and
Other
Applications
of
Applied
Intellig
ent
Systems
,
2021,
pp.
47–58,
doi:
10.1007/978-3-030-79463-7
5.
[29]
T
.
T
.
H.
Nguyen,
T
.
B.
Nguyen,
N.
P
.
Pham,
Q.
T
ruong,
T
.
L.
Le,
and
C.
M.
Luong,
“T
o
w
ard
human-friendly
ASR
systems:
reco
v
ering
capitalization
and
punctuation
for
vietnamese
te
xt,
”
IEICE
TRANSA
CTIONS
on
Information
and
Systems
,
v
ol.
104,
no.
8,
pp.
1195–1203,
2021.
[30]
F
.
Batista,
D.
Caseiro,
N.
Mamede,
and
I.
T
rancos
o,
“Reco
v
ering
punctuation
marks
for
automatic
speech
recognition,
”
in
Inter
-
speec
h
2007
,
ISCA:
ISCA,
2007,
pp.
2153–2156,
doi:
10.21437/Interspeech.2007-581.
[31]
W
.
Lu
and
H.
T
.
Ng,
“Better
punctuation
prediction
with
dyna
mic
conditional
random
elds,
”
in
EMNLP
2010
-
Confer
ence
on
Empirical
Methods
in
Natur
al
Langua
g
e
Pr
ocessing
,
2010,
pp.
177–186.
[32]
E.
Cho,
J.
Niehues,
and
A.
W
aibel,
“NMT
-based
se
gmentation
and
punctuation
insertion
for
real-time
spok
en
language
translation,
”
in
Inter
speec
h
2017
,
ISCA:
ISCA,
Aug.
2017,
pp.
2645–2649,
doi:
10.21437/Interspeech.2017-1320.
[33]
T
.
Pham,
N.
Nguyen,
Q.
Pham,
H.
Cao,
and
B.
Nguyen,
“V
ietnamese
punctuation
prediction
using
deep
neural
netw
orks,
”
in
SOFSEM
2020:
Theory
and
Pr
actice
of
Computer
Science
,
2020,
pp.
388–400,
doi:
10.1007/978-3-030-38919-2
32.
[34]
N.
Nik
entari
and
H.
L.
W
ei,
“M
ulti-task
learning
using
non-linear
autore
gressi
v
e
models
and
recurrent
neural
netw
orks
for
tide
le
v
el
forecasting,
”
International
J
ournal
of
Electrical
and
Computer
Engineering
,
v
ol.
14,
no.
1,
pp.
960–970,
2024,
doi:
10.11591/ijece.v14i1.pp960-970.
[35]
R
.
M.
Samant,
M.
R.
Bachute,
S.
Gite,
and
K.
K
otecha,
“Frame
w
ork
for
deep
learning-based
language
models
using
multi-task
learn-
ing
in
natural
language
understanding:
a
systematic
literature
re
vie
w
and
future
directions,
”
IEEE
Access
,
v
ol.
10,
pp.
17078–17097,
2022,
doi:
10.1109/A
CCESS.2022.3149798.
[36]
T
.-N.
Phung,
M.
C.
Luong,
and
M.
Akagi,
“
An
in
v
estig
ation
on
perceptual
line
spectral
frequenc
y
(PLP-LSF)
tar
get
stability
ag
ainst
the
v
o
wel
neutralization
phenomenon,
”
in
2011
3r
d
International
Confer
ence
on
Signal
Acquisition
and
Pr
ocessing
(ICSAP
2011)
,
2011,
pp.
512–514.
[37]
P
.
T
.
Nghia,
L.
C.
Mai,
and
M.
Akagi,
“Impro
ving
the
naturalness
of
concatenati
v
e
vietnamese
speech
synthesis
under
limited
data
conditions,
”
J
ournal
of
Computer
Science
and
Cybernetics
,
v
ol.
31,
no.
1,
2015,
doi:
10.15625/1813-9663/31/1/5064.
[38]
V
.
L.
Phung,
H.
K.
Phan,
A.
T
.
Dinh,
and
Q.
B.
Nguyen,
“Da
ta
processing
for
optimizing
naturalness
of
vietnamese
te
xt-
to-speech
system,
”
in
2020
23r
d
Confer
ence
of
the
Oriental
COCOSD
A
International
Com
mittee
for
the
Co-or
dination
and
Standar
disation
of
Speec
h
Databases
and
Asses
sment
T
ec
hniques
(O-COCOSD
A)
,
IEEE,
2020,
pp.
1–6,
doi:
10.1109/O-
COCOSD
A50338.2020.9295025.
[39]
H.
T
.
M.
Nguyen,
Q.
T
.
Ngo,
L.
X.
V
u,
V
.
M.
T
ran,
and
H.
T
.
T
.
Nguyen,
“VLSP
shared
task:
named
entity
recognition,
”
J
ournal
of
Computer
Science
and
Cybernetics
,
v
ol.
34,
no.
4,
pp.
283–294,
2019,
doi:
10.15625/1813-9663/34/4/13161.
[40]
C.
H.
Ng
a,
C.-T
.
Li,
Y
.-H.
Li,
and
J.-C.
W
ang,
“
A
surv
e
y
of
vietnamese
automatic
speech
recognition,
”
in
2021
9th
Inter
national
Confer
ence
on
Or
ang
e
T
ec
hnolo
gy
(ICO
T)
,
IEEE,
2021,
pp.
1–4,
doi:
10.1109/ICO
T54518.2021.9680652.
BIOGRAPHIES
OF
A
UTHORS
Phuong-Nhung
Nguy
en
recei
v
ed
the
master
de
gree
from
the
Uni
v
ersity
of
T
echnology
-
Hanoi
National
Uni
v
ersity
,
majoring
in
information
technology
.
Her
resea
rch
interests
include
data
mining,
machine
learning,
soft
computing,
and
fuzzy
computing.
She
is
currently
a
researcher
at
the
Institute
of
Information
T
echnology
,
V
ietnam
Academy
of
Science
and
T
echnology
.
She
can
be
contacted
at
email:
ntpnhung@ioit.ac.vn.
Int
J
Artif
Intell,
V
ol.
14,
No.
2,
April
2025:
1605–1615
Evaluation Warning : The document was created with Spire.PDF for Python.