IAES
Inter
national
J
our
nal
of
Articial
Intelligence
(IJ-AI)
V
ol.
14,
No.
3,
June
2025,
pp.
2358
∼
2366
ISSN:
2252-8938,
DOI:
10.11591/ijai.v14.i3.pp2358-2366
❒
2358
Depr
ession
and
post
traumatic
str
ess
disorder
analysis
with
multi-modal
data
Rajalakshmi
Si
v
anaiah,
Angel
Deborah
Suseelan,
Krupa
Elizabeth
Thannickal,
Sanmati
Pirabahar
Department
of
Computer
Science
and
Engineering,
Sri
Si
v
asubramaniya
Nadar
Colle
ge
of
Engineering,
Chennai,
India
Article
Inf
o
Article
history:
Recei
v
ed
Sep
3,
2024
Re
vised
Feb
14,
2025
Accepted
Mar
15,
2025
K
eyw
ords:
Articial
intelligence
Deep
learning
Ensemble
learning
Mental
health
T
e
xt,
video
and
audio
data
T
ransformer
models
ABSTRA
CT
W
ith
an
increasing
global
population
and
more
people
li
ving
to
the
age
when
major
depressi
v
e
disorder
(MDD)
or
post
t
raumatic
stress
disorder
(PTSD)
com-
monly
occurs,
the
number
of
those
who
suf
fer
from
such
di
sorders
is
rising.
Studies
ha
v
e
also
sho
wn
a
high
lik
elihood
of
comorbidity
between
these
2
dis-
orders.
This
comorbidity
ca
n
w
orsen
s
ymptoms,
increase
the
risk
of
chronic-
ity
,
and
complicate
treatment,
signicantly
impacting
patients’
emotional
well-
being
and
social
and
occupational
functioning.
There
is
a
need
to
enable
f
aster
and
reliable
diagnosis
methods
,
while
taking
into
account
the
subjecti
vity
of
indi
viduals
and
the
role
of
beha
vioural
cues.
The
proposed
approach
analyses
the
combination
of
audio,
video
and
te
xt
input
features
(multi-modal
data)
of
the
subject
to
determine
the
se
v
erity
class
of
MDD
and
PTSD.
The
DistilBER
T
transformer
is
used
for
learning
and
b
uilding
a
model
with
the
te
xtual
modality
and
random
forest
classiers
for
the
audio
and
video
modal
ities.
An
ensemble
of
thes
e
3
models
from
3
modalities
performs
better
in
the
nal
classication
of
MDD
and
PTSD
when
compared
to
indi
vidual
models.
This
w
ork
also
co
v
ers
a
comparison
of
the
models
with
dif
ferent
splits
on
the
dataset.
This
ensembled
system
sho
ws
an
impro
v
ed
accurac
y
of
2%
to
7%
for
the
MDD
and
PTSD
multi
class
classication
o
v
er
the
models
tested
on
indi
vidual
modalities.
This
is
an
open
access
article
under
the
CC
BY
-SA
license
.
Corresponding
A
uthor:
Rajalakshmi
Si
v
anaiah
Department
of
Computer
Science
and
Engineering,
Sri
Si
v
asubramaniya
Nadar
Colle
ge
of
Engineering
Chennai
-
603110,
T
amil
Nadu,
India
Email:
rajalakshmis@ssn.edu.in
1.
INTR
ODUCTION
The
role
of
machine
learning
and
deep
learning
to
automate
disease
diagnosis
and
treatment
is
e
v
er
-
increasing.
One
such
area
where
automation
could
be
of
immense
help
is
in
mental
health
diagnosis.
According
to
the
W
orld
Health
Or
g
anization
(WHO),
mental
health
disorders
are
the
major
contrib
utors
to
years
li
v
ed
with
disability
globally
.
The
rst
step
to
w
ards
helping
such
patients
is
ob
viously
the
proper
and
correct
identication
of
the
disorder
the
y
might
be
suf
fering
from
in
init
ial
stage
itself.
But
as
per
a
WHO
2022
report
[1],
nearly
50%
of
the
global
population
resides
in
countries
where
the
ratio
of
psychiatrists
to
peopl
e
is
approximately
one
psychiatrist
for
e
v
ery
200,000
indi
viduals
or
more.
The
tw
o
disorders
this
paper
focuses
on
are
depression,
also
kno
wn
as
major
depressi
v
e
disorder
(MDD)
and
post
traumatic
stress
disorder
(PTSD).
Bre
win
et
al.
[2]
ha
v
e
sho
wn
high
correlation
between
MDD
and
PTSD,
with
approximately
50%
of
people
with
PTSD
also
suf
fering
from
MDD.
Current
methodology
to
assess
depression
and
PTSD
is
using
self-report
questionnaires,
lik
e
patient
health
questionnaire
(PHQ)
PHQ-8
and
PTSD
Check
List
-
Ci
vilian
v
ersion
(PCL-C)
respecti
v
ely
,
follo
wed
by
clinical
intervie
ws
and
a
re
vie
w
of
medical
his
tory
.
This
system
relies
on
se
v
eral
v
ariable
f
actors
J
ournal
homepage:
http://ijai.iaescor
e
.com
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
2359
and
is
not
v
ery
reliable
due
to
multiple
subjecti
v
e
issues
and
beha
vioural
cues
of
the
indi
vidual.
Also,
present
psychosocial
intervie
ws
tak
e
longer
time,
while
an
automated
s
ystem
lik
e
Ellie,
that
w
as
used
to
retrie
v
e
the
Extended
Distress
Analysis
Intervie
w
Corpus
(E-D
AIC)
dataset,
tak
es
lesser
time
to
eng
age
the
patient.
As
in
the
case
of
most
medical
treatment,
an
early
diagnosis
increases
chances
of
a
f
aster
reco
v
ery
and
can
pre
v
ent
the
onset
of
w
orse
conditions.
This
is
the
most
pressing
reason
to
identify
the
co-e
xistence
of
depression
and
PTSD
as
ongoing
research
suggests
[2]
a
link
between
the
tw
o
conditions
could
inuence
treatment
approaches.
One
of
the
areas
where
the
International
Audio/V
isual
Emot
ion
Challenge
and
W
orkshop
(A
VEC)
has
attempted
to
spur
interest
i
s
in
depression
detection.
In
2019,
the
E-D
AIC
dataset
described
in
section
2.1.
w
as
pro
vided
along
with
the
results
of
a
baseline
model
to
participants
[3].
The
baseline
model
in
this
challenge
considered
a
binary
classication
for
depression
detection
using
a
g
ated
recurrent
unit
(GR
U)
-
recurrent
neural
netw
ork
(RNN)
model.
One
of
the
more
successful
attempts
from
A
VEC
2019
w
as
of
a
multi-le
v
el
attention
netw
ork
that
used
a
fusion
approach
to
select
the
most
im-
portant
features
of
each
input
modality
[4].
These
are
gi
v
en
to
a
series
of
feed-forw
ard
netw
orks
whose
nal
outputs
are
ag
ain
fused
in
a
stack
ed
bidirectional
long
short-term
memory
(BLS
TM)
model.
The
resulting
system
succeeded
in
outperforming
the
challenge
baseline
root
mean
square
error
(RMSE)
by
17.52%,
while
also
identifying
the
relati
v
e
importance
of
the
dif
ferent
input
data
types
.
The
study
utilizes
databases
from
mul-
tiple
languages
for
acoustic
feature
s
election
and
then
e
xtracts
te
xt-based
features
for
depression
assessment
[5].
Gi
v
en
the
small
amount
of
traini
n
g
data
a
v
ailable,
ef
fecti
v
e
data
selection
is
crucial,
and
the
proposed
multi-lingual
method
outperformed
baseline
algorithms
in
feature
selection,
impro
ving
depressi
on
assessment
accurac
y
.
Research
has
also
been
done
for
a
predicti
v
e
model
from
te
xt
using
a
combination
of
a
long-short
term
memory
(LSTM)
model
and
a
RNN
[6].
The
RNN
is
trained
on
te
xtual
data
to
recognize
depression
based
on
semantics
and
writt
en
content.
The
proposed
frame
w
ork
has
a
99.0%
accurac
y
rate,
whi
ch
is
higher
than
frequenc
y-based
deep
learning
models
for
te
xt-based
detection,
and
also
has
a
lo
wer
f
alse
positi
v
e
rate.
Za-
v
orina
and
Makaro
v
describe
a
transformer
encoder
netw
ork
in
their
study
on
v
oice-base
depression
detection
[7],
compar
ed
to
top
baseline
approaches
using
the
E-D
AIC
dataset.
T
o
handle
the
small
size
of
the
a
v
ailable
dataset,
lo
w-le
v
el
features
are
e
xtracted
f
rom
audio
recordings
and
then
augmented.
Their
netw
ork
attains
a
recognition
accurac
y
of
73.51%
on
the
E-D
AIC
database,
which
is
better
than
the
accurac
y
of
65.85%
and
66.35%
obtained
by
traditional
approaches.
Research
on
visual
f
eature
selection
[8]
sho
ws
that
the
high
di-
mensionality
of
these
features
poses
a
challenge
in
identifying
informati
v
e
and
discriminati
v
e
ones.
T
o
address
this,
a
h
ybrid
dimensionality
reduction
approach
is
introduced
that
combines
lter
and
wrapper
methods.
Using
the
technique
of
mobile
cro
wd
sensing
combined
with
a
task-based
approach,
a
design
for
an
end-to-end
ML
pipeline
from
data
collection
to
binary
classication
has
also
been
proposed
[9].
This
study
e
xperimented
with
v
arious
features
from
multi-modalities,
feature
selection
techniques,
fused
features,
and
machine
learning
classiers
such
as
logistic
re
gression
and
support
v
ector
machines
(SVM).
The
SVM
algorithm
produced
the
best
accurac
y
,
reaching
86%,
when
features
from
all
three
modalities
were
fused.
The
re
vie
w
on
the
need
and
the
methods
u
s
ed
for
PTSD
are
discussed
in
[10],
[11].
Apart
from
the
PHQ-8
scale
for
depression,
the
Hamilton
Depression
Rating
Scale
(HDRS)
is
also
used
in
research
to
train
models
for
detecting
depression
se
v
erity
[12].
This
technique
emplo
ys
deep
neural
netw
orks
to
detect
video,
speech,
and
te
xt
modalities,
and
combines
the
modalities
using
weighted
fusion
to
calculate
the
total
points
in
the
HDRS.
Results
sho
wed
that
the
video
modality
achie
v
ed
an
accurac
y
of
66%,
the
speech
modalit
y
achie
v
ed
81%,
and
82%
for
the
te
xt
modality
.
Apart
from
the
E-D
AIC
and
D
AIC-W
oZ
dat
aset,
studies
ha
v
e
been
carried
out
on
the
D-Vlog
dataset
[13],
compri
sing
961
Y
ouT
ube
vlogs.
This
research
de
v
eloped
models
based
on
the
non-v
erbal
beha
vior
of
indi
viduals
in
real-w
orld
scenarios,
using
only
the
acoustic
and
visual
features.
A
cross-attention
mechanism
w
as
used
to
disco
v
er
the
connection
between
features,
follo
wed
by
comparisons
with
baseline
models.
There
is
a
scanty
of
e
xisting
w
ork
a
v
ailable
for
depression
detection
system
with
multi-modal
data.
Either
the
y
use
a
unimodal
[14]
or
bi-modal
[15]
data
whereas
we
concentrate
on
e
xtracting
features
from
multi-modal
data
of
a
particular
user
for
better
understanding
of
the
user’
s
beha
viour
.
Mostly
all
the
w
orks
are
done
for
binary
cl
assication
(depression
or
not)
and
we
ha
v
e
considered
a
multi-class
scenario
(5
classes
to
sho
w
the
le
v
el
of
depression).
The
co-occurrence
of
both
MDD
and
PTSD
in
an
indi
vidual
is
also
not
analysed.
W
e
ha
v
e
aimed
to
concentrate
on
these
issues
and
de
v
elop
a
multi-modal
system
to
detect
the
depression
and
PTSD.
This
section
e
xplains
the
need
and
current
research
in
depression
and
PTSD
detection.
Section
2
describes
the
dataset
and
methodologies
used
to
solv
e
the
problem.
Section
3
e
xplains
the
e
xperimental
results
and
the
discussions.
Section
4
concludes
the
ndings
and
future
w
ork.
Depr
ession
and
post
tr
aumatic
str
ess
disor
der
analysis
with
multi-modal
data
(Rajalakshmi
Sivanaiah)
Evaluation Warning : The document was created with Spire.PDF for Python.
2360
❒
ISSN:
2252-8938
2.
METHOD
2.1.
Dataset
description
The
E-D
AIC
dataset
consists
of
275
intervie
ws
of
participants
collected
both
in
a
W
izard
of
Oz
mode
(human-controlled
agent)
and
with
a
purely
AI
agent
named
Ellie
[16].
Each
of
the
participant
data
labelled
as
XXX
P
has
been
or
g
anized
into
a
directory
with
ra
w
audio
les
for
each
intervie
w
(XXX
Audio.w
a
v).
T
anscripts
of
each
audio
le
(XXX
T
ranscript.csv),
pose,
g
aze
and
action
units
from
video
collected
with
a
time
step
of
0.033
seconds
and
PHQ-8
and
PCL-C
scores
of
participants.
These
scores
are
used
to
determine
the
se
v
erity
class
labels
of
the
samples
as
described
in
subsection
2.2.
2.2.
Modules
2.2.1.
Cleaning,
featur
e
extraction
and
labelling
Pre
vious
studies
ha
v
e
pointed
out
some
kno
wn
issues
with
the
ra
w
audio
les
in
the
E-D
AIC
dataset
[17].
F
or
e
xample,
there
are
long
intermittent
pauses
in
some
intervie
ws
which
should
be
re
mo
v
ed.
T
o
han-
dle
these
the
ra
w
audio
is
trimmed
by
remo
ving
re
gions
ha
ving
v
olumes
less
than
33dB.
The
OpenSMILE
toolkit
is
used
to
e
xtract
the
Compare2016
set
of
65
features
from
the
trimmed
ra
w
audio.
A
frame
size
of
0.1
second
w
as
used
for
this.
Gi
v
en
the
v
arying
length
of
indi
vidual
intervie
ws,
Compare2016
features
and
pose,
g
aze
and
action
units
for
video
are
also
v
ariable
in
length
for
each
participant.
T
o
deal
with
this,
a
simple
a
v
eraging
of
the
e
xtracted
features
is
done,
so
that
this
is
reduced
to
a
single
feature
frame
each
for
audio
and
video.
F
or
each
intervie
w
,
the
te
xt
transcript
is
split
into
timestamps
as
separate
lines
or
phrases
of
con
v
ersation.
All
te
xts
for
a
single
participant
are
concatenated
into
one
single
ro
w
of
te
xt.
The
labelling
w
as
done
for
the
cleaned
sample
based
on
the
range
in
which
the
corresponding
PHQ-8
and
PCL-C
score
f
alls
under
.
The
ranges
used
for
labelling
are
menti
oned
in
T
ables
1
and
2,
adopted
from
the
e
xisting
standards
for
MDD
and
PTSD
classication
[18],
[19].
The
dataset
comprises
of
the
PHQ-8
score
and
PCL-C
score
only
and
we
ha
v
e
con
v
erted
the
score
to
multiple
class
labels
based
on
the
se
gre
g
ation
v
alues
mentioned
in
T
ables
1
and
2
using
[20],
[21].
Figure
1
sho
ws
the
data
distrib
ution
for
the
MDD
dataset
for
trai
n
i
ng,
de
v
el-
opment
and
test
splits.
The
outer
circle
denotes
the
training
dataset,
middle
circle
denotes
the
de
v
elopment
dataset
and
the
inner
circle
denot
es
the
t
est
dataset.
Fi
gure
2
sho
ws
the
data
distrib
ution
for
PTSD
dat
aset
for
training,
de
v
elopment
and
test
datasets.
T
able
1.
MDD
se
v
erity
labels
from
PHQ-8
scores
PHQ-8
score
Se
v
erity
class
0-4
Absence
5-9
Mild
10-14
Moderate
15-19
Moderate
to
se
v
ere
>
=20
Se
v
ere
T
able
2.
PTSD
se
v
erity
labels
from
PCL-C
scores
PCL-C
score
Se
v
erity
class
17-27
Absence
28-29
Maybe
mild
30-44
Moderate
to
high
45-85
Se
v
ere
2.2.2.
Multimodal
data
-
indi
vidual
models
After
conducting
se
v
eral
e
xperiments
with
v
arious
machine
learning
(logistic
re
gression,
decision
trees,
random
forest,
SVM,
k
nearest
neighbours,
g
aussian
nai
v
e
Bayes)
[22]
and
deep
learning
models
[23]
(BER
T
[24],
RNN,
CNN,
LSTM,
Roberta,
DistilBER
T
[25])
the
nal
decision
w
as
made
to
choose
the
Dis-
tilBER
T
m
odel
for
te
xt
and
the
support
v
ector
and
random
forest
classiers
for
audio
and
video
data,
as
these
models
e
xhibited
a
higher
accurac
y
rate
compared
to
other
models.
The
DistilBER
T
model
used
is
a
transformer
model
under
Python’
s
torch
library
,
where
the
encoded
input
te
xt
is
passed
through
a
pre-trained
“distilbert-
base-uncased”
model
that
generates
a
sequence
of
conte
xtualized
embeddings
for
each
tok
en
in
it.
This
is
follo
wed
by
a
dropout
l
ayer
ha
ving
a
dropout
rate
of
0.1
to
reduce
o
v
ertting,
and
a
linear
classication
layer
that
maps
the
hidden
state
size
of
the
DistilBER
T
model
[26]
to
the
number
of
output
classes.
This
produces
the
logits,
which
are
ra
w
scores
indicating
the
lik
elihood
of
each
input
belonging
to
each
output
class.
Int
J
Artif
Intell,
V
ol.
14,
No.
3,
June
2025:
2358–2366
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
2361
The
random
forest
classier
is
used
for
MDD
audio
and
video
training
and
for
PTSD
video
modelling
[27].
The
support
v
ector
classier
w
as
chosen
for
PTSD
audio
training
as
it
displayed
a
higher
accurac
y
than
the
corresponding
random
forest
model.
The
training
w
as
carried
out
for
indi
vidual
models
under
te
xt,
audio,
and
video
data
using
the
cleaned
and
labelled
samples
and
sa
v
ed
to
be
used
later
for
the
ensemble
.
These
sa
v
ed
models
were
loaded
for
ensemble
learning,
where
the
outputs
from
multiple
models
are
combined
to
impro
v
e
the
o
v
erall
prediction
accurac
y
.
2.2.3.
Multimodal
data
-
ensemble
models
F
or
each
participant
of
the
testing
sam
ples,
the
nal
class
is
decided
based
on
majority
v
oting.
The
class
that
is
gi
v
en
by
2
out
of
3
m
od
a
lities
is
set
to
be
the
label
for
that
particular
participant.
If
all
three
modal-
ities
resulted
in
dif
ferent
labels,
the
nal
label
w
as
set
as
an
a
v
erage
of
the
numerical
v
alues
corresponding
to
the
labels.
Figure
3
sho
ws
the
o
v
erall
architecture
of
the
proposed
system
to
detect
MDD
and
PTSD
using
multimodal
data
in
the
form
of
audio,
video
and
te
xt
input
of
the
user
.
Machine
learning
models
are
b
uilt
for
the
indi
vidual
modalites
of
E-D
AIC
dataset
and
the
y
are
ensembled
to
produce
the
nal
output
class
label.
The
rst
step
of
the
system
uses
Python’
s
librosa
and
soundle
libraries
to
trim
parts
of
ra
w
audio
where
decibel
le
v
els
were
less
than
33
dB.
From
this
trimmed
audio,
65
features
were
e
xtracted
under
the
Compare2016
set
using
the
OpenSMILE
toolkit.
These
were
deri
v
ed
using
a
frame
size
of
0.1
seconds.
The
set
of
video
features
used
from
the
E-D
AIC
dataset
were
the
49
pose,
g
aze
and
action
units.
These
had
been
dervied
using
the
OpenF
ace
toolkit
with
a
frame
size
of
0.033
seconds.
A
simple
a
v
eraging
of
the
e
xtracted
features
for
both
audio
and
video
w
as
then
carried
out
for
each
of
the
participant
sample
les.
The
te
xt
tran-
script
for
each
indi
vidual
participant
from
their
corresponding
csv
les
w
as
combined
into
one
single
ro
w
of
te
xt.
It
w
as
observ
ed
that
the
training,
testing
and
v
alidation
split
pro
vided
in
the
E-D
AIC
dataset
gi
v
en
in
Figures
1
and
2
were
not
uniformly
distrib
uted
across
the
ne
w
labels
assigned
for
the
multi-class
approach.
In
order
to
compare
performances,
a
ne
w
split
up
w
as
created
in
the
ratio
of
70:15:15
for
training,
testing
and
v
alidation.
This
ne
w
split
is
visualized
in
Figures
4
and
5
for
MDD
and
PTSD
respecti
v
ely
.
Figure
1.
MDD
multi-class
distrib
ution
of
E-D
AIC
dataset
o
v
er
train,
de
v
and
test
splits
Figure
2.
PTSD
multi-class
distrib
ution
of
E-D
AIC
dataset
o
v
er
train,
de
v
and
test
splits
Depr
ession
and
post
tr
aumatic
str
ess
disor
der
analysis
with
multi-modal
data
(Rajalakshmi
Sivanaiah)
Evaluation Warning : The document was created with Spire.PDF for Python.
2362
❒
ISSN:
2252-8938
Figure
3.
System
architecture
Figure
4.
MDD
multi-class
distrib
ution
o
v
er
ne
w
train,
de
v
and
test
splits
Figure
5.
PTSD
multi-class
distrib
ution
o
v
er
ne
w
train,
de
v
and
test
splits
3.
RESUL
TS
AND
DISCUSSION
This
paper
discusses
the
k
e
y
ndings
on
detecting
the
depression
le
v
el
and
PTSD
le
v
el
as
a
multi-cl
ass
problem,
while
the
earlier
studies
deal
this
as
a
re
gression
or
binary-class
problem.
W
e
ha
v
e
de
vised
the
rules
for
classifying
the
scores
into
multiple
cate
gories
and
e
xperimented
with
the
need
for
multiple
modalaties
in
detecting
the
le
v
el
of
MDD
and
PTSD.
The
e
v
aluation
of
the
three
trained
models
for
indi
vidual
modalities
and
the
ensemble
model
for
MDD
and
PTSD
includes
the
calculation
of
accurac
y
and
F1-score.
The
results
for
all
four
models
and
their
performance
measured
using
F1
score
are
presented
in
T
ables
3
and
4
for
MDD
and
PTSD
respecti
v
ely
.
T
able
5
depicts
the
performance
of
the
proposed
ensemble
approach
compared
to
other
algorithms.
W
e
ha
v
e
observ
ed
that
the
ne
w
split
created
sho
ws
better
results
than
the
gi
v
en
E-D
AIC
split
in
terms
of
performance.
Our
study
demonstrates
that
the
ensemble
model
with
all
t
hree
modalities
(te
xt,
audio
and
video)
gi
v
es
better
results
than
the
indi
vidual
models
for
both
MDD
and
PTSD
classication.
Another
performance
metric
that
visualizes
and
summarizes
the
performance
of
the
multi-class
classication
is
a
confusion
matrix
that
sho
ws
the
number
of
correctly
classied
and
misclassied
samples
under
each
class.
The
matrices
for
MDD
and
PTSD
classication
are
displayed
in
T
ables
6
and
7
respecti
v
ely
.
Int
J
Artif
Intell,
V
ol.
14,
No.
3,
June
2025:
2358–2366
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
2363
T
able
3.
F1-Score
and
accurac
y
of
MDD
classication
Modality
F1
E-D
AIC
F1-
Ne
w
Acc
E-D
AIC
Acc
Ne
w
T
e
xt
0.34
0.38
35.7
40.5
Audio
0.31
0.35
39.3
45.2
V
ideo
0.25
0.29
35.7
45.8
Ensemble
0.38
0.42
42.9
47.8
T
able
4.
F1-Score
and
accurac
y
of
PTSD
classication
Modality
F1
E-D
AIC
F1-
Ne
w
Acc
E-D
AIC
Acc
Ne
w
T
e
xt
0.42
0.35
42.9
40.5
Audio
0.21
0.43
37.5
47.6
V
ideo
0.38
0.42
46.1
50.0
Ensemble
0.47
0.48
48.6
52.4
T
able
5.
Accurac
y
of
dif
ferent
machine
learning
algorithms
T
ask
DT
RF
SVM
KNN
Ensemble-Ne
w
MDD
32
33
33
33
47.8
PTSD
32
32
37
33
52.4
T
able
6.
Confusion
matrix
for
MDD
ensemble
Labels
Abs
Mild
Mod
ModSe
v
Se
v
Abs
17
2
0
0
0
Mild
9
7
0
0
0
Mod
6
3
0
0
0
ModSe
v
8
2
0
0
0
Se
v
1
0
1
0
0
T
able
7.
Confusion
matrix
for
PTSD
ensemble
Labels
Abs
Mild
Mod
Se
v
Abs
20
0
0
1
Mild
2
0
0
0
Mod
9
0
1
1
Se
v
16
0
1
5
From
the
confusion
matrix
of
MDD
sho
wn
in
T
able
6,
out
of
19
“
Abse
nce”
(Abs)
labe
lled
test
data
samples
17
are
correctl
y
classied
and
2
are
misclassied
as
“Mild”
class.
Similarly
the
classication
is
sho
wn
for
other
class
labels.
Ho
we
v
er
,
it
is
w
orth
noting
that
neither
of
the
tw
o
test
data
samples
of
“Se
v
ere”
(Se
v)
class
are
correctly
classied.
One
sample
is
misclassied
as
“
Absence”
and
the
other
one
as
“Moderate”
(Mod).
Same
scenario
e
xists
for
“Moderate
to
se
v
ere”
(ModSe
v)
cate
gory
.
These
f
alse
ne
g
ati
v
e
classication’
s
need
to
be
reduced
in
the
case
of
depression
detection.
From
the
confusion
matrix
of
PTSD
sho
wn
in
T
able
7,
one
test
data
sample
of
“
Absence”
is
misclassied
as
“Se
v
ere”
while
16
test
data
samples
of
“Se
v
ere”
are
misclassied
as
“
Absence”.
Both
of
these
f
alse
ne
g
ati
v
es
need
attention.
In
our
future
w
ork
we
are
w
orking
on
reducing
these
f
alse
ne
g
ati
v
es
using
boosting
algorithms.
Classwise
accurac
y
from
the
confusion
matrices
are
determined
for
the
indi
vidual
classes
of
both
MDD
and
PTSD
as
sho
wn
in
T
ables
8
and
9
respecti
v
ely
.
T
able
8.
Accurac
y
for
MDD
multi-class
labels
Class
Accurac
y
Absence
89.47
Mild
43.75
Moderate
0.0
Moderate
to
Se
v
ere
0.0
Se
v
ere
0.0
Depr
ession
and
post
tr
aumatic
str
ess
disor
der
analysis
with
multi-modal
data
(Rajalakshmi
Sivanaiah)
Evaluation Warning : The document was created with Spire.PDF for Python.
2364
❒
ISSN:
2252-8938
T
able
9.
Accurac
y
for
PTSD
multi-class
labels
Class
Accurac
y
Absence
95.24
Mild
0.0
Moderate
to
high
9.09
Se
v
ere
22.73
4.
CONCLUSION
From
the
training
of
these
model,
it
w
as
observ
ed
that
initially
all
the
samples
are
set
to
belong
to
the
‘
Absence’
class
in
both
MDD
and
PTSD.
The
reason
for
this
might
be
that
the
model
be
gins
to
distinguish
between
classes
only
in
later
epochs
or
stages
of
training.
Ho
we
v
er
,
because
of
the
dataset
imbalance
in
a
multi-class
approach,
the
results
f
ail
to
impro
v
e
with
training.
This
imbalance
has
resulted
in
fe
w
classes
being
identied
while
other
classes
are
not
recognized
by
the
ensemble
model.
Another
reason
for
this
is
deri
v
ed
from
the
comparison
the
performance
of
models
on
the
E-D
AIC
split
v
ersus
the
ne
w
split
up
of
data.
Since
the
ne
w
split
distrib
uted
more
samples
for
training,
the
accurac
y
is
seen
to
be
a
bit
better
for
models
trained
on
this
di
vision.
Secondly
,
despite
the
e
xtensi
v
e
features
e
xtracted,
the
limited
size
of
the
dataset
to
275
participants
on
the
whole,
hinders
the
accurate
identication
in
a
multi-class
scenario.
If
the
dataset
could
be
augmented
to
pro
vide
more,
and
equal
number
of
samples
for
all
classes,
it
is
possible
that
the
results
might
impro
v
e.
It
is
also
possible
that
by
identifying
the
most
rele
v
ant
features
for
a
participant,
models
could
be
trained
better
,
b
ut
this
might
require
assistance
from
e
xperts
in
the
particular
subject
area.
There
might
be
scope
for
better
results
depending
on
the
algorithms
and
approaches
used.
While
audio
feat
ures
ha
v
e
been
e
xtracted
using
the
Compare2016
set
in
the
proposed
system,
feeding
the
ra
w
audio
directly
into
the
model
can
allo
w
the
model
to
disco
v
er
its
o
wn
patterns
and
feature
sets.
Using
other
feature
sets
for
audio
training
could
also
yield
better
results.
On
the
te
xtual
front,
the
v
arying
length
of
transcripts
can
inuence
the
model
and
ho
w
much
of
the
actual
content
it
focuses
on.
Choosing
appropriate
weights,
or
cleaning
the
dataset
of
irrele
v
ant
te
xt,
identifying
spelling
errors,
grammatical
mistak
es,
and
other
inconsistencies
can
help
to
impro
v
e
the
performance.
W
e
hope
that
our
system
will
thereby
enable
further
research
into
the
comorbidity
spectrum
of
these
tw
o
disorders
as
this
can
directly
impact
medical
research
and
related
treatments.
Early
detection
of
the
stress
using
multi-modal
data
and
prescribing
for
treatments
can
reduce
the
casuality
of
MDD
and
PTSD.
FUNDING
INFORMA
TION
Authors
state
no
funding
in
v
olv
ed.
A
UTHOR
CONTRIB
UTIONS
ST
A
TEMENT
This
journal
uses
the
C
on
t
rib
utor
Roles
T
axonomy
(CRediT)
to
recognize
indi
vidual
author
contrib
u-
tions,
reduce
authorship
disputes,
and
f
acilitate
collaboration.
Name
of
A
uthor
C
M
So
V
a
F
o
I
R
D
O
E
V
i
Su
P
Fu
Rajalakshmi
Si
v
anaiah
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Angel
Deborah
Suseelan
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Krupa
Elizabeth
Thannickal
✓
✓
✓
✓
✓
✓
✓
Sanmati
Pirabahar
✓
✓
✓
✓
✓
✓
✓
C
:
C
onceptualization
I
:
I
n
v
estig
ation
V
i
:
V
i
sualization
M
:
M
ethodology
R
:
R
esources
Su
:
Su
pervision
So
:
So
ftw
are
D
:
D
ata
Curation
P
:
P
roject
Administration
V
a
:
V
a
lidation
O
:
Writing
-
O
riginal
Draft
Fu
:
Fu
nding
Acquisition
F
o
:
F
o
rmal
Analysis
E
:
Writing
-
Re
vie
w
&
E
diting
CONFLICT
OF
INTEREST
ST
A
TEMENT
Authors
state
no
conict
of
interest.
Int
J
Artif
Intell,
V
ol.
14,
No.
3,
June
2025:
2358–2366
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Artif
Intell
ISSN:
2252-8938
❒
2365
D
A
T
A
A
V
AILABILITY
The
data
that
support
the
ndings
of
this
study
are
a
v
ailable
on
request
and
signing
an
agreement
form
present
in
the
website
https://dcapsw
oz.ict.usc.edu/.
REFERENCES
[1]
M.
Freeman,
“The
w
orld
mental
health
report:
transforming
mental
health
for
all,
”
W
orld
Psyc
hiatry
,
v
ol.
21,
no.
3,
pp.
391–392,
Oct.
2022,
doi:
10.1002/wps.21018.
[2]
C.
R.
Bre
win,
L.
Atw
oli,
J.
I.
Bisson,
S.
Galea,
K.
K
oenen,
and
R.
Le
wis-Fern
´
andez,
“Post-traumatic
stress
disorder:
e
v
olv-
ing
conceptualization
and
e
vide
nce,
a
nd
future
research
directions,
”
W
orld
Psyc
hiatry
,
v
ol.
24,
no.
1,
pp.
52–80,
Feb
.
2025,
doi:
10.1002/wps.21269.
[3]
F
.
Ringe
v
al
et
al
.,
“
A
VEC
2019
w
orkshop
and
challenge:
State-of-mind,
detecting
depression
with
AI,
and
cross-cultural
af
fect
recognition,
”
A
VEC
2019
-
Pr
oceedings
of
the
9th
Inte
rnational
A
udio/V
isual
Emotion
Challeng
e
and
W
orkshop,
co-located
with
MM
2019
,
pp.
3–12,
Jul.
2019,
doi:
10.1145/3347320.3357688.
[4]
A.
Ray
,
S.
K
umar
,
R.
Reddy
,
P
.
Mukherjee,
and
R.
Gar
g,
“Multi-le
v
el
attention
netw
ork
using
te
xt,
audio
and
video
for
depression
prediction,
”
A
VEC
2019
-
Pr
oceedings
of
the
9th
International
A
udio/V
is
ual
Emotion
Challeng
e
and
W
orkshop,
co-located
with
MM
2019
,
pp.
81–88,
Sep.
2019,
doi:
10.1145/3347320.3357697.
[5]
C.
Demiroglu,
A.
Bes
¸
irli,
Y
.
Ozkanca,
and
S.
C
¸
elik,
“Depression-le
v
el
assessment
from
multi-lingual
con
v
ersational
speech
data
using
acoustic
and
te
xt
features,
”
EURASIP
J
ournal
on
A
udio,
Speec
h,
and
Music
Pr
ocessing
,
v
ol.
2020,
no.
1,
Dec.
2020,
doi:
10.1186/s13636-020-00182-4.
[6]
A.
Amanat
et
al.
,
“Deep
learning
for
depression
detection
from
te
xtual
data,
”
Electr
onics
,
v
ol.
11,
no.
5,
Feb
.
2022,
doi:
10.3390/elec-
tronics11050676.
[7]
E.
Za
v
orina
and
I.
Makaro
v
,
“Depression
detection
by
person’
s
v
oice,
”
in
Analysis
of
Ima
g
es,
Social
Networks
and
T
e
xts
,
2022,
pp.
250–262,
doi:
10.1007/978-3-031-16500-9
21.
[8]
S.
Rathi,
B.
Kaur
,
and
R.
K.
Agra
w
al,
“Selection
of
rele
v
ant
visual
feature
sets
for
enhanced
depression
detection
using
incremental
linear
discriminant
analysis,
”
Multimed
T
ools
Applications
,
v
ol.
81,
no.
13,
2022,
doi:
10.1007/s11042-022-12420-2.
[9]
R.
P
.
Thati,
A.
S.
Dhadw
al,
P
.
K
umar
,
and
S.
P
,
“
A
no
v
el
multi-modal
depression
detection
approach
based
on
mobile
cro
wd
sensing
and
task-based
mechanisms,
”
Multimed
T
ools
Applications
,
v
ol.
82,
no.
4,
pp.
4787–4820,
2023,
doi:
10.1007/s11042-022-12315-2.
[10]
D.
Nila
weera
et
al
.,
“Lifetime
posttraumatic
stress
disorder
as
a
predictor
of
mortality:
a
systematic
re
vie
w
and
meta-analysis,
”
BMC
Psyc
hiatry
,
v
ol.
23,
no.
1,
Apr
.
2023,
doi:
10.1186/s12888-023-04716-w
.
[11]
G.
I.
Al
Jo
wf,
Z.
T
.
Ahmed,
R.
A.
Reijnders,
L.
de
Nijs,
and
L.
M.
T
.
Eijssen,
“T
o
predict,
pre
v
ent,
and
manage
post-traumatic
stress
disorder
(PTSD):
A
re
vie
w
of
pathoph
ysiology
,
tre
atment,
and
biomark
ers,
”
International
J
ournal
of
Molecular
Sciences
,
v
ol.
24,
no.
6,
Mar
.
2023,
doi:
10.3390/ijms24065238.
[12]
H.
V
.
M.
Sudhan
and
S.
S.
K
umar
,
“Multimodal
depression
se
v
erity
detection
using
deep
neural
netw
orks
and
depression
assessment
scale,
”
i
n
Pr
oceedings
of
International
Confe
r
ence
on
Computational
Intellig
ence
and
Data
Engineering
,
v
ol.
99,
2022,
pp.
361–375,
doi:
10.1007/978-981-16-7182-1
29.
[13]
J.
Y
oon,
C.
Kang,
S.
Kim,
and
J.
Han,
“D-vlog:
Multimodal
vlog
dataset
for
depression
detection,
”
Pr
oceedings
of
the
AAAI
Confer
-
ence
on
Articial
Intellig
ence
,
v
ol.
36,
no.
11,
pp.
12226–12234,
Jun.
2022,
doi:
10.1609/aaai.v36i11.21483.
[14]
S.
Hemtanon,
S.
Aekw
arangk
oon,
and
N.
Kittiphattanaba
w
on,
“Proacti
v
e
depression
detection
from
F
acebook
te
xt
and
beha
v-
ior
data,
”
International
J
ournal
of
Electrical
and
Computer
Engineering
(IJECE)
,
v
ol.
12,
no.
5,
pp.
5027-5035,
Oct.
2022,
doi:
10.11591/ijece.v12i5.pp5027-5035.
[15]
A.
Ashraf,
T
.
S.
Guna
w
an,
B.
S.
Riza,
E.
V
.
Haryanto,
and
Z.
Janin,
“On
the
re
vie
w
of
image
and
video-based
depression
detection
using
machine
learning,
”
Indonesian
J
ournal
of
Electrical
Engineering
and
Computer
Science
,
v
ol.
19,
no.
3,
pp.
1677–1684,
2020,
doi:
10.11591/ijeecs.v19.i3.pp1677-1684.
[16]
D.
DeV
ault
et
al
.,
“SimSensei
kiosk:
A
virtual
human
intervie
wer
for
healthcare
decision
support,
”
in
13th
International
Confer
ence
on
A
utonomous
Ag
ents
and
Multia
g
ent
Systems,
AAMAS
2014
,
2014,
pp.
1061–1068.
[17]
A.
Baile
y
and
M.
D.
Plumble
y
,
“Gender
bias
in
depression
detection
using
audio
features,
”
in
2021
29th
Eur
opean
Signal
Pr
ocessing
Confer
ence
(EUSIPCO)
,
IEEE,
Aug.
2021,
pp.
596–600,
doi:
10.23919/EUSIPCO54536.2021.9615933.
[18]
I.
Moshe
et
al
.,
“Digital
interv
entions
for
the
treatment
of
depression:
A
meta-analytic
re
vie
w
,
”
Psyc
holo
gical
Bulletin
,
v
ol.
147,
no.
8,
pp.
749–786,
2021,
doi:
10.1037/b
ul0000334.
[19]
A.
J.
Angleman,
V
.
B.
V
an
Hasselt,
and
B.
B.
Schuhmann,
“Relationship
between
posttraumatic
stress
symptoms
and
cardio
v
ascular
disease
risk
in
reghters,
”
Behavior
Modication
,
v
ol.
46,
no.
2,
pp.
321–351,
2022,
doi:
10.1177/01454455211061320.
[20]
J.
Gratch
et
al.
,
“The
distress
analysis
intervie
w
corpus
of
human
and
computer
i
ntervie
ws,
”
in
Pr
oceedings
of
the
9th
International
Confer
ence
on
Langua
g
e
Resour
ces
and
Evaluation
,
LREC
2014,
2014,
pp.
3123–3128.
[21]
I.
A.
Mures
,
anu
et
al.
,
“Ev
aluation
of
post-traumatic
stress
disorder
(PTSD)
and
related
comorbidities
in
clinical
studi
es,
”
J
ournal
of
Medicine
and
Life
,
v
ol.
15,
no.
4,
pp.
436–442,
Apr
.
2022,
doi:
10.25122/jml-2022-0120.
[22]
V
.
S.
Sathvika,
S.
V
aishna
vi,
S.
A.
Deborah,
S.
Rajalakshmi,
and
T
.
T
.
Mirnalinee,
“The
Ma
v
ericks@L
T
-EDI-2023:
Detection
of
signs
of
depression
from
social
media
te
xts
using
na
vie
Bayse
approach,
”
in
Pr
oceedings
of
the
Thir
d
W
orkshop
on
Langua
g
e
T
ec
hnolo
gy
for
Equality
,
Diver
sity
and
Inclusion
,
2023,
pp.
244–249.
[23]
K.
Anantharaman,
S.
Rajalakshmi,
S.
A.
Deborah,
M.
Saritha,
and
R.
S.
Milton,
“SSN
MLRG1@L
T
-EDI-A
CL2022:
Multi-
class
classication
using
BER
T
models
for
detecting
depression
signs
from
social
media
T
e
xt,
”
in
L
TEDI
2022
-
2nd
W
ork-
shop
on
Langua
g
e
T
ec
hnolo
gy
for
Equality
,
Diver
sity
and
Inclusion,
Pr
oceedings
of
the
W
orkshop
,
2022,
pp.
296–300,
doi:
10.18653/v1/2022.ltedi-1.44.
[24]
V
.
O.
Y
enumulapalli,
R.
V
.
Ara
vindh,
S.
Rajalakshmi,
and
S.
A.
Deborah,
“T
echSSN1@L
T
-EDI:
Depression
detection
and
classica-
tion
using
BER
T
model
for
social
media
te
xts,
”
in
L
TEDI
2023
-
3r
d
W
orkshop
on
Langua
g
e
T
ec
hnolo
gy
for
Equality
,
Diver
sity
and
Inclusion,
associated
with
the
14th
International
Confer
ence
on
Recent
Advances
in
Natur
al
Langua
g
e
Pr
ocessing
,
RANLP
2023
-
Pr
oceedings
,
2023,
pp.
149–154,
doi:
10.26615/978-954-452-084-7
022.
Depr
ession
and
post
tr
aumatic
str
ess
disor
der
analysis
with
multi-modal
data
(Rajalakshmi
Sivanaiah)
Evaluation Warning : The document was created with Spire.PDF for Python.
2366
❒
ISSN:
2252-8938
[25]
V
.
Sanh,
L.
Deb
ut,
J.
Chaumond,
and
T
.
W
olf,
“DistilBER
T
,
a
distilled
v
ersion
of
BER
T
:
smaller
,
f
aster
,
cheaper
and
li
ghter
,
”
arXiv-
Computer
Science
,
pp.
1-5,
2019.
[26]
K.
E.
Thannickal,
P
.
Sanmati,
S.
Rajalakshmi,
and
S.
A.
Deborah,
“T
echSSN4@L
T
-EDI-RANLP2023:
Depression
sign
detection
in
social
media
postings
using
DistilBER
T
model,
”
in
L
TEDI
2023
-
3r
d
W
orkshop
on
Langua
g
e
T
ec
hnolo
gy
for
Equality
,
Diver
sity
and
Inclusion,
associated
with
the
14th
International
Confer
ence
on
Recent
Advances
in
Natur
al
Langua
g
e
Pr
ocessing
,
RANLP
2023
,
2023,
pp.
239–243,
doi:
10.26615/978-954-452-084-7
036.
[27]
S.
P
.
Pimpalkar
,
A.
A.
Cha
v
an,
S.
V
.
Ga
w
ali,
and
S.
S.
Dalvi,
“
Adv
ancements
in
depression
se
v
erity
prediction:
A
multi-modal
machine
learning
approach
inte
grating
MC
Qs
and
audio
data,
”
in
2nd
IEEE
International
Confer
ence
on
Advances
in
Information
T
ec
hnolo
gy
,
ICAIT
2024
-
Pr
oceedings
,
2024,
pp.
1–8,
doi:
10.1109/ICAIT61638.2024.10690826.
BIOGRAPHIES
OF
A
UTHORS
Rajalakshmi
Si
v
anaiah
is
w
orking
as
Assistant
Professor
in
Department
of
Computer
Science
and
Engineering,
SSN
Colle
ge
of
Engineering,
Chennai,
India.
She
has
more
t
han
19
years
of
e
xperience
in
teaching.
Her
area
of
interest
includes
data
mining,
machine
learning,
de
ep
learn-
ing,
data
science,
natural
language
processing,
and
internet
technologies
.
She
completed
Ph.D
in
recommendation
systems
under
Anna
Uni
v
ersity
,
Chennai,
India.
She
is
consistently
taking
part
in
the
International
W
orkshop
on
“Semantic
e
v
aluation
(SemEv
al)”
or
g
anized
by
Association
of
Com-
putational
Linguistics
from
2018,
MediaEv
al
and
L
T
-EDI
tasks.
She
has
published
papers
in
v
arious
conferences
and
journals
and
granted
with
1
patent.
She
has
completed
an
inter
nally
funded
project
on
ne
grained
emotion
detection.
She
can
be
contacted
at
email:
rajalakshmis@ssn.edu.in.
Angel
Deborah
Suseelan
is
Assistant
Professor
in
the
Department
of
Computer
Science
has
11
years
and
6
months
of
teaching
and
research
e
xperience.
She
recei
v
ed
her
Ph.D.
from
Anna
Uni
v
ersity
.
She
is
consistently
taking
part
in
the
SemEv
al,
MediaEv
al,
and
L
T
-EDI
tasks.
She
has
published
papers
in
v
arious
conferences
and
journals
and
granted
with
1
patent.
Her
research
areas
includes
machine
learning,
natural
language
processing,
data
science,
e
mbedded
systems
and
internet
of
things.
She
has
completed
an
internally
funded
project
on
ne
grained
emotion
det
ection.
She
can
be
contacted
at
email:
angeldeborahs@ssn.edu.in.
Krupa
Elizabeth
Thannickal
graduated
with
a
Bachelor
of
Engineering
(B.E)
de
gree
in
Computer
Science
and
Engineering.
Currently
emplo
yed
in
the
capacity
of
Associate
Softw
are
Engineer
at
Informatica,
she
applies
her
technical
skills
in
de
v
eloping
softw
are
for
process
inte
gration
and
data
management.
Her
areas
of
interest
include
programming
in
Python,
cryptograph
y
,
and
graphic
design.
She
can
be
contacted
at
email:
krupa19054@cse.ssn.edu.in.
Sanmati
Pirabahar
is
a
skilled
professional
with
a
Ba
chelor’
s
de
gree
in
Computer
Science
from
SSN
Colle
ge
of
Engineering,
passionate
about
operating
systems,
Ja
v
a
programming,
micro-
processors,
and
back
end
de
v
elopment.
Currently
emplo
yed
as
a
Softw
are
Engineer
at
Informatica,
she
applies
her
e
xpertise
and
enthusiasm
t
o
contrib
ute
ef
fecti
v
ely
in
v
arious
technological
projects.
W
ith
a
k
een
interest
in
e
xploring
adv
ancements
in
these
areas,
she
remains
dedicated
to
enhancing
her
skills
and
dri
ving
inno
v
ation
in
the
eld
of
technology
.
She
can
be
contacted
at
email:
san-
mati19098@cse.ssn.edu.in.
Int
J
Artif
Intell,
V
ol.
14,
No.
3,
June
2025:
2358–2366
Evaluation Warning : The document was created with Spire.PDF for Python.