Indonesian
J
our
nal
of
Electrical
Engineering
and
Computer
Science
V
ol.
37,
No.
2,
February
2025,
pp.
1150
∼
1167
ISSN:
2502-4752,
DOI:
10.11591/ijeecs.v37.i2.pp1150-1167
❒
1150
Educational
impact
and
ethical
considerations
in
using
Chatbots
in
Academia
Dina
M.
Ibrahim,
Njood
K.
Al-harbi,
Amal
A.
Al-Shar
gabi
Department
of
Information
T
echnology
,
Colle
ge
of
Computer
,
Qassim
Uni
v
ersity
,
Buraydah,
Saudi
Arabia
Article
Inf
o
Article
history:
Recei
v
ed
Jun
11,
2024
Re
vised
Sep
13,
2024
Accepted
Sep
30,
2024
K
eyw
ords:
Chatbots
ChatGPT
-4
En
vironmental
education
research
Ethical
considerations
Independent
learning
Learning
tools
Quality
of
education
ABSTRA
CT
Chatbots
are
getting
bet
ter
e
v
ery
day
due
to
the
adv
ancements
in
their
capa-
bilities
in
today’
s
technological
age.
This
study
aims
to
assess
the
ef
cac
y
of
ChatGPT
-4
and
Gemini
in
producing
scientic
a
rticles.
T
w
o
types
of
prompts
are
gi
v
en:
direct
questions
and
com
plete
scenarios.
Subsequently
,
we
e
v
aluate
the
educational
and
ethical
aspects
of
the
produced
material
by
emplo
ying
sta-
tistical
analysis.
W
e
v
erify
the
credibility
of
references,
detect
an
y
instances
of
plagiarism,
and
ensure
the
precision
of
the
articles
gene
rated
by
the
chatbot.
In
addition,
we
utilize
topic
modeling
to
assess
the
e
xtent
to
which
the
content
of
the
articles
corresponds
to
the
specied
topic.
According
to
the
ndings,
Gemini
outperformed
ChatGPT
-4,
specically
in
scena
rio
questions,
where
it
achie
v
ed
an
accurac
y
rate
of
85%,
while
ChatGPT
-4
only
achie
v
ed
35%
accurac
y
.
This
is
an
open
access
article
under
the
CC
BY
-SA
license
.
Corresponding
A
uthor:
Dina
M.
Ibrahim
Department
of
Information
T
echnology
,
Colle
ge
of
Computer
,
Qassim
Uni
v
ersity
Buraydah,
Saudi
Arabia
Email:
d.hussein@qu.edu.sa
1.
INTR
ODUCTION
The
present-day
era’
s
f
ast
scientic
adv
ancement
means
that
people
from
all
w
alks
of
life
encounter
and
learn
about
ne
w
technical
marv
els
daily
[1].
Who
kne
w
that
a
chatbot
could
help
people
sa
v
e
time
and
w
ork
more
ef
ciently?
An
articial
intelligence
chatbot
called
ChatGPT
,
which
stands
for
”generati
v
e
pre-
trained
transformer
,
”
has
lately
captured
the
interest
of
man
y
in
the
technological
community
.
The
articial
intelligence
(AI)
research
and
publishing
rm
OpenAI
created
it
and
releas
ed
it
to
the
public
in
No
v
ember
2022.
Construction
be
g
an
with
OpenAI’
s
GPT
-3.5
and
GPT
-4
lar
ge
language
model
(LLM)
f
amilies,
and
it
w
as
ne-tuned
using
supervised
and
reinforcement
learning
approaches,
which
is
a
transfer
learning
approach
[2].
In
addition
to
ChatGPT
,
another
articial
intelligence
chatbot
called
Gemini,
pre
viously
kno
wn
as
Bard,
is
designed
to
simulate
con
v
ersations
with
users.
Google
serv
es
as
its
po
wer
source
and
it
also
depends
on
LLM.
Enhanced
and
streamlined
v
ersion
of
LaMD
A.
It
is
lik
e
ChatGPT
,
e
xcept
that
it
is
current
and
obtains
its
information
directly
from
the
web
.
Gemini
is
the
of
cial
monik
er
of
the
ne
w
v
ersion,
which
w
as
pre
viously
in
the
de
v
elopment
phase.
In
addition,
mass
i
v
e
multitask
language
understanding
(MMLU)
w
as
the
k
e
y
f
actor
during
the
de
v
elopment
[3].
The
emer
gence
of
AI
technologies,
lik
e
ChatGPT
-4,
Gemini,
and
other
chatbots,
grants
signicant
capabilities
and
imposes
substantial
constraints.
Hence,
it
is
imperati
v
e
to
tak
e
into
account
ethical
consider
-
ations.
While
it
may
be
adv
antageous
f
o
r
certain
repetiti
v
e
tasks,
such
as
editing,
it
presents
notable
inte
grity
concerns.
Furthermore,
her
rapidity
in
composing
research
papers
presents
a
threat
to
scientic
inte
grity
.
The
present
study
aims
to
in
v
estig
ate
t
he
inuence
of
chatbots,
specically
ChatGPT
-4
and
Gemini,
on
academia.
J
ournal
homepage:
http://ijeecs.iaescor
e
.com
Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian
J
Elec
Eng
&
Comp
Sci
ISSN:
2502-4752
❒
1151
As
well
as
e
xplore
methods
for
educating
students
and
researchers
on
the
ethical
and
inte
grity
aspects
of
scien-
tic
research.
Currently
,
researchers
are
in
v
estig
ating
the
follo
wing
questions:
1.
Can
eng
aging
in
a
con
v
ersation
with
a
chatbot
generate
v
aluable
scientic
data
for
researchers
in
the
aca-
demic
community?
Alternati
v
ely
,
does
it
gi
v
e
rise
to
ethical
considerations?
2.
What
are
the
potential
threats
associated
with
chatbots?
3.
Does
the
chatbot’
s
content
possess
authenticity
and
dependability?
T
o
address
the
questions
of
the
study
,
specic
objecti
v
es
must
be
achie
v
ed,
which
encompass:
1.
Comparati
v
e
analysis
of
the
functionalities
and
operations
of
ChatGPT
-4
and
Gemini.
2.
Conduct
e
xperiments
with
chatbots,
including
ChatGPT
-4
and
Gemini.
3.
Assess
the
credibility
of
the
bibliograph
y
.
4.
Instruct
chatbots
to
produce
original
articles
through
tw
o
methods:
posing
direct
questions
and
presenting
complete
scenarios,
follo
wed
by
conducting
plagiarism
checks.
5.
Examine
and
analyze
the
ethical
challenges.
Research
holds
importance
in
maintaining
the
inte
grity
of
scientic
research
and
in
educating
s
tu-
dents
and
researchers
about
the
ethical
aspects
of
scientic
research.
Therefore,
it
is
crucial
to
emplo
y
AI
tools
responsibly
and
ethically
when
conducting
academic
research
and
publishing.
Moreo
v
er
,
it
is
crucial
to
consider
cop
yright,
authorship,
and
accurate
citation
of
information
sources.
This
study
in
v
estig
ated
the
ef
fects
of
ChatGPT
-4
and
Gemini
to
produce
original
articles
through
tw
o
methods:
posi
ng
direct
questions
and
pre-
senting
complete
scenarios,
follo
wed
by
conducting
plagiarism
checks.
While
earlier
studies
ha
v
e
e
xplored
the
impact
of
normal
questions,
the
y
ha
v
e
not
e
xplicitly
addressed
its
inuence
on
chatbots,
including
ChatGPT
-4
and
Gemini.
The
follo
wing
sections
of
the
paper
are
structured
as
follo
ws:
section
2
of
fers
a
summary
of
the
rele
v
ant
literature,
section
3
outlines
the
methodologies
used
for
in
v
estig
ation,
section
4
presents
the
results
of
this
research,
section
5
pro
vides
a
discussion
of
the
results,
and
nally
,
section
6
presents
the
conclusion
and
suggests
recommendations
for
future
research.
2.
RELA
TED
W
ORKS
This
section
will
pro
vide
a
brief
o
v
ervie
w
of
the
literature
on
ChatGPT
,
follo
wed
by
literature
on
Gemini,
and
conclude
with
a
discussion
on
the
ethical
implications
of
chatbots
in
general.
2.1.
Literatur
e
about
ChatGPT
-4
Zhai
[4]
predicts
ChatGPT
will
af
fect
all
aspects
of
society
.
The
y
wrote
an
academic
article
to
t
est
ChatGPT
.
The
results
sho
w
that
ChatGPT
can
help
researchers
write
a
cogent
(partial)
report
t
h
a
t
is
correct,
insightful,
and
methodical
in
tw
o
to
three
hours
despite
their
minimal
professional
e
xperience.
Based
on
user
e
xperience,
the
author
considers
ChatGPT
and
other
AI
tools’
potential
ef
fects
a
n
d
proposes
changing
the
learning
objecti
v
es
to
teach
students
ho
w
to
use
them
and
focus
on
creati
vity
and
critical
thinking,
which
AI
tools
cannot
replace.
Recent
research
addressed
ChatGPT’
s
ethical
implications
for
scientic
research,
including
t
rans-
parenc
y
,
bias,
informed
consent,
pri
v
ac
y
,
responsibilit
y
,
and
inte
grity
.
Khlaif
[5],
concluded
that
researchers
must
declare
and
ackno
wledge
their
usage
of
ChatGPT
in
their
research
methods
while
sticking
to
research
ethics
and
inte
grity
.
In
an
additional
study
e
xamining
the
inuence
of
AI
on
ethical
concerns,
specically
in
the
eld
of
medical
publishing
practices,
Liebrenz
et
al.
[6]
requested
the
participation
of
ChatGPT
to
pro
vide
a
commentary
on
AI
and
the
ethical
considerat
ions
surrounding
medical
publishing
for
lancet
digital
health.
The
y
inquired
about
ho
w
the
editorial
team
can
ef
fecti
v
ely
manage
the
AI-generated
academic
content,
seek-
ing
advice
from
ChatGPT
.
According
to
ChatGPT’
s
response
to
their
question,
lancet
digital
health
should
“carefully
consider
the
ethical
implications
of
publishing
articles
produced
by
AI.
”
In
a
subsequent
publication
[7],
the
author
ask
ed
ChatGPT
,
“When
streptozotocin-induced
dia
betes
is
prepared
in
gro
wing
rats,
can
you
predict
its
ef
fect
on
the
f
acial
bone
gro
wth
pattern?”
ChatGPT
answered.
ChatGPT
quickly
g
a
v
e
a
thorough
list
of
references,
including
their
authors,
when
he
ask
ed
them.
After
v
erify-
ing
these
referrals,
the
researcher
found
them
all
to
be
f
ak
e.
Ho
we
v
er
,
the
study
sho
ws
that
ChatGPT
impro
v
ed
English
grammar
well.
The
researcher
found
that
ChatGPT’
s
no
v
el
ideas
should
be
v
alidated
before
publica-
tion.
Additionally
,
AI
in
v
olv
ement
should
be
stated.
Continuing
from
the
last
ar
ticle
discussing
the
authenticity
Educational
impact
and
ethical
consider
ations
in
using
Chatbots
in
Academia
(Dina
M.
Ibr
ahim)
Evaluation Warning : The document was created with Spire.PDF for Python.
1152
❒
ISSN:
2502-4752
of
ChatGPT
references.
King
[8]
uses
ChatGPT
to
complete
the
essay
without
an
y
in
v
olv
ement
or
editing
from
humans.
Re
grettably
,
after
a
thorough
e
xamination,
it
w
as
disco
v
ered
that
the
references
requested
by
the
author
from
ChatGPT
were
f
ak
e.
Ray
[9]
addresses
ChatGPT’
s
history
,
applications,
chal
lenges,
bias,
ethics,
limitations,
and
future.
Data
pri
v
ac
y
,
bias,
transparenc
y
,
autonomy
,
human
agenc
y
,
emotional
manipulation,
persuasion,
and
AI-
generated
material
are
among
the
ethics
co
v
ered.
ChatGPT
contrib
uted
to
scholarly
research
in
linguistically
coherent
te
xt
generation,
is
grammatically
accurate,
and
has
the
potential
to
alter
the
discipline
pro
vided
its
obstacles
and
ethical
issues
are
solv
ed.
2.2.
Literatur
e
about
Gemini
Bard
recently
released
a
ne
w
v
ersion
called
Gemini,
as
a
result
of
the
authors’
latest
research,
which
analyzed
Bard.
It
is
no
w
being
de
v
eloped,
and
it
has
been
sho
wn
that
t
he
content
pro
vided
by
Bard
in
the
past
is
not
as
accurate
when
compared
to
ChatGPT
-4
[10].
Ne
v
ertheless,
the
recently
launched
Gemini
Ultra
is
the
initial
model
to
surpass
human
e
xperts
in
MMLU.
This
achie
v
ement
is
accomplished
by
utilizing
a
blend
of
57
dis
ciplines,
including
math,
ph
ysics,
history
,
la
w
,
medicine,
and
ethics,
to
assess
both
global
kno
wledge
and
problem-solving
skills,
with
a
score
of
90.0%
o
v
er
ChatGPT
-4,
which
has
86.4%
[11].
The
study’
s
author
[12]
analyzed
three
widely
used
chatbots:
ChatGPT
,
Gemini,
and
Copilot.
The
results
of
the
analysis
were
e
xamined.
Most
platforms,
particularly
Google
Gemini,
ha
v
e
sho
wn
a
positi
v
e
response
to
the
personal
response
prompt.
Ov
erall,
the
y
indicated
that
AI
chatbots
are
not
dependable
for
generating
ideas.
Imran
and
Almusharraf
[13],
stated
Google
Gemini
is
useful
for
education.
T
o
minimize
f
airness
issues,
its
algorithms
must
be
thoroughly
e
v
aluated
a
nd
monitored
to
ensure
the
y
are
unbiased
and
do
not
prejudice
certain
learners.
Along
with
other
AI
chatbots
and
tools,
Gemini
must
protect
data.
This
requires
protecting
user
data
and
foll
o
wing
data
collection
ethics.
These
steps
are
necessary
for
AI
inte
gration
in
educa-
tion.
Google
Gemini
is
a
major
educational
technology
competitor
.
Its
multimodal,
reasoning,
and
generating
skills
pro
vide
tailored
learning,
eng
aging
training,
and
dynamic
e
v
aluation.
T
o
appropriately
utilize
this
genera-
ti
v
e
AI
technology
,
ethical
issues
must
be
consi
dered,
responsible
adv
ancement
must
be
made,
and
transparent
e
x
ecution
must
be
ensured.
Thus,
by
prioritizing
human-centered
design,
ackno
wledging
and
correcting
bi-
ases,
and
maintaining
high
ethical
standards,
Google
Gemini
could
create
a
future
where
technology
allo
ws
tailored
learning
e
xperiences
for
e
v
eryone.
The
researcher
in
the
study
[14],
utilizes
ChatGPT
and
Google
Gemini
as
technology
resources
to
de
v
elop
lesson
plans
for
7th-grade
mathematics,
science,
literature,
and
social
studies
courses.
The
chatbots
were
prompted
to
generate
lesson
plans
tailore
d
to
the
specied
course,
subject,
and
le
v
el.
The
results
indicate
that
the
l
esson
plans
generated
by
both
chatbots
closely
resemble
human-authored
instructional
content
in
terms
of
language
structures,
class
acti
vities,
and
assessments.
While
the
lesson
plans
designated
teachers
as
f
acilitators
and
pro
vided
part
ially
producti
v
e
acti
vities,
it
w
as
disco
v
ered
that
the
technology-inte
grated
acti
vities
were
se
v
erely
restricted.
Chatbots
in
healthcare
education
are
e
xamined
in
the
article
[15].
ChatGPT
-4
and
Gemini
are
tested
on
V
irology
multiple-choice
questions
(MCQs)
in
English
and
Arabic,
as
well
as
content
quality
.
ChatGPT
-4
and
Gemini
sho
wed
pedagogical
promise.
ChatGPT
-4
had
80%
English
accurac
y
and
65%
Arabic
accurac
y
,
outper
-
forming
Gemini’
s
62.5%
and
55%,
respecti
v
ely
.
Their
performance
v
aried
across
languages,
underlining
the
need
for
continued
w
ork
to
inte
grate
AI
into
healthcare
education
w
orldwide.
The
aim
of
the
study
introduced
by
[16],
is
to
determine
the
procienc
y
of
ChatGPT
-4
and
Google
Gemini
in
analyzing
detailed
glaucoma
case
descriptions
and
suggesting
an
accurate
sur
gical
plan.
ChatGPT
-4
demonstrated
a
commendable
le
v
el
of
anal-
ysis
procienc
y
when
it
came
to
both
typical
and
dif
cult
glaucoma
sur
gical
cases.
Ho
we
v
er
,
Google
Gemini
e
xhibited
signicant
constraints
in
this
conte
xt,
with
a
high
occurrence
of
inaccurate
or
unanswered
responses.
2.3.
Literatur
e
about
ethical
aspect
In
No
v
ember
1981,
Omni
Mag
azine
at
trib
uted
the
authorship
of
a
te
xt
to
a
computer
program
called
Racter
[17].
In
1984,
Racter’
s
book
w
as
published,
making
it
the
rst
book
e
v
er
created
using
a
computer
program
[18].
Subsequently
,
as
a
result
of
the
de
v
elopment
of
Racter
and
AI,
cop
yright
issues
ha
v
e
g
arnered
signicant
attention
[19].
Can
a
chatbot
be
considered
an
author
,
considering
the
rise
of
AI
and
chatbots
and
their
e
xtensi
v
e
use
in
se
v
eral
indust
ries,
particularly
in
education
a
n
d
student
support?
The
author
cited
an
article
[20]
that
asserted
that
utilizing
an
AI
te
xt
generator
without
proper
ackno
wledgment
w
ould
be
deemed
plagiarism.
Plagiarism
refers
to
the
unpermitted
utilization
of
someone
else’
s
w
ork
or
ideas
without
pro
viding
proper
attrib
ution.
According
to
the
authors
in
the
papers
[5],
[20],
it
is
necessary
to
menti
o
n
a
source
reference
Indonesian
J
Elec
Eng
&
Comp
Sci,
V
ol.
37,
No.
2,
February
2025:
1150–1167
Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian
J
Elec
Eng
&
Comp
Sci
ISSN:
2502-4752
❒
1153
when
utilizing
AI
technologies,
re
g
ardless
of
whether
the
information
is
generated
by
humans
or
AI.
Ne
v
erthe-
less,
esteemed
publications
lik
e
Nature
and
Science
ha
v
e
e
xplicitly
declared
that
AI
chatbots
are
ineligible
to
be
credited
as
authors
of
w
orks
published
in
their
journals.
The
reason
for
this
is
that
the
editorial
policies
of
these
publications
e
xplicitly
indicate
that
“
AI
chatbots
do
not
currently
meet
our
criteria
for
authorship”
[21].
AI
chatbots
cannot
function
as
writers
because
of
their
inability
to
match
the
present
criteria,
not
because
of
their
lack
of
human
qualities.
In
the
future,
AI
chatbots
could
be
recognized
as
authors
of
scholarly
w
orks
pro
vided,
the
y
fulll
the
necessary
conditions
[17].
Another
f
actor
that
pre
v
ents
it
from
being
recognized
as
an
author
is
its
inability
to
grant
authorization
for
the
publication
of
these
papers,
which
is
the
basis
of
the
cop
yright
pri
v
ac
y
ar
gument
[22].
The
follo
wing
ethical
issues
arise
when
chatbots
create
academic
content:
-
Systematic
decision-making
errors
that
yield
unf
air
consequences
are
called
bias.
Data
g
athering,
algorithm
design,
and
human
interpretation
can
cause
bias.
Machine
learning
models,
an
articial
intelligence
system,
can
learn
and
repeat
bias
patterns
in
training
data,
resulting
in
unf
air
and
biased
results
[23].
-
Human
subjects
research
ethics
require
informed
consent.
It
in
v
olv
es
getting
a
participant’
s
permission
after
presenting
them
with
enough
information
about
the
research,
its
risks
,
adv
antages,
and
alternati
v
es,
as
well
as
the
chance
to
ask
questions
and
clarify
[24].
Ho
we
v
er
,
researchers
using
AI-generated
te
xt
lack
AI
permission.
Thus,
for
transparenc
y
and
informed
consent,
researchers
must
kno
w
their
techniques
to
get
accurate
and
reliable
results
[5].
-
Pri
v
ac
y
concerns:
Chatbots
generate
te
xt
from
enormous
datasets
that
may
contain
sensiti
v
e
material.
As
chatbot-generated
content
becomes
more
common
in
scientic
research,
participant
data
must
be
protected.
Sharing
this
information
infringes
on
pri
v
ac
y
and
creates
ethical
concerns
[5].
-
Research
inte
grity
,
chatbots
lik
e
Gemini
and
ChatGPT
may
pro
vide
decepti
v
e
information
and
data.
F
or
e
xample,
ChatGPT
for
ged
references
for
this
w
ork
[8].
This
sho
ws
that
chatbots
are
unreliable
and
unsuited
for
use
and
that
researchers
are
responsible
for
their
research
quality
and
reliability
.
In
addition,
Lund
et
al.
[25],
e
xamine
the
impact
of
GPT/ChatGPT
in
comparison
to
other
language
paradigms,
specically
i
n
the
conte
xt
of
ethical
concerns
surrounding
technology
use
in
academia,
scholarly
research,
and
publishing.
GPT3
has
demonstrated
its
v
ersatility
,
ef
fecti
v
eness,
and
ability
to
produce
lan-
guage
that
closely
resembles
human
speech,
rendering
it
v
aluable
for
tasks
such
as
translation,
annotation,
and
question-answering.
Moreo
v
er
,
ChatGPT
has
the
capacity
to
impro
v
e
search
ef
fecti
v
eness
and
the
caliber
of
scholarly
articles.
One
aspect
of
the
discussion
focused
on
the
ethical
implications
that
must
be
considered,
including
the
o
wnership
of
the
content.
It
is
currently
unclear
who
holds
the
rights
to
the
generated
te
xt,
which
raises
concerns
about
cop
yright
issues.
3.
MA
TERIALS
AND
METHODS
The
present
study
methodology
is
comprised
of
four
distinct
e
xperi
ments.
F
or
each
e
xperiment,
request
the
chatbot
to
generate
tw
o
types
of
questions:
direct
questions
and
scenario
questions.
3.1.
Dir
ect
questions
By
posing
direct
questions
or
making
requests
to
the
chatbots,
which
are
ChatGPT
-4
and
Gemini,
we
will
inquire
without
pro
viding
further
details
to
observ
e
their
respecti
v
e
responses
and
assess
their
accurac
y
.
The
follo
wing
are
v
e
direct
requests
for
scientic
articles.
Each
article
has
a
distinct
title
b
ut
belongs
to
the
same
topic
and
has
the
same
number
of
w
ords.
The
articles
should
include
bibliographies,
b
ut
it
is
not
necessary
to
specify
whether
the
y
should
be
real
or
f
ak
e.
-
Request
1:
I
w
ant
you
to
write
a
1,000-w
ord
scientic
article
on
machine
learning
for
nancial
forecasting
and
include
bibliographies.
-
Request
2:
I
w
ant
you
to
write
a
1,000-w
ord
scientic
article
on
machine
learning
for
image
and
video
processing
and
include
bibliographies.
-
Request
3:
I
w
ant
you
to
write
a
1,000-w
ord
scientic
article
on
machine
learning
for
medical
diagnosis
and
include
bibliographies.
-
Request
4:
I
w
ant
you
to
write
a
1,000-w
ord
scientic
article
on
machine
learning
for
natural
language
processing
and
include
bibliographies.
-
Request
5:
I
w
ant
you
to
write
a
1,000-w
ord
scientic
article
on
machine
learning
for
social
media
analysis
and
include
bibliographies.
Educational
impact
and
ethical
consider
ations
in
using
Chatbots
in
Academia
(Dina
M.
Ibr
ahim)
Evaluation Warning : The document was created with Spire.PDF for Python.
1154
❒
ISSN:
2502-4752
3.2.
Scenario
questions
In
scenario
requests,
we
will
ask
the
chatbots
for
additional
information
re
g
arding
the
title
and
mak
e
the
request
more
accurate
by
pro
viding
data
about
the
requester
,
such
as
“I
am
a
master’
s
student
specializing
in
information
technology
.
I
require
an
in-depth
scientic
article
encompassing
an
abstract,
introduction,
literature
re
vie
w
,
met
hodology
,
results,
discussion,
and
a
conclusion
with
future
w
ork
on
the
topic
about
...
”.
Further
-
more,
the
arti
cle
includes
the
same
amount
of
w
ords
in
direct
questions
and
pertains
to
the
same
topic,
b
ut
the
bibliographies
will
e
xplicitly
indicate
the
requirement
for
reliable
sources.
The
follo
wing
request
consists
of
v
e
scenario
inquiries
from
ChatGPT
-4
and
Gemini.
-
Request
1:
I
am
a
master’
s
student
specializing
in
information
technology
.
I
require
an
in-depth
full
scientic
article
with
at
least
1,000
w
ords
encompassing
an
abstract,
introduction,
literature
re
vie
w
,
methodology
,
results,
discussion,
and
a
conclusion
with
future
w
ork
on
the
topic
of
machine
learning
for
nancial
forecasting
in
stock,
and
you
should
include
real
bibliographies.
-
Request
2:
I
am
a
master’
s
student
specializing
in
information
technology
.
I
require
an
in-depth
full
scientic
article
with
at
least
1,000
w
ords
encompassing
an
abstract,
introduction,
literature
re
vie
w
,
methodology
,
results,
discussion,
and
a
conclusion
with
future
w
ork
on
the
topic
of
machine
learning
for
image
and
video
pro-
cessing
for
the
detection
of
f
aces,
and
you
should
include
real
bibliographies.
-
Request
3:
I
am
a
master’
s
student
specializing
in
information
technology
.
I
require
an
in-depth
full
scientic
article
with
at
least
1,000
w
ords
encompassing
an
abstract,
introduction,
literature
re
vie
w
,
methodology
,
results,
discussion,
and
a
conclusion
with
future
w
ork
on
the
topic
of
machine
learning
for
medical
diagnosis
for
the
detection
of
viruses,
and
you
should
include
real
bibliographies.
-
Request
4:
I
am
a
master’
s
student
specializing
in
information
technology
.
I
require
an
in-depth
full
scientic
article
with
at
least
1,000
w
ords
encompassing
an
abstract,
introduction,
literature
re
vie
w
,
methodology
,
results,
discussion,
and
a
conclusion
with
future
w
ork
on
the
topic
of
machine
learning
in
natural
language
process-
ing
for
sentiment
analysis
of
te
xtual
data,
and
you
should
include
real
bibliographies.
-
Request
5:
I
am
a
master’
s
student
specializing
in
information
technology
.
I
require
an
in-depth
full
scientic
article
with
at
least
1,000
w
ords
encompassing
an
abstract,
introduction,
literature
re
vie
w
,
methodology
,
results,
discussion,
and
a
conclusion
with
future
w
ork
on
the
topic
of
machine
learning
in
social
media
adv
ertising
analysis,
and
you
should
include
real
bibliographies.
Then,
e
v
aluate
the
results
using
tw
o
criteria:
statistical
analysis
and
topic
modeling.
The
stati
stical
analysis
comprises
the
quantication
of
w
ord
count,
the
total
number
of
bibliographies,
the
identication
of
real
bibliographies,
the
identication
of
f
ak
e
bibliographies,
and
the
utilization
of
the
plagiarism
check
er
program,
Plagiarism
Check
er
X.
Then
the
accurac
y
will
be
computed
after
performing
a
comparison
based
on
specied
criteria
presented
as
(1).
Accur
acy
=
cor
r
ectedbibl
iog
r
aphies
T
hetotal
number
of
bibl
iog
r
aphies
(1)
T
opic
modeling
is
an
unsupervised
machine-learning
technique
that
entails
analyzing
a
lar
ge
number
of
documents,
articles,
feedback,
or
emails.
This
st
udy
will
specically
e
xamine
articles
generated
by
chatbots.
The
main
goal
of
this
technique
is
to
detect
patterns
of
w
ords
and
phrases
in
the
articles,
without
considering
their
relation
to
the
topic,
b
ut
rather
focusing
on
their
underlying
semantic
structure.
The
pro
vided
entity
can
be
characterized
as
an
assemblage
of
w
ords
lacking
an
y
particular
arrangement
or
or
g
anization
[26].
The
utilization
of
latent
dir
ichlet
allocation
(LD
A)
for
analysis
is
a
computational
paradigm
that
shares
similarities
with
latent
semantic
analysis
(LSA).
Ne
v
ertheless,
LD
A
and
LSA
di
v
er
ge
in
their
methodology
for
allocating
subjects.
LD
A
utilizes
w
ord
order
to
allocate
topics,
intending
to
comprehend
the
structure
of
themes
inside
documents
[26].
Only
LD
A
will
be
utilized
in
the
current
study
.
Figure
1
pro
vides
a
comprehensi
v
e
and
complete
representation
of
each
e
xperiment.
Indonesian
J
Elec
Eng
&
Comp
Sci,
V
ol.
37,
No.
2,
February
2025:
1150–1167
Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian
J
Elec
Eng
&
Comp
Sci
ISSN:
2502-4752
❒
1155
Figure
1.
Our
e
xperiments
steps
4.
RESUL
TS
AND
DISCUSSION
4.1.
Results
This
section
pro
vides
a
detailed
comparison
between
the
chatbots
ChatGPT
-4
and
Gemini.
Addi
tion-
ally
,
it
discusses
the
use
of
a
topic
modeling
technique
in
the
articles
listed
in
T
ables
1,
2,
3,
and
4,
which
were
subsequently
discussed.
Educational
impact
and
ethical
consider
ations
in
using
Chatbots
in
Academia
(Dina
M.
Ibr
ahim)
Evaluation Warning : The document was created with Spire.PDF for Python.
1156
❒
ISSN:
2502-4752
4.1.1.
ChatGPT
-4
When
requesting
bibliographies
without
specifying
reliable
sources
in
Chat
GPT
-4,
you
will
recei
v
e
f
ak
e
bibliographies,
as
sho
wn
in
T
able
1.
The
LD
A
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
ChatGPT
-4
(direct
request)
are
illustrated
in
T
able
2.
The
topic
modeling
results
for
5
topics
and
4
w
ords
for
ChatGPT
-4
(direct
request)
are
sho
wn
in
Figure
2.
T
able
1.
Comparison
of
the
academic
articles
generated
by
chatbot-4
in
the
same
elds
(direct
request)
Article
title
Field
Number
of
w
ords
The
actual
w
ord
with
a
bibliograph
y
Number
of
bibliograph
y
T
rue
bibliograph
y
F
alse
bibliograph
y
Plagiarism
check
er
X
1.
Adv
ancements
in
machine
learning
for
nancial
fore-
casting:
a
compre-
hensi
v
e
o
v
ervie
w
T
echnology
1,000
488
4
0
4
14%
2.
Enhancing
image
and
video
processing
through
machine
learning:
techniques
and
applications
T
echnology
1,000
557
4
0
4
17%
3.
Machine
learning
in
medical
diagnosis:
current
applications
and
future
prospects
T
echnology
1,000
534
4
0
4
16%
4.
Adv
ancements
in
machine
learning
for
natural
language
proc-
essing:
a
comprehen-
si
v
e
re
vie
w
T
echnology
1,000
542
4
0
4
9%
5.
Le
v
eraging
machine
learning
for
social
media
analysis:
insights
and
inno
v
ations
T
echnology
1,000
606
4
0
4
18%
T
able
2.
LD
A
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
ChatGPT
-4
(direct
request)
T
opic
number
T
erm
s
Rate
1
Application
Section
Ef
cienc
y
Accurac
y
0.240
0.240
0.040
0.040
2
Section
Predicti
v
e
Ef
cienc
y
Accurac
y
0.171
0.171
0.171
0.171
3
Section
Application
Ef
cienc
y
Accurac
y
0.067
0.067
0.067
0.067
4
Ethical
Language
Implication
Bias
0.157
0.086
0.086
0.086
5
Section
Application
Ef
cienc
y
Accurac
y
0.067
0.067
0.067
0.067
Indonesian
J
Elec
Eng
&
Comp
Sci,
V
ol.
37,
No.
2,
February
2025:
1150–1167
Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian
J
Elec
Eng
&
Comp
Sci
ISSN:
2502-4752
❒
1157
T
able
3.
Comparison
of
the
academic
articles
generated
by
chatbot-4
in
the
same
elds
(direct
request)
Article
title
Field
Number
of
w
ords
The
actual
w
ord
with
a
bibliograph
y
Number
of
bibliograph
y
T
rue
bibliograph
y
F
alse
bibliograph
y
Plagiarism
check
er
X
1.
Enhancing
nancial
forecasting
in
stock
mark
ets
through
machine
learning
techniques
T
echnology
1,000
568
5
1
4
14%
2.
Adv
ancements
in
machine
learning
for
f
ace
detection
in
image
and
video
processing
T
echnology
1,000
643
5
3
2
in
the
publication
year
9%
3.
Adv
ancing
viral
detection
in
clinical
diagnostics
through
machine
learning
techniques
T
echnology
1,000
565
3
0
3
14%
4.
Le
v
eraging
machine
learning
in
natural
language
processing
for
sentiment
analysis
of
te
xtual
data
T
echnology
1,000
588
4
2
2
in
the
publication
year
17%
5.
Enhancing
social
media
adv
ertising
through
machine
learning:
an
analytical
approach
T
echnology
1,000
574
3
1
2
11%
T
able
4.
LD
A
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
ChatGPT
-4
(scenario
request)
T
opic
number
T
erms
Rate
1
Result
Discussion
Base
Detection
0.015
0.015
0.015
0.015
2
Sentiment
Base
Detection
Ho
we
v
er
0.060
0.033
0.033
0.033
3
Model
Application
Machine
Practical
0.044
0.024
0.024
0.024
4
Discussion
Dealing
Remain
Social
0.082
0.014
0.014
0.014
5
T
rend
Despite
Challenge
Dimensionality
0.047
0.047
0.047
0.047
When
requesting
bibliographies
with
rel
iable
sources
specied
in
ChatGPT
-4,
you
will
recei
v
e
r
eal
and
f
ak
e
bibliographies,
as
sho
wn
in
T
able
3.
LD
A
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
ChatGPT
-4
(scenario
request)
are
illustrated
in
T
able
4.
The
topic
modeling
results
for
5
topics
and
4
w
ords
for
ChatGPT
-4
(scenario
request)
are
sho
wn
in
Figure
3.
Educational
impact
and
ethical
consider
ations
in
using
Chatbots
in
Academia
(Dina
M.
Ibr
ahim)
Evaluation Warning : The document was created with Spire.PDF for Python.
1158
❒
ISSN:
2502-4752
Figure
2.
T
opic
modeling
results
for
5
topics
and
4
w
ords
for
ChatGPT
-4
(direct
request)
Figure
3.
T
opic
modeling
results
for
5
topics
and
4
w
ords
for
ChatGPT
-4
(scenario
request)
4.1.2.
Gemini
When
requesting
bibliographies
in
Gemini
without
specifying
reliable
sources,
you
may
not
recei
v
e
bibliographies
for
certain
articles.
These
bibliographies
can
contain
a
mix
of
accurate
and
partially
inaccurate
information,
as
indicated
in
T
able
5.
The
LD
A
of
the
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
Indonesian
J
Elec
Eng
&
Comp
Sci,
V
ol.
37,
No.
2,
February
2025:
1150–1167
Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian
J
Elec
Eng
&
Comp
Sci
ISSN:
2502-4752
❒
1159
Gemini
(direct
request)
is
sho
wn
in
T
able
6,
while
the
topic
modeling
results
for
5
topics
and
4
w
ords
for
GEMINI
(direct
request)
are
presented
in
Figure
4.
When
requesting
bibliographies
specifying
reliable
sources
in
Gemini,
you
will
recei
v
e
real
and
f
ak
e
bibliographies
as
sho
wn
in
T
able
7.
The
LD
A
of
the
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
Gemini
(direct
request)
is
sho
wn
in
T
able
8,
while
the
topic
modeling
results
for
5
topics
and
4
w
ords
for
GEMINI
(direct
request)
are
presented
in
Figure
5.
T
able
5.
Comparison
of
the
academic
articles
generated
by
Gemini
in
the
same
elds
(direct
request)
Article
title
Field
Number
of
w
ords
The
actual
w
ord
with
a
bibliograph
y
Number
of
bibliograph
y
T
rue
bibliograph
y
F
alse
bibliograph
y
Plagiarism
check
er
X
1.
Machine
learning
for
nancial
fore-
casting:
un
v
eiling
hidden
patterns
and
predicting
mark
et
trends
T
echnology
1,000
1,587
7
7
1
&
1
in
the
publication
year
19%
2.
Un
v
eiling
the
potential:
machine
learning
for
image
and
video
processing
T
echnology
1,000
1,180
7
7
0
22%
3.
Machine
learning
for
medical
diag-
nosis:
re
v
olutionizing
healthcare
through
pattern
recognition
T
echnology
1,000
1,121
8
6
2
19%
4.
Un
v
eiling
the
po
wer
of
language:
machine
learning
for
natural
language
processing
T
echnology
1,000
1,333
6
4
1
&
1
in
the
publication
year
27%
5.
Un
v
eiling
the
social
pulse:
machine
learning
for
social
media
analysis
T
echnology
1,000
1,054
5
4
1
16%
T
able
6.
LD
A
most
rele
v
ant
terms
for
v
e
topics
and
four
w
ords
for
Gemini
(direct
request)
T
opic
number
T
erms
Rate
1
Ev
ent
Black
Mark
et
V
olatile
0.165
0.113
0.113
0.062
2
Speech
Con
v
ert
Speak
Recognition
0.143
0.143
0.143
0.143
3
Recognition
Con
v
ert
Language
Speak
0.059
0.059
0.059
0.059
4
Language
Recognition
Speak
Speech
0.059
0.059
0.059
0.059
5
Speak
Recognition
Language
Con
v
ert
0.059
0.059
0.059
0.059
Educational
impact
and
ethical
consider
ations
in
using
Chatbots
in
Academia
(Dina
M.
Ibr
ahim)
Evaluation Warning : The document was created with Spire.PDF for Python.