Inter
national
J
our
nal
of
Electrical
and
Computer
Engineering
(IJECE)
V
ol.
11,
No.
2,
April
2021,
pp.
1549
1560
ISSN:
2088-8708,
DOI:
10.11591/ijece.v11i2.pp1549-1560
r
1549
Help
thr
ough
demonstration
and
automation
f
or
interacti
v
e
computing
systems:
A
sur
v
ey
of
r
ecent
w
orks
P
edr
o
Rodrigues
1
,
J
os
´
e
Lu
´
ıs
Silv
a
2
1
Instituto
Uni
v
ersit
´
ario
de
Lisboa
(ISCTE-IUL),
Lisboa,
Portug
al
2
ITI
/
LARSyS
/
Instituto
Uni
v
ersit
´
ario
de
Lisboa
(ISCTE-IUL),
IST
AR-IUL,
Lisboa,
Portug
al
Article
Inf
o
Article
history:
Recei
v
ed
Jan
17,
2020
Re
vised
Aug
25,
2020
Accepted
Sep
10,
2020
K
eyw
ords:
Automation
Demonstration-based
help
Help
systems
Learning
Picture-dri
v
en
computing
ABSTRA
CT
Usability
is
v
ery
important
ho
we
v
er
,
it
is
still
dif
ficult
to
de
v
elop
interacti
v
e
computing
systems
that
meet
all
user’
s
specificities.
Help
systems
should
be
a
w
ay
of
bridging
this
g
ap.
This
paper
presents
a
general
surv
e
y
on
recent
w
orks
(b
uilding
upon
pre
vious
surv
e
ys)
related
to
impro
ving
applications’
help
through
demonstration
and
automa-
tion
and,
identifies
which
technologies
are
acting
as
enablers.
The
mai
n
contrib
utions
are,
identifying:
i
)
which
are
the
recent
e
xisting
solutions;
ii)
which
aspects
must
be
in
v
estig
ated
further;
and
iii)
which
are
the
main
dif
ficulties
that
are
pre
v
enting
a
f
aster
progress.
This
is
an
open
access
article
under
the
CC
BY
-SA
license
.
Corresponding
A
uthor:
Jos
´
e
Lu
´
ıs
Silv
a
ITI
/
LARSyS
/
Instituto
Uni
v
ersit
´
ario
de
Lisboa
(ISCTE-IUL)
IST
AR-IUL,
1649-026,
Lisboa,
Portug
al
E-mail
address:
jlcsa@iscte-iul.pt
1.
INTR
ODUCTION
User
interf
aces
(UIs)
ha
v
e
a
major
role
in
an
y
interacti
v
e
computing
system
(ICS).
The
life
c
ycle
of
technology
proposed
by
Norman
[1]
state
that
when
technology
satisfy
basic
user
needs
then
the
y
become
more
interested
in
ef
ficienc
y
,
con
v
enience
and
pleasure.
Therefore,
usability
and
user
e
xperience
became
crucial.
Indeed,
usability
is
vital
for
websites,
products
and
services
to
achie
v
e
success.
Ho
we
v
er
,
although
all
adv
ances
on
user
-centered
approaches
and
design,
some
interf
aces
are
still
dif
ficult
to
use,
causing
frustration
to
users
[2].
Managers
and
de
v
elopers
often
recognize
it
b
ut
still
lea
v
e
it
apart
from
the
product-de
v
elopment
process
[3,
4].
T
o
address
user’
s
dif
ficulties
some
ICS
pro
vide
help
systems
ho
we
v
er
,
it
is
still
v
erified
a
major
dif-
ficulty
in
making
people
use
them.
Dw
orman
et
al
.,
[5]
state
that
this
beha
viour
is
e
xplained
by
either
users
not
recognizing
help
systems
e
xistence,
not
w
anting
to
pause
their
current
tasks
to
search
help
or
because
the
y
w
ant
to
find
the
solution
on
their
o
wn.
Ne
v
ertheless,
the
design
of
ef
fecti
v
e
user
-centered
help
systems
is
still
seen
as
a
weak
ne
gotiator
point
by
the
b
usiness
because
it
is
f
aced
as
a
secondary
functionality
[4,
6]
as
sho
wn
in
Figure
1.
T
o
combat
user’
s
resistance
to
use
help
sys
tems
and
still
mak
e
users
f
amiliar
with
tools,
especial
ly
the
ones
with
some
comple
xity
,
training
programs
focused
on
task-modelling
were
proposed
[7].
The
y
aim
to
understand
which
are
the
k
e
y
aspects
that
depend
on
the
user
.
This
modelling
has,
yet,
the
potential
to
impro
v
e
systems
design
if
considered
in
the
implementation
process
[8].
Ho
we
v
er
,
this
modelling
has
a
certain
le
v
el
of
comple
xity
and
some
tools
propose
to
o
v
erlap
it
by
analyzing
the
task
being
accomplished
and
defining
it
[9].
Objectifying
the
impro
v
ement
of
aid
systems,
conte
xtual
help
pro
v
ed
to
be
ef
fecti
v
e
in
learning
us
er
interf
aces.
Conte
xtual-help
defines
the
paradigm
of
help
gi
v
en
in
the
conte
xt
where
the
doubt
has
risen
up,
J
ournal
homepage:
http://ijece
.iaescor
e
.com
Evaluation Warning : The document was created with Spire.PDF for Python.
1550
r
ISSN:
2088-8708
e.g.
a
certain
tool-tip
associated
to
a
b
utton
[10].
This
concept
is
similar
to
the
situated
action
theory
which
states
that
kno
wledge
can
only
be
interpreted
in
its
conte
xt.
From
this
concept
help
tools
ha
v
e
arose,
lik
e
AIDE
which
through
communic
ation
with
users,
understanding
the
conte
xt
of
the
doubt,
disco
v
ers
and
solv
es
the
issues
[11].
Similar
approaches
are
demonstrat
ion-based
help
that
in
addition
to
w
orking
in
the
conte
xt
of
the
application
demonstrates
ho
w
a
certain
task
is
to
be
e
x
ecuted.
F
or
instance,
through
automation
scripts
[12]
and,
model-dri
v
en
help
[10]
that
firstly
models
the
system
and
details
interactions
on
it
and
then
starts
the
automation/e
xplanation
design.
Dif
ferent
approaches
to
pro
vide
help
ha
v
e
been
de
v
eloped.
Either
using
video,
tutorials,
functional
ities
as
automation
and
picture-dri
v
en
computing
(PDC),
among
others
[11,
13-16].
Some
also
suggest
approaches
based
on
cro
wd
learning,
lik
e
help
forums,
ar
guing
that
people
learn
better
when
discussing
subjects
and
an
man
y-to-one
support
is
more
ef
ficient
[17,
18].
Man
y
current
help
systems
are
still
based
on
user
manuals
or
do
not
pro
vide
inter
-application
support.
The
major
barriers
to
an
o
v
erall
impro
v
ement
are
the
increase
on
systems’
comple
xity
[19]
and
impossibility
to
access
and
change
source
codes.
Ov
ercoming
user’
s
issues,
this
paper
re
vie
ws
ho
w
e
xisting
help
systems
are
being
implemented
and
which
technologies
are
being
used.
The
re
vision
aims
to
identify
e
xisting
approaches
to
de
v
elop
help
systems
and
what
can
still
be
impro
v
ed
on
them.
Additionally
,
understanding
ho
w
f
ar
hel
p
tools
are
connected
to
the
application
(e.g.
can
be
separated),
whether
approaches
lik
e
continuous
learning
[20]
either
supervised
or
unsupervised
are
applicable
and,
ho
w
f
ar
could
a
cro
wd-sourced
system
be
beneficial.
The
re
vie
w
includes
applications
that
implement
help
mechanism
o
r
automate
certain
(sub)tasks
to
o
v
ercome
user’
s
issues
(e.g.
usability).
This
paper
is
or
g
anized
as
follo
ws.
Section
2
present
s
topics
related
to
most
recent
help
solutions
(e.g.
picture-dri
v
en
computing,
automation).
Section
3
re
vie
ws
e
xisting
help
solutions
and
sort
them
into
se
v
eral
high-le
v
el
cate
gories.
In
Section
4,
both
comparison
between
identified
help
solutions
and
critical
analysis
are
made.
Finally
,
conclusions
and
directions
for
future
w
ork
are
presented
in
Section
5.
Figure
1.
Usability
study
in
a
lar
ge
compan
y
focusing
on
problems
and
barriers
to
usability
[4]
2.
B
A
CKGR
OUND
This
section
introduces
important
concepts
associated
with
most
recent
help
and
automation
tools.
The
section
starts
by
describing
picture-dri
v
en
computing
(PDC),
a
paradigm
that
enables
computer
program-
ming
based
on
screen
shots,
not
just
code.
The
tw
o
main
approaches
based
in
this
paradigm
are
presented
(sikuli
and
robotic
process
automation).
Then
automation,
a
mechanism
to
reduce
users’
ef
fort
by
automating
steps
to
accomplish
some
task
and,
task-models
are
presented.
2.1.
Pictur
e-dri
v
en
computing
K
ourousias
and
Bonfiglio
present
the
visual
information
in
a
2-dimensional
spati
al
domain
as
nuclear
data
of
the
PDC
paradigm.
This
visual
information
is
what
intends
to
be
visible
on
a
graphical
output
de
vice
[21].
This
paradigm
is
useful
for
task
automation
of
systems
already
de
v
eloped
and/or
without
access
to
their
Int
J
Elec
&
Comp
Eng,
V
ol.
11,
No.
2,
April
2021
:
1549
–
1560
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Elec
&
Comp
Eng
ISSN:
2088-8708
r
1551
source
code
[11].
One
w
ay
to
implement
PDC
is
by
using
the
Sikuli
[21]
tool.
This
is
an
MIT
open-source
program-
ming
en
vironment
de
v
eloped
using
p
ython
and
using
OpenCV
(OpenCV
:
https://opencv
.or
g
(last
accessed:
14
January
2020))
packages
for
finding
images
on
the
screen.
Sikuli
w
as
de
v
eloped
to
empo
wer
help
designers
in
the
general
public
(computer
-skill
independent)
to
create
conte
xtual
help
[15].
The
tool
allo
ws
for
standard
programming
functions
(e.g.
loops,
conditions,
v
ariables)
b
ut
with
the
addition
of
tar
get
patterns
identification
on
the
screen.
T
o
use
the
tool,
users
can
tak
e
adv
antage
of
the
SikuliX
inte
grated
de
v
elopment
en
vironment
(IDE)
that
is
a
basic
script
editor
to
load,
edit,
sa
v
e
and
run
scripts.
In
what
re
g
ards
non-picture
related
func-
tions
users
must
ha
v
e
pre
vious
basic
computer
programming
basis.
On
Figure
2
an
e
xample
of
a
Sikuli
script
de
v
eloped
on
the
IDE.
Figure
2.
Sikuli
script,
e
xample
of
a
script
de
v
eloped
using
sikuli
IDE
In
the
conte
xt
of
de
v
eloping
help
systems,
sikuli’
s
user
can
select
an
element
from
the
interf
ace
by
capturing
its’
screenshot
and
selecting
the
type
of
conte
xtual
help
to
add
(a
v
ailable
action
to
perform
on
spec-
ified
UI
elements)
to
that
element.
Some
e
xampl
es
are
highlights,
clicks,
and
type
into
(writes
a
specified
message
when
an
element
is
pressed).
When
presenting
the
resulting
help
system
to
users,
pix
els
on
the
screen
are
searched,
elements
located
and
the
corresponding
act
ion
reproduced.
S
ee
the
w
ork
of
Y
eh
et
al.
[15]
for
additional
information
about
creating
conte
xtual
help
using
screenshots.
Another
w
ay
to
implement
PDC
solutions
is
done
using
robotic
process
automation
(RP
A).
As
s
tated
by
Schatsk
y
et
al.
[22]
RP
A
softw
are
automate
repetiti
v
e,
rule-based
processes
usually
performed
by
people
sitting
in
front
of
computers.
The
interaction
with
computers
is
done
by
mimicking
the
interaction
between
humans
and
computers
and
therefore
robots
can
open
email
attachments,
complete
e-forms,
and
record
and
replay
data
[22].
T
o
mimic
user
interactions,
current
robots
operate
in
tw
o
main
modes.
Either
the
y
are
f
amiliar
with
applications,
usually
through
communication
plugins,
and
can
interact
directly
with
all
UI
elements
(e.g.
b
uttons,
te
xt
box
es,
drop
menus,
combo
box
es)
or
the
y
operate
through
PDC
using
also
screenshots
and
image
recognition
techniques.
On
Figure
3
a
synthesis
of
RP
A
goals
and
its
description
is
presented.
This
information
w
as
g
athered
Help
thr
ough
demonstr
ation
and
automation
for
...
(P
edr
o
Rodrigues)
Evaluation Warning : The document was created with Spire.PDF for Python.
1552
r
ISSN:
2088-8708
from
one
of
the
mark
et
RP
A
applications.
The
goals
emphasized
are
increased
customer
satisf
action,
better
resource
utilization,
increased
accurac
y
and
impro
v
ed
producti
vity
.
Figure
3.
RP
A
goals,
Synthesis
of
RP
A
goals
retrie
v
ed
from
Nice
[23]
Most
common
RP
A
applications
are
UIP
ath
[24],
BluePrism
[25],
and
Automation
An
ywhere
[26].
The
y
are
designed
mainly
for
companies
which
recognize
the
e
xistence
of
tasks
that
pose
as
good
candidates
to
be
automated
[27].
2.2.
A
utomation
and
task-models
This
section
describes
automation
and
concepts
related
to
it
(e.g.
tasks
and
task-models).
Before
automating
it
is
important
to
properly
understand
the
task
that
will
be
automatically
performed.
This
can
be
achie
v
ed
with
a
detailed
design
and
analysis.
Adequate
notations
for
this
purpose
are
also
described
in
this
section.
Automation
consists
on
the
transformation
of
systems’
functions
into
automatic
processes
to
af
fect
it
follo
wing
a
specific
tar
get
[19].
Some
basic
concepts
associated
to
automation
are:
Business
process-consists
of
sequences
of
acti
vities
Acti
vity-discrete
process
step
performed
either
by
m
achines
or
humans
(acti
vities
may
consist
of
one
or
more
tasks)
W
orklist-set
of
tasks
to
be
performed
by
a
user
in
a
w
orkflo
w
system
W
orkflo
w-is
the
automation
of
a
b
usiness
process
in
whole
or
in
part,
during
which
documents,
informa-
tion
or
tasks
are
passed
through
participants
follo
wing
a
set
of
rules
[28,
29]
One
w
ay
to
describe
automation
le
v
el
of
a
system
corresponds
to
the
identification
of
the
proportion
of
automatic
actions
[19].
The
y
are
three
dif
ferent
types,
full
automation,
semi-automation
or
manual.
Fully
automated
are
fully
conducted
by
a
computer
and
all
possible
outcomes
are
predicted.
Semi-automated
requires
an
interaction
between
the
computer
and
a
human
that
v
alidate,
compare,
or
add
information
to
the
w
orkflo
w
.
Manual
are
usually
comple
x
and
requires
comprehensi
v
e
kno
wledge
and
skills
to
tak
e
adequate
decisions
[28].
Other
w
ay
to
describe
automation
l
e
v
el
is
using
a
scale
from
one
to
ten.
In
this
scale
a
v
alue
of
one
represents
b
usiness
process
on
which
the
computer
of
fers
no
assistance
and
humans
tak
e
all
decisions
and
actions.
A
v
alue
of
ten
means
that
the
computer
decides
e
v
erything,
acting
autonomously
and
ignoring
human
inputs
[30].
On
critical
w
orkflo
ws
a
semi-automated
approach
is
best
suited
making
the
system
to
w
ork
on
a
multi-
agent
basis
(an
human
agent
interacting
with
a
softw
are
agent)
[31].
Ho
we
v
er
,
some
problems
and
concerns
might
arise:
When
the
automation
requires
user
to
input
some
v
ariables
to
set
the
session
the
automation
can
be
put
at
risk.
This
happen
either
because
the
input
parameters
are
not
the
e
xpected
or
because
the
responsible
might
for
get
that
the
automation
program
w
as
w
aiting
for
the
parameter
[19]
The
f
act
that
people
cannot
recognize
repetiti
v
e
tasks
w
orth
y
of
automating
leading
to
bad
automation
task
selection.
User
intervie
ws
re
v
ealed
that
tasks
were
identified
as
repetiti
v
e
if
manually
performed
numerous
times
in
a
ro
w
or
if
comple
x
b
ut
infrequent
[32]
Int
J
Elec
&
Comp
Eng,
V
ol.
11,
No.
2,
April
2021
:
1549
–
1560
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Elec
&
Comp
Eng
ISSN:
2088-8708
r
1553
Those
problems
might
be
mitig
ated
with
an
accurate
process
definition
that
can
be
accomplished
with
task-models
and
proper
task
definition
[7,
33].
Po
werful
analytical
tools
such
as
Petri
[34]
or
concur
task
trees
(CTT)
[35]
pro
vide
a
thorough
pro-
cess
(specifying
tasks)
definition
and
enable
formal
w
orkflo
w
analyze
(e.g.
correctness
and
consis
tenc
y).
A
v
aluable
aspects
of
these
tools
is
the
decomposition
of
tasks
in
its
sub-tasks
enabling
a
proper
understanding
of
required
steps
to
accomplish
them
[7].
T
o
define
a
task,
it
is
necessary
to
identify
the
objects
and
actions
that
define
the
communication.
Objects
consist
on
entities
that
are
manipulated
to
perform
tasks
(e.g.
menus,
icons,
state
requests
of
some
application,
etc.)
[35].
Those
tools
also
pro
vide
hierarchical
logical
structure
allo
wing
for
dif
ferent
abstraction
le
v
el
and
process
refinement.
A
v
ariety
of
temporal
relationships
operators
supporting
interacti
v
e
beha
viors
and
representation
of
rele
v
ant
relationship
is
also
pro
vided.
T
o
e
xpress
temporal
rela-
tionship
among
tasks
LO
T
OS
[36]
concurrent
notation
is
used.
CTT
editor
and
HAMSTERS
[37]
enable
the
definition
of
task-models
being
possible
to
specify
the
type
of
each
task:
User
task-performed
entirely
by
the
user
Application
task-completely
e
x
ecuted
by
the
system
Interacti
v
e
task-performed
by
user
interactions
with
the
system
Abstract
task-represents
high
le
v
el
task
including
a
set
of
tasks
of
dif
ferent
types
The
possibility
to
represent
concurrent
t
asks
mak
es
CTT
dif
ferent
from
unique
sequential
relati
on
s
hip
a
v
ailable
on
GOMS
[38],
despite
their
similarity
on
hierarchical
tasks
disposition
[30,
39].
Figure
4
pres
ents
a
partial
task-model
(in
the
conte
xt
of
a
flight
softw
are
board)
de
v
eloped
with
HAMSTERS.
T
ask-models
might
represent
a
barrier
for
task
automation
mainly
because
the
additional
ef
fort
to
the
o
v
erall
de
v
elopment
[29,
32].
Aiming
at
reducing
the
ef
fort
required
for
the
de
v
elopment
of
task
models,
Machado
et
al.
proposed
a
tool
that
constructs
CTT
task
models
from
the
e
x
ecution
of
tasks
[9].
Figure
4.
T
ask-model,
Example
of
a
task-model
de
v
eloped
with
HAMSTERS
[30]
3.
LITERA
TURE
REVIEW
This
section
presents
approaches
which
aim
to
impro
v
e
daily
interactions
between
users
and
com-
puters.
The
section
starts
by
presenting
the
methodology
used
in
the
selection
process
and
then
presents
the
identified
w
ork
grouped
by
interaction
core
used
by
the
approaches,
i.e.
ho
w
interaction
with
applications
is
performed
(e.g.
script
and
automation-based,
video-based,
GUI
elements
based).
The
intent
w
as
to
perform
a
(not
systematic)
literature
re
vie
w
of
recent
w
ork
adding
to
e
xisting
surv
e
ys.
Help
thr
ough
demonstr
ation
and
automation
for
...
(P
edr
o
Rodrigues)
Evaluation Warning : The document was created with Spire.PDF for Python.
1554
r
ISSN:
2088-8708
3.1.
Methodology
f
or
tool
selection
The
search
for
this
re
vie
w
w
as
made
through
digital
repositories.
The
repositories
from
which
most
information
w
as
tak
en
are
A
CM
Digital
Library
,
Springer
,
IEEE
Xplore
and
Else
vier
.
Additionally
,
information
from
AAAI
Digital
Library
,
W
ile
y
Online
Library
,
T
aylor
and
Francis
Online
and
Scientific
Research
publisher
w
as
also
tak
en.
The
range
of
presented
tools
focus
mainly
in
the
2010-2018
period.
This
choice
w
as
due
to
the
identification
of
a
re
vie
w
made
by
Grossman
and
Fi
tzmaurice
[40]
that
presents
help
tools
pre
vious
to
2010
based
on
video
and
animated
documentation,
conte
xtual
assistance,
and
conte
xtual
video
assistance.
T
w
o
additional
w
orks
where
included
because
the
y
present
interesting
approaches
which
with
the
addition
of
current
technology
capabilities
w
ould
pose
as
good
candidates
on
the
help-tools
umbrella.
The
k
e
yw
ords
used
on
the
research
were
‘action
repetition’,
‘conte
xtual
help’,
‘cro
wdsourced
help’,
‘demonstration
help’,
‘GUI
automation’,
‘kno
wledge
sharing’,
‘model-dri
v
en
help’,
‘program
ming
by
demon-
stration’,
‘softw
are
support’,
‘w
orkflo
w
automation’,
‘picture-dri
v
en
computing’,
‘tutorial’
and
‘user
support’.
Rele
v
ant
cited
papers
were
also
included.
3.2.
Scripts
and
automation-based
Sheepdog
[20]
is
a
programming-by-demonstration
system
that
w
orks
in
dif
ferent
phases.
First,
it
learns
from
multiple
e
xperts,
each
performing
the
same
procedure
(set
of
steps
to
achie
v
e
a
goal)
directly
on
a
W
indo
ws
desktop.
W
ith
the
recording,
Sheepdog
models
the
procedure
allo
wing
dif
ferent
configurations.
These
m
odels
can
be
reproduced
on
an
end-use
r
de
vice
via
an
e
x
ecutabl
e
file.
When
generating
the
models,
Sheepdog
abstracts
from
lo
w-le
v
el
W
indo
ws
e
v
ents
i
nto
clean
high-le
v
el
representations
and
uses
input/output
hidden
Mark
o
v
models
to
determine
the
odds
of
occurrence
of
the
ne
xt
node.
The
i
n
t
erf
ace
where
the
system
automation
is
mapped
should
be
collaborati
v
e
to
allo
w
the
user
(e
xpert)
to
add
e
xtra-input.
When
replaying
the
script,
step
by
step,
the
system
tak
es
a
snapshot
and
from
it
determines
and
indicates
what
is
the
ne
xt
step.
CoScripter
,
pre
viously
called
K
oala
[41],
is
a
collaborati
v
e
scripting
en
vironment
for
recording,
au-
tomating
and
sharing
web-based
processes.
It
has
tw
o
main
modules,
a
plug-in
that
allo
ws
users
to
record
and
play
actions
and
a
repository
where
users
can
share
their
scripts,
rate
and
comment
other
scripts.
The
actions
recorded
are
recognized
and
replayable
via
HTML
interpretation.
ActionShot
[42]
is
an
e
xtension
to
Firefox
web
bro
wser
b
uil
t
on
top
of
the
CoScripter
web
record-
ing/playback
platform.
Unlik
e
CoScripter
,
ActionShot
continuously
records
users’
bro
wser
history
grouping
actions
by
pages’
host
name.
It
also
pro
vides
a
visual
interf
ace
that
allo
ws
users
to
use
or
share
an
y
action
the
y
ha
v
e
e
v
er
performed.
The
description
of
each
module
is
based
on
te
xt
tak
en
from
HTML
code
(usually
the
te
xt
displayed
on
widgets).
CoCo
[43]
is
a
system
that
automates
web
tasks.
It
tak
es
adv
antage
of
a
repository
with
pre
vious
web
scripts
and
users’
personal
web
bro
wsing
history
.
Pe
o
pl
e
ask
questions
via
T
witter
and
the
system
can
either
respond
asking
for
more
information
or
with
an
answer
.
The
system
can
co
v
er
each
user
needs.
T
askT
racer
[44]
is
able
to
track
users
beha
vior
when
using
applications
lik
e
Microsoft
Of
fice,
e-mail,
and
Internet
Explorer
(an
e
xtra
add-in
has
to
be
installed)
being
able
to
reco
v
er
the
full
conte
xt
of
the
task
if
desired.
T
o
classify
the
task,
the
user
must
indicate
its
be
ginning
and
ending.
Authors
ackno
wledge
it
as
an
e
xtra
b
urden.
The
y
aim
at
the
ability
to
restore
all
applications
associated
with
a
process,
the
documents
used
and
e
v
en
the
indication
of
the
last
changes.
Smart
W
eb
T
utor
[18]
proposes
an
approach
to
enhance
massi
v
e
open
online
courses
(MOOCs)
with
education
resources
de
v
elopment
for
instructors
and
students,
allo
wing
realtime
collaborati
v
e
learning
for
stu-
dents.
As
for
the
recording
phase,
it
is
based
on
the
DOM
structure,
common
on
web-sites,
and
w
orks
on
three
main
phases,
recording,
optimizing,
replaying.
A
similarity
formula
that
considers
platforms
v
ariability
is
also
considered
as
dif
ferent
systems
may
require
dif
ferent
steps
to
accomplish
a
task.
F
or
real-time
collaborati
v
e
learning,
authors
propose
a
communication
protocol
to
send
each
step
on
a
synchronous
w
ay
,
making
possible
to
a
group
of
people
to
w
atch
a
procedure
e
x
ecuting
on
real-time.
3.3.
V
ideo-based
Ambient
Help
[45]
w
orks
on
a
secondary
monitor
.
While
people
are
perform
ing
a
task
on
the
primary
monitor
Ambient
Help
is
automatically
displaying
vi
d
e
os
or/and
te
xtual
information
rele
v
ant
to
the
process.
Whene
v
er
people
need
help
or
w
ant
to
see
Ambient
Help
suggestions
the
y
focus
on
the
secondary
monitor
and
search
the
desired
help-content.
Int
J
Elec
&
Comp
Eng,
V
ol.
11,
No.
2,
April
2021
:
1549
–
1560
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Elec
&
Comp
Eng
ISSN:
2088-8708
r
1555
P
ause-and-Play
[14]
proposes
a
learning
impro
v
ement
via
video
tutorials.
The
approach
consists
on
recei
ving
a
video-tutorial
and
recognizing
its
main
steps
via
computer
-vision
algorithms.
Then
to
trace
user
beha
vior
,
it
is
poss
ible
(with
the
addition
of
plugins)
to
pause
and
play
the
video
tutoria
l
according
to
user
performance
on
the
application.
In
addition,
a
functionality
which
allo
ws
easy
na
vig
ation
through
the
most
important
parts
of
the
video
w
as
de
v
eloped.
T
oolClips
[13]
is
a
conte
xtual-based
help
system
that
aims
to
replace
tooltips
associated
to
UI
el
e-
ments.
I
n
s
tead
of
standard
tooltips
(e.g.
with
icons’
name),
when
the
cursor
pass
o
v
er
an
UI
element,
the
user
can
access
video
or
te
xt
with
i
nformation
about
the
elements’
use.
W
ith
this
approach
users
are
presented
with
e
xtra-information
inside
the
application.
3.4.
GUI
elements
based
LemonAid
[17,
46]
is
a
cro
wdsourced
conte
xtual
help
system.
It
pro
vides
technical
help
based
on
the
selection
of
GUI
elements
(label,
widget,
link,
and
image)
instead
of
k
e
yw
ords.
People
get
help
by
selecting
an
UI
element
and,
from
there,
LemonAid
presents
the
fi
v
e
most
ask
ed
questions
by
the
community
.
Users
can
use
the
search
bar
to
enter
k
e
yw
ords,
refine
the
questions
sho
wn,
mak
e
ne
w
questions
or
see
the
answers
to
e
xisting
questions.
T
echnically
,
it
w
orks
as
a
layer
abo
v
e
a
web
application
UI,
independent
from
back
and
front-end
implementations.
Authors
claim
that
one-to-man
y
support
is
more
ef
ficient
and
pro
vides
greater
cost
sa
vings.
DemoHelp
[47]
is
an
help
system
that
tak
es
adv
antage
of
PDC
and
automation
(using
Sikuli
scri
pts)
and
is
to
be
used
side
by
side
with
an
application.
Users
select
an
action
to
e
x
ecute
in
the
help
system
and
the
procedure
is
triggered
on
the
application.
T
o
f
acilitat
e
the
creation
of
Sikuli
scripts
users
can
enrich
CTT
task
models
with
applications’
UI
elements
and
select
possible
scenarios.
Based
on
this
w
ork,
an
help
tool
(FEL)
[48]
w
as
de
v
eloped.
An
user
study
w
as
perf
o
r
med
comparing
the
help
pro
vided
by
the
de
v
eloped
tool
and
guidebooks.
Results
sho
wn
benefits
using
FEL
on
first-time
task
e
x
ecution.
Interacti
v
e
systems
inte
gration
tool
(ISI)
[12]
is
a
tool
that
also
w
orks
with
enriched
task
m
od
e
ls
(CTT)
and
scenario
selection
(using
a
tool
from
the
Human
Interf
aces
in
Information
Systems
Laboratory
[49])
to
automatically
create
Sikuli
scripts.
The
tool
remo
v
es
comple
xity
from
original
GUI,
presenting
a
simplified
GUI
with
comple
x
tasks
automated
and
ready
to
be
reproduced.
The
tool
can
mer
ge
multiple
interf
aces
from
dif
ferent
applications
into
only
one
GUI.
When
the
user
selects
a
task,
all
steps
are
e
x
ecuted
by
Sikuli
on
a
virtual
machine
being
presented
to
the
user
only
the
final
result
(hidding
the
original
GUI).
The
approach
w
orks
with
an
y
application/platform.
Help,
it
looks
confusing
(HILC)
[50]
is
a
tool
that
relies
on
compute
r
vision
techniques
rather
than
accessibility
APIs
being
therefore,
domain
independent
.
It
focus
on
personal
automation
and
script
genera-
tion
from
non-programmer
users
and
automates
procedures
by
demonstration.
When
demonstrating,
the
tool
produces
the
needed
screenshots
and
corresponding
mouse-k
e
yboard
e
v
ents.
Scripts
can
be
made
from
video
records,
with
specialized
screen
casting
softw
are,
or
with
a
snif
fer
program.
Inputs
are
con
v
erted
to
log
files
that
pass
through
a
classifier
algorithm.
Ho
we
v
er
,
the
authors
state
that
pure
programming-by-demonstration
is
still
unrealistic.
The
y
v
erified
that
for
small
procedures
the
classifier
algorithm
does
not
ha
v
e
great
performance
yet.
Conte
xtual
help
for
adapti
v
e
interf
aces
(CHAIN)
[10]
w
as
de
v
eloped
to
pro
vide
model
dri
v
en
con-
te
xtual
help
to
adapti
v
e
user
interf
aces
(interf
a
ces
that
change
their
pres
entation
at
runtime).
The
adaptation
to
change
is
made
by
an
association
between
concrete
elements
and
scene
elements.
Concrete
elements
are
defined
by
specific
parameters
lik
e
the
concrete
id
on
the
code,
and
scene
elements
are
associated
to
them
(e.g.
a
menu
is
a
concrete
element
and
its
‘inner’
options
are
the
scene
elements).
CHAIN
w
as
de
v
eloped
using
Ja
v
aScript
for
the
web,
C#
with
.Net
Frame
w
ork
for
desktop
and
C#
with
Xamarin
for
mobile.
When
source-
code
is
not
a
v
ailable
it
is
able
to
use
the
PDC
paradigm
interacting
in
dif
ferent
w
ays
depending
on
where
the
application
is
r
u
nni
ng.
F
or
web
applications
it
uses
Selenium
capabilities
and
for
desktop
applications
it
mak
es
use
of
an
UI
automation
frame
w
ork.
The
help
is
gi
v
en
by
the
definition
of
a
model
on
Cedar
Studio
[51],
which
also
allo
ws
for
the
importation
of
Selenium
scripts.
The
focus
of
CHAIN
is
pro
viding
proper
annotations
on
the
tar
get
interf
ace,
being
also
able
to
insert
some
v
alues-see
Figure
5.
The
authors
state
(based
on
their
study)
that
one-shot
learning
is
something
appreciated.
Help
across
se
v
eral
applications
w
as
something
that
the
y
did
not
reach
and
cro
wd-sourcing
w
ould
be
an
interesting
trait.
T
able
1
(see
in
appendix)
summarizes
and
sorts
into
three
cate
gories
(automation,
automati
on/help,
and
help)
the
stated
w
orks.
The
cate
gorization
is
based
on
the
goal
of
the
w
ork
(i.e.
to
impro
v
e
processes
Help
thr
ough
demonstr
ation
and
automation
for
...
(P
edr
o
Rodrigues)
Evaluation Warning : The document was created with Spire.PDF for Python.
1556
r
ISSN:
2088-8708
through
automation
or
to
help
users).
Each
w
ork
w
as
then
classified
into
six
f
actors
considered
important
to
describe
and
understand
their
purpose.
The
y
are:
i)
domain;
ii)
interaction
core
(ho
w
interaction
with
applications
is
performed);
iii)
cro
wdsourced-based;
i
v)
uses
picture-dri
v
en
computing
interaction;
v)
enables
demonstration-based
learning
(learn
with
user
demonstration);
vi)
enables
continuous
learning
(impro
v
es
with
user
interactions).
Those
f
actors
emer
ged
from
the
i
nterconnection
between
our
goals
and
identified
w
orks.
From
them,
one
can
identify
,
solutions
independent
of
application’
s
source
code,
ho
w
interaction
is
performed,
check
con-
tinuous
learning
viability
,
identify
interf
ace
interaction
challenges,
and
if
it
w
orks
on
a
cro
wdsourced-based
w
ay
.
Figure
5.
CHAIN,
e
xample
of
an
interf
ace
with
annotations
made
with
CHAIN
[10]
4.
DISCUSSION
This
section
describes
ho
w
the
presented
w
ork
addresses
the
three
contrib
utions
presented
in
the
ab-
stract.
Consideri
ng
the
identi
fication
of
re
cent
solutions
(1),
se
v
eral
w
orks
were
found
and
presented.
The
y
were
cate
gorized
and
classified
into
dif
ferent
f
actors
to
f
acilitate
their
comparison
as
sho
wn
in
T
able
1
(see
in
appendix).
From
the
table
man
y
dif
ferences
and
similarities
between
w
orks
can
be
identified
b
ut
more
interesting
are
the
patterns
that
seem
to
emer
ge.
Starting
with
the
main
goal(s)
of
solutions,
only
one
goal
tends
do
be
addressed
(only
tw
o
recent
w
orks
address
both
automation
and
help).
In
relation
to
domain,
no
pattern
stands
out,
being
al
l
web,
plugin
dependent
and
domain
independent
present
with
the
same
frequenc
y
.
Ho
we
v
er
,
in
what
concerns
automation
solutions,
a
pre
v
alence
for
web
stands
out.
One
reason
for
this
might
the
DOM
structure
a
v
ailability
making
de
v
elopment
easier
.
Another
reason
might
be
related
to
the
higher
applicability
as
typical
users
use
to
utilize
mainly
web
applications.
A
similar
conclusion
can
be
dra
w
re
g
arding
interaction
core,
i.e.
a
pattern
seems
to
emer
ge
only
for
solutions
with
focus
on
automation-the
use
of
automation
scripts.
Cro
wdsourced
is
being
frequently
used
in
automation
b
ut
more
frequent
usage
should
be
reached
when
the
goal
of
solutions
is
help.
PDC
and
one-shot
learning
started
to
be
used
in
solutions
to
pro
vide
help.
T
rends
were
not
identified
re
g
arding
continuous
learning.
From
the
comparison
made
some
problems
and
possible
impro
v
ements
were
identified
(2).
Some
tools
present
benefits
of
both
source
code
and
domain
independence
b
ut
o
v
ercoming
all
aspects
still
needs
research.
PDC
is
a
recent
technology
that
f
acilitates
the
o
v
erall
implementation
of
e
xtra-application
capabilities
ho
we
v
er
,
Int
J
Elec
&
Comp
Eng,
V
ol.
11,
No.
2,
April
2021
:
1549
–
1560
Evaluation Warning : The document was created with Spire.PDF for Python.
Int
J
Elec
&
Comp
Eng
ISSN:
2088-8708
r
1557
after
the
decomposition
of
an
GUI
(see
the
w
ork
of
Dubro
vina
et
al.
[52]
or
Pref
ab
[53]
for
further
details)
it
is
still
dif
ficult
to
properly
interact
with
its
elements
atomical
ly
(e.g.
Dixon
et
al.
[54]
reports
that
clickable
“tar
get”
are
often
ambiguous
and
cannot
be
re
v
erse
engineered
without
human
interv
ention).
T
o
accomplish
full
domain
independence
icons
and
widgets
should
be
fully
identified
e
v
en
without
accessing
source
code.
Solutions
based
on
artificial
intelligence
(AI)
might
be
able
to
address
this
challenge.
Being
able
to
w
ork
in
groups
and
share
kno
wledge
is
also
a
v
aluable
aspect
that
most
tools
still
lea
v
e
apart
(cro
wdsourced-help).
Continuous
learning,
through
the
implementation
of
learning
algorithms,
may
also
be
an
adv
antage
impro
ving
the
interaction
of
the
us
er
as
the
steps
that
are
chosen
to
automate
can
be
better
selected.
Additionally
,
the
ability
to
implem
ent
cont
inuous
learning,
allo
wing
the
tools
to
k
eep
learning
from
users,
is
another
interesting
aspect
still
missing.
The
joint
between
PDC,
automation,
cro
wdsourced-help,
with
the
ability
to
capture
tasks
from
a
single
demonstration
are
desired.
The
main
barrier
for
f
aster
de
v
elopment
(3),
is
related
to
domain
independenc
y
,
either
from
source
code
and/or
e
xtra-plugins.
Not
requiring
the
user
to
install
e
xtra-plugins
or
to
restrict
the
automations/
demon-
strations
to
a
set
of
applications
is
something
that
brings
man
y
benefits
lik
e
inter
-application
help
possibilities.
T
o
achie
v
e
this
independenc
y
a
system
that
does
not
depend
on
applications’
code
or
st
ructure
is
a
must.
PDC
has
the
capabilities
required
ho
we
v
er
,
we
belie
v
e
it
is
still
not
being
lar
gely
used
because
it
has
some
limita-
tions
yet
(e.g.
element’
s
identification
ambiguity).
The
de
v
elopment
of
solutions
mer
ging
PDC
with
AI
will
o
v
ercome
this
challenge.
F
or
instance,
the
w
ork
of
Dubro
vina
et
al.
[52]
presents
a
step
in
this
direction
pre-
senting
a
solution
for
GUI
object
classification
using
a
support
v
ector
machine
classifier
.
Solutions
lik
e
this
are
essential
(a
US
patent
already
e
xists
[55])
for
image-based
softw
are
automation/demonstration
tools
ho
we
v
er
,
this
is
an
area
that
has
still
not
got
the
deserv
ed
attention.
5.
CONCLUSIONS
This
paper
presents
a
lit
erature
re
vie
w
of
recent
tools
that
aim
to
f
acilitate
the
w
ay
tasks
are
ac
com-
plished
via
help
and/or
automation.
The
most
rele
v
ant
solutions
were
identified,
described,
compared
and
sorted
into
cate
gories.
The
cate
gories
that
came
up
are
related
to
the
w
ay
tools
implement
help
and/or
automation.
The
cate
gories
are,
Scripts
and
automated-based,
V
ideo-based,
GUI
elements
based.
From
the
comparison
made
as
sho
wn
in
T
able
1
it
can
be
concluded
that
both
automating
tasks
to
pro
vide
help
is
something
being
addressed
in
man
y
recent
w
orks
and,
most
w
orks
are
still
restricted
to
a
specific
domain.
In
what
concerns
continuous-learning
it
is
an
aspect
that
has
not
recei
v
ed
much
attention
yet.
This
is
true
also
for
cro
wdsourced
concepts
allo
wing
people
to
easily
w
ork
and
share
their
de
v
elopments
in
groups.
PDC
and
one-shot
l
earning
are
implemented
in
some
tools
and
sho
w
interesting
results
b
ut
are
concepts
still
on
a
rising
phase.
Impro
ving
the
w
ay
people
get
help
and/or
making
tasks’
accomplishment
easier
are
major
adv
ant
ages.
This
is
true
for
both
corporate
w
ork
ers
in
their
daily
acti
vities
where
the
y
must
be
producti
v
e
and,
for
singular
computer
users
that
man
y
times
feel
the
y
cannot
e
x
ecute
some
tasks
(usually
attrib
uting
wrongly
the
f
ailure
to
their
lack
of
skills
instead
of
recognizing
softw
are
deficiencies).
De
v
eloping
a
tool
that
w
ould
consider
the
implementation
of
cro
wdsourced
collaboration,
i
nteraction
through
picture-dri
v
en
computing,
that
w
ould
easily
capture
each
steps
of
a
task
and,
that
could
learn
with
users’
e
x
ecutions
seems
t
o
be
an
interesting
path
to
follo
w
.
Help
thr
ough
demonstr
ation
and
automation
for
...
(P
edr
o
Rodrigues)
Evaluation Warning : The document was created with Spire.PDF for Python.
1558
r
ISSN:
2088-8708
APPENDIX
T
able
1.
Identified
w
orks
cate
gorized
re
g
arding
their
main
goal
and
classified
into
six
f
actors
Main
goal
Domain
Interaction
core
Cro
wd
uses
PDC
enables
enables
Name
sourced-
interaction
Demonstration-
Continuous
based
based
Learning
Learning
Automation
W
eb
Automation
scripts
Y
es
No
No
CoScripter
W
eb
Autom
ation
scripts
Y
es
No
No
ActionShot
W
eb
Autom
ation
scripts
Y
es
No
Y
es
CoCo
question-
answer
analysis
Plugin
Autom
ation
scripts
–
No
No
T
askT
racer
dependent
–
Automation
Independent
Learn-
–
No
Y
es
Y
es
Sheepdog
/Help
Independent
replay
procedure
–
Independent
Enriched
task
No
No
No
No
ISI
models
and
PDC
Help
W
eb
Dialogues
with
users
No
No
No
Y
es
AIDE
W
eb
Scripts-W
eb’
DOMs
Y
es
No
Y
es
No
Smart
W
eb
T
utor
–
V
ideos
or/and
No
No
No
No
T
oolClips
–
te
xtual
help
Plugin
V
ideos
No
No
No
No
P
ause-and-Play
dependent
Plugin
GUI
Elements
Y
es
No
No
No
LemonAid
dependent
Independent
V
ideos
or/and
No
No
No
No
Ambient
Help
te
xtual
help
Independent
Enriched
task
No
Y
es
No
No
DemoHelp
models
and
PDC
Independent
GUI
Elements
No
Y
es
Y
es
Y
es
HILC
Independent
Interaction
models
No
Y
es*
No
No
CHAIN
(through
Cesar
Studio)
A
CKNO
WLEDGEMENTS
This
w
ork
w
as
supported
by
Fundac
¸
˜
ao
para
a
Ci
ˆ
encia
e
a
T
ecnologia
(FCT
,
Portug
al),
t
hrough
projects
UIDB/50009/2020,
UIDB/04466/2020
and
UIDP/04466/2020.
The
authors
w
ould
lik
e
to
thank
R
´
uben
Pereira
for
the
gi
v
en
support.
REFERENCES
[1]
Norman,
D.
A.,
”The
in
visible
computer:
wh
y
good
products
can
f
ail,
the
personal
computer
is
so
com-
ple
x,
and
information
appliances
are
the
solution,
”
MIT
press,
1998.
[2]
Lazar
J,
Jones
A,
and
Shneiderman
B,
”W
orkplace
user
frustration
with
computers:
An
e
xploratory
in
v
es-
tig
ation
of
the
causes
and
se
v
erity
,
”
Beha
viour
and
Information
T
echnology
,
v
ol.
25,
no.
03,
pp.
239-251,
2006.
[3]
Rosenbaum
S,
Rame
y
J,
and
Redish
JG.,
”Current
issues
in
assessing
and
impro
ving
information
usabil-
ity
,
”
In
CHI’10
Extended
Abstracts
on
Human
F
actors
in
Computing
Systems,
pp.
3155-3158,
2010.
[4]
W
inter
J,
R
¨
onkk
¨
o
K,
Rissanen
M.,
”Identifying
or
g
anizational
barriers-A
case
study
of
usabil
ity
w
ork
when
de
v
eloping
softw
are
in
the
automation
industry
,
”
Journal
of
Systems
and
Softw
are,
v
ol.
88,
pp.
54-73,
2014.
[5]
Dw
orman
G,
Rosenbaum
S.,
”Helping
users
to
use
help:
impro
ving
interaction
with
help
systems,
”
CHI’04
e
xtended
abstracts
on
Human
f
actors
in
computing
systems,
pp.
1717-1718,
2004.
[6]
Dw
orman
G,
”Arbitration
of
a
help
system,
”
Interactions,
v
ol.
14,
no.
1,
pp.
39-42,
2007.
[7]
Martinie
C,
P
alanque
P
,
Na
v
arre
D,
W
inckler
M,
Poupart
E.,
”Model-based
training:
an
approach
sup-
porting
operability
of
critical
interacti
v
e
systems,
Proceedings
of
the
3rd
A
CM
SIGCHI
symposium
on
Engineering
interacti
v
e
computing
systems,
pp.
53-62,
2011.
Int
J
Elec
&
Comp
Eng,
V
ol.
11,
No.
2,
April
2021
:
1549
–
1560
Evaluation Warning : The document was created with Spire.PDF for Python.